SQLScript in BW transformation routines

SQLScript in BW transformation routines
Jörg Brandeis

(C) Brandeis Consulting.

Demo AMDP Create

Task
Create a method GET_COUNTRIES_TEXT that reads all texts of the countries from the table T005T for a given language.

Steps

  1. implement in ABAP
    Create a test program that calls the method and outputs the result. 2.
  2. implementation in SQLScript
(C) Brandeis Consulting.

Demo: Call another procedure in AMDP

Task
Call the method from the previous demo with language E in a new AMDP method GET_COUNTRIES_TEXT_EN.

(C) Brandeis Consulting.

SAP HANA execution of a DTP

If a transformation meets certain conditions, it can be executed directly on the SAP HANA database as of Release 7.40 SPS05. The most important prerequisite for this is that no ABAP routines are present.

Whether an execution in SAP HANA is possible can be seen under 7.40/7.50 in the runtime status field:

(C) Brandeis Consulting.

ABAP execution of a DTP (illustration)

(C) Brandeis Consulting.

SAP HANA execution of a DTP (illustration)

(C) Brandeis Consulting.

Performance gain with HANA execution

The higher performance with SAP HANA execution has several reasons:

  • No copying of data between ABAP and HANA.
  • Larger data packages (typically: 1,000,000 vs. 50,000) reduce overhead for status management and monitoring.
  • By processing directly on the SAP HANA database, data processing can be well parallelised.
  • Existing routines are implemented in SQLScript for SAP HANA transformations. These can then be optimised well.
(C) Brandeis Consulting.

Tools

  • The implementation of AMDP routines requires the use of the ABAP Development Tools (ADT) in Eclipse.
  • The modelling of the BW transformation should be done with the BW Modelling Tools (BW-MT) in Eclipse, as the jump to the routine implementation works better here.
  • The routines can be debugged with the AMDP Debugger.
  • With the HANA SQL Console, transformation routines can be tested or simulated directly.
(C) Brandeis Consulting.

Transformation routines as AMDP

All routines of a transformation must be implemented in the same technology: Either ABAP or AMDP.

(C) Brandeis Consulting.

Successive transformations and mixed execution

  • Several transformations may lie between a source and a target if an InfoSource has been modelled in between.
  • The runtime environment may differ as long as the lower transformations are executable on HANA.
  • An ABAP transformation may never lie below a HANA transformation.
    In case of mixed execution, no error processing is possible.
  • SAP recommends a maximum of two InfoSources per data flow, see Blog by Thorsten Kessler
(C) Brandeis Consulting.

The generated AMDP class

BW on HANA

  • For each routine, a separate AMDP class is generated in the $TMP package with a cryptic GUID name in the /BIC/ namespace.
  • The connection from routine to transformation is stored in the transformation.
  • This always contains exactly one method with the name PROCEDURE.

BW/4HANA

  • For each transformation, two AMDP classes are generated in the /BIC/ namespace.
  • The TRF name is contained in the class name.
  • The classes have the ending _M(=Modified) and _A (=Active).
  • The classes contain one method per routine.

(C) Brandeis Consulting.

Demo & Exercise: Creating an End Routine

Exercise 0 - Creating a transformation routine for deriving CAL characteristics.

(C) Brandeis Consulting.

Simulation of a transformation routine in the console.

Testing transformations repeatedly is time consuming. Yet it is relatively easy to test the code of a transformation in the SQL console.

Procedure:

  • Create an anonymous block: DO BEGIN..END
  • Insert the transformation code
  • Define and fill the table INTAB at the beginning of the block
  • Query the OUTTAB at the end
DO BEGIN
  intab = SELECT *, 
                 '' AS record, 
                 '' AS sql__procedure__source__record
            FROM "/BIC/A<DSO name>2"            
            --if necessary WHERE condition or LIMIT 1000 to keep the amount of data small 
--------------------------------
  --Here comes the routine code
--------------------------------
  SELECT * FROM :outtab;
END;
(C) Brandeis Consulting.

Transport of the routine

  • The AMDP class is local ($TMP) and therefore cannot be transported.
  • Instead, the source code is stored in the metadata of the transformations (table RSTRANSTEPROUT).
  • When the transformation is activated, the class is then regenerated in the target system.
(C) Brandeis Consulting.

Signature of the AMDP methods

The interface of the PROCEDURE method is generated to match the transformation. There are the same parameters for all four routine types:

  • Input Table INTAB - The data is passed to the routine via the IN parameter with the name INTAB.
  • Output table OUTTAB - The result of the routine is written to the table parameter OUTTAB. To activate the AMDP class, it is necessary that this parameter is filled with an appropriate SELECT query.

As of BW/4HANA, only if error processing is switched on:

  • Output table ERRORTAB - Incorrect records are written to this table. It consists of two fields:
    • ERROR_TEXT - Description of the error
    • SQL__PROCEDURE__SOURCE__RECORD - Field from the corresponding record of the INTAB.
  • scalar parameter I_ERROR_HANDLING - This parameter has the value TRUE if the current procedure call queries the error table ERRORTAB.
(C) Brandeis Consulting.

Extended signature (as of BW/4HANA)

parameter description
I_REQ_DTP_NAME Technical DTP Name
I_REQ_LOGSYS Name of the logical source system
I_REQ_SRC_NAME Source object of the DTP
I_REQ_SRC_TYPE Type of source object
I_REQ_TGT_NAME Target object of the DTP
I_REQ_TGT_TYPE type of target object
I_REQ_REQUID Request ID - A timestamp that is the same for all packets
(C) Brandeis Consulting.

The RECORD field in the tables

The RECORD field is a 56 character string.

After transformation, records are sorted by this field.
.

This may be relevant for the order of insertion into the target of the transformation. Initially, the RECORD field contains the concatenated REQUESTID, DATAPAKID and RECORD fields from the source of the transformation.

For successive transformations, it contains the value of the RECORD field of the previous transformation.

(C) Brandeis Consulting.

Changing the 'RECORD' field

  • If the sort order is relevant in a scenario, you as the developer of the AMDP routine must ensure that the RECORD field is filled with an appropriate content.
  • For example, if the data is to be sorted by a date field and a time field, you can write both of these concatenated in the RECORD column.

Code example / Demo

You can also create a consecutive number with the window function ROW_NUMBER(). Please note that the field has the data type string and that an alphanumeric sorting takes place. So that the 10 does not come before the 2, you must add leading zeros with LPAD ( <LfdNr.>, 56, '0').

outtab = SELECT ...          
                LPAD( ROW_NUMBER() OVER (ORDER BY <sort>), 56, '0') AS RECORD,
                SQL__PROCEDURE__SOURCE__RECORD
       FROM :intab ;
(C) Brandeis Consulting.

SQL__PROCEDURE__SOURCE__RECORD

The field SQL__PROCEDURE__SOURCE__RECORD contains a unique key for records in the table INTAB.

This key can be used to assign entries in the error table ERRORTAB to the respective record from INTAB. If new data records are generated in a routine, the value of the respective source data record is to be taken over.

(C) Brandeis Consulting.

Demo: Test programme for an AMDP routine

Optional, if execution in the SQL console is not possible.

REPORT Z_TEST_AMDP.

DATA lt_intab TYPE /BIC/GJXH4BF0CQHVTB53PRRU_M=>tn_t_in_global_end.

SELECT * FROM /bic/azbr_e1_s2
    INTO CORRESPONDING FIELDS OF TABLE lt_intab UP TO 100 ROWS.

new /BIC/GJXH4BF0CQHVTB53PRRU_M( )->GLOBAL_END(
  EXPORTING
    I_REQ_DTP_NAME = ''
    I_REQ_LOGSYS = ''
    I_REQ_SRC_NAME = ''
    I_REQ_SRC_TYPE = ''
    I_REQ_TGT_NAME = ''
    I_REQ_TGT_TYPE = ''
    I_REQ_REQUID = ''
    intab = lt_intab
  IMPORTING
    outtab = data(lt_outtab)
).

cl_demo_output=>display( lt_outtab ).
(C) Brandeis Consulting.

Assignment of the output tables

The output table OUTTAB and possibly also ERRORTAB must be assigned in the SQLScript source code, otherwise there will be syntax errors. This is also necessary if you do not want to perform error processing.

 METHOD global_end BY DATABASE PROCEDURE FOR HDB LANGUAGE SQLSCRIPT OPTIONS READ-ONLY.
    OUTTAB = SELECT * FROM :INTAB;
    ERRORTAB = SELECT '' AS ERROR_TEXT,
                      '' AS SQL__PROCEDURE__SOURCE__RECORD
                 FROM DUMMY WHERE DUMMY <> 'X';
  ENDMETHOD.

This code corresponds to the initial code of the generated method of an end routine on a BWoH 7.50, although it has been formatted a little more attractively in the example.

  • The table OUTTAB is filled 1:1 from the INTAB.
  • The table ERRORTAB obviously remains empty.
ERRORTAB = SELECT * FROM :ERRORTAB;
(C) Brandeis Consulting.

Access database tables with USING.

If you use other database tables, views, AMDP procedures or functions, you must add them first with the USING clause in the METHOD statement:

 METHOD global_end BY DATABASE PROCEDURE FOR HDB LANGUAGE SQLSCRIPT 
  OPTIONS READ-ONLY USING /bi0/aepm_adso32.
    OUTTAB = SELECT
                    IT.epm_po,
                    IT.epm_ipos,
                    ...
 		        	lookup.epm_langu,
                    lookup.epm_poovs,
                    it.record,
                    it.sql__procedure__source__record
                    FROM :INTAB as IT
                    left outer join "/BI0/AEPM_ADSO32" as lookup
                    on it.epm_po = lookup.epm_po
                    and it.epm_ipos = lookup.epm_ipos;
  ENDMETHOD.
  • In ABAP, the table is written without quotation marks.
  • In SQL the table is written with quotation marks!
(C) Brandeis Consulting.

Tables & Views of the ADSOs

The table and view names for the ADSOs are formed according to the following scheme:
/BIC/A<DSO name><suffix>

Tables

Regardless of the type of ADSO, three tables are always created. However, depending on the type, sometimes only some of them are used:

  • Input tables (1)
  • Active table (2)
  • Change table (3)

The tables are not released for public use SAP Note 1682131
"While these cases are supported, testing and update/upgrade compatibility is the responsibility of the customer."

Views

In addition to the tables, two to three views are created:

SAP Note 2723506

"The external SAP HANA SQL view provides a robust interface that allows official and authorised access to the data contained in an aDSO."

(C) Brandeis Consulting.

Reading from other DDIC tables

When reading from non-BIC/ tables, the mandate may need to be taken into account. For this, there are the session variables that contain the necessary information.

 SELECT ...
  FROM <DDic-Table>
  WHERE mandt = session_context( 'CLIENT' )

If the routine accesses the mandt from the Session Context, the routines cannot be simulated so easily in the console.
For this purpose, the corresponding parameter Session Client must be maintained in the user administration of HANA:

(C) Brandeis Consulting.

NOT NULL

There must be no NULL values in the result tables. If there are, an error message will be displayed:

The automatic initialisation exists from BW/4HANA 2021:

(C) Brandeis Consulting.

Avoidance of NULL

  • Columns that are filled by an OUTER JOIN should always be secured here with the COALESCE or IFNULL function.
  • The SQL function COALESCE returns the first (leftmost) value from a series of values that is not NULL.
  • The SQL function IFNULL works the same way, but with exactly two parameters
  • CASE expressions should always have an ELSE branch, even if it is theoretically never called.
(C) Brandeis Consulting.

Start routines

The start routine is run at the beginning of the transformation, i.e. before the assignment of the individual fields and their processing with the field rules takes place. Accordingly, the two table parameters 'INTAB' and 'OUTTAB' include the fields of the data source.

Typical tasks of the start routines are:

  • Filtering records if the selection criteria in the DTP are not sufficient, e.g. because they can only be determined by a join.
  • Adding data records, which are then processed in the further steps of the transformation exactly like the original data.

Both can be done in the same way in the end or expert routine!

(C) Brandeis Consulting.

End routines

The end routine is performed as the last step of the transformation. Therefore, the field list of 'INTAB' and 'OUTTAB' contains the field names of the data target.
The result of the end routine is passed directly to the data target. Accordingly, it can override all other processing steps already carried out at this point.

Typical use cases for end routines:

  • Filtering of data to the result of previous processing steps.
  • Calculation or derivation of several fields with (partially) common logic. DRY principle
  • Dependencies among the fields to be calculated.
(C) Brandeis Consulting.

Expert routines

The expert routine replaces the entire transformation logic. So if an expert routine is created, field rules (1:1 mapping, formulas, lookups, etc.) are no longer possible. The table 'INTAB' contains only the fields of the source and the table 'OUTTAB' the fields of the target of the transformation.

Typical use cases

  • Fields from source and target are needed
  • The logic would otherwise be spread over many places (field rules, start and end routine)
  • Trigger DSOs
(C) Brandeis Consulting.

Field routines

The field routines determine the values for exactly one target field. One or more source fields are used for input, which are assigned in the definition of the rule details

  • An AMDP field routine is called only once.
  • All source fields of the rule are available in the `INTAB'.
  • The result is written to the OUTTAB.
  • The result is merged with the rest of the data via `VERTICAL_UNION
  • No records may be lost or added.
  • The contents of the RECORD and SQL__PROCEDURE__SOURCE__RECORD fields must not be changed.

(C) Brandeis Consulting.

Error processing and error stack

TLDR - It does not work cleanly!

Error processing can occur in all four types of routines. This means that the routine writes the values of the SQL__PROCEDURE__SOURCE__RECORD field of the erroneous records from the INTAB to the ERRORTAB table. There, a suitable error text can also be entered in the ERROR_TEXT column.
Please test error handling thoroughly

In the past, the area of error handling and error stack had repeatedly caused problems. It is advisable to test these functions thoroughly before using them, even after importing new SPs.

(C) Brandeis Consulting.

SAP Note 2580109 - Error Handling

SAP Note 2580109 - Error Handling

Generally, major performance restrictions are to be expected, ... may result in longer loading times than with the ABAP runtime"_.

In general, scenario 2 is recommended. The use of scenario 1 must be thoroughly tested.
==> Errorstack is not recommended by SAP!

(C) Brandeis Consulting.

Processing sequence in the data transfer process

Every transformation, and therefore every routine, is run through twice when executing a data transfer process.
In the first run, the incorrect data records are determined and in the second run, the actual processing takes place.

h=400

h=400

# Semantic Grouping in the HANA version With BW/4HANA the semantic grouping works, With BW on HANA (7.40/7.50) not ;-(

Alternatives are:

  • Large package sizes. However, one must consider that with under also data must be completely rebuilt. Then the package size of the daily loads may no longer be sufficient...
  • Re-reading the complete source data for the keys:
 new_intab = select *
                from "/BIC/AZBR_E2_S2"
                WHERE ( "BUDAT", "ACCOUNT" ) in (select budat,
                                                         account
                                                    from :intab);
(C) Brandeis Consulting.

AMDP Debugging

The status of the AMDP debugger

A breakpoint is shown in the AMDP debugger, as in the ABAP debugger, as a small round circle to the left of the line number:

(C) Brandeis Consulting.

The status of the breakpoints

The circle of the breakpoint changes colour depending on its status:

colour meaning
Blue Only shortly after the breakpoint is set. The routine is compiled again for debugging in the background.
Green The breakpoint is confirmed and the debugger is active.
Grey The debugger is inactive. It can be activated through the context menu on the breakpoint.
White The breakpoint is deactivated. It can also be activated through the context menu on the breakpoint.

After 10 minutes of inactivity, the debugger deactivates itself.

(C) Brandeis Consulting.

Debugging BW transformation routines

Typical sources of errors during debugging:

(C) Brandeis Consulting.

Debug Mode vs. Optimised Mode

In order for procedures to be analysed with the AMDP debugger, they are recompiled in the so-called debug mode. This means that some optimisations are not performed that are taken into account in the normal, optimised mode.

After debugging, the routine remains in the debug version for a while (about 1h).

(C) Brandeis Consulting.

Caution - routines can disappear

Even on current BW4 systems, it can always happen that routines disappear after activating the transformation or DTP!

Recommendation therefore:

  • Save the source code to the clipboard or Notepad.
  • Close the object, open it again and check which code is there now.
  • Make dummy changes in the transformation before activating it, e.g. insert spaces in the description. -
  • Then save and only then activate.
(C) Brandeis Consulting.

In BW on HANA: Orphaned AMDP classes

If you create an AMDP routine from the transformation in transaction RSA1, an ABAP class is generated and saved for it. However, the transformation is not yet saved at this point - this only happens when you click on Save in the transformation.

It happens again and again that you implement the AMDP method PROCEDURE in the generated class, but do not save the corresponding transformation. This means that the logical assignment between transformation and AMDP class has disappeared.

You can use the table 'TADIR' to determine the old class name and thus copy the code.

(C) Brandeis Consulting.

Tips

The following are a few tips to make life easier with Eclipse ;-)

(C) Brandeis Consulting.

Block marking

Alt+Shift+A - Very useful to copy the field lists from the declaration.

(C) Brandeis Consulting.

Create field list in the editor

  1. copy the definition from the Info element
  2. In Notepad++, replace: " type.*" with ",".

This can also be recorded as a macro.

You can also create the special notation with quotation marks.

(C) Brandeis Consulting.

Element Info with F2 or as View

With F2, details about method definitions can be displayed in AMDP routines.

(C) Brandeis Consulting.

Linking the project view with the editors on the right

(C) Brandeis Consulting.

Code completion with 'Ctrl + Space

(C) Brandeis Consulting.

Automatic syntax check on and off

(C) Brandeis Consulting.

Compare with Local History

(C) Brandeis Consulting.

Set background colour for SQL script

(C) Brandeis Consulting.