(2022-May-31)
Personal notes of one of the recent ADF JSON pipeline development to remember how to use it next time.
Requirements:
- Sourcing data comes from a SQL Server database
- The destination is a PostgreSQL database table
- Transformation logic is to aggregate several rows from a sourcing table and populate the resulting JSON structured document into a single row JSONB type column
Things to remember:
- It does help to prepare a resulting JSON document in a sourcing query as a string expression with all the required JSON elements within to save time for validation in the ADF pipeline
- T-SQL STRING_AGG function is a nice alternative to combine multiple table rows into a single text value, originally I would rely on a SELECT … FOR JSON PATH T-SQL statements for this.
- A good way to validate the sourcing data is to extract it into blob storage and then pass this extracted further toward the ETL pipeline. Don’t forget to set ‘No…’ for Escape character and Quote character settings in the blob storage souring dataset.
- Special characters to pay attention to:
- ADF Escape data flow expression allows to format an input string to a particular format, however, I thought to control the construction of my JSON document in the sourcing query myself without using this command. This also saved me some time to troubleshoot several data transformation issues.
- CR = Carriage Return (0x0D in hexadecimal, 13 in decimal) and LF = Line Feed (0x0A in hexadecimal, 10 in decimal), could be replaced with r, and n, however, I made a decision to remove them in my sourcing query.
- TABs can also be replaced with t, but I just simply replace them with space ‘ ‘ in the sourcing query as well.
- Alter row transformation in the mapping data flow is a great way to support multiple transformation scenarios (Insert, Update, Delete, Upsert) in a single data flow stream.
- JSONB data column in PostgreSQL provides its own data validation during the ETL process. When my ADF pipeline tried to push data to that column and I was getting this error message, “StatusCode": "DFExecutorUserError”, that meant that my final JSON document is not well-formatted and doesn’t meet JSON type requirements.