October 7, 2024 at 5:34 pm
I am trying to build an ETL for an Existing Table. The ETL (SSIS) Takes data from Excel, Convert it into CSV.
The Data in Excel Source is:
INSERT INTO YourTableName (
[Fiscal year/period],
[Company Code],
[Contra Company Code],
[Account],
[Level 1],
[Level 2],
[Level 3],
[Level 4],
[Generic Material],
[Metric],
[Unit],
[Value]
)
VALUES (
'004.2025', -- Fiscal year/period
'SA10', -- Company Code
'AE10', -- Contra Company Code
'100778134', -- Account
'2700041843', -- Level 1
'Kimberly Clark (PARENT)', -- Level 2
'101020439', -- Level 3
'MENA MANF CATEGORY RGP', -- Level 4
'101020439', -- Generic Material
'rDP (out)', -- Metric
'EA', -- Unit
407 -- Value
);
Ouput in CSV:
INSERT INTO YourTableName (
[Fiscal year/period],
[Company Code],
[Contra Company Code],
[Account],
[Level 1],
[Level 2],
[Level 3],
[Level 4],
[Generic Material],
[Metric],
[Unit],
[Value]
)
VALUES (
'001.2024', -- Fiscal year/period
'SA10', -- Company Code
'AE10', -- Contra Company Code
'100778134', -- Account
'2700041843', -- Level 1
'Kimberly Clark (PARENT)', -- Level 2
'101020434', -- Level 3
'Non-Food Category RP', -- Level 4
'101020439', -- Generic Material
'rEP (out)', -- Metric
'EA', -- Unit
415 -- Value
);
The existing Table that needs to be updated:
INSERT INTO [IMETA_FINAL].[Prod].[IMETA - Global Field Flows and Holdings (Period and client)_model_SAP BW] (
[Fiscal year period],
[Company],
[Intra Company],
[Account],
[Account name],
[Level 1],
[Level 1 name],
[Level 2],
[Level 2 name],
[Level 3],
[Level 3 name],
[Level 4],
[Level 4 name],
[Metric],
[Material Code],
[Value],
[Loaddate],
[Month]
)
VALUES (
'001.2024', -- Fiscal year period
'SA10', -- Company
'AE10', -- Intra Company
'100778134', -- Account
'Kimberly Clark (PARENT)', -- Account name
'2700041843', -- Level 1
'Non-Food Category RP', -- Level 1 name
'101020434', -- Level 2
'Non-Food Category RP', -- Level 2 name
'101020439', -- Level 3
'MENA MANF CATEGORY RGP', -- Level 3 name
'101020439', -- Level 4
'MENA MANF CATEGORY RGP', -- Level 4 name
'rEP (out)', -- Metric
'1', -- Material Code
415, -- Value
GETDATE(), -- Loaddate (current date)
'2023-12-07' -- Month (format as needed)
);
Errors;
[CSV Data Source - Created and updated format (Global Field Flows) [2]] Error: An error occurred while skipping data rows.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on CSV Data Source - Created and updated format (Global Field Flows) returned error code 0xC0202091. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
Code to INSERT:
INSERT INTO [I_FINAL].[Prod].[IMETA_Global_Field_Flows_Backup]
SELECT
CONVERT(VARCHAR(20),[Fiscal year period]) AS [Fiscal year period]
,CONVERT(VARCHAR(8),[Company Code]) AS [Company Code]
,CONVERT(VARCHAR(8),[Contra Company Code]) AS [Contra Company Code]
,CONVERT(VARCHAR(50),[Account]) AS [Account]
,CONVERT(VARCHAR(50),[Contra Account]) AS [Contra Account]
,CONVERT(VARCHAR(25),[Generic Material]) AS [Generic Material]
,CONVERT(VARCHAR(225),FF.[Metric]) AS [Metric]
,[IMETA_FINAL].[PROD].(FF.[Metric Value]) AS [Metric Value]
,CONVERT(DATE,FF.[Loaddate]) AS [Loaddate]
,CONVERT(DATE,FF.[Month]) AS [Month]
FROM
(SELECT
[Fiscal year period]
,[Company Code]
,[Contra Company Code]
,[Account]
,[Contra Account]
,[Generic Material]
,[sPD (Wal-Mart reissue)]
,[rDP (Wal-Mart returns to TPM WM)]
,
,[v (Chep Plant to TPM WM)]
,[sPQ (WM TPM to E & D)]
,[sPD (Issues to WM- D)]
,[rDP (Returns from TPM WM to Chep)]
,[rEP (E to WM TPM)]
,[rDP (Other D <>WM to WM TPM)]
,[sPD - (Issues to SAMs)]
,[Other Non-Revenue Issues]
,[Non revenue Issues]
,
,[v]
,[x]
,[y (Based on P Ord )]
,[y (star) (Based on P Ord )]
,[zP]
,[zE]
,[zD]
,[z]
,[sPE (out)]
,[sPD (out)]
,
,[sP(out) Other ]
,[sPE (star) (out)]
,[sPD (star) (out)]
,
,[sP (star) (out) Other ]
,[dDE (out)]
,[d (out ]
,[d (out) Other ]
,[dDE (star) (out)]
,[d (star) (out)]
,[d (star) (out) Other ]
,[dDD (out)]
,[dDO (out)]
,[dED (out)]
,[dEE (out)]
,[dEO (out)]
,[dOD (out)]
,[dOE (out)]
,[dOO (out)]
,[tEE (out)]
,[tDD (out)]
,[tDE (out)]
,[tED (out)]
,[tED (Unknown)]
,[tED (Variance)]
,[tED (Out of Pool)]
,[t flow (out) Other -excl Admin]
,[tDO (out)]
,[tEO (out)]
,[tOD (out)]
,[-tEO (tOE(out))]
,[tOO (out)]
,[rEP (out)]
,[rDP (out)]
,[r (out)]
,[r (out) Other ]
,[rEP (star) (out)]
,[rDP (star) (out)]
,[r (star) (out)]
,[r (star) (out) Other ]
,[qE (in) - (sPE (in) (incl y))]
,[qD (in) - (sPD (in) (incl y))]
,
,[sP (in) Other ]
,[qE (star) (in) - (sPE (star) (in) (incl y))]
,[qD (star) (in) - (sPD (star) (in) (incl y))]
,
,[sP (star) (in) Other ]
,[dDE (in)]
,[dDE (in) (Other)]
,[dDE (star) (in)]
,[dDE (star) (in) (Other)]
,[dDO (in)]
,[dED (in)]
,[dEE (in)]
,[dDD (in)]
,[dEO (in)]
,[dOD (in)]
,[dOE (in)]
,[dOO (in)]
,[tEE (in)]
,[tDD (in)]
,[tDE (in)]
,[tED (in)]
,[t flow (in) Other ]
,[tDO (in)]
,[tEO (in)]
,[tOE (in)]
,[tOD (in)]
,[tOO (in)]
,[rEP (in)]
,[rDP (in)]
,[r (in)]
,[r (in) Other ]
,[rEP (star) (in)]
,[rDP (star) (in)]
,[r (star) (in)]
,[r (star) (in) Other ]
,[Admin to Emitter]
,[Admin to Distributors]
,[Admin to Customer Other ]
,[Admin to Plant]
,[Material Transfer]
,[Plant Stock Transfer]
,[Customer Stock Transfer]
,[Admin Stock Transfer]
,[Plant to Admin Flow (excl Scrap)]
,[w Unidentified returns]
,[Go Live Write Off Write On]
,[Total Posting Quantity]
,[Others (Not Captured)]
,
,
,
,
,[Processed Stock (Change)]
,[Processed Stock (Balance)]
,Loaddate
,[Month]
FROM [I_FINAL].[Stag].[IMETA_Global_Field_Flows_Staging2]
AS FF WITH (NOLOCK)
) AS FF
UNPIVOT
([Metric Value] FOR [Metric] IN
([sPD (Wal-Mart reissue)],[rDP (Wal-Mart returns to TPM WM)],,[v (Chep Plant to TPM WM)],[sPQ (WM TPM to E & D)],[sPD (Issues to WM- D)],[rDP (Returns from TPM WM to Chep)],[rEP (E to WM TPM)],[rDP (Other D <>WM to WM TPM)],[sPD - (Issues to SAMs)],[Other Non-Revenue Issues],[Non revenue Issues],,[v],[x],[y (Based on P Ord )],[y (star) (Based on P Ord )],[zP],[zE],[zD],[z],[sPE (out)],[sPD (out)],,[sP(out) Other ],[sPE (star) (out)],[sPD (star) (out)],,[sP (star) (out) Other ],[dDE (out)],[d (out ],[d (out) Other ],[dDE (star) (out)],[d (star) (out)],[d (star) (out) Other ],[dDD (out)],[dDO (out)],[dED (out)],[dEE (out)],[dEO (out)],[dOD (out)],[dOE (out)],[dOO (out)],[tEE (out)],[tDD (out)],[tDE (out)],[tED (out)],[tED (Unknown)],[tED (Variance)],[tED (Out of Pool)],[t flow (out) Other -excl Admin],[tDO (out)],[tEO (out)],[tOD (out)],[-tEO (tOE(out))],[tOO (out)],[rEP (out)],[rDP (out)],[r (out)],[r (out) Other ],[rEP (star) (out)],[rDP (star) (out)],[r (star) (out)],[r (star) (out) Other ],[qE (in) - (sPE (in) (incl y))],[qD (in) - (sPD (in) (incl y))],,[sP (in) Other ],[qE (star) (in) - (sPE (star) (in) (incl y))],[qD (star) (in) - (sPD (star) (in) (incl y))],,[sP (star) (in) Other ],[dDE (in)],[dDE (in) (Other)],[dDE (star) (in)],[dDE (star) (in) (Other)],[dDO (in)],[dED (in)],[dEE (in)],[dDD (in)],[dEO (in)],[dOD (in)],[dOE (in)],[dOO (in)],[tEE (in)],[tDD (in)],[tDE (in)],[tED (in)],[t flow (in) Other ],[tDO (in)],[tEO (in)],[tOE (in)],[tOD (in)],[tOO (in)],[rEP (in)],[rDP (in)],[r (in)],[r (in) Other ],[rEP (star) (in)],[rDP (star) (in)],[r (star) (in)],[r (star) (in) Other ],[Admin to Emitter],[Admin to Distributors],[Admin to Customer Other ],[Admin to Plant],[Material Transfer],[Plant Stock Transfer],[Customer Stock Transfer],[Admin Stock Transfer],[Plant to Admin Flow (excl Scrap)],[w Unidentified returns],[Go Live Write Off Write On],[Total Posting Quantity],[Others (Not Captured)],,,,,[Processed Stock (Change)],[Processed Stock (Balance)])
) AS FF
WHERE
LEN(FF.[Metric Value]) <> 0
Script Task C# Code for conversion:
#region Help: Introduction to the script task
/* The Script Task allows you to perform virtually any operation that can be accomplished in
* a .Net application within the context of an Integration Services control flow.
*
* Expand the other regions which have "Help" prefixes for examples of specific ways to use
* Integration Services features within this script task. */
#endregion
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Data.OleDb;
using System.IO;
#endregion
namespace ST_f3a71be1589b40b0a4b219ee358ea033
{
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region Help: Using Integration Services variables and parameters in a script
/* To use a variable in this script, first ensure that the variable has been added to
* either the list contained in the ReadOnlyVariables property or the list contained in
* the ReadWriteVariables property of this script task, according to whether or not your
* code needs to write to the variable. To add the variable, save this script, close this instance of
* Visual Studio, and update the ReadOnlyVariables and
* ReadWriteVariables properties in the Script Transformation Editor window.
* To use a parameter in this script, follow the same steps. Parameters are always read-only.
*
* Example of reading from a variable:
* DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
*
* Example of writing to a variable:
* Dts.Variables["User::myStringVariable"].Value = "new value";
*
* Example of reading from a package parameter:
* int batchId = (int) Dts.Variables["$Package::batchId"].Value;
*
* Example of reading from a project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].Value;
*
* Example of reading from a sensitive project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
* */
#endregion
#region Help: Firing Integration Services events from a script
/* This script task can fire events for logging purposes.
*
* Example of firing an error event:
* Dts.Events.FireError(18, "Process Values", "Bad value", "", 0);
*
* Example of firing an information event:
* Dts.Events.FireInformation(3, "Process Values", "Processing has started", "", 0, ref fireAgain)
*
* Example of firing a warning event:
* Dts.Events.FireWarning(14, "Process Values", "No values received for input", "", 0);
* */
#endregion
#region Help: Using Integration Services connection managers in a script
/* Some types of connection managers can be used in this script task. See the topic
* "Working with Connection Managers Programatically" for details.
*
* Example of using an ADO.Net connection manager:
* object rawConnection = Dts.Connections["Sales DB"].AcquireConnection(Dts.Transaction);
* SqlConnection myADONETConnection = (SqlConnection)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Sales DB"].ReleaseConnection(rawConnection);
*
* Example of using a File connection manager
* object rawConnection = Dts.Connections["Prices.zip"].AcquireConnection(Dts.Transaction);
* string filePath = (string)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Prices.zip"].ReleaseConnection(rawConnection);
* */
#endregion
/// <summary>
/// This method is called when this script task executes in the control flow.
/// Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
/// To open Help, press F1.
/// </summary>
public void Main()
{
// TODO: Add your code here
string ConnString;
ConnString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + Dts.Variables["User::Var_ExcelFilePath_1"].Value.ToString() +
";Extended Properties=\"Excel 12.0;HDR=No;IMEX=1\";";
var conn = new OleDbConnection(ConnString);
conn.Open();
string query = "SELECT * FROM [Global Field Flows$]";
var command = new OleDbCommand(query, conn);
OleDbDataAdapter adap = new OleDbDataAdapter(command);
var datatable = new DataTable();
adap.Fill(datatable);
//Create csv File
using (var sw = new StreamWriter(Dts.Variables["User::Var_CsvFilePath_1"].Value.ToString()))
{
for (int row = 0; row < datatable.Rows.Count; row++)
{
var strRow = "";
for (int col = 0; col < datatable.Columns.Count; col++)
{
strRow += datatable.Rows[row][col].ToString() + "|";
}
//remove last , from row
strRow = strRow.Remove(strRow.Length - 1);
//write row to file
sw.WriteLine(strRow);
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
October 7, 2024 at 5:57 pm
if your C# code above is on your "Data source" on the dataflow Image that is your issue - a Datasource HAS to output a datarow - not write to a CSV file and expect it to magically appear as a row on the output of a data source.
plenty of examples on google of how to generate a CSV file (which you have above) - that would be for a script component.
also plenty of examples of reading excel as part of a Data source step on a DataFlow - search for these and adapt your code to do what needs to be done.
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply