January 12, 2014 at 8:11 am
I am struck in one point as we gets files in unix box and we don't have any windows server .
Issue - We get pipe'|' separated text file, usually this can be bcp in to table if the structure exactly matching with table .
Here structure is same but some of the rows having issue with row delimiters which cause overlapping to next line.
This file opens in excel perfectly no issue.
I need to find a solution how to fix this. This we cannot say forrmat issue as this issue with only on unix env. Is there any way we can use ssis ? please advice if any other solution.
January 12, 2014 at 8:18 am
Is there any way we can use ssis ?
To do what, precisely?
As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 12, 2014 at 8:47 am
Phil Parkin (1/12/2014)
Is there any way we can use ssis ?
To do what, precisely?
As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?
Sample data looks like below, for 3rd row rowdelimiter is not correct.
|USD|1542.52|14536|xxxx
||1542.52|14536|
EUR|1542.52|14536|xxxx
I m not sure how does SSIS Fit into Unix, please advice me if SSIS fits into unix.
I can create an ksh scripts ,is ssis can be called from ksh ?
January 12, 2014 at 9:00 am
sql crazy kid (1/12/2014)
Phil Parkin (1/12/2014)
Is there any way we can use ssis ?
To do what, precisely?
As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?
Sample data looks like below, for 3rd row rowdelimiter is not correct.
|USD|1542.52|14536|xxxx
||1542.52|14536|
EUR|1542.52|14536|xxxx
I m not sure how does SSIS Fit into Unix, please advice me if SSIS fits into unix.
I can create an ksh scripts ,is ssis can be called from ksh ?
Sorry but I know very little about Unix. But I'm pretty sure that SSIS can be used to process and push data into a Unix database, assuming the connectivity can be resolved.
But SSIS needs to be installed on a Windows server - is that a show-stopper for you? It's also quite expensive to justify if you are not already using SQL Server.
Does your sample data represent two or three rows of actual data?
If three rows, it looks like the column delimiters (not row delimiters) are incorrectly placed. That's just bad data and it should be fixed at source, not by you - in my opinion.
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 12, 2014 at 9:39 am
Phil Parkin (1/12/2014)
sql crazy kid (1/12/2014)
Phil Parkin (1/12/2014)
Is there any way we can use ssis ?
To do what, precisely?
As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?
Sample data looks like below, for 3rd row rowdelimiter is not correct.
|USD|1542.52|14536|xxxx
||1542.52|14536|
EUR|1542.52|14536|xxxx
I m not sure how does SSIS Fit into Unix, please advice me if SSIS fits into unix.
I can create an ksh scripts ,is ssis can be called from ksh ?
Sorry but I know very little about Unix. But I'm pretty sure that SSIS can be used to process and push data into a Unix database, assuming the connectivity can be resolved.
But SSIS needs to be installed on a Windows server - is that a show-stopper for you? It's also quite expensive to justify if you are not already using SQL Server.
Does your sample data represent two or three rows of actual data?
If three rows, it looks like the column delimiters (not row delimiters) are incorrectly placed. That's just bad data and it should be fixed at source, not by you - in my opinion.
Here is few more details
Backend - SQL server 2008 R2,SSIS installed
OS - Unix Env
We have many process running on unix ,connecting to sql server , BCP into sql tables , all are working fine .
Above sample data is a representation of actual data . But really there is a correction required on data file , but in case we are using this in windows env this not an issue . Either with SSIS or Informatica this works perfectly with no issue as we have column to column mapping . Since that we cannot ask to fix that issue.
January 12, 2014 at 9:57 am
Here is few more details
Backend - SQL server 2008 R2,SSIS installed
OS - Unix Env
We have many process running on unix ,connecting to sql server , BCP into sql tables , all are working fine .
Above sample data is a representation of actual data . But really there is a correction required on data file , but in case we are using this in windows env this not an issue . Either with SSIS or Informatica this works perfectly with no issue as we have column to column mapping . Since that we cannot ask to fix that issue.
OK. Then please answer my questions about your sample data. At the moment, it looks like:
c1 (blank),c2,c3,c4,c5
c1 (blank),c2 (blank),c3,c4,c5 (blank)
c1 (missing),c2,c3,c4,c5
My point? If columns can be missed out randomly from your source data, there is NO WAY ON EARTH to get it fixed by SSIS, which is lacking the necessary Paranormal Transformation.
If you are saying that only certain columns are missing, under certain circumstances, you need to explain the logic of what needs to be done please.
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 12, 2014 at 10:21 am
Phil Parkin (1/12/2014)
Here is few more details
Backend - SQL server 2008 R2,SSIS installed
OS - Unix Env
We have many process running on unix ,connecting to sql server , BCP into sql tables , all are working fine .
Above sample data is a representation of actual data . But really there is a correction required on data file , but in case we are using this in windows env this not an issue . Either with SSIS or Informatica this works perfectly with no issue as we have column to column mapping . Since that we cannot ask to fix that issue.
OK. Then please answer my questions about your sample data. At the moment, it looks like:
c1 (blank),c2,c3,c4,c5
c1 (blank),c2 (blank),c3,c4,c5 (blank)
c1 (missing),c2,c3,c4,c5
My point? If columns can be missed out randomly from your source data, there is NO WAY ON EARTH to get it fixed by SSIS, which is lacking the necessary Paranormal Transformation.
If you are saying that only certain columns are missing, under certain circumstances, you need to explain the logic of what needs to be done please.
Columns are not missing here . I have 5 column here ,looks like this.
[Just a sample data ,but this is the format we load]
col1 col2 col3 col4 col5
USD1542.5214536xxxx
1542.5214536
EUR 1542.5214536xxxx
January 12, 2014 at 10:50 am
Columns are not missing here . I have 5 column here ,looks like this.
[Just a sample data ,but this is the format we load]
col1 col2 col3 col4 col5
USD1542.5214536xxxx
1542.5214536
EUR 1542.5214536xxxx
What would this sample data look like if there were no problems with delimiters?
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 12, 2014 at 10:55 am
Phil Parkin (1/12/2014)
Columns are not missing here . I have 5 column here ,looks like this.
[Just a sample data ,but this is the format we load]
col1 col2 col3 col4 col5
USD1542.5214536xxxx
1542.5214536
EUR 1542.5214536xxxx
What would this sample data look like if there were no problems with delimiters?
I am getting output as like below when i do bcp command on unix.
C1 C2 C3 C4 C5
<NULL>USD 1542.5214536xxxx
<NULL> <NULL> 1542.5214536EUR|1542.52|14536|xxxx
January 12, 2014 at 10:57 am
sql crazy kid (1/12/2014)
Phil Parkin (1/12/2014)
Columns are not missing here . I have 5 column here ,looks like this.
[Just a sample data ,but this is the format we load]
col1 col2 col3 col4 col5
USD1542.5214536xxxx
1542.5214536
EUR 1542.5214536xxxx
What would this sample data look like if there were no problems with delimiters?
I am getting output as like below when i do bcp command on unix.
C1 C2 C3 C4 C5
USD1542.5214536xxxx
1542.5214536EUR|1542.52|14536|xxxx
I understand, but that is not the question I am asking.
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 12, 2014 at 11:00 am
Phil Parkin (1/12/2014)
sql crazy kid (1/12/2014)
Phil Parkin (1/12/2014)
Columns are not missing here . I have 5 column here ,looks like this.
[Just a sample data ,but this is the format we load]
col1 col2 col3 col4 col5
USD1542.5214536xxxx
1542.5214536
EUR 1542.5214536xxxx
What would this sample data look like if there were no problems with delimiters?
I am getting output as like below when i do bcp command on unix.
C1 C2 C3 C4 C5
USD1542.5214536xxxx
1542.5214536EUR|1542.52|14536|xxxx
I understand, but that is not the question I am asking.
If there is no problem,output would be
C1 C2 C3 C4 C5
NULLUSD1542.5214536xxxx
NULL NULL1542.5214536NULL
EUR NULL 1542.52 14536 xxxx
January 12, 2014 at 11:16 am
If there is no problem,output would be
C1 C2 C3 C4 C5
NULLUSD1542.5214536xxxx
NULL NULL1542.5214536NULL
EUR NULL 1542.52 14536 xxxx
Great, that helps.
Now my question is: why in the above is there a NULL after EUR in row 3?
That, surely, is a missing column, if the source is
EUR|1542.52|14536|xxxx
?
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 12, 2014 at 11:28 am
Phil Parkin (1/12/2014)
If there is no problem,output would be
C1 C2 C3 C4 C5
NULLUSD1542.5214536xxxx
NULL NULL1542.5214536NULL
EUR NULL 1542.52 14536 xxxx
Great, that helps.
Now my question is: why in the above is there a NULL after EUR in row 3?
That, surely, is a missing column, if the source is
EUR|1542.52|14536|xxxx
?
Thanks for getting into deeper
Few of the rows , first column coming as empty . But do you think in SSIS or Excel this will be a issue as there is a column delimiter to treat the 1st column as empty.
January 12, 2014 at 11:47 am
Thanks for getting into deeper
Few of the rows , first column coming as empty . But do you think in SSIS or Excel this will be a issue as there is a column delimiter to treat the 1st column as empty.
You've avoided my question again!
If your question can be restated as:
Sometimes there will be four columns in a row, sometimes five columns.
Will SSIS know which of the columns is missing?
The answer is no.
But if you know that there is a rule such as: if there are four columns in a row, add a fifth as Col1 and set its value to NULL, that can be done.
I'm going to sign off now, by saying that if you can come up with the necessary logical steps to process your file (and I don't think that you're there yet), it will be possible to implement those steps in SSIS. It might require some scripting.
The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin
January 13, 2014 at 10:16 am
Phil Parkin (1/12/2014)
Thanks for getting into deeper
Few of the rows , first column coming as empty . But do you think in SSIS or Excel this will be a issue as there is a column delimiter to treat the 1st column as empty.
You've avoided my question again!
If your question can be restated as:
Sometimes there will be four columns in a row, sometimes five columns.
Will SSIS know which of the columns is missing?
The answer is no.
But if you know that there is a rule such as: if there are four columns in a row, add a fifth as Col1 and set its value to NULL, that can be done.
I'm going to sign off now, by saying that if you can come up with the necessary logical steps to process your file (and I don't think that you're there yet), it will be possible to implement those steps in SSIS. It might require some scripting.
Thank you for your time and effort put in. As you said this is a data issue ,we are done is our end corrected file and bcp into the table. Another think this is hard to connect SSIS from Unix .
Viewing 15 posts - 1 through 14 (of 14 total)
You must be logged in to reply to this topic. Login to reply