Can't upload all files from the S3 bucket

Question

Post reply

Can't upload all files from the S3 bucket

christi1711

SSC-Addicted

Points: 441

October 17, 2021 at 8:51 pm

#3941393

Hi all,

I have a package that uploaded files from S3 bucket.

Currently I need to add two another files. All previous files started with the phrase FA. The files that I need to add started from the phrase FARAN. And contain the name of the country in the format "Country_" or "_Counrty".

The S3 bucket has folders with the number of each month and the number of each day. All files are stored separately. For example:

2021/05/01 - Data of the 1st of May 2021

2021/05/02 - Data of the 2nd of May 2021

I changed C# code for this:


public void Main()
{
string accessKey = Convert.ToString(Dts.Variables["User::AWS_Access_Key"].Value);
string secretKey = Convert.ToString(Dts.Variables["User::AWS_Secret_Key"].Value);
string startKey = Convert.ToString(Dts.Variables["User::LastFileName"].Value);
string bucketName = Convert.ToString(Dts.Variables["User::S3_Bucket"].Value);
string countryCode = Convert.ToString(Dts.Variables["User::Country"].Value);
string sameFolder = startKey.Substring(0, startKey.LastIndexOf('/') + 1);
var sameFolderFiles = Convert.ToString(Dts.Variables["User::SameFolderFiles"].Value).Split(',');

try
{
AmazonS3Client s3Client = new AmazonS3Client(accessKey, secretKey, RegionEndpoint.EUWest1);
System.Collections.ArrayList files = new System.Collections.ArrayList();
ListObjectsRequest listRequest = new ListObjectsRequest
{
BucketName = bucketName,
//Prefix = countryCode + "/"
};

ListObjectsResponse listResponse;
do
{
// Get a list of objects
listResponse = s3Client.ListObjects(listRequest);
foreach (S3Object obj in listResponse.S3Objects)
{
if (
(((startKey.CompareTo(obj.Key) < 0) && (!obj.Key.StartsWith(sameFolder))) || ((obj.Key.StartsWith(sameFolder)) && (!sameFolderFiles.Contains(obj.Key))))
&& (obj.Key.LastIndexOf('/') != obj.Key.Length - 1)
&& ((obj.Key.Contains("_" + countryCode)) || obj.Key.Contains(countryCode + "_"))
&& ((obj.Key.Contains("FA")) || obj.Key.Contains("FARAN"))

)
files.Add(obj.Key.ToString());
}


listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);

Dts.Variables["User::Collection"].Value = files;
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
}
}?

I have two variables: LastFileName, SameFolderFiles.

This variables take data from the Audit using this code:

DECLARE @FILEPROCESSKEYPREVIOUS INT = 0
DECLARE @LASTFILENAME VARCHAR(500) = ''

SELECT TOP 1 
@FILEPROCESSKEYPREVIOUS = ISNULL(File_Processing_Key,0),
@LASTFILENAME = ISNULL([FileName], '')
FROM Audit_File 
WHERE Data_Source = 'Bucket'
AND Area = @Target_TableName
AND COUNTRY = @COUNTRYCODE 
ORDER BY 1 DESC

DECLARE @FILEPATH VARCHAR(500) = substring(@LASTFILENAME, 0, len(@LASTFILENAME) - charindex('/', reverse(@LASTFILENAME), 0) + 2)

DECLARE @SAMEFOLDERFILES VARCHAR(1000) = ''

SELECT @SAMEFOLDERFILES += ISNULL([FileName], '') + ','
FROM Audit_File (NOLOCK) 
WHERE Data_Source = 'Bucket'
AND Area = @Target_TableName
AND COUNTRY = @COUNTRYCODE 
AND [FileName] LIKE @FILEPATH + '%'

SELECT @FILEPROCESSKEYPREVIOUS, @LASTFILENAME, SUBSTRING(@SAMEFOLDERFILES,0,LEN(@SAMEFOLDERFILES))

I still haven't uploaded these files.

Could you help me with advice?

Viewing 6 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic. Login to reply

Phil Parkin SSC Guru Points: 246960 More actions · Answer 1

You've given us lengthy pieces of code and a description of what you are trying to do, which is good.

But you have not described the problem in enough detail. What error messages are you seeing? Have you stepped through the code in debug mode? Which line of code is not working as intended?

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

christi1711 SSC-Addicted Points: 441 More actions · Answer 2

Hi Phil,

yes, I debugged the code and see that when the file with the name FARAN comes the package is down.

Looks like this code doesn't work good.

Phil Parkin SSC Guru Points: 246960 More actions · Answer 3

christi1711 wrote:

Hi Phil,
yes, I debugged the code and see that when the file with the name FARAN comes the package is down. Looks like this code doesn't work good.

Did you read my previous message? It contains two questions, of which you have answered precisely zero.

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

christi1711 SSC-Addicted Points: 441 More actions · Answer 4

Hi Phil,

1. I see only the message "Script was failed".

2.Not for sure how correctly I can debug C# code. I put breakpoint in the main function in C# code, but after that Studio is closing..

Phil Parkin SSC Guru Points: 246960 More actions · Answer 5

Not sure whether I know how to debug C# 'correctly', but this is how I do it.

Edit the code in the Script Task and put a breakpoint near the beginning of the code. In your case, I'd put the breakpoint on the line (string accessKey = Convert.ToString(Dts.Variables["User::AWS_Access_Key"].Value);). Position your cursor anywhere on the line and press F9 to enter the breakpoint.
A red dot should appear in the left margin and the row of code should be permanently highlighted.
Close the Script Task. After closing the Script Task, a red dot should be shown on the Script Task icon on the Control Flow.
Run the package. When processing the Script Task, you should go into debug mode, with processing paused at the breakpoint created in (1). At this point, you have several options available in terms of how you step through the code. Of these, F10 is where I would start, this allows you to step through the code one line at a time. Press F10, and you should see execution advance to the next row.
One of these presses of F10 will result in an error – now you know which line of code has caused the error and you can home in on that.
Control should be passed to the 'Catch' section of the code. View the properties of the ex (exception) variable at that point to get more information about the error (particularly the 'innerException' property).

Start with this and see how you get on.

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin