Cozyroc Parallel Loop Task and the Foreach Loop File Enumerator

Question

Cozyroc Parallel Loop Task and the Foreach Loop File Enumerator

Paul Hernández

SSCarpal Tunnel

Points: 4961
More actions
February 8, 2013 at 6:15 am

#144214

Hi friends,
I'm trying to improve the the performance of a foreach loop task, that reads around 20.000 pdf files each day and registers them in a log table.
I found the Cozyroc Parallel Task but I'm not able to make it work.
I found a couple of Sample Packages, but one uses an Item Enumerator and the other a ADO Net Enumerator. Both declare a variable called "Iter" twice, one with a package scope and the other with a foreach loop scope. I soppose the name of the variable is not relevant.
This Iter variable is map as the foreach loop iterator. In my case, the file enumerator return to the enumeration variable the file name, in a particular format depending on the selected values (Fully qualify, name and extension, etc.).
It seems like the foreach and parallel loop need a "clean variable" to iterate, and I'm wondering if it´s possible to use these two components to improve the file iteration.
Any comment would be appreciated.
Paul Hernández

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply

Phil Parkin SSC Guru Points: 246960 More actions · Answer 1

Please explain what you mean by "registers them in a log table" - are you actually processing the files or just recording their names?

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

Paul Hernández SSCarpal Tunnel Points: 4961 More actions · Answer 2

Hi Phil,

Thanks for your quick response.

I need only to log the filename, path and modify date in a table.

Background: I receive digital card request in pdf format. The name of the file is a barcode containing a prefix, the card number and a checksum. I need to extract the card number, look up the contact_Id in a Microsoft Dynamics CRM. If the contact_Id is found I move the file to a specific directory, if move to another. Finally I update a custom CRM entity with the file name, path, modify date and contact_Id (only for the files that matched a contact).

I made a design that works, but I would be nice if I can improve the process.

Kind Regards,

Paul Hernández

Phil Parkin SSC Guru Points: 246960 More actions · Answer 3

Have you considered a pure (ish!) SQL solution?

Haven't got much time right now, but have a look here[/url] & see whether it might be useful.

The absence of evidence is not evidence of absence
- Martin Rees
The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
- Phil Parkin

Paul Hernández SSCarpal Tunnel Points: 4961 More actions · Answer 4

Thanks so much again Phil,

I followed your suggestion and that makes the difference. I reduced the execution time from more than one hour using the foreach file enumerator, to just a few seconds, around 15 seconds. I just needed to adjust the code in your link to meet my requirements, because I need not only the file name but also the modified date, but at the end it works like a charm.

Kind Regards

Paul Hernández