Post reply

Best practices for Data Integration

  • October 27, 2008 at 2:50 pm

    #127204

    We get nightly data extracts from the databases hosted at different service provider sites through FTP in flat file format.

    Existing system can be unstable during FTP downloads at times and data quality is also questionable.

    Now we have an opportunity to modify the whole process.

    Would you be able to share your best practices for data integration in batch format. Would it be an overkill to have a VPN connection with an API. How about using XML file to improve data quality.

    Thanks for sharing your ideas...

  • October 28, 2008 at 7:05 am

    #890591

    If you want to transfer your data securely I would advise that you setup FTPS or SFTP transfers. You can find SSIS components for it here. Setting up VPN is too heavy in my opinion.

    Regarding the data quality, I don't know how XML format will help you. The data quality depends on your provider, not the transfer format you have in use. What XML format can bring is more more metadata , but keep in mind your data size will increase and XML parsing is much slower process.

    ---
    SSIS Tasks Components Scripts Services | http://www.cozyroc.com/

Viewing 2 posts - 1 through 1 (of 1 total)

You must be logged in to reply to this topic. Login to reply