How to pull data from an Internet URL

• Brian Carlson

  SSCarpal Tunnel

  Points: 4170

  More actions

  March 18, 2009 at 7:35 am

  #127170

  I have a dynamic URL that will provide me with public information that I need to download from the Internet into my database. I'm using SSIS but am not sure how to proceed. This URL is to a .csv file, it is not a web service. I can get a single file to work with the Flat File Connection Source but what it does is automatically download it to the Temporary Internet Files directory and then connect to it.

  I have 500 of these to do and plan on putting it in a ForEach Loop. If anybody could suggest a way to do this I would appreciate it.

  For reference, here is the URL.

  http://ichart.finance.yahoo.com/table.csv?s=HD&a=00&b=23&c=1960&d=02&e=17&f=2009&g=d&ignore=.csv

  The s=HD is the portion that will be replaced. I'm trying to download the S&P 500 historical data to do some data mining.

  Thanks,

  Brian.

• Brian Carlson

  SSCarpal Tunnel

  Points: 4170

  More actions

  March 18, 2009 at 9:11 am

  #961998

  I found a link for this.

  http://sqlblogcasts.com/blogs/greenmtnsun/archive/2007/09/06/http-authenticatoin-download-of-file-in-ssis-to-csv-file.aspx

• Frank Bazan

  SSCertifiable

  Points: 5378

  More actions

  March 20, 2009 at 3:19 am

  #963374

  Hi Brian,

  I blogged[/url] about this a little while back as I couldn't find much info on it at the time. It may help you too. You'd need to manipulate the variable for the changing URL but it should work.

  Hope this helps

  Kindest Regards,

  Frank Bazan