Post reply

A Whole Lot of Data

  • May 31, 2006 at 3:50 pm

    #168662

    I can tell you one thing: I'm glad I don't work for an ISP, especially these days as that industry seems to be becoming more and more regulated. It seems the FBI and US Attorney are urging ISPs to keep more data that could be used to check on users' activities.

    I don't know how many ISPs you've had in your life, but I've been using the Internet in various forms since 1990. In that time, 16 years, I've had 4 ISPs. One of them, Qwest, I've had twice because of moves, but if an ISP were looking to retain my history until a year after my account closed, that's a lot of data. Even a simple time stamp, an IP, and my accountID would occupy lots of rows. Add to that the fact that monitoring would be looking at all the http requests, not pages, and there's probably a million rows a month for just me!

    This isn't required yet, but the amount of traffic that goes across any particular ISP's network is staggering and I cannot imagine trying to somehow get this into a database. I think SQL Server 2005 is robust, but I'm not sure who wants to be the DBA to deal with all this data.

    If it were up to me I'd store it in flat files, straight linear logging and drop that on tape on a regular basis. Then if it were required in response to a subpoena, just give them a tape. And let the lawyers have fun loading it into a database and building queries.

    If they asked, I'd point them to Jamie Thomson and Conchango. Let those guys get the big bucks building SSIS packages to import and transform it.

    Steve Jones

  • June 1, 2006 at 10:22 am

    #641231

    This definately sounds like a poor idea to me.  For them to log enough information to actually identify a person and what that person was doing would be an insane amount of data.  The larger ISP's might be able to swing it on their own steam; but some of the really small ISP's would never be able to affort a storage system capable of that even if their users were significantly less.

    Also I don't care what flag they are pushing this under be it child porn or something equally bad.  In the end this would just turn into another method of getting our personal information.

Viewing 2 posts - 1 through 1 (of 1 total)

You must be logged in to reply to this topic. Login to reply