These guys must have some database!
-
How'd you like to work on a PB-sized database?
I'm not sure how large this database is, but it's got to be up there. Actually you can imagine that there are probably multiple databases at work here with applications rolling information up or providing extensive searching capabilities. Still the figure quoted in the article is:
"Some intelligence data sources grow at a rate of four petabytes per month now, and the rate of growth is increasing"
Wow! And I thought the Microsoft showcase, stating 1GB growth a day, was impressive. 4PB a month?!?!!!?
I'm sure that much of this data is archived onto other media and it's no all online. Or at least not immediately available. There could be some sort of HSM system at work here to move images and other media onto optical or tape formats.
Still that's got to post some challenges as all VLDBs do: indexing, backups, query tuning, development and more. Imagine you are the guy that forgets to include a WHERE clause on a delete statement?
Do you work on your resume before or after you start the restore?
Steve Jones
-
Big is as big does. Back in 2000 I was involved in some work for the UK Land Registry. Back then their main system was growing at about 10 GB per day, and replicated to their hot standby site. Everything needed to be on line, so their main database was then about 15TB. They were talking to me about a new application... "It is small, the database will only be 2TB".
There are some big systems out there. Windows has a long way to go before it can cope with workloads considered normal on mainframes and *nix.
Original author: https://github.com/SQL-FineBuild/Common/wiki/ 1-click install and best practice configuration of SQL Server 2019, 2017 2016, 2014, 2012, 2008 R2, 2008 and 2005.
When I give food to the poor they call me a saint. When I ask why they are poor they call me a communist - Archbishop Hélder Câmara
-
I stand by my skewed belief that the value of data decreases exponentially as the scale of the data increases. Anything that increases at 4PB a month must be collecting information no one on the planet could possibly put in context. Better to have a few K of data generating quality information than a GB or more generating head-scratches. But...then...I never have been that good at reading seven dictionaries simultaneously either.
All silliness aside, I hope that the data is something large like digital imaging, because otherwise the scale of that data must make sensible retrieval and analysis almost impossible. And being a programmer, I fear any system where a machine is relied upon for analysis. Flawed logic, large data sets, and desperation make for a volatile mix.
This entire opinion is opinion, even the bits that refer to other opionions...but then that's only my opinion. 😉
-
Whatever it takes to prevent another 9/11.... moreover, search engines are in their infancy. Eventually with 128-bit or 256-bit operating systems, advanced search algorithms and the ubiquitous brilliance of the human mind, we may be able to detect threats to our way of life, which though imperfect, is the best we have, and in doing so, give researchers of various disciplines by-products to help advance knowledge, cure diseases and probe deeper.
-
A criminal attack does not change our *way* of life. The real threats to our way of life come from an ever intrusive governments who feel themselves justified in monitoring the private lives of citizens --that is the threat to our actual *way* of life.
In truth, also, huge databases, extensive snooping will not really make us more secure. There is far more data that is ambiguous or useless, and the sheer size of the process will produce thousands, likely millions of false positives for every true positive. Sort of Stalin's approach.
...
-- FORTRAN manual for Xerox Computers --
-
It's not so bad to forget the where clause, as long as you forget the commit.
jg
-
Brave thought Jay...BUT
When a former child molester moves in next door, you DO change your way of life by watching your children more closely, you might stop walking down the street to avoid gangs, that does change your way of life. You probably haven't seen anything in your lifetime that this government has done that has possibly hampered your lifestyle. In the US, criminal activity still rates as the number 1 enemy, not the government. Board a plane in the past few years? Why is all this security needed? Because of criminal activity, not the government. Brave thoughts, but hardly true in 2005. With your attitude, don't complain when the next 911 hits us, people will be happy to give up "rights" to be safe. Sure things can get out of hand, but it's not happened yet, we need to see that it doesn't. But never forget 911, and all the "people" chatting "kill" over some Danish cartoons. They're not the US government at all.
-
"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." ~Albert Einstein
I don't believe every response can be to Remember 9/11, the Alamo etc. But we must be ever diligent! Fool me once, shame on you, fool me twice shame on me.
-
(Adorning flame proof suit...)
"people will be happy to give up "rights" to be safe"
I have a real problem with this statement. Why is it that we, as US citizens are so ready to die fighting both at home and abroad in the name of protecting our freedoms and way of life, yet when we feel threatened we're so willing to just just lay those very same hard won freedoms at our government's feet? What is wrong with this picture? To me this is a gross affront to those that have sacrificed life and limb to protect those same freedoms for the rest of us.
I have read and listened to literally hours of conversation about the NSA spying scandal and a recurring theme always seems to be "Well, if you don't have anything to hide, you don't have anything to worry about". And each and every time I just want to shake and yell at the person saying this and tell them: "You are completely missing the whole point!". It's not about 'nothing to hide', it's about our goverment doing something that is in clear violation of the Constitution - the very same Constitution that they all took a sacred oath of office to uphold.
Taking off my shoes while being screened at an airport is a completely different thing than the possibility of having private conversations or correspondence illegally monitored by my own government. The right to air travel is not guaranteed by the Constitution. The right to privacy and protection from unwarranted searches is.
You cannot pretend to defend the law of this land unless you yourself are willing to uphold its principles while doing so.
My hovercraft is full of eels.
-
if you read a bit of bruce schneier's website, you'll see that the problem is not the government's lack of enough data, but rather that they have too much data to analyze efficiently. after the fact they can look up everything about everyone that was involved (or even nearby, or having a similar name to the terrorist's ex-roommate), but it doesn't help prevent terrorism.
"Those who would give up essential liberty to purchase a little temporary safety, deserve neither liberty nor safety." -- Benjamin Franklin
personally I'd rather deal with the miniscule chance that i'll be the victim of a terrorist plot over having my government strip away my rights a bit more every iteration.
-- Stephen Cook
-
> if you read a bit of bruce schneier's website, you'll see that the problem is not the government's lack of enough data, but rather that they have too much data to analyze efficiently.
Perfectly put. Back in graduate school, for one of my computer courses I wrote a research paper on information overload. One of the articles I used as a source said that the term "information overload" is a misnomer; "data overload" would be far more correct. Data is data, just that and no more. It doesn't become useful information until it's been appropriately analyzed. Even then, there might not be any information to be had, if the data is garbage in the first place.
Viewing 11 posts - 1 through 10 (of 10 total)
You must be logged in to reply to this topic. Login to reply