January 14, 2011 at 9:11 am
Being an OLTP developer and not an actual DBA, I really don't see what the big problem is, it that I would expect personnel (DBA) resources and hardware to increase in number and speed with data requirements. From my perspective, I would imagine that a Lead DBA would spend most of his time directing subordinates rather then being solely responsible for the entire architecture. Of course, having a good BlackBerry, IPhone, or other smart communication device would be a requirement in directing 24/7 subordinates.
"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." -- Martin Fowler
January 14, 2011 at 9:18 am
Ron Kunce (1/14/2011)
I would expect personnel (DBA) resources and hardware to increase in number and speed with data requirements.
In the real world though this often not the case....particularly in this streamlined economy of today. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
January 14, 2011 at 9:48 am
I made the jump from OLTP DBA to DataWarehouse DBA 4 years ago and couldn't be happier. The BI system I support has a footprint of approximately 40 TerraBytes and actually feeds an even larger downstream system around 2 PetaBytes. There have been two keys to my success as DW DBA: 1) A generous hardware budget which allows me to scale up when needed and 2) I automate every repetitve manual task no matter how complex the task may be.
Not everyone gets presented with a blank check for hardware, so I know that's quite unique. The automation of tasks is absolutely essential in order to maintain a worklife balance. Its always a big time investment up front to write and stabilize the code, but it always pays off in teh end. My children were 7 and 11 when I took this position and I have been able to remain active in their lives because of the 2 keys above.
January 14, 2011 at 11:21 am
I am on Zune BI. In our biggest data mart table we are getting a row whenever anyone with a Zune device, Zune PC Client or one on Xbox plays a song or watches a video. That adds up to significant numbers really fast, and we keep a three years tail.
Our biggest challenge is not how to handle the volumes in the data mart, it is design of the cubes that the data mart feeds. MDX queries against our Transactions and Usage cubes typically take minutes to process. In cubes we keep only "educated aggregates," e.g., if you play say twenty songs, in the cube we keep only the info that you played twenty songs. Each song you played also gets the daily count of plays bumped up by one.
My challenge is not how to handle big volumes (although they are in TBs allright) but how to reduce the data by aggregation so that my business users do not have to take a vacation while they wait for their query.
I could really use a solid state drive for our Dashboard cube, but I have to wait until proces drop to about one fifth of what they are now.
January 14, 2011 at 1:39 pm
A decade ago, the first large SQL Server project that I had ownership of was 300 GB with SQL Server 7.0 running on a 2 CPU server with 4 GB RAM. It was a datamart for a government agency, which received quarterly data feeds from about 65 US states and territories. My approach was to partition the tables horizontally, one for each state. Also, I partitioned the databases by period year. This worked out well, because submission files were imported seperately for each state, and reporting was typically done on a per state basis for a specific period year. There was a need for national level reporting and trend reports spanning multiple period years, and I handled that using summary tables which got inserted with "group by" resultsets after each table load. A collection of views wrapped it all up nicely. That was ten years ago, and today SQL Server provides partioned tables and materialized views, which would make the same project even more managable.
The bottom line is that it doesn't really matter how much data you have contained in the database or server instance, so long as you approach the problem as an architect in any other field would; first take time to understand how the data is segmented and how it will be used. Next, logically and physically partitioning out the data in a way that distributes the workload and facilitates the query requirements of the application.
100MB, 100GB, or 100TB: you just need a data warehousing strategy that's scalable.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
January 14, 2011 at 2:05 pm
The most important thing with databases that size is to not give the job to inexperienced database designers and developers. No amount of hardware resources will make up for a bad design.
Of course, you need plenty of hardware power. Especially disk storage, lots and lots of it. There are many issues that are not really a problem with smaller databases, so they have to be carefully planned in advance, and you can’t just wing it.
A vendor recently proposed selling us a system with databases totaling 10 TB in size and they will use Transparent Data Encryption, so there is no way to compress the backups. The backup guys were horrified when I said I would need in excess of 25 TB of space for backups.
January 14, 2011 at 2:30 pm
Michael Valentine Jones (1/14/2011)
There are many issues that are not really a problem with smaller databases, so they have to be carefully planned in advance, and you can’t just wing it.
True story Michael! Disaster Recovery scenarios take on a whole new dimension when you are dealing with VLDB databases. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
January 14, 2011 at 2:52 pm
I like working with large (1TB+) databases. It gives an opportunity to do some things that are not necessary in smaller databases. Small databases are nice to work with too for different reasons. Both help to round out the DBA.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
January 15, 2011 at 8:55 am
MB, TB, whatever. The hours relate more to how much of your life you are willing to commit. I spent more hours managing a single server single db installation than I did as one of 4 DBA's managing 53 servers with over 350 db's. It depends on the relationship to your employer. If you have to constantly give over-and-above without adequate resources and compensation, you need to examine yourself and your employer.
January 16, 2011 at 10:26 pm
I read all the comments and I think these people must be one of two things. Either very dumb or very brave. No thanks, I always wanted to become a dba until I was given a database to administer (you see, I am a developer) and the size of this db is a meazly 20 GB and I say that's more 'n enough. Actually I would rejoice if they get someone else but that would mean I am out of a job so I'll just shurrup and carry on.
I LOVE developing new applications with new databases not sit in a dark little room and just care for the db's all the time.
:-PManie Verster
Developer
Johannesburg
South Africa
I can do all things through Christ who strengthens me. - Holy Bible
I am a man of fixed and unbending principles, the first of which is to be flexible at all times. - Everett Mckinley Dirkson (Well, I am trying. - Manie Verster)
January 17, 2011 at 6:33 am
I LOVE developing new applications with new databases not sit in a dark little room and just care for the db's all the time.
Just remember though, without the data your bright shiny new application don't mean squat.:-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
January 17, 2011 at 10:31 am
rick-507511 (1/15/2011)
MB, TB, whatever. The hours relate more to how much of your life you are willing to commit. I spent more hours managing a single server single db installation than I did as one of 4 DBA's managing 53 servers with over 350 db's. It depends on the relationship to your employer. If you have to constantly give over-and-above without adequate resources and compensation, you need to examine yourself and your employer.
Excellent point.
Viewing 12 posts - 16 through 26 (of 26 total)
You must be logged in to reply to this topic. Login to reply