May 11, 2015 at 9:24 pm
Comments posted to this topic are about the item Opening Up Data
May 12, 2015 at 12:28 am
My concern, as a developer who tries to keep abreast with the database development world, is that we are moving towards a time where there is so much more to learn in the database development world that it is going to be hard enough for data professionals to do so but near on impossible for those of us who it is merely a facet where our own areas have the same amount of diversity. Some developers specialize in areas and can limit what they have to keep in touch with, however, many are generalists. Exactly the same is true for DBAs who develop as well.
I don't see this changing but as a challenge that we have evolved into. Basically, I have to "suck it up".
My concern is that we will have some people who are stretched on a limited number of technologies so may become dangerous in the vast ocean of technologies being demanded.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
May 12, 2015 at 6:28 am
Yet Another DBA (5/12/2015)
@Gary: Thats the whole of IT and why the polygot approach seems to be popular with some. Its can be the mindset of I can juggle more balls than you, rather than I have more skills/tricks then you.
I get it and have thrived in such a culture to some degree, however, my concern is that there is a whole lot of people who do not and cannot cope.
They are most likely to be the sort who would only visit sites like these to find a solution to a single issue and never to return until their next problem.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
May 12, 2015 at 7:30 am
From article:
. I think SSIS will continue to be one of the most valuable tools in SQL Server (along with lots of demand for work), but it still needs improvement and enhancement to catch up to other ETL tools.
SSIS/SSDT needs a lot of improvement before I can recommend its use.
1. The issue where SSIS/SSDT will cache destination/source table metadata and will NOT release the cache without intervention is a show stopper for me. Nothing like changing something in a table and then having the tool you are using ignore the change. Making any schema changes to an existing SSIS/SSDT project requires careful QA to make sure that the tool actually is doing what you tell it to.
2. Having to use a wysiwyg type tool, that then requires you to assign variables in one place, put your SQL in another and then write C# code somewhere else (why do I need to write code for any of this?) is beyond crazy. There is code everywhere and the structure of SSIS/SSDT makes troubleshooting a massive pain in the butt.
3. Transforms/tools/widgets that just plainly don't work. I am looking at you merge. I can manually write a merge that will vastly out perform the merge in SSIS/SSDT. Truthfully, where I work, we ignore most of the SSIS/SSDT widgets and use stored procedures instead because we have found so many of the widgets to be poor.
I've said this before and I'll say it again, I can create an ETL 'package' in KSH/SQL faster than I (and I dare say many of the people that I work with) can create a SSIS/SSDT package and my 'package' will be more readable and easier to troubleshoot because you only have one or two pieces of code to look at instead of six or seven different places to look for to find the errors in SSIS/SSDT. My package just won't have the boxes and lines, which if you are an experienced ETL developer won't matter.
I agree about data portability.
Working in a health care setting, I can confirm that data between vendors/applications is not portable.
Even with standards like HL7 or X12, it can be very difficult to have two data sets 'talk' to each other.
May 12, 2015 at 7:30 am
Companies that originate large databases that would be of public interest (FaceBook, Amazon, NetFlix, etc.) do open up their data, but for a number of obvious reasons they prefer do it using an SaaS API calls, not massive data dumps. So in that way data is exchanged in a standardized way.
Also, data interchange (ETL) becomming more and more a specialized role all to itself. We don't really need to know JSON or R to be a SQL Server DBA or developer, even if our data originates from third parties. For example, even if our vendor catalog data originates from 'Johnson & Johnson', BMW, or 100 seperate vendors, we want the data in our own relational Products table. As the DBA or developer, we don't care what the original data format was. There's maybe only one guy in the IT department who needs to know that.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
May 12, 2015 at 8:50 am
podmate (5/12/2015)
SSIS/SSDT needs a lot of improvement before I can recommend its use.
Agree with the first half. There is work to be done. However, for many, many people, I think the tool does the job they need it to do.
May 12, 2015 at 8:51 am
I think for the DBAs/developers, who perform the ETL stuff, we do need to think about knowing JSON, XQuery, and more to help us transform data into formats that we can use. Or we need to consider the places where we can make a SOAP or other API call to just "get" data in real time.
It's an exciting time to work with data.
May 12, 2015 at 9:29 am
Steve Jones - SSC Editor (5/12/2015)
I think for the DBAs/developers, who perform the ETL stuff, we do need to think about knowing JSON, XQuery, and more to help us transform data into formats that we can use. Or we need to consider the places where we can make a SOAP or other API call to just "get" data in real time.It's an exciting time to work with data.
That's a great job, if you can find it. The problem is that in IT departments with more than 100 employees; roles, projects, and teams tend to be compartmentalized and the DBA is a spectator (if not a complete outsider) when it comes to the development and integration side of things.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
May 12, 2015 at 2:12 pm
I switched to the new job for more learning opportunities, so learning new / different things is very important to me. However, too much of a good thing is not good. I had gone to the following trainings in the past 12 months: DB2, MySQL, MongoDB, AWS (EC2 + RDS), Netezza, Linux, and Attunity coming up in 2 weeks. I have lost my focus. Things I would like to improve in existing SQL Server environments have been put on the back burners and neglected. I just don't have time to expand vertically. It is not fun.
May 12, 2015 at 2:21 pm
TrailRunner (5/12/2015)
I switched to the new job for more learning opportunities, so learning new / different things is very important to me. However, too much of a good thing is not good. I had gone to the following trainings in the past 12 months: DB2, MySQL, MongoDB, AWS (EC2 + RDS), Netezza, Linux, and Attunity coming up in 2 weeks. I have lost my focus. Things I would like to improve in existing SQL Server environments have been put on the back burners and neglected. I just don't have time to expand vertically. It is not fun.
Wow, it typically takes about a year to become proficient in (1) new platform, at least when filling a support role during the day and learning on the side or during downtime. Going from desert to floodplane is just life. Maybe if you're taking some good bootcamp training courses it will work out.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
May 13, 2015 at 7:29 am
TrailRunner (5/12/2015)
I switched to the new job for more learning opportunities, so learning new / different things is very important to me. However, too much of a good thing is not good. I had gone to the following trainings in the past 12 months: DB2, MySQL, MongoDB, AWS (EC2 + RDS), Netezza, Linux, and Attunity coming up in 2 weeks. I have lost my focus. Things I would like to improve in existing SQL Server environments have been put on the back burners and neglected. I just don't have time to expand vertically. It is not fun.
It has to stay fun. I hope you can find your way back to that.
May 13, 2015 at 7:55 am
Iwas Bornready (5/13/2015)
TrailRunner (5/12/2015)
I switched to the new job for more learning opportunities, so learning new / different things is very important to me. However, too much of a good thing is not good. I had gone to the following trainings in the past 12 months: DB2, MySQL, MongoDB, AWS (EC2 + RDS), Netezza, Linux, and Attunity coming up in 2 weeks. I have lost my focus. Things I would like to improve in existing SQL Server environments have been put on the back burners and neglected. I just don't have time to expand vertically. It is not fun.It has to stay fun. I hope you can find your way back to that.
I'd put IT development in the same category as teaching, art, acting, or professional sports; you have to really love and believe in what you do in your current position, there has to be a creative aspect to it, or else it begins to reflect negatively in your work.
To paraphrase Henry David Thoreau: Most IT folks lead lives of quiet desperation and go to the grave with that cool project they've always dreamed about still in them.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
May 13, 2015 at 10:11 am
Thanks guys, I totally agree with you. It has be fun with passion. Learning the Open Source side of things open up opportunities in the Internet companies and start-ups. But I have to keep up with SQL Server as well, as I do love the product. I guess that is life, something has to give. So the topic is that DBAs has to expand their knowledge outside of RDBMS or SQL Server, but what if your company gives you too much to learn and to support all at once that you can't keep up? Or may be there is no such thing as too much learning? One has to just suck it up?
May 13, 2015 at 10:46 am
TrailRunner,
You mentioned training for DB2, MySQL, MongoDB, AWS (EC2 + RDS), Netezza, Linux, and Attunity within the past year. Are all these platforms leveraged on the same project, or do you work for a consulting firm and they're just wanting to cover a lot of different bases?
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
Viewing 15 posts - 1 through 15 (of 18 total)
You must be logged in to reply to this topic. Login to reply