Data Sprawl

  • Steve:

    Didn't mean to imply it's easy to do, but rather if you've used *nix and windows and OSX, you move between them without an issue. There's no "reset".

    I do move with ease, but then I've been interested in operating systems since the start. I run Fedora on the desktop at home and Windows on the laptop. I've used up to 10 totally different OSes at a job at one time. I use a lot of OSS tools at the current Windows position since the budget is tight.

    As far as resumes, I haven't seen that. I've put a variety on my resume in the past and get calls (and interviews) for them all. I have tended to shrink down and limit what I put on the resume to what I want to do. However I have had friends with .NET and Java on their resumes, and get lots of calls. Maybe your industry or area?

    It's the area. DFW has a awful lot of Microsoft only shops and a few militant OSS shops. Hiring managers are looking for more narrow, focused skill sets, so us Jack-of-All trades types are a tough match for them and recruiters. We do have a semi-decent user group environment.

  • The last 3 companies I've talked to who are using the Microsoft stack for their applications have told me they are strongly considering NoSQL as an option. While they are looking for that Microsoft stack expert, they have a huge need for NoSQL in the future, just maybe not right this minute.

    I think this is primarily because they are already running so much in the so called Cloud and they have many more options in terms of database solutions they can spin up and use now. Each with their own pros and cons on handling issues they are facing with cheaper and at greater ease than possibly what they are currently using.

    Just looking at Amazon Web Services for reference with some of these companies. I have few options here for Hadoop. I can spin up a few EC2 instances where I can set up my own Hadoop cluster from a range of Hadoop vendors or I can use Amazon Elastic MapReduce (Amazon EMR) under a more managed service. This is on top of my already MySQL, Oracle, PostgreSQL, and Microsoft SQL Server choices. This is not counting Amazon Redshift, which is a fast, fully managed, petabyte-scale data warehouse solution for all.

    The possibilities are growing so vast these days.

  • xsevensinzx (9/23/2015)


    The last 3 companies I've talked to who are using the Microsoft stack for their applications have told me they are strongly considering NoSQL as an option. While they are looking for that Microsoft stack expert, they have a huge need for NoSQL in the future, just maybe not right this minute.

    I think this is primarily because they are already running so much in the so called Cloud and they have many more options in terms of database solutions they can spin up and use now. Each with their own pros and cons on handling issues they are facing with cheaper and at greater ease than possibly what they are currently using.

    Just looking at Amazon Web Services for reference with some of these companies. I have few options here for Hadoop. I can spin up a few EC2 instances where I can set up my own Hadoop cluster from a range of Hadoop vendors or I can use Amazon Elastic MapReduce (Amazon EMR) under a more managed service. This is on top of my already MySQL, Oracle, PostgreSQL, and Microsoft SQL Server choices. This is not counting Amazon Redshift, which is a fast, fully managed, petabyte-scale data warehouse solution for all.

    The possibilities are growing so vast these days.

    Give us some more detail about how much data, what type of data, and how it's accessed.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • Eric M Russell (9/23/2015)


    xsevensinzx (9/23/2015)


    The last 3 companies I've talked to who are using the Microsoft stack for their applications have told me they are strongly considering NoSQL as an option. While they are looking for that Microsoft stack expert, they have a huge need for NoSQL in the future, just maybe not right this minute.

    I think this is primarily because they are already running so much in the so called Cloud and they have many more options in terms of database solutions they can spin up and use now. Each with their own pros and cons on handling issues they are facing with cheaper and at greater ease than possibly what they are currently using.

    Just looking at Amazon Web Services for reference with some of these companies. I have few options here for Hadoop. I can spin up a few EC2 instances where I can set up my own Hadoop cluster from a range of Hadoop vendors or I can use Amazon Elastic MapReduce (Amazon EMR) under a more managed service. This is on top of my already MySQL, Oracle, PostgreSQL, and Microsoft SQL Server choices. This is not counting Amazon Redshift, which is a fast, fully managed, petabyte-scale data warehouse solution for all.

    The possibilities are growing so vast these days.

    Give us some more detail about how much data, what type of data, and how it's accessed.

    I didn't really dive into too much specifics on why mainly because currently, they are very much invested in SQL Server and that was the topic of conversation. The only reason it got brought up is because I talk a lot about how I'm trying to leverage NoSQL with my own data with SQL Server. And because I touch on the subject, that's when mentions of NoSQL are brought up as serious considerations on their end too.

    From my end, it's more about having a centralized data platform that can scale, has distributed processing, greater access to cheaper column store options as well another system to offload processes intense ETL systems that SQL Server is supporting.

  • a day late and a dollar short comment 🙂 i read steve's editorial while waiting for the kids at the bus stop, so this is the first time I had to respond. I came here to see how others felt, and it seems the ones who have commented feel similar to me. I love learning new technology. At work i'm now the go to person to figure out new tech, but i'm starting to have a love/hate relationship to learning new things. Maybe i'm getting old? Do I see myself as I saw the 40+ crowd when I first started out? Just waiting for retirement or to hit the lottery after being beaten to submission by the corporate culture. I'm at the point where I'm desperately trying to fight the desire to become one who fights against new technology to the point where I view all new stuff as a possibility. However 2 things drive me crazy

    1. always looking at the future. I can barely work on my today's problem without checking out 2016 and what's new in that and how will it help, etc...and as soon as 2016 hits, i gotta start looking at 20xx, and that's just not sql server, but any technology related to data.

    2. how many times do i spend so much time and energy to learn something new then a. someone thinks i'm the expert after downloading the install package or b. they put it on the shelf to never come back to it or worse, expect me to still be that expert after moving on with other things 6 months later.

    This isn't 1980, so i guess research depts are no longer the norm?

  • nawillia:

    Just waiting for retirement or to hit the lottery after being beaten to submission by the corporate culture.

    I'll be waiting for retirement by waiting for my databases to recover. IT minion "accidently" killed UPS power to half of my servers this morning...

    :angry:

    As for learning, I love learning, new or old technology, what ever the job requires or my curiosity desires...

    Although if I did win the lottery, my personal server room would be a bit more robust than my clients or employers...

    😛

  • For why many developers are looking to NoSQL, look no further than JSON. So many of the new tools and languages have built some support for JSON storage, and the serializing/deserializing. It's easy to work with for many developers. I think this is some of the reason that NoSQL solutions are intriguing.

    Is this an issue for your application? It depends. I think that some of these JSON stores will become problematic when applications look to evolve and reorder the data, or report on it in any scale. If that's not an issue, or if you have an application that won't change its data model (or that doesn't matter), then perhaps it's fine.

    I think that for new, greenfield projects, there's a temptation to use a simpler store to move quicker. Whether that works for the longer term, hard to say. There certainly are domains of problems where the NoSQL stores work well.

  • chrisn-585491 (9/22/2015)


    ...Additionally, listing OSS experience on a resume for a Windows position or vice-versa is a instant disqualification or red flag when apply for a good percentage of jobs. (Bitter pill of personal experience...) There's too many entrenched managers that still see the software world as religious battleground. (You can thank Ballmer, Stallman and the fan boys for that...)...

    Shame!!! I haven't come across that from either side of the interview table.

    Gaz

    -- Stop your grinnin' and drop your linen...they're everywhere!!!

  • Abrar Ahmad_ (9/22/2015)


    [font="Verdana"]Learning new technologies/tools but related ones is good but as per suitability of business/job...[/font]

    Definitely!!!

    We cannot apply a universal rule to learning except that keeping ones skills up is essential. Which skills? New ones? Existing ones? Closely related ones? Disparate ones?

    It so depends on so many factors. Keep moving (without overdoing it and burning out) is a sure way to keep relevant.

    Gaz

    -- Stop your grinnin' and drop your linen...they're everywhere!!!

  • Gary Varga (11/16/2015)


    Abrar Ahmad_ (9/22/2015)


    [font="Verdana"]Learning new technologies/tools but related ones is good but as per suitability of business/job...[/font]

    Definitely!!!

    We cannot apply a universal rule to learning except that keeping ones skills up is essential. Which skills? New ones? Existing ones? Closely related ones? Disparate ones?

    It so depends on so many factors. Keep moving (without overdoing it and burning out) is a sure way to keep relevant.

    After learning the full stack for your platform (T-SQL, SSIS, SSAS, SSRS), then it would focus on diagonal topics like Agile, SCRUM, UML, or even public speaking.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

  • I think it only appropriate that "data sprawl" accompany the "Tower of Babel" that a lot of people end up using.

    We had a similar problem way back in 1996 were we received a bunch of gobilty-gook from multiple different vendors (thank goodness, no XML back then for us). Rather than create a "Tower of Babel", we simply converted the data to a common format before importing it. And, we didn't only have the challenge of converting the data but we also had to work with different media types. FTP, CD, Diskette, downloadable text, and even reel-to-reel tapes (AT&T).

    It's all simple when you peel just one potato at a time and never let anything but potatoes into your "potato bin".

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.


    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • I agree, do your data scrubbing and transformation as near upstream as possible. Transform it once in staging, or transform it 1,000x in the application layer. Many organization don't get how that's the most important aspect of a data warehouse: achieving conformity of enterprise data.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

Viewing 12 posts - 16 through 26 (of 26 total)

You must be logged in to reply to this topic. Login to reply