Big Data Problems

Question

Big Data Problems

Steve Jones - SSC Editor

SSC Guru

Points: 736255
More actions
September 9, 2015 at 11:44 pm

#320515

Comments posted to this topic are about the item Big Data Problems

Viewing 13 posts - 1 through 12 (of 12 total)

You must be logged in to reply to this topic. Login to reply

akljfhnlaflkj SSC Guru Points: 76202 More actions · Answer 1

I love this piece, thanks Steve. Part of my job is just looking at the data. I look for things that jump out at me as possible problems. So I guess what I am doing is trying to see if there is a signal within the noise. I'll use that phrase in the future. Thanks again.

kpatel2 Grasshopper Points: 13 More actions · Answer 2

Hello Steve, Thanks for "simple words to describe complex issue".

Jeff Moden SSC Guru Points: 1004356 More actions · Answer 3

I think the biggest problem that people have with the analysis of any size data is that they have some preconceived notion of what they want to see rather than what the data is telling them. It's why I frequently refer to BI as the world's greatest oxymoron. 😀

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

cbeels 84210 SSC Journeyman Points: 84 More actions · Answer 4

Steve said:

I wish there were more depth since many of these topics could be the subject of a book all by themselves.

It's worth pointing out that they are indeed a book, Nate Silver's The Signal and the Noise. It's a very engaging read.

http://www.amazon.com/The-Signal-Noise-Predictions-Fail-but/dp/0143125087

Mitch C. Stein SSChasing Mays Points: 658 More actions · Answer 5

I am fascinated with the paradoxes businesses are willing to embrace right now (Ex: everyone is supposed to rush to put everything in the cloud at the exact moment providers are just starting to gouge for moving data up and down). Your point about understanding has been one of the ones that's stuck in my craw for the last several years.

Business seems at a rush to embrace outsourcing at a time when understanding what makes data meaningful is a fruit that is truly ripening. The momentum of outsourcing seems to be fueled in part by the idea that paying for someone who resides in-house and understands what is meaningful to the organization is no longer necessary. I think the magic that is supposed to make the situation workable resides in the idea that once IT is out of the end-user's way, they will finally be able to really get things done on their own. Yet, corporate landholders need a proficient shepherd to help them determine which are their sheep and which are merely bushes with white flowers. More than ever, the dust of the technologically capable is being eaten by those who are only pretending to understand. The resentment towards those nerds that do "know things" may also be part of the trend to separate the function to some other office the cool kids don't have to actually see anymore.

Does anyone think that this is shortsighted and that those building strong teams internally will later be acknowledged as the ones who built an advantage?

Steve Jones - SSC Editor SSC Guru Points: 736255 More actions · Answer 6

Jeff Moden (9/10/2015)
I think the biggest problem that people have with the analysis of any size data is that they have some preconceived notion of what they want to see rather than what the data is telling them. It's why I frequently refer to BI as the world's greatest oxymoron. 😀

I hear you say this a lot, Jeff, but I think that's what should happen. You should have some hypothesis as you look at the data. Otherwise how can you tell in which direction to go and ask questions?

Certainly some people want to cling to their idea, regardless of what the data says, but just as many people, in my experience, revise their belief as they learn more from analysis.

Steve Jones - SSC Editor SSC Guru Points: 736255 More actions · Answer 7

cbeels 84210 (9/10/2015)
Steve said:
I wish there were more depth since many of these topics could be the subject of a book all by themselves.
It's worth pointing out that they are indeed a book, Nate Silver's The Signal and the Noise. It's a very engaging read.
http://www.amazon.com/The-Signal-Noise-Predictions-Fail-but/dp/0143125087

Thanks. I'll have to pick it up.

Steve Jones - SSC Editor SSC Guru Points: 736255 More actions · Answer 8

mitchellcstein (9/10/2015)
I am fascinated with the paradoxes businesses are willing to embrace right now (Ex: everyone is supposed to rush to put everything in the cloud at the exact moment providers are just starting to gouge for moving data up and down). Your point about understanding has been one of the ones that's stuck in my craw for the last several years.
Business seems at a rush to embrace outsourcing at a time when understanding what makes data meaningful is a fruit that is truly ripening. The momentum of outsourcing seems to be fueled in part by the idea that paying for someone who resides in-house and understands what is meaningful to the organization is no longer necessary. I think the magic that is supposed to make the situation workable resides in the idea that once IT is out of the end-user's way, they will finally be able to really get things done on their own. Yet, corporate landholders need a proficient shepherd to help them determine which are their sheep and which are merely bushes with white flowers. More than ever, the dust of the technologically capable is being eaten by those who are only pretending to understand. The resentment towards those nerds that do "know things" may also be part of the trend to separate the function to some other office the cool kids don't have to actually see anymore.
Does anyone think that this is shortsighted and that those building strong teams internally will later be acknowledged as the ones who built an advantage?

I think you're conflating two things. Embracing outside consultants and embracing outsouring. Neither necessarily has anything to do with the other.

Outsourcing has been looked at my entire career, in a variety of industries. This is almost always a cost saving measure. It does devalue the internal knowledge that your employees have, and I do think it can be a short-sighted decision. It really varies depending on the functions being outsourced.

In terms of looking at knowledge outside the organization as more valuable than that within has been around since business has existed. When managers and executives do not have faith in their employees, or they are seduced by the prospect of a better decisions by a consultant, they often listen there.

However I think some of the internal knowledge of employees is overstated. We have employees that think they know more than they do. We also have plenty of managers looking for excuses to do or not make a decision, or to pay someone else to make the decision (and take some of the blame). It's a very complex topic.

If your managers don't listen to you, why? What are you not doing that you can control? I have seen many people fail to make a change because their presentation is poor, their approach off-putting. A lot of your impact comes from your skill, and how you present that skill.

Mitch C. Stein SSChasing Mays Points: 658 More actions · Answer 9

Steve Jones - SSC Editor (9/10/2015)
I think you're conflating two things. Embracing outside consultants and embracing outsouring. Neither necessarily has anything to do with the other.
Outsourcing has been looked at my entire career, in a variety of industries. This is almost always a cost saving measure. It does devalue the internal knowledge that your employees have, and I do think it can be a short-sighted decision. It really varies depending on the functions being outsourced.
In terms of looking at knowledge outside the organization as more valuable than that within has been around since business has existed. When managers and executives do not have faith in their employees, or they are seduced by the prospect of a better decisions by a consultant, they often listen there.
However I think some of the internal knowledge of employees is overstated. We have employees that think they know more than they do. We also have plenty of managers looking for excuses to do or not make a decision, or to pay someone else to make the decision (and take some of the blame). It's a very complex topic.
If your managers don't listen to you, why? What are you not doing that you can control? I have seen many people fail to make a change because their presentation is poor, their approach off-putting. A lot of your impact comes from your skill, and how you present that skill.

I am lucky to have managers who do listen to me, because I have been fortunate enough to land positions with managers who strive for knowledgeable employees. I have also held positions where I am not sure to what degree my own presentation prevented the easily provable value of knowledge to be understood.

However, the number of conversations online involving moaning over the scattering of good IT teams to the wind does speak a touch to what I am saying. I think that advances have made the ability to spit out canned outsource solutions far easier than in the past. Remote work is far easier than in the past. There's room for every approach, and the future will tell us which had the greatest advantage.

chrisn-585491 SSCoach Points: 16006 More actions · Answer 10

If your managers don't listen to you, why? What are you not doing that you can control? I have seen many people fail to make a change because their presentation is poor, their approach off-putting. A lot of your impact comes from your skill, and how you present that skill.

It doesn't matter how good an engineering staff you have, if upper management is clueless about technology and makes bad decisions.

See HP and the last 4 CEOs. Now they are laying off thousands and will "save the company" through more "RightSourcing". Both H&P are probably spinning in their graves enough to effect the earths rotation.

xsevensinzx One Orange Chip Points: 25560 More actions · Answer 11

Big problem for me is knowing what the end user wants. Another is preventing them from pulling huge datasets into a client application and letting me do the work for them on the backend.

I don't have too many issues with storage or access.

Gary Varga SSC Guru Points: 82166 More actions · Answer 12

Steve Jones - SSC Editor (9/10/2015)
Jeff Moden (9/10/2015)
I think the biggest problem that people have with the analysis of any size data is that they have some preconceived notion of what they want to see rather than what the data is telling them. It's why I frequently refer to BI as the world's greatest oxymoron. 😀
I hear you say this a lot, Jeff, but I think that's what should happen. You should have some hypothesis as you look at the data...

Yes but also try to disprove ones own hypothesis. Demonstrating a hypothesis is only as good as the attempts to disprove it.

Also, as already said, analysis should not be limited to preconceived ideas.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!