The Struggles to Keep Up with Growing Systems

Question

Post reply

The Struggles to Keep Up with Growing Systems

Steve Jones - SSC Editor

SSC Guru

Points: 736478
More actions
March 3, 2023 at 12:00 am

#4154755

Comments posted to this topic are about the item The Struggles to Keep Up with Growing Systems

Viewing 4 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply

Jeff Moden SSC Guru Points: 1004406 More actions · Answer 1

NM

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

David.Poole SSC Guru Points: 76035 More actions · Answer 2

"But its not web scale". Well neither is your market, or your share of the market. Your market is mature rather than emerging so that isn't going to change any time soon. When it does change it is likely to be in decline.

Years ago I joked in response to someone extolling the power of the latest and greatest hardware. "I'm sure normal development practises will bleed off any excess performance". Many a true word said in jest.

In the data world you hear people talking about the "massive amount of data we have to process". Spinning up Spark clusters to process what they consider to be a vast amount of data. Or, what a modern DBA would call a sod allth of data. Don't misunderstand me, Apache Spark is an impressive piece of engineering and once you start working in the high TB range it is a very useful tool. When you are in the sub-TB range it is likely to be overkill.

There seems to be received wisdom that what an organisation needs in its IT infrastructure is micro-services, NOSQL Dbs, Elastic Search, Kubernetes, complex orchestration tools and God knows what else. All that comes at a cost and I think parts of the moderns stack exist to demand that other parts of the modern stack also exist.

Does it work? Well at a cost, yes, but is it necessary for what you actually have to do and at the scale you operate?

When I look at tools I ask

What does this do that I couldn't do before?
For the stuff I already do, does this make it easier?
What is my return on investment for adopting this?

There are tools for which those questions get strong positive answers, but there is complexity and necessary complexity and frankly the former is in the majority and is our fault.

LinkedIn Profile

Steve Jones - SSC Editor SSC Guru Points: 736478 More actions · Answer 3

We saw some of this years ago with Hadoop. Neat tech, still in use underneath lots of modern stuff, but far too few people needed "big data" to make this work. Most of us still have GB, mostly sub 100GB systems. TB are common, but modern SQL, if written well, can usually handle query processing in low TBs.