Realities of Predictive Analysis

Or for short, “Why you have to play the game”

“Don't tell me the odds”, cried out Han Solo just before doing something that seemed impossible. How on Coruscant did he do that? Well, as a certain director said about a certain floating door in a galaxy we all call home. ”It was in the script.”

The real world has no script

Are you surprised when there’s a 10% chance of rain and you get drenched leaving work? Or watching a sports ball game using super fancy stats where a player scores, even though they only were given a .7 % chance of that occurring during the match? What if that player scored on a rainy day where there was 0% chance of precipitation and was given a 0% likelihood of scoring that day. Should their score even count?

Of course it should and does, but how could the predictive analysis get it so wrong? For the clear reason that the only thing you can predict with 100% accuracy is the past. And to add to that argument, most people can’t even agree on exactly what happened in the past. (If you want some fun, watch or read a few documentaries on the same subject written by different people.)

As we get better and better with stats and sensors, we absolutely get better at predicting the future. Weather forecasts are far more dependable than they were even a few years ago. Predicting the value of stocks, markets, etc. is easier than ever. Game winners are typically picked before the game with startling accuracy. Yet there are always surprises. Why is this?

You have to play the game

I will admit, I tried doing some reading on this subject. I read (over) this article, and this one, and when I got here, I had learned one thing: there is a lot of math involved. But one thing they all tended to say is this: “we don’t have enough data”.

There are two general things I have noticed. Because we will never have enough sensors to monitor everything, there is no way we ever get enough data to know exactly what happens next. Especially not being able to know what is in the hearts and minds of human beings. Humans do things sometimes that seem supernatural. A catch that looked over their heads and already in the goal, “wait, they got that!?! The GOT THAT!” These same players also make mistakes they shouldn’t. Sometimes a top caliber player is gently handed a ball on the final play and all they had to do to win the game was not drop the ball. Any other player on the field, or even any person watching them from the stands, would not have failed to hold on to the ball. But this person did.

Why? If you could have answered that question before it happened, you would be rich beyond belief. Problem is that the player may have had a hand spasm. Or a sneeze coming on. Or to take it to another place altogether, the player may have been thinking “Yes, we are going to win right here! I am so great!” and took their eye off the most important thing (the ball, you probably guessed that!) It is also possible that as the play started, they got a little hungry… for pie.

“I want pie after the game. Maybe rhubarb. What the heck is rhubarb? It looks like celery, but they put it in sweets on Bake Off. I mean it is freaking purple. Is it purple? I think it is a bit more red. And was Paul Hollywood’s last name really Hollywood growing up? Wonder what his parents names are?” Etc.

Before you careen yourself off thinking about that last question and don’t finish reading my editorial. Yes, it apparently was.

What does this mean for me?

In my years supporting marketers, I was always amazed at how well you could predict what a group of people would do as a whole. Ask in the right way, and an expected percentage of people would typically answer positively. Pre-computers, they just knew to do the same thing because it worked before. Over time, as a method stopped working, they tweaked the formula and tried to adjust until they got the results they wanted.

On a large scale, it was easy to predict that a campaign would work. But predicting existing how ONE customer would behave on that one day you solicited from them? No way. They may have had a bad day. They may have needed work on their chimney that day (don’t try to sell me something today, by the way). Or they may have really hated your company for some random reason that you could not predict without a sensor in their brain (or at least access to everything they saw on TV and Social Media that month.)

So don’t let the stats tell you what you can and cannot do

Let them guide you, and help you understand your strengths and weaknesses, because that is what they are, a guide. A prediction that if you behave like you did last time, this is what will happen. There will never be a time when life is so programmed that statistics can predict that you will buy a product no matter what the salesperson says, no matter how great (or not so great) the product. Or that you are going to win the game no matter what, so why not just sit on the bench today and take a load off. At least not until all the people are replaced by robots... but that isn't predicted to happen for a while.

Stats do a great job of predicting the future because they have knowledge of the past, when you were likely working your hardest. They are not magic!

Louis Davidson (@drsql)

Join the debate, and respond to the editorial on the forums

DevOps in a Day is hitting the road. Join us for hands-on workshops hosted by Redgate’s Huxley Kendell and Microsoft Data Platform MVP, Grant Fritchey. We are visiting three cities: 11th June - Dublin 13th June - Glasgow 14th June - Manchester If you can't make it, don't fear, we will be touring more cities in September.

The Clean command resets a database to its initial state, before any Flyway migrations were applied. In other words, it empties the database. This can be useful for any development task that requires that you recreate the database structure, or for tearing down a test harness. It also allows you to try out experiments and alternative strategies within an isolated feature branch, and then reverse out of them.

Kendra Little is happy: I’ve spent a bit of time with Microsoft’s new database watcher tool for Azure SQL recently. There are a lot of things I…

Aaron Bertrand continues a series on threading: In part 2 of this series, I showed an example implementation of distributing a long-running workload in parallel, in…

In a world awash with data, the challenge lies in ...

The 13th cumulative update release for SQL Server 2022 RTM is now available for download at the Microsoft Downloads site. Please note that registration is no longer required to...

Manvendra Singh talks about backups: This article will explain backup storage redundancy for Azure Cosmos DB. Backups are a critical feature to keep copies of…

This article will explain backup storage redundancy for Azure Cosmos DB. Backups are a critical feature to keep copies of our data to ensure data protection and recoverability in...

Today, I worked on a service request that our cust...

Data API builder (DAB) is Microsoft's open-source engine designed to expose Azure databases to client applications and customers through secure, feature-rich REST and GraphQL endpoints. It's cross-platform, language-independent, and...

I was chatting with a client’s DBA about this thought-provoking blog post about data governance in the age of generative AI. The DBA’s concern was, “What if we hook...

On the era of SaaS, I see many companies hiring a lot of SaaS services. They create a dependency on them for their production operations. It may seems the...

I got a really interesting comment on a blog post talking about query optimization. The context was that working with dates is much easier for a query engine than...

Eitan Blumin talks indexing: SQL Server performance optimization is not a simple topic, and index design plays a pivotal role in it, determining the efficiency…

Andy Brownsword gives us a blast from the past: Configurations for Integration Services packages allow us to tailor their execution without needing to redeploy. There…

Learn how to control access to OneLake data the right way in Microsoft Fabric with these security tips. Keep your data safe and secure with these best practices! OneLake...

In the previous articles this series, I demonstrated various ways to retrieve document data from a MongoDB database, using both MongoDB Shell and MongoDB Compass. In this article, my focus shifts from retrieving data to updating data, which is an essential skill to have when working with MongoDB. Whether you access the data directly in a MongoDB database or build applications that rely on the data, you should have a good foundation in how to modify a collection’s documents. This article can help you build that foundation.

Hugo Kornelis digs into one type of cursor: I have used the GLOBAL scope option, so that it is possible to step through the code…

Today, I worked on a service request that our customer needs to monitor and optimize queries that can significantly impact performance. This includes both queries running in parallel and those...

Shorty This is a short post. I know you’ve been somewhat spoiled by longer posts and videos lately! Just kidding, y’all don’t pay attention (with the exception of Kevin...

Like All Assumptions You and Me might feel like the lower back end of a thing if we’re tuning a query that has other problems. Perhaps it’s running on one...

In part 30 of the plansplaining series, we’ll continue our discussion of cursor processing. I recommend first reading the previous post, where I explain all the necessary basics. Sample...

One year later, another special unplugged video where we write DAX measures with the new ChatGPT-4o! Previous video with ChatGPT-4: https://www.sqlbi.com/tv/writing-dax-with-chatgpt-4-unplugged-50/ Note: the first 20 seconds and the end...

Since the announcement in March that Power BI Premium P-SKUs are being retired and that customers will need to migrate to F-SKU capacities intead I have been asked the...

For edition 174 T-SQL Tuesday, Kevin Feasel asks us to write about our favorite job interview questions. I don’t like job interviewing. Never have, never will. And while the...

Happy T-SQL Tuesday! This month’s topic comes from Kevin Feasel (b | t), who wants to know: What is your favorite job interview question? As an interviewer, I’m going...

This month is a great topic to me. I think growing and improving your career is a skill that most of  us could improve, especially in our younger years.... The...

Welcome to another edition of T-SQL Tuesday. This month, Kevin Feasel asks us to think about our favorite job interview questions. He provides some examples from both sides of...

Steven Sanderson goes hunting for strings: Today I want to discuss a common task in data manipulation: selecting columns containing a specific string. Whether you’re…

This post came around when I was at a loose end one evening, and just started poking at a local sandpit database, and it got me...

SQL aggregate functions are your go-to tools for transforming raw data into actionable insights. They condense multiple rows into single, meaningful values, making it easier to grasp the big... The...

In last week’s post, I gave you a trigger that populated a history table with all changes to the Users.AboutMe column. It was your job to write a T-SQL...

The presentation layer of a headless CMS is separated from the content management system itself, making it a backend-only system for managing, creating, and storing material. Content presentation (how the content is shown on websites or applications) and content creation are handled by the content management system in a standard CMS.

There are many packages and tools that you can use to facilitate your API development with Rust. Rust has a rich third-party ecosystem of crates for building APIs, including web packages like Actix and Rocket and ORMs like Diesel and SeaORM. This article delves into using Actix and Diesel to build web applications. You’ll learn by building a CRUD API with persistence through Diesel on a SQLite database.