Working with data has proven to be a challenge for most of my career. It's been fun, and certainly fulfilling, but there are constant challenges involved. Let's take away the hardware and admin challenges of keeping systems running, backed up, and performing well. I'm thinking today of the struggles of just data.

There is a short article that talks about three common data issues, and these are some of the same ones I've struggled with for most of my career. Are these challenges any different than they were 20 or 30 years ago? I'm not sure, and I was working with databases and software nearly 30 years ago.

ETL is a constant challenge, even today with tools like SSIS and Biml that make it much easier to build flows that migrate data from one database to another. ETL is such a challenge that many people make a very comfortable living helping organizations meet their every changing needs to move and prepare information for end users.

The other challenges noted in the article are getting a complete data picture because of missing data and not trusting or believing in data. The latter hasn't been as much of a problem for me.  I might describe it differently as more often we aren't sure what weight to place on certain data. The world is messy, and often we collect data that we think might be valuable only to realize later that it doesn't mean what we thought or our our hypothesis was incorrect in the first place.

I think the challenges are part of what makes this work interesting. Our employers and clients might view the effort and time involved as frustrating, and I wish I had solutions to make our process quicker and smoother. Actually, I do think that advances like SSIS have made things quicker, but the world has grown more complex. We deal with more data from more systems, in the still chaotic, messy formats of the world.

What are your challenges with data today? Are they getting better, worse, or still the same as they always were?