Most of us know that sending extra data back in a result set is a waste of resources. This can certainly cause delays and poor response in our apps, and across slow networks, this can even cause timeouts. Good developers try to limit the data transfer to just the data necessary, without pulling back extra fields. This is one reason a "SELECT *" is a bad idea.
What about when we are sending just the data we need, but that data is so voluminous that we can't easily transmit it across a network. That's one of the ideas that is noted in this piece, which asks for innovation in the way that we move data around, especially IoT data. This type of data is growing, and potentially growing across networks that are bandwidth constrained, such as satellite and cellular links.
Do many of you find that your network links are constrained? I certainly do at times, though I live in a place where the broadband and cellular networks are not very robust. Even so, most of the time I never notice issues with moving data around. Even when using a real time application, such as streaming data.
Perhaps there are tricks that buffer data and hide network issues, but even if that is the case, is there a problem? After all, that's part of what our applications should do: isolate and protect us from the inherent unreliability of hardware.
Relational databases continue to store and process larger amounts of data all the time. Size of data operations, which grow increasingly cumbersome as data volumes grow, are a place where many vendors are trying to improve their offerings. Microsoft is one of these, and some of the improvements in SQL Server 2019, with Big Data Clusters, Accelerated Database Recovery, and scale out architectures are designed to precisely help us cope with more data.
I have confidence that we will continue to solve these problems, and that networks will keep growing to support the demands of our ever increasing database sizes. We just need to be sure we continue to evolve and grow our applications to take advantage of these improvements.
