May 12, 2013 at 10:34 pm
Comments posted to this topic are about the item Better Coding, More Savings
May 12, 2013 at 11:25 pm
You DO have a knack for hitting the "sweet spot" with me, Steve. 😀
I've found that the causes of writing high-resource-usage code can be boiled down to 3 things.
1. Demanding schedule which leaves no time to check for alternative methods that may lead to less resource usage.
2. The "just get if off my plate as quickly as possible" syndrome exhibited by many developers even when not under schedule pressure.
3. Lack of knowledge and data to test with.
The latter reason (#3) is inexcusable considering how easily correctable it is. You're being paid to do a job and to do it well. A lack of knowledge in the area(s) that are required to actually do your job is your fault. Please, no whining about the company not paying for your education. Most people just starting out in the job market didn't have a company pay for their education. If the company you work for won't pay for education, suck it up and get your own education. A great education in SQL Server is relatively inexpensive to obtain. A cheap desktop computer, a $60 USD (including shipping) copy of the Developer's Edition of SQL Server, and a $60 USD book or two are most of what you need. If you want to officially document your education, taking an MS Certification test or two will only set you back a couple of hundred bucks. The rest is simply taking the time to study and practice your trade on your own. Additionally, reading about and solving problems on forums will give you a world of practical problems not available in any book. Studying other people's solutions to these real world problems will give you an incredible education not easily duplicated and usually not available in any book.
The justification of "having a life", especially if you have kids, is a very strong one. Just remember, when your kids get hungry, they can't eat you. You'll need to have a good job to pay for their food and maybe even their education so they won't have to work as hard as you do. 😉
Part of the lack of knowledge are the skills necessary to make large quantities of test data especially when shrink-wrapped tools to do so aren't available. Here's an opportunity to suck it up and learn something on you own. Please see the following two articles. Whether you read and learn anything from them or not, you can no longer say that a lack of data is the reason you didn't test for scalability and resource usage.
http://www.sqlservercentral.com/articles/Data+Generation/87901/
http://www.sqlservercentral.com/articles/Test+Data/88964/
As to the first reason (#1), you managers need to learn to make better schedules. You need to stop over promising in a vain attempt to look good, learn to properly estimate jobs, and learn the lesson that every manager should learn. That is "If you want it real bad, that's the way you'll normally get it". If a bad schedule comes to you "from the top", you need to grow some hair and push back a little because delivering a bad product to the customer is much worse than missing a deadline. And to be sure, delivering a bad product to the customer is the fault of only one person… yours.
Last but certainly not least, you need to learn how to hire good people and pay them well enough to keep them. Treat them with the respect they've earned. Enable your team. Don't try to lead them. Rather, set an example. Follow those suggestions and they'll make you look good without you even trying.
As to the second reason (#2), you people need have a serious change of attitude or find a different career. The only good thing about you is that your continued actions guarantee that people who are in the business of cleaning up performance challenged code will have a lifetime of high paying work.
--Jeff Moden
Change is inevitable... Change for the better is not.
May 13, 2013 at 5:07 am
Being both a VB.NET developer (yes, VB.NET; coming from a C and C++ background I'd prefer C# but other team members don't) and a SQL Server DBA, spending also one or two days a week on BI solutions with the Microsoft stack, I'm kind of stuck in the middle.
The days you knew in advance what you were building are long gone now. A very efficient solution in terms of processor usage, memory usage, network bandwidth and disk access degrades slowly as more and more changes are piled upon it. It's easy to blame the developer but he didn't plan it that way, neither did the manager or anyone else. With so many software providers to choose from a company must do whatever the customer asks to avoid loosing another valueable account. Only after seeing the nearly finished application that customer will tell you what other features he can't live without. A relational database is not the most optimal tool to support these constant changes in requirements but it is a very stable environment to store data.
Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers. LINQ is a beautiful integrated query tool on sets, but most developers still prefer those old-school For Each loops. If your lucky they understand the benefits of joins and use them appropriately. They're good at other things but they're no good at databases. Efficient algorithms are at the base of efficient code, but for most developers getting the job done without subtle flaws or hidden 'features' is already hard enough. I know a lot of performance tuning on many aspects of applications, but I'm generally called AFTER an application becomes too slow to be useful.
Most programmers like to call themselves developers. Most companies want to hire developers while paying only a programmers salary. Most errors will only surface after the solution has been deployed. The use of inefficient techniques are no exception to that rule. Good programmers are hard to find, good developers even harder. Would you be able to learn those so-called developers how to implement their solutions with those efficient techniques? If not, please stop complaining ...
May 13, 2013 at 7:07 am
vliet (5/13/2013)
Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.
Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.
🙂
May 13, 2013 at 8:00 am
dmbaker (5/13/2013)
vliet (5/13/2013)
Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.
🙂
I completely agree. Broad brush indeed! Vliet is hanging around the wrong devs.
May 13, 2013 at 8:08 am
When I was doing development work on a regular basis, I always kept my old computer as long as I could, as they were cycling out for the latest and greatest new PCs for the end-users.
Why? Because if I could get acceptable performance from the app/DB on my PC with half the memory and CPU of everybody else, then the performance would be fine in production.
Jeff Moden (5/12/2013)
3. Lack of knowledge and data to test with.
These are really two separate things in my view. Education, training, seminars, etc. have no real correlation to test data.
And the problem with the way your scripts generate data is that while random data is great, sometimes it has to be hand entered so the data is consistent, hence small test databases. For example the latest software I've worked with is HIPPA patient data. So it is entered row, by row in a specific way by the end users. (Patient demographics, admission data, insurance info, then allergies and diagnoses, etc.) It all starts off the with the patient and has to be consistent to itself to work, such as the 50 states and valid zip codes, a limited set of diagnosis codes, a limited set of insurers, etc. So using a customer's database and using it is problematic because the data is hard to sanitize, and we can only hold unsanitized data for 180 days.
So our developers were always working with these small fake DBs and then weren't understanding why the SW would break and time out in the real world.
----------------
Jim P.
A little bit of this and a little byte of that can cause bloatware.
May 13, 2013 at 8:34 am
What I have observed living as a DBA in the worlds of Sybase ASE, SQL Server, and Oracle:
Developers, for the most part, are focused on meeting functional specifications and deadlines as a result of managers who impose rigid timelines on them and then less than optimally peforming code falls in the DBA lap to solve. The managers impose the timelines because they are scrambling to meet less than flexible demands from the business-users and are unwilling to push back, most likely due to concerns about their career path (or even their job path).
--
I see the world of IT as having become more and more complex, with expectations of faster and faster response times andor throughputs, but an unwillingness to allow those hired as subject matter expects to perform the jobs they have been asked to do. I submit That extra up-front time to allow the IT folks to think it through is more likely to result in a better end-result than not.
There seems to be a continual need for the Jerry Maguire "Help me help you" up and down the org chart as a starting point to get everyone focused on the ideas that "we are all in this togther" and therefore can accomplish more by working together than apart.
May 13, 2013 at 9:21 am
dmbaker (5/13/2013)
vliet (5/13/2013)
Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.
🙂
Would SQL query and update statements (without looping and other RBAR constructs) really even be qualified to be called "algorithms" of any sort?
May 13, 2013 at 9:39 am
patrickmcginnis59 10839 (5/13/2013)
dmbaker (5/13/2013)
vliet (5/13/2013)
Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.
🙂
Would SQL query and update statements (without looping and other RBAR constructs) really even be qualified to be called "algorithms" of any sort?
I was wondering the same thing, that's why I used "quotey fingers".
Google definition:
al·go·rithm
/'alg??riT?H?m/Noun
A process or set of rules to be followed in calculations or other problem-solving operations, esp. by a computer.
If you think of a SQL query as a "set of rules" then I think "algorithm" absolutely applies. The "process" part may be abstracted away (being performed all or in part by the engine), but you still set up the rules in your query.
I dunno.
May 13, 2013 at 9:59 am
I like this editorial. What I like even better is how I saw it differently to a degree than anyone else. The first word that jumped to my mind when Steve said using resources most efficiently was "spaghetti". Often in the old days the most efficient code was rarely readable and understandable. People avoided calling subs/procedures because they took up extra memory when a simple yet devastating GOTO would do the trick. I know that once I had to do a material safety process and I only had 32k of memory to work with. Wow was it ugly code but it worked and fit into the limited memory (that included storing the data sheet). I remember when we had to upgrade it when memory limits were removed, I was definitely badmouthed! 🙂
I'm thinking the most efficient code may not always be the most elegant code.
May 13, 2013 at 10:15 am
There are developers building solutions from a combination of application code (in C# for example) and stored procedures. Actualy, I am one of them. But this combination of skills is rather rare. In most cases stored procedures that operate on more than a single row are either build with cursors by programmers or are implemented separately by a DBA with some knowledge of the data contained in the database. Lately, code generated by an ORM is more and more poluting the database, allowing developers without ANY knowledge of good database design to use SQL server for data storage and retrieval.
Far too often I have to improve code that does all operations on a nested row-by-row base. Or even worse, code that first fetches all columns of all rows and then filters out the needed rows, finaly using less than a handful of columns. If you haven't come across such code in .NET, you may consider yourself a very lucky DBA. Did you ever come across such code and tried to explain to the developer responsible for that code how he (or she) could improve it by making better use of the capabilities of the database engine? For most developers, a relational database is just a bunch of tables. They might understand the purpose of adding domain and referential integrity constraints, but generally they will not know how and where to add them.
I'll expect some future version of SQL server will be tailored to work with code-first databases with a decent performance. After all, Micorosft is promoting this way of building solutions in nearly all examples of applications using a database and the Entity Framework. Some .NET developers may still have enough knowledge about relational databases to produce efficient code, but most programmers will asume that the performance and integrity of the data access layer is not their concern. If any of you did succeed in educating these programmers, please let me know, I'd love to hear more about that ...
May 13, 2013 at 10:17 am
I completely agree developers (or programmers) are pressured by deadlines and unrealistic expectations. They are struggling to keep up. I get it.
That doesn't excuse not learning and improving, as Mr. Moden noted. It's not that you have to build the perfect app here, or get this particular model/method/query to be efficient.
But learn to do something better.
If you love for loops, that's fine. Make an effort to learn what a set based solution looks like for that problem. Maybe you can't implement it here, but the improvement in your code, and your skills, comes with practice over time.
I don't expect it all to be efficient and amazing today. I don't expect it all poorly written in 5 years. I expect improvement, small, slow, and incremental, along the way.
May 13, 2013 at 11:47 am
Steve Jones - SSC Editor (5/13/2013) ... I expect improvement, small, slow, and incremental, along the way.
And this is a reasonable expectation.
We need to not only learn from our mistakes in development but also from our successes. It is not a failure to look back and see that you could have done something better. This is how we learn. We need to celebrate when we find a better way to do things. Then we need to institute doing processes this new way until even better again comes.
How many have looked back at a project and said something like "If I knew then what I know now I would have done it differently." And once we have those revelations about better ways to do things we must alter our path and make the changes to do it differently, better, and more efficiently. How could we live with ourselves if we didn't do this?
M.
Not all gray hairs are Dinosaurs!
May 15, 2013 at 8:43 am
If you're going to work in the cloud, you better learn to code more efficiently
Kudos. I fully agree.
I'd take it a step further (especially after dealing with hairy execution plans that are ugly and > 5mb):
If you're going to be a vendor, you better learn to code more efficiently.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
May 22, 2013 at 1:58 am
It cannot be forgotten that sometimes maintainability must take an equal footing to performance. For some, being able to adjust a system to ever changing business rules is as important as getting the results quickly.
As always there is not one single facet of systems delivery that has an overriding lead in business requirements. As such, compromise often needs to be found. Having said that, the compromise must be an educated choice i.e. proactively choosing a design that is maintainable but slightly slower that encompasses efficient algorithms will probably satisfy the requirements for an enterprise application. Of course that leads us to the differing emphasis various systems have...
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
Viewing 15 posts - 1 through 15 (of 25 total)
You must be logged in to reply to this topic. Login to reply