June 6, 2016 at 9:54 pm
Comments posted to this topic are about the item Abstraction
June 7, 2016 at 5:28 am
My creations these days are exclusively abstracted via views and procedures. Some time ago I learned how to use Visual Studio build configurations to quickly switch between Dev, Test and Production connections. Admittedly never considered a read-only connection. Recently I sat down and got to know Service Broker to provide abstraction for certain inter- and intra-database processes.
Too often developers want to tightly couple their application to the underlying table structure.
Microsoft isn't being too helpful in this regard. They're heavily pushing the "Code First" development model using EF to create/modify tables for objects and linking directly to them. In fact Code First will be the ONLY paradigm available in future versions of VS as I understand.
It always makes me scratch my head when developers who practically quote chapter and verse on loose coupling, abstraction and programming to an interface scream bloody murder when DBAs suggest doing precisely that with regard to the database.
____________
Just my $0.02 from over here in the cheap seats of the peanut gallery - please adjust for inflation and/or your local currency.
June 7, 2016 at 6:35 am
lshanahan (6/7/2016)
Microsoft isn't being too helpful in this regard. They're heavily pushing the "Code First" development model using EF to create/modify tables for objects and linking directly to them.
The "code first" model for Entity Framework isn't about tying directly to tables, it's about using code, and only code, rather than designers, to design your Entity Framework interface.
Entity Framework actually works pretty well with views and stored procedures too, by the way, if you have the discipline to actually use them. I think that's more the problem, personally.
June 7, 2016 at 7:40 am
lshanahan (6/7/2016)
It always makes me scratch my head when developers who practically quote chapter and verse on loose coupling, abstraction and programming to an interface scream bloody murder when DBAs suggest doing precisely that with regard to the database.
This. Over and over.
I think it's about loss of control. It's okay to do it (from a developer's perspective) because they are the ones choosing the exact details of the abstraction, but when somebody else (DBA) does it to them the sky falls.
Hakim Ali
www.sqlzen.com
June 7, 2016 at 8:39 am
The more abstracted your application developers are from the database, then the more abstracted your logical model needs to be. I've worked on projects where the application developer was sitting across the table from me coding the front end in tandem as I developed the database, we were both coding SQL directly against the tables, and daily refactoring was situation normal. Ideally you want to have a layer of views, stored procedures, and role based security. If you're exposing your database to an external user base (ie: SOA), then you need to use web services based API. However, that would probably be over engineering (and counterproductive) for internal line of business database applications.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
June 8, 2016 at 12:24 am
hakim.ali (6/7/2016)
This. Over and over.
I think it's about loss of control. It's okay to do it (from a developer's perspective) because they are the ones choosing the exact details of the abstraction, but when somebody else (DBA) does it to them the sky falls.
So if they're frightened of losing control let them write stored procedures themselves.
If they don't know how, teach them.
If you have access to the source control repository then you can work with them to refactoring stuff later
June 8, 2016 at 4:17 am
hakim.ali (6/7/2016)
lshanahan (6/7/2016)
It always makes me scratch my head when developers who practically quote chapter and verse on loose coupling, abstraction and programming to an interface scream bloody murder when DBAs suggest doing precisely that with regard to the database.This. Over and over.
I think it's about loss of control. It's okay to do it (from a developer's perspective) because they are the ones choosing the exact details of the abstraction, but when somebody else (DBA) does it to them the sky falls.
On one project I gave the developers a logical schema and stored procedure access only. They could not see the tables nor the code in the sprocs. The Tech Lead of the project was more than happy with this. But every meeting the same developers wanted to know what the tables were and what the code was doing and demanded dbo access to the database. No real reason was ever given that made either the Tech Lead or myself think about changing the permissions.
Other than the whining from the same developers the project was done quicker than expected, and was more stable due to the abstraction/insulation layer of the sprocs, testing was easier, and performance tuning a magnitude easier. The only downside was the rattles being thrown out of the pram by a couple of developers.
The Tech Lead and myself were going to repeat the same pattern on future projects but then development was then outsourced lock stock and barrel. Oh well, it helps to be stoic at times :satisfied:
June 9, 2016 at 4:17 am
I find abstraction overdone, but I am firstly a developer and secondly a DBA. In fact, we write a lot of code precisely to replace tedious DBA involvement, leaving to deal with higher functionality. I certainly would never abstract the DB name and insist that test and dev environments change it. I have seen too many disasters where test environments end up over production sets. We also prefer to reduce SP usage on the grounds that it splits business logic into two quite separate management environments. No I do not want source control over my DB and frankly, our code is used to manage SPs, views, keys and constraints. I do not want humans touching these things or thinking they know better than the designers. Abstraction results in more complex code and more expensive maintenance.
I think you have to review the environment. If the DB is tightly bound to an application then reduce abstraction. If it is uncoupled and is used in BI and other frameworks, abstract as far as you want or need. But we must be careful not run a mantra that abstraction is all good all the time. Like all things, moderation is good.
June 9, 2016 at 4:41 am
geoff-532101 (6/9/2016)
I find abstraction overdone, but I am firstly a developer and secondly a DBA. In fact, we write a lot of code precisely to replace tedious DBA involvement, leaving to deal with higher functionality. I certainly would never abstract the DB name and insist that test and dev environments change it. I have seen too many disasters where test environments end up over production sets. We also prefer to reduce SP usage on the grounds that it splits business logic into two quite separate management environments. No I do not want source control over my DB and frankly, our code is used to manage SPs, views, keys and constraints. I do not want humans touching these things or thinking they know better than the designers. Abstraction results in more complex code and more expensive maintenance.I think you have to review the environment. If the DB is tightly bound to an application then reduce abstraction. If it is uncoupled and is used in BI and other frameworks, abstract as far as you want or need. But we must be careful not run a mantra that abstraction is all good all the time. Like all things, moderation is good.
Out of curiosity what examples can you give of "...we write a lot of code precisely to replace tedious DBA involvement..". Not trying to start an argument, just intrigued.
June 9, 2016 at 4:55 am
Prior to the EF gaining prominence in our applications, our code base managed all aspects of table structure, indexes, primary and foreign keys, functions and stored procs. It did the backups, ran the index analytics (to choose when to reindex) and managed all aspects of access security, transaction management and locking. Code always managed table structure, not the other way around. Otherwise we were forever fixing data sets that broke because someone changed the column order or column name or the data type or suddenly disallowed nulls. It was too easy: the application needed something different, just change the code, run the app and differencing engine (with rights etc) rebuilt the table or the object. In an often changing environment, this works really well and the rise of EF has only enforced this model more strongly. The caveat is that the DB is tightly bound to an application but this is more often than not the normal way of things.
June 12, 2016 at 10:34 am
When the app dictates the database structure and the app is the only thing using the database structure then you are in a happy place. Every decision you make is for the benefit of the application alone.
Data modelling in Apache Cassandra is aligned to this. It works on the principle that you design your tables (column families) for the way that the app wishes to consume them. Even if this means data duplication and the headache of maintaining consistency of data between the tables (as opposed to consistency across the nodes in the Cassandra ring). Used correctly this can be extremely effective. It seems to work well for iTunes which has a multi-petabyte Cassandra cluster.
Where things get tricky is when someone decides that the data in your application would be jolly useful for a variety of purposes especially when some of those purposes require the same level of data consistency as the app that spawned the database in the first place. In other words, the implications of "Data is Shared" become apparent.
There is no perfect world. It's a case of which set of compromises are most important to your organisation. Is one particular app the king? Are the apps more or less equal partners? What is the value put on data beyond the obvious value to the original application? How fast does the data have to flow between the different applications?
These things have a profound affect on the data modelling decisions.
Perhaps today we have to take a decision that we know will have to be reconsidered if the application becomes successful. To give an example today the app is King. We have to find out if the app gives customer value and delivers a profit. We will simply copy the data as is from the front-end data store to the data store used for reporting. People doing the reporting will just have to cope with what they are given.
Tomorrow people want more than simple reporting, they want analytics and machine learning. Translation of data from a front-end application data structure to one suited to machine learning and analytics becomes more burdensome and eventually proves to be a bottleneck in what the business wishes to do. Unless the business completely turns on its head it is unlikely that the back end needs will dictate the front-end structures. What I have seen is an evolution towards a master representation of the data with the transforms to front-end/back-end being a much smaller change in either direction. In effect both front and back end applications use abstractions of master data. Yes the app loses some of the rapidity of change but does not come to a shuddering halt.
June 13, 2016 at 12:21 am
Yes, wise words and completely relevant. But mostly, desktop apps run in environments that don't have the luxury of onsite DBAs and even the bigger companies we work with I wouldn't let the DBAs near our app databases. You are right in that I think we all expect data to become more valuable if more end points are able to connect to it and use it but I would still counsel carefully about the degree of abstraction. My motto is to keep reviewing both the need and structure and the business processes attached to the data. As long as both eyes are open to change then additional abstraction can be added as its needed. It isn't as traumatic as some think to post-apply.
July 5, 2016 at 4:06 am
It's an old thread, and I feel anxious to chime in among these well thought comments. I've learned and reflected on things today. Which is good.
I do, however, have a perspective on the topic "Abstraction" which hasn't reached this debate (yet):
In my shop, it is the one thing which kills performance and auditing the most - and raises maintenance and development costs the most.
It has taken me almost 5 years to terminate the majority of the leftovers from previous developers who bought in on the concepts of loose coupling, data access layers, etc., but never left a note on their decisions, setups, or designs.
Without proper systems set-up (and awareness) and without impeccable documentation, it is a nightmare to debug / change an application where you have no clue where the data comes from and why it suddenly looks the way it does. When you don't know and have no one to ask, you face the choice between writing your query from scratch (which is fairly fast and easy) or go looking for an interface doing exactly what you need in a lake of deprecated and not maintained objects (which is prominently slow and frustrating).
December 15, 2016 at 9:04 am
Abstraction is just demarcation of resposibilities to different collections of code that can cause some people to complain about the lack of transparancy. Clear and concise coding helps alleviate this!!!
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
December 15, 2016 at 2:29 pm
....that and as hjp dais, good documentation.
I'm a developer so I have a genetic predisposition against DBAs (Can't help it sorry 🙂 )> We have suffered DBAs going in to tables and objects and changing data and calcs at will. Sometimes they leave comments, often not. If anything is invisible to auditors it is direct access to data.
So where does that leave us? We need to have a cooperative eco-system where each party plays by the rules but I still have a leaning to making code systems control data rather than humans. At least the resulting actions are consistent.
Geoff
Viewing 15 posts - 1 through 14 (of 14 total)
You must be logged in to reply to this topic. Login to reply