Bad Breaches
-
I saw a slideshow about the worst data breeches and there are some interesting ones on there. Most of them are hacks, inadvertent exposures, or loss/theft of some device (hard drive, tape, laptop, etc).
This story, however, is more disconcerting: an employee sabotaging data. Apparently a former engineer and manager hacked into his former employer's network after resigning over a negative performance evaluation and physically deleted data from servers. As if this wasn't bad enough, a few days before he did this, he connected to the system and disabled backup jobs.
While I think this is criminal in and of itself, it took place at a health care clinic, his former employer, and affected patient appointments, diagnosis, treatment plans, etc. That's reprehensible and this guy deserves time in jail. And if anyone was hurt at the clinic before of this, he should be liable for that as well. It's one thing to hold a grudge and even want to harm your employer (or former employer). I'm sure many people have thoughts of sabotaging their company to get back at them. Attacking someone's health, especially innocent patients, is just malicious. It's beyond unethical.
As DBAs, we're in charge of data, and we're often charged with ensuring it's protected not only from accidental harm or disasters, but also from malicious attacks. However we depend on others, the network and tape backup folks, security provisions, etc., to be sure that we can perform our jobs.
This case highlights one of the main problems with data security: there's too many ways to attack our data, whether it'd a deliberate change or an unauthorized copy. As DBAs, just as with other security professionals, a strong auditing solution is probably the most important thing you can do. Encryption, rules, they'll all get bypassed at some point. At least if you know something happened, you can begin to work on a solution, contact the authorities, or notify customers. Probably all of these are required.
I know this will create a mountain of data and it will be hard to detect "unauthorized changes", especially if they're real users making changes. Hopefully the BI and data mining guys will give us some models that can learn what looks like regular access and what doesn't.
So as you build new systems and architect changes, be sure you have a strong, independent auditing solution you can use to keep track of your data and detect problems first before someone else notifies your boss you've missed something.
Lastly, respect the security rules at your company. They're a pain, you want to do favors, the guy who just quit might ask to log in so he can get his MP3s or something else he forgot. Remember that while most of these are innocent, but you never know. Don't give anyone access they shouldn't have.
People should be able to take their legitimate data with them, the digital equivalent of cleaning out their office. Just be sure someone vets the data to be sure that it's really something they should be allowed to take.
-
'Word data breeches' would have to include the tartan trews worn by our Scottish DBA. They are extremely unethical!
-
I don't know if I'm just being old-fashioned, but I seem to be seeing a lack of personal responsibility in a lot of these stories. If I get a bad performance appraisal then I have a couple of choices none of which is sabatoge. I either appeal to higher authority (my supervisor is not the last word) or I learn from it and leave the organization or fix what I did wrong. Possibly causing harm to innocents is so far out of the question. Sorry, just my 2 cents worth.
-
Along the lines of responsibility, why didn't anyone notice the backups had been shut off? Doesn't anyone check these things to insure they completed correctly? As a developer, I have several systems that I check every morning to insure that the data loads happened correctly, and the reports didn't fail. I would expect the same kind of quality control on the DBA side of things. Now maybe this guy was the only DBA, but his boss should have known to have someone cover critical functions after the guy left - especially given that it was a "hostile" leave-taking!
And hey, we all may wear different hats from time to time, but they all go well with data breeches (except the funny plaid ones the golfers wear - must be a Scottish thing).
Here there be dragons...,Steph Brown
-
Stephanie,
While I agree that someone should have noticed backups failing, I can see the problem there. How does a network admin check the backups?
he looks for failures. You'd need some reporting system that checks for an absence of something being done. Sounds like an article series to me
-
Hi Steve,
Great point. I know that everyone in an organization thinks that their corner is the one that should get the most resources, but your editorial does make the case that at the very least, data backups need more protection and scrutiny. There probably needs to be a meeting of managers to point out that data and backups are pretty much the lifeblood of the company. I'm sure there are various technical ways of implementing the requirements, but the requirements should handle whatever level of data safety the company needs to stay in business. If it means surviving a disgruntled employee, then there needs to be a level where two or more people need to sign off on changes (such as backup schedules) as well as notifications of whether things are running - or not running - etc.
Just my two cents.
webrunner
-------------------
A SQL query walks into a bar and sees two tables. He walks up to them and asks, "Can I join you?"
Ref.: http://tkyte.blogspot.com/2009/02/sql-joke.html -
In my case, my backups go to a SNAP server which I check on a regular basis, so I would see that my backups are no longer running. When I was the sole network admin MANY years ago (can you say 3+Share?), I logged the space backed up on a daily basis, so I would have noticed the absence of numbers or the same number coming up twice in a row.
It's a bit of a puzzler, discovering the non-error absence of something. If you have your jobs set to log on completion, not just success or failure, a smart grep could possibly reveal their absence. I could also theoretically grab a directory listing, suck it into a table, and parse it against my normal backup jobs. You'd probably want to do this from a private server or MSDE-type engine so that the potential perp couldn't tweak your system to keep you from noticing. Come to think of it, I know some backup systems use MSDE as their log repository and possibly their scheduler/job system, so that probably wouldn't be too bad to inspect.
Definitely an interesting situation. And definitely unethical.
-----
[font="Arial"]Knowledge is of two kinds. We know a subject ourselves or we know where we can find information upon it. --Samuel Johnson[/font] -
I have to agree with both Steve and Robert, it's one thing to be mad at someone but to willfully put others at risk just to get back at them has got to be about as low as you can get.
This isn't the first time I have heard of this sort of thing happening, much as I wish every organization implemented best practices I know it will never be perfect. It was really just the reasoning (or malicious lack thereof) behind this attack that caused it to stand out to me.
-
The DBA, or whoever in charge, should have noticed. But yes, if I leave there really would be anyone that could check that everything was running 100%. I do have a checklist of items for people to check when I am absent, so there is some backup.
On top of that I have SQL Sentry set up on a different server that emails me whenever there is a change to any job on my production server. I also have a job that emails me a list of the last full backup of all the databases, with olderst backups first. That way I can in 5 seconds make sure that the backups ran for all the databases.
-
I also agree that intentionally destroying data should be a felony offense (if it isn't already).
The backup program we use also e-mails when problems occur. In fact, I'm chasing an issue now and hate to see that e-mail every morning
A side story to this is that the VAR that set up our ERP and eventually brought us into the SQL world, set it up so that all users could use Crystal Reports without any problem.
He set them all up with SA access.
Yep, they worked without any problem.
Now you have a problem where an innocent user could cause great harm without malicious intent. But only if they stumbled into an area they should have not been.
I discovered and fixed that problem after a server upgrade and yes, some reports still do not run well for all users but none have SA authority either.
-
Shame on the network engineer ... he deserves what he gets.
Shame on the healthcare provider for not changing their VPN setups and passwords after the network engineer was terminated - does it sound like a few people should get disciplined or even fired ???
More shame on the network engineer for endangering the lives of the healthcare providers patients - no matter what criminal and/or civil penalties he receives nothing can really equal the potential for harm he/she may have caused.
More shame on the healthcare provider for not having any type of monitoring in place for system maintenance - again some people are in for time off or termination.
The biggest offender of all though is the healthcare provider itself - most of them use antiquated hardware and software, outmoded or no architecture at all, and even worse still treat IT as a cost center as opposed to a revenue enhancement,
I may sound jaded becaue I am. I worked a a similar type of provider with a similar number of locations. Everything I have spoken of was the 'state of the union' when I started there as part of the 'new regime' When I departed 3 years later the provider was in state of the art IT shop !
RegardsRudy KomacsarSenior Database Administrator"Ave Caesar! - Morituri te salutamus."
-
It's good to see people that believe this is as horrible as I think. There's definitely blame to go around as passwords should have been changed ASAP when someone in an admin or other control position changes.
Getting a list of backups is good, but it's not that easy. If someone stops a backup job, like a tape job, then you may or may not notice. I can appreciate you checking backups, but it's a boring, tedious job, and I bet that if we had a list of backups on each of our databases, we'd miss noticing if the last backup was 20070812 or 20070813 sometimes. It's just an easy thing to do.
It's a good reason to make sure you build reports and monitoring that look for exceptions. It's easy to overlook this and expect to be notified of "problems".
-
I really don't understand aspects of this. You would think that the network engineer would know enough about his craft to know that unless he were exceptionally skilled and careful, he was likely to get caught. The first thing that the investigators would look for would be recently terminated employees, especially those that had bad performance reviews.
So now he's a convicted felon, and what is he going to do when he gets out of prison? Would you hire a network engineer who violated his previous employer's system and deleted data? So what's left for him, working for organized crime (which isn't always all that organized) or a total change in career?
One thing that we don't know is all of the details. It's possible that the employer acted properly and changed logins and such and that the engineer left himself a back door that went undetected. We don't know. With all of the requirements of HIPAA and such, one would hope that the employer would have been more proactive about such, but it's possible they were operating under a small shop mentality.
-----
[font="Arial"]Knowledge is of two kinds. We know a subject ourselves or we know where we can find information upon it. --Samuel Johnson[/font] -
Some would change passwords just before telling the person they are terminated so they won't even have time to put in any backdoors, etc.
webrunner
-------------------
A SQL query walks into a bar and sees two tables. He walks up to them and asks, "Can I join you?"
Ref.: http://tkyte.blogspot.com/2009/02/sql-joke.html -
Wouldn't have helped in this situation: he resigned two months after a poor performance review.
When the dot bombs started falling, it wasn't unusual to be called into the office of the boss, while in there, network admins disable your account and security packs all your personal items. I think that is rather cold-hearted, but they were quite ruthless when that went down.
It was SOP at a previous company where I worked that when you turned in your two week's notice, you were given a check for those two weeks, your account was deactivated, passwords were changed, and you were walked out the door. I think that is overkill for an employee in good standing, but that was the policy. None of the jobs that I worked in government were that bad.
-----
[font="Arial"]Knowledge is of two kinds. We know a subject ourselves or we know where we can find information upon it. --Samuel Johnson[/font]
Viewing 15 posts - 1 through 15 (of 18 total)
You must be logged in to reply to this topic. Login to reply