October 14, 2011 at 8:08 pm
Greetings all :-D.
First let me say...thanks for a great PASS Summit, it was my first time and I was very grateful for meeting so many new people (as I am a chronic isolationist!) and learning so much!
Now on to my s-storm of a week! I manage a decent sized Datawarehouse that my company does BI off of. Cubes, Reporting Services, etc. I should state here, that we have NO test or dev environment, although I have been asking for one since I started. Now with this in mind, we have had a failure in the nightly load 4 out of the 5 days this week, while at PASS none the less (since I live just north of Seattle, I was up at 5am anyways!)
So despite having a great time at PASS I was having a horrible time with work. Our main issue was an FTP file we get from across the seas. I am not sure why this particular place cannot create a simple csv flat file for database consumption, but suffice to say for 4 months we had problems. So they finally got it finished and I tested this on Saturday (note that this testing is in production) with no issues. Come Monday our load fails because they put an invalid date in the file which failed the check constraints. So frantically Monday I attempt to solve the problem. I resolve this, but for the next 3 days we have continued issues...why? Because this is all in production. In hind sight I should have just dedicated a few hours I needed in order to resolve the problem thoroughly but the fixes I had thought I had made did not completely hold up to all issues. This morning, we had one more failure due to an unrelated issue was easily resolvable.
Now I would like to say that despite these failures, no one was without data or reporting, they simply did not have the most recent updated data. I realize this is important but the system was still online. Also, I resolved the issue and have everyone's data to them within 30 minutes or less (which is better than pizza hut!). So the 'outage' was really very minimal. I am already beyond frustrated with myself in not handling this as well as I could, and should have but, this morning I received an email from my boss (who has only worked her for 3 months) stating that this was highly embarrassing and that I need to dedicate next week to ensuring my system is much more resilient.
Now I tend to be a bit more emotional than most, and I took quite a great deal of offense to this. I apologized, but the damage to my confidence had already been done. I know what I am doing and feel that I have done the best I can with the systems given to me. I realize that despite that, that this week is still my fault however, if I had the proper environment 'test/dev' then this week would have never happened. I feel like I am getting the negative attention and critique for some things that are beyond my control.
Also, it is not like this system fails very often. I had one hiccup 2 months ago and since then it has been fine for quite a while before that. I could understand this is an uncommon issue, but I felt that the email from my boss was very unprofessional, and plainly insulting (well to me anyways). My last boss would have taken the time to understand the issue, and given me support to resolve it the best I could. I feel like this new boss is more concerned with how everyone else views her than actually supporting her team.
So right now I am considering sending a detailed email to her boss (my vp, who I know fairly well) but I am conflicted as to what the best approach to this situation would be. I do not want to jeopardize my career, but I also feel I should have better support from my leader.
What are your thoughts?
Link to my blog http://notyelf.com/
October 17, 2011 at 2:58 pm
Well I guess I am glad I am not the only one that sees this as a difficult situation 🙂
Link to my blog http://notyelf.com/
October 17, 2011 at 3:22 pm
There are lot of things possible here. I would suggest NOT sending a message to your boss's boss (and circumventing the chain of command) as your first step. It is very possible that your boss has gotten some heat over this and remember which way that stuff rolls.
I would talk to your boss and explain calmly that this type of situation is exactly the reason you have been trying to get a proper dev/test/live environment. Then explain the only way you can make a system resilient is to test things and that testing on the live box is NOT testing. That is hoping it works.
This is crappy situation to say the least but if you proceed carefully and professionally you may be able to end up with a better situation.
_______________________________________________________________
Need help? Help us help you.
Read the article at http://www.sqlservercentral.com/articles/Best+Practices/61537/ for best practices on asking questions.
Need to split a string? Try Jeff Modens splitter http://www.sqlservercentral.com/articles/Tally+Table/72993/.
Cross Tabs and Pivots, Part 1 – Converting Rows to Columns - http://www.sqlservercentral.com/articles/T-SQL/63681/
Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs - http://www.sqlservercentral.com/articles/Crosstab/65048/
Understanding and Using APPLY (Part 1) - http://www.sqlservercentral.com/articles/APPLY/69953/
Understanding and Using APPLY (Part 2) - http://www.sqlservercentral.com/articles/APPLY/69954/
October 17, 2011 at 3:29 pm
shannonjk (10/14/2011)
I am already beyond frustrated with myself in not handling this as well as I could, and should have but, this morning I received an email from my boss (who has only worked her for 3 months) stating that this was highly embarrassing and that I need to dedicate next week to ensuring my system is much more resilient
Highly annoying when that happens, but will happen a lot with non-technical managers. Hold that thought...
I realize that despite that, that this week is still my fault however, if I had the proper environment 'test/dev' then this week would have never happened. I feel like I am getting the negative attention and critique for some things that are beyond my control.
You are, but a different 'beyond your control'. I usually avoid saying 'blame the vendor', but in this case their poor file building mechanics ruined your process. Dev/QA will not save you when the file-builder blunders unless you're running every file in D/Q first. The best you can do is build failure traps in these cases, but you can't manually review their files before loads. It needs to be understood though that you didn't waste resources to 'gold-plate' a process that was expected to be automated with rare occassions of problems due to file corruption in transit.
I could understand this is an uncommon issue, but I felt that the email from my boss was very unprofessional, and plainly insulting (well to me anyways).
Without seeing the email, I wouldn't find it insulting out of the gate. Their being embarassed for something out of our control entirely is not my failure, but that's their choice. Making the system more resilient... no problem boss, one gold-plating to help work with others' mistakes coming up.
My last boss would have taken the time to understand the issue, and given me support to resolve it the best I could. I feel like this new boss is more concerned with how everyone else views her than actually supporting her team.
Unfortunately not a rare scenario, but not a pleasant one. It may just be that she has to get her peer-confidence up first before she can fire back going "Get your client to do things right and my staff isn't up at 4AM." It may also be she is an aggressive climber and only sees this team as a stepping stone. I can't answer that.
However, what you CAN do is provide a detailed, executive level summary that she can forward on and use as ammunition against anyone outside of your purview as to the causes and reasons these failures occurred and what is being done to make sure that their failures have less of a chance to interrupt your load systems.
So right now I am considering sending a detailed email to her boss (my vp, who I know fairly well) but I am conflicted as to what the best approach to this situation would be. I do not want to jeopardize my career, but I also feel I should have better support from my leader.
That needs to be an in-person discussion. Not email, not phone. If you try to nail this down on paper (initially) only one personality type is okay with it, and they're not who you're dealing with here. I would START, however, directly with your boss, not the VP, and have that conversation. Find out what she needs to keep from being 'embarassed'. Try to work with her and she will probably work with you.
If that fails... take the next step.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
October 17, 2011 at 3:29 pm
Wait several days before you send anything...
Re-evalute the things that make you say:
I am already beyond frustrated with myself in not handling this as well as I could
and put together your own "action" plan on how you can handle this type of situation better going forward. That way, when you're asked, you can say that you've isolated the problematic areas and have the following "plan" in place in the "unlikely" event something similar should happen again.
There's a saying "When I do right no one remembers, when I do wrong, no one forgets" (or something like that)...it doesn't matter much to upper management if your system hasn't been down in 2 months, all that matters is that it went down when they needed something. I feel your pain (really) but that's the way it is just about everywhere.
Take your boss' advice and spend next week thoroughly going through your system, pinpointing problematic areas/improvments, and build a "document" outlining all the issues taking the time to explain how you can get around them in the future...making sure you clearly outline what "you" need to adress and fix the issue. Leave opinions out. Stick to the facts.
If you boss understands what you need to get the job done and there's a strong business case for it, you never know what you'll get. If you don't get it, and things go south again...at least you'll have things clearly documented and can produce that to help CYOA...
______________________________________________________________________________Never argue with an idiot; Theyll drag you down to their level and beat you with experience
October 17, 2011 at 4:19 pm
Wow thanks everyone.
I think just the support and the related experience helps me a but :-D.
I haven't sent anything yet as I tend to try and be patient with things like this. The information so far has been great, and I like the suggestions. I will probably begin on them immediately (as ironically my boss is out of town this week).
Thank you again I appreciate the responses. Building a detailed document now :). Also I have already been spending all today isolating every possible issue that can arise from these 2 situations in hopes that I can at least prevent the same issue from happening again.
Link to my blog http://notyelf.com/
October 17, 2011 at 4:40 pm
shannonjk (10/17/2011)
Wow thanks everyone.I think just the support and the related experience helps me a but :-D.
I haven't sent anything yet as I tend to try and be patient with things like this. The information so far has been great, and I like the suggestions. I will probably begin on them immediately (as ironically my boss is out of town this week).
Thank you again I appreciate the responses. Building a detailed document now :). Also I have already been spending all today isolating every possible issue that can arise from these 2 situations in hopes that I can at least prevent the same issue from happening again.
A side note, you can create a generic 'failure dump' in SSIS using a row failure to a VARCHAR(MAX) field in an errors table. At the tail of the job, have something check the rowcount to alert you to errors having been entered to this table (Rowcount > 0) so you can handle it without stopping the process entirely.
This of course will depend on inheritance of data from the file, etc etc, but it's a catchall for easy robustness that you can then start doing gold-plating for whatever occurs at a reasonable speed, and you can even kick them back to the sender if necessary.
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
October 17, 2011 at 4:50 pm
I have been in a similar situation and am glad to be out of it. It was eerily just like yours. Vendor supplied files changed, caused the nightly process to fail, I fixed it quickly but got read the riot act in public rather than via email (email followed with same terminology). Water off a ducks back and I explained the whole situation to her. It did no good. I kept my cool and continued to make the systems more resilient and better overall.
The other eery thing is that you were told to dedicate a week to solidify your process. I was told the same thing - but never got the time to do it. Don't be surprised to not get the opportunity to make things more resilient. Oh, and document the patterns (verbal, non-verbal, and all communications) for future self-preservation.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
October 17, 2011 at 6:11 pm
Good god yes, as SQL RNNRPointed out, keep track of any and all emails to your boss about the process! When things come up that will take you away from the dedicated week to "fix yer crap" so-to-speak 😉 make certain you follow those things up with an email to your boss/requesters stating how much time the additional requests will take to complete, meaning you'll need more time to complete the resiliency project, and be sure to get a response before starting anything new.
Cover your own a$$...
______________________________________________________________________________Never argue with an idiot; Theyll drag you down to their level and beat you with experience
October 18, 2011 at 3:47 am
Absolutely an exercise in CYA as MyDoggieJessie said.:cool:
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
October 19, 2011 at 8:28 am
1. Like everyone else said, document everything. It takes a little time, but it is the only way to defend yourself. Rest assured your boss is documenting.
2. Keep your documentation (or copies) on something you can access outside of your IT system. If called upon to tell your story, you may not be given access to internal systems. I use a thumb drive.
3. Going over your boss' head in writing is job suicide, and it doesn't matter if you're judged right or wrong. My Dad used to say, "never leave anything in writing on your boss' desk (or their boss for that matter). Everyone can be replaced and there's no sense making it easy for them."
4. Keep your resume updated and your knowlege base current. If nothing else it will help you feel more confident.
5. Write a business case for a development environment. That's how upper management communicates needs.
6. Write an action plan for dealing with bad data from vendors. Make it clear that dealing with bad data is a pain in the a$$. That's how upper management evaluates issues.
7. Never let your manager be suprised. When issues occur talk to her immediately. It'll make her look better to her boss when her boss jumps on her about something. That's how management thinks you have their back.
October 20, 2011 at 6:24 am
We had issues similar to yours with our import process, albeit more frequently I imagine. Users would complain, even though there was nothing we could do. The thing that bothers people isn't so much that the data is a little late, it's that they don't know what's happening. It's a trust thing and the easiest way to build trust is with transparency. So we added a screen on the site that before you get to the main page, if the daily process hasn't finished yet, it says what parts have finished but allows you to move to the main site as long as you understand that yesterday's details aren't in yet.
That being said, posting about your issues in a forum is a great way to at least deal with the frustration. It's always helpful to know that there are other people out there who face the same sorts of issues you do.
October 20, 2011 at 8:37 am
Thanks again for your replies everyone, very good stuff that I am finding very useful (and therapeutic).
JP that is quite a list and very useful thanks!
Ian, thank you for sharing your story, that helps as well. We do have something similar but it is not good enough for our users. Despite having an entire warning system I devised that mails people early in the morning if anything fails, and having a notification on our internal BI site that says if the load for today is successful, I still get messages at 5am staging 'SOMETHING IS WRONG THE CUBES DIDN'T LOAD!!'.
It's at this point I try to take a deep breath and reply calmly and fix the problem. Despite this however, everyone still has all their data, its simply yesterdays data that is not there. I know that yesterday is important as everyone probably wants to see how the day went, but is not going to be any major decisions based on yesterdays data that will be made that can't wait 30 minutes I am sure.
That is just my 2 cents though I could be completely wrong, but in any cause I think people are definitely over reacting without understanding the situation.
Link to my blog http://notyelf.com/
October 20, 2011 at 2:36 pm
shannonjk (10/14/2011)
(since I live just north of Seattle, I was up at 5am anyways!)<snip other stuff>
So right now I am considering sending a detailed email to her boss (my vp, who I know fairly well) but I am conflicted as to what the best approach to this situation would be. I do not want to jeopardize my career, but I also feel I should have better support from my leader.
What are your thoughts?
My thoughts are that since you live near Seattle you know the job market around here is red hot.
Don't stress it over your career. You could have another job within days, likely with a raise, and you should know that.
Send the email to your VP if you know that person well and explain it just like you did here.
Explain that you don't have a dev/test environment and if they are requesting you to make changes on the fly in a production database, then that is by it's very nature "not a good idea".
October 20, 2011 at 5:19 pm
Actually James, I didn't know that 😀
I get a lot of emails (I haven't really updated my resume in a while though) of like 6 month contractor jobs, but nothing that is really Full time. To be honest I am not that great in knowing the 'market' or how to get myself into a good role ;-).
I have a feeling I will start learning this here shortly and I have been updating my resume these last few days, but perhaps you could send me a few tips ;-).
Link to my blog http://notyelf.com/
Viewing 15 posts - 1 through 15 (of 25 total)
You must be logged in to reply to this topic. Login to reply