January 13, 2014 at 8:31 pm
Comments posted to this topic are about the item Double Checks
January 14, 2014 at 2:31 am
It is just the IT equivalent of double entry book keeping; without the check one cannot be certain of the result. We have plenty of technical examples e.g. checksums. This is just bringing the same principle to larger/longer/more complex processes.
The double check not only can check that the process has run and is not tampered with but also that the process itself hasn't definitely moved into being invalid.
NOTE: It does not prove that it is correct and not tampered with but MAY highlight when it is incorrect or has been tampered with.
EDIT: Spell checker was not working. Now run for a (slightly) easier read 🙂
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
January 14, 2014 at 4:32 am
I am a firm believer in having an independent automated check. Until recently, I was responsible for the support of
an ePayments system for a local authority. We support multiple clients on a shared service basis and we do not sit on a client's server checking it constantly.
This client, however was frequently frustrated when they discovered the system had a problem before we did. I wrote scripts for nearly all the live servers to monitor
what they did. I could check the availability of SQL Server by logging into it and running a test query periodically, and also checked the status of the relevant SQL Server processes. I also monitored availability of disk space, percentage of CPU usage, sent test transactions to a card payment server, monitored the mounted status of an encrypted disk drive, checked web servers were serving pages, and that telephony software was running on automated telephone payment servers.
Faults and test results out of limits were reported in the application event log on each occurrence, and notified to me by email hourly until fixed. All this was done in VBScript. The scripts were configured to run as services, so that they continue running when everyone is logged out (the normal mode for a live server). The scripts ran every 5 minutes, although some tests were run on a rotation basis (one of a set of say, 5 tests would be run on each cycle).
Before anyone says "why didn't you use BMC Patrol?" - we did, on every server.
Obviously, the tests can be tailored to address the risk you perceive. If you are worried about a script being altered, monitor its Last Edited date/time and byte count, for example.
January 14, 2014 at 7:41 am
Every ETL process needs a dashboard application and email alert process. For example, for one my larger data warehouse projects, the dashboard has one screen that lists a complete history of each ETL run with a record containing total record count, total distinct member count, etc. in addition to other things like runtime duration hh:mm:ss. If a run has one or more summary counts or indicators that fall outside a standard deviation, it is highlighted in red, and an email notification is triggered and sent out to operations. This covers technical failures, data quality issues, and even helps trend runtime performance.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
Viewing 4 posts - 1 through 3 (of 3 total)
You must be logged in to reply to this topic. Login to reply