May 23, 2008 at 11:25 am
Comments posted to this topic are about the item Sorting the Sheep from the Goats
May 23, 2008 at 1:36 pm
I've never been very fond of multiple-choice tests for validating anything. What they really measure, in my experience, is the ability to take tests.
Personality tests are a laugh. I took a DiSC analysis (Google it if you haven't taken it), which claims to be extremely accurate, to increase productivity, blah blah blah. According to the results on the test, I have absolutely no ability nor inclination when it comes to technical skills, like computer programming. I could be mistaken, but I think some people on this site would back me up on the claim that I have at least some aptitude in the area of software and computer work. (The detailed analysis of my results on the test basically said, in a somewhat diplomatic manner, that the only thing I'm good for is being the office gossip.)
Tests can help eliminate the obviously incompetent, if they are well-written. They can help narrow down a field of applicants, etc. But they certainly can't be the final arbiter on anything that matters.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
May 24, 2008 at 2:34 pm
GSquared (5/23/2008)
(The detailed analysis of my results on the test basically said, in a somewhat diplomatic manner, that the only thing I'm good for is being the office gossip.)
ROTFL.
I took an aptitude test some years back (beginning highschool) and the results said that I had little aptitude for maths or science but was good at learning foreign languages.
I studied French for 2 years, Zulu for 2, Sesotho for 3 and Afrikaans for 12. The only language I can manage more that 1 sentence in is English. I did maths and physics to university levels
Much as I would love otherwise, I think the accurate aptitude test is going to remain a pipedream. After all, people often don't know themselves what they are good at.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
May 25, 2008 at 7:24 pm
I agree with Gus... multiple-choice questions make a test easy to grade and that's all. My "favorite" multiple choice question looks something like the following...
Which of the following roles allows the user to use Bulk Insert?
bulkadmin
bulk admin
bulk_admin
bulk administrator
Shear memorization... and totally useless for telling whether or not someone has actually done a Bulk Insert or not.
The really bad part about tests like that is that 2 people, hired by my company, were walking, talking, MS certified, bloody Books on Line dictionaries... you could ask them virtually any question from BOL, and they knew the answer...
... but they couldn't hit the ground with their hat when it came to writing code. Even though they got over 90% on their respective exams, they couldn't do simple things like strip the time from a date or delete dupes from a staging table without using a cursor. Worse yet, these two guys supposedly had years of SQL Server and T-SQL experience.
On the subject of "psychometric" tests... people know when they're taking one... what do you expect them to answer? Of course, they're going to answer what they think you want them to. Besides, who says that a particular situation won't change someone? It's amazing what happens to a person when they snap. We recently had a killing in Troy where some well thought of guy got laid off from work and showed up the next day with the shot-gun blazing! Rumor has it that he passed his "psych" test with flying colors, too!
I've seen people absolutely smoke some intelligence tests... you know the ones... the first half of the test is where they have 4 boxes with patterns and they have to identify the next pattern from 4 other boxes. Then, the second half is a bunch of word problems like "John has twice as many fish as Jerry. If John gives one fish to Jerry, they'll have the same number of fish. How many fish does each have"? That's all very well and good, but can ya write some SQL to figure out a running total that executes on a million rows without pegging 4 CPU's for 2 days? Do you know what a bloody "Numbers" or "Tally" table is? Do you know how to create a BCP Format File?
We (actually, I strongly recommended against it) recently hired a guy with a Phd in mathematics... surely, such a guy who claims to also be a Ninja in Java and SQL would be worth hiring, right? I asked him two simple questions... How to you keep someone from hitting the "Save" or "Submit" button more than once and what is 1416 converted to Decimal? He couldn't answer either one... but they still hired him because of his supposed "pedigree" on his resume! He was a miserable failure and it's good that he quit because he was getting ready to be let go.
Everyone that hires, stresses "team work" and being a "team player". Yet, many of these companies won't even talk to someone who can't pass a supposed intelligence test or doesn't happen to like the smell of massive deoderent failure in a tight room. Get real folks... if you want to get the job done right, take all comers and interview them! Check for attitude, ability to understand and communicate, ability to think, personal hygene, and skill. It normally takes me about 3 questions to decide... all the rest are for confimation one way or the other only.
I thought the title of the editorial was appropriate... "Sorting the Sheep from the Goats"... that's usually all people look for with the types of tests we're talking about. Not me, though... I'm not looking for either a Sheep or a Goat... I want a "Wolf" or a "Fox" when it comes to writing SQL or Java that can still get along with the people on the team. There's only one way to do that... the interview. Stop wasting your money on a bunch of tests that don't and can't tell you anything real about a person for actual programming skills or the ability to work on a team... do the interview.
--Jeff Moden
Change is inevitable... Change for the better is not.
May 27, 2008 at 1:58 am
I suppose that tests may have some function. I started my career in IT following a quick test by the Civil Service, which prompted them to interview me. I knew nothing - but I guess my enthusiasm shone through. It's not what you know... you can always look it up, and hey, I've got a memory like a sieve - everything falls through.
But, what I do, and I'm not a SQL expert or guru (but competent), is to look at the requirement, question the requirement with the customers (often and early, not afraid of repeating questions) and deliver. Almost always on time, mostly before. And the most important thing?
When you're stuck, ask! The only stupid question is the one you didn't ask...
So,
Tests may screen, qualifications are good on your cv/resume - they get you the interview in which you may shine. Then you've got a short time in which to prove your value. And I'm gloriously happy in my current role despite the 30% pay drop from my last - and that counts for so much more.
May 27, 2008 at 5:03 am
The thing is, though, that validation will demonstrate that a test is or is not doing what is asked of it. However, the problem is that people need to know what they want to ask of said test. The test is a tool; nothing more.
If you were able to clearly define what "good DBA" actually means in your organisation, I have no doubt a set of tests could be developed that would provide pretty accurate assessments against that definition for all your applicants (although finding someone who can develop effective tests once they have that explicit definition is still a huge challenge in itself ;)). However, those tests would be worthless in another company whose definition of "good DBA" is subtly different.
Eventually, it's another example of garbage in, garbage out. You can't validate against a moving target, and everyone's perception of "good DBA" is different.
Semper in excretia, suus solum profundum variat
May 27, 2008 at 5:23 am
After years of software development, it is not suprising that I have found that a great many that walk the walk and talk the talk, but can't do the do.
I have programmed in many languages from C thru Java and now C# and of course SQL from varous vendors. And the conclusion? You use 15% of the language 99% of the time. There are two real tricks,
1) To have the right approach to programming i.e. functional decomposition, methods and stored procedures that are small, easy to read and commented. I always use the phrase "If you get run over by a bus, then how easy is it for someone to pickup the pieces" (and I don't mean of the unfortunate individual)
2) Know what the language is capable of, and where to find the knowledge for those bits you don't remember.
In fact, whenever I have interviewed people, my questions are always of the nature of how do you solve a problem and implement a solution. This gives you an idea of a candidates thought processes. Coding tests are a waste of time, except maybe for the initial speration of the men from the boys.
And the problem for our industry is a huge percentage are poor at best. Which leads to management thinking outsourcing is the key, whereupon you oftern have the same rubbish produced for less money instead of addressing what the real problems are; poor methods, poor managment, misdirection and the lack of customer expectation mangement.
Anyway, I have had my rant. I'll be here all week....
May 27, 2008 at 5:43 am
I'm a card-carrying dyslexic. Real, genuine. I find any sort of simple-minded multiple-choice questionnaire that merely tests ones ability to remember facts to be extraordinarily difficult. Naturally, I failed the British Government's test at 11 to assess the suitability of a child for an academic secondary education. It didn't bother me much once I'd got my MSc and a brace of professional qualifications. Despite having programmed commercially, and successfully, for a quarter of a century, in several languages, I routinely fail any HR test put in front of me. It is galling to be told by HR people that I have none of the SQL Server or .NET skills that I profess to.
I'm on the militant anti-testing wing of opinion. I will argue for hours that they are a confidence trick on the industry, without any scientific justification. They are an abomination. If you are looking for someone with a skill, you test that skill. If you want a woodcarver, for example, you give someone a block of wood and a set of chisels to see if they can do it. If you want someone to do OLAP programming, you might ask the candidate to do one or more of this sort of test (Taken from Sean Kelly's excellent textbook on data warehousing) ...
• Ranking — e.g., for each product show the top 100 customers in the first quarter of the fiscal/calendar year.
• Variations — e.g., show the variation in sales of product X by reference to target sales in each year for the past three years.
• Percentages — e.g., show the current years sales of product X as a percentage of product Y.
• Calculations — e.g., list all those customers which have generated more than 1000 sales of product X in month Y where the customers' annual purchases exceed 5000 units of product.
• Comparisons — e.g., show the mean average number of sales of product X in each region compared to product Y.
• Relationships — e.g., show all those customers who purchased product X within three months of purchasing product Y.
• Statistics — e.g., calculate the standard deviation in the sales of product X over a two-year period.
Best wishes,
Phil Factor
May 27, 2008 at 6:20 am
Those are great questions, Phil... Now, all I have to do is find someone that knows how to get the current system date and time without using a GUI! 😛 Seriously... I had a guy with a (supposedly) Masters in Computer Science and claimed to be a 9 out of 10 in both SQL Server and Oracle... but he couldn't tell me how to get the date and time out of either system. Couldn't answer any of the other "break the ice and loosen up" questions, either. Interview lasted about 10 minutes...
--Jeff Moden
Change is inevitable... Change for the better is not.
May 27, 2008 at 6:30 am
Phil, Thanks!
I'd struggle with many of those questions, but they are extraordinarily good.
I'll be consistent, and stick with an initial screening test. Which should be a simple IQ test, and not one that involves words.
And I heartily agree that it's the approach to problem solving that brings results.
Not book knowledge.
Paraphrase this.
It's not what you know, it's not who you know. It's what you know about who you know.
Or simply - Have confidence in yourself, don't be afraid to ask, don't be afraid to say I don't know. Ask those who do, listen and learn. Then people will start asking you with confidence. Great team building thing.
May 27, 2008 at 6:51 am
I've been fortunate to be on the opposite side of some repliers here; I typically trounce BrainBench and other so-called "technical aptitude" tests (despite being a mediocre test taker overall). But it's laughable.
These tests are virtually irrelevant to the work that I do and represent a poor attempt by people who don't understand the work (HR) to screen candidates. I sympathize with their efforts, but they should invest in different methods. I join Phil's vote that employer's should measure what is expected.
---------------------------
|Ted Pin >>
May 27, 2008 at 6:55 am
I tend to be able to tell which new hires are going to be great and which aren't based on the orientation I provide in 45 minutes within their first couple of weeks on the job. Of course, they've already been hired so it's pretty much a moot point. When you start explaining to them at a high level how certain things work and their eyes just glaze over, you know that this person isn't going to be a great developer. However, when they ask questions, especially to the point that I almost get winded answering them because they really want to understand, then you've usually got a winner.
The quality of the questions asked by a potential developer is a great indicator. Regurgitating BOL? Probably about as useful a skill as reciting lines from movies from memory.
May 27, 2008 at 7:23 am
Definitely agree with this editorial. Taking the MS Certification tests is painful because it feels like we are spitting back BOL answers for a good portion of the entries. Even worse when there are the trick answers or the ones that are so incredibly similar. I know that I'm not the best SQL Dev out there, but I know what I don't know and how to find the answers when I need them. I also know how to write relatively efficient queries and have some excellent co-workers to keep us all in balance. At this point, I have no intention of taking another Cert test unless I absolutely have to just because they are almost completely useless once you have experience.
I can say one great thing about my company - we test our applicants for the knowledge that they should have for the job they will perform. No personality tests or IQ tests, but tests to see how much they know and what they do when they don't know the answers. At my last job, the head of Development would drill down into whatever subject was being discussed until the person either couldn't answer anymore or showed that he/she really understood the topic. He was quite entertained when an "expert" showed up and immediately answered one of the questions with "well, if you want the textbook answer, it's ...". His attitude showed that he thought assessment questions were beneath him. That opened him up to asking the harder questions to see how you'd actually do something in real life. IIRC, that candidate left a little more humble than he came in. 🙂
Now if we could truly revise the Certification system to reflect actual proficiency.....
May 27, 2008 at 8:50 am
Aptitude testing and general interviewing are somewhat useful tools in the assessment toolbox, but pale into insignificance when compared with putting the candidate in front of a keyboard and getting them to complete some tasks. When interviewing candidates I always do some paired technical task with them and it's absolutely amazing to watch someone with an apparently wonderful resume crumble into dust when faced with simple tasks. It's great for figuring out that the 5 years of C# on the resume is theoretical and they can't remember a jot of syntax because they just did a year of Visual Basic (true story).
I don't think I'd want to accept a technical job where the interview process failed to test me technically. Who knows what other duffers have already joined that organization?
May 27, 2008 at 9:12 am
It seems that "what you know" seems to be less important than "how fast you can learn". Unfortunately, "what you know" is much easier to test. Testing with scenarios or fuzzy-situations has always been my favourite interview method. Sit someone down at a table with a situation that routinely comes up with your product or project and see how they would handle it technically and personally.
The interviewee that starts off with solving the problem without asking any questions is a skeptical hire for me.
(Of course, I don't get to interview very often - and back when I did, my opinion was usually ignored :angry: )
Mia
Mia
I have come to the conclusion that the top man has one principle responsibility: to provide an atmosphere in which creative mavericks can do useful work.
-- David M. Ogilvy
Viewing 15 posts - 1 through 15 (of 40 total)
You must be logged in to reply to this topic. Login to reply