I remember the first time I interviewed for a front-end programming position and got asked how to do something in JavaScript on a white board. The specifics are vague, but it’s crystal clear how stupid it made me feel and how little it had to do with the actual job.
Since then I’ve rarely heard a kind word about the parlor tricks of programmer hiring, but I’ve heard plenty of scorn. There’s certainly a minority of puzzle solvers who love to get their fancy tickled like this, but I sure wasn’t one of them and neither have most programmers I’ve met.
I’ve known fabulous programmers flame out in the quizzing cage and terrible ones excel. So unless you’re specifically hiring someone to design you the next sorting algorithm, making them do so on the white board is a poor gauge of future success.
The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written, talking through bigger picture issues, and, if all that is swell, trying them out for size.
(If you need help posting a comment, feel free to use any of these samples: “You make todo lists, you don’t need real software engineers”, “Math is actually really important, you know!”, “Google is worth one gajillion dollars and they use quizzes, so there!”)
Joe Sak
on 05 Jan 12I once did not get a job because I couldn’t do vanilla JS on paper, and I couldn’t solve a faux columns problem in pure CSS without hacks or something… I forget now.
I’m glad I have the job I have now, I know for certain it’s way better than that other company would have been.
Anyway… I remember saying to the guy, “I don’t know how to do this but in a normal situation I would spend 5 minutes googling for it, find the solution, and implement it. After a couple of regular tasks, I’d probably have it memorized for a while.”
It’s like, dude: No one works like this. No one is forced to write code on whiteboards and paper in front of their boss without the internet to look things up. That would be very counterproductive.
Seb
on 05 Jan 12But last post about Anton Koldaev, you said “we gave a traditional operations task to complete” looks like a puzzle to solve isn’t it?
nick
on 05 Jan 12A large company like Google or Amazon has to hire, what, hundreds of developers a year?
I am not saying Google or Amazon has it right. But the challenges they face in hiring are way different than the challenges a shop like 37signals face.
So as bizarre or stupid you find their hiring techniques, I am sure they may find yours insufficient to hiring for their companies.
I think the lesson is – don’t model your hiring processes off large companies if you are a small company.
Seb
on 05 Jan 12BTW, I’m 100% agree with you. We don’t work as we work years ago mainly because of internet. I know a lot of stuff and I know how to instant refresh my mind if I don”t remember something.
Developing without an internet access does make sense to me. I don”t say copy/paste code NO DON’T DO THAT. But documentation great articles, twitter developer community is a huge help.
Carlo
on 05 Jan 12It’s dismissive to call them “quizzes” because it implies that all you care about is the correct answer, when you (should) actually care about the thought process in arriving at the answer, and the problem-solving abilities that they demonstrate in order to address additional concerns that you (should) bring up after they get to the initial answer.
Alex
on 05 Jan 12This is all well and good but the next time you have to count to 9 minutes and all you have are a 3-minute and 5-minute hourglass, don’t go crying to your employees for help…
Matthew
on 05 Jan 12Refreshing to hear – thanks, David.
Nick, I don’t think David’s point has anything to do with the size of the company. I don’t think asking a programmer to perform monkey tricks under pressure has anything to do with compentence, qualification for most any position, or more importantly the question: “do I think he/she is a good fit for this job?”
Dmitry Nikolaev
on 05 Jan 12Most important thing that drives interviewers to give some sort of puzzles, code on paper etc. is fear. Fear about getting person that would may have problems with current company tasks. Most often is not about to get good worker, is about not to get bad one.
Paul
on 05 Jan 12In general if you state on your resume that you are an expert in a subject, I’m going to ask you questions about it.
If I’m working at a company that likes to discuss things in the language of design patterns, i’m going to feel out how well a candidate is versed in that subject.
we do a basic coding test that is to build a simple project. In that we don’t focus on the result but on the thought process into why they are solving the problem the way they are solving it.
all of this is to get towards finding the person to “try out”
i’m sure all of this reeks of having other problems that we are masking. however basic api questions, language questions for the experts, and seeing how a person solves a problem whether a puzzle or program all help in evaluating a candidate.
However I agree, eventually you have to pick one and give them a try out.
hmans
on 05 Jan 12When interviewing for a job, I find these quizzes extremely useful—because they tell me instantly that I don’t want to work for the place, and I should run as fast as I can.
MJ
on 05 Jan 12Completely agree. Being pretty introverted and uncomfortable trying to sell myself, the interview is already a nervous experience for me. Shoulder surfing me while I try to solve some goofy problem is annoying. I would much rather have the client send me a real problem in some code or git branch and have me solve it, or ask me to implement some functionality for them.
nick
on 05 Jan 12“Getting to know a candidate and their capabilities really well,” is obviously a better strategy.
But, my point is, it may not be an option for a large company that has to hire a lot programmers or a lot of employees who are young or just out of school.
Jesper
on 05 Jan 12Quiz questions might be useful as a simple way of catching people off their game and seeing how they react. Additionally, as others have said, I think the original aim was just to see how people approach and break down problems, but aside from whether it’s being reused for that purpose, you’re also correct that you should do that in a more domain-specific way. Unless that case is getting ten people across a river in a three-man boat before the lake floods, in which case run.
redneck joe
on 05 Jan 12Facebook is worth ten billion dollars and they use quizzes, so there!
Tony
on 05 Jan 12A few years ago, an interviewer at Amazon asked me-on the phone-how I’d implement a distributed key-value store. Since then, such things have become much better known, of course. Even so, my wild-assed guesses about how I (with no experience then in distributed anything) would implement one could hardly say much about me, or my work ethic.
Adam
on 05 Jan 12People who use word or math puzzles in interviews are making up for their inability to judge character or carry on a conversation. They are, thankfully, rare.
Whiteboard programming is, on the other hand, almost universal, even amongst competent managers. It is only slightly less bad. All my previous employers have used it and all of them misjudged my aptitude. Usually they overestimate, which makes them meek later on, particularly when discussing estimates for projects. That does neither of us any good. The one time an employer underestimated, I regretted working for them immediately. You don’t want to be employed by someone who thinks you’re an idiot but hired you anyway because they were desperate, no matter how qualified you actually are.
First impressions are important in all relationships. I’ve never done contract-to-hire, but it sounds like a great idea. Employers would get a more accurate sense of ability and come across less like the bridgekeeper from Monty Python.
hmans
on 05 Jan 12What Jesper said.
walkindude
on 05 Jan 12Aaahh, some truth. How refreshing.
I was almost sure I was the only one to think of quizzes in this way, and feeling a bit bad about it.
Christopher Bruno
on 05 Jan 12At will employment. Use the interview to get a feel for a candidate but don’t make it so high stakes. If they turn out to be a lemon, just let them go. I could understand these quizzes if programmers were like federal employees and were near impossible to fire, then you’d want make sure each new hire is a “ninja”.
There are good and hard working engineers who have lives outside of programming. These quizzes tend to favor those who spend most of their waking hours thinking about programming – I’m not sure these people are that much more productive than a smart person with a life – especially with tools like google and Oreilly safari.
Either way, at will employment makes the choice less of an issue.
Jamie Lawrence
on 05 Jan 12I recently applied for a job and they emailed back a specification for a little simulation/game/CA thing that I had to write within 48hrs. Technically, no problem. But I was slightly offended that they assumed I’d have spare time to engage in their puzzles. Anyway, I sent it back 24hrs later.
Then came the interview. I was fully expecting the interviewers to question my coding style, choice of architecture, or which aspects I’d written unit tests for. Surely we could use that code as a talking point? Nope. Nothing. Nada. Not even an acknowledgement that they’d seen my code. So I’ve withindrawn my job application without even waiting for an offer.
The one good thing about that test: it made me realise how uninterested I am in writing Java code again. I wanted to write that thing in Ruby so this year I’m going to get out there and start doing freelance Ruby/Javascript work.
GeeIWonder
on 05 Jan 12Math ‘riddles’ are not parlour tricks. It’s a shame your (well earned, no doubt) success has validated these types of biases in your mind. Please quit disseminating this nonsense.
The point is usually to see how people’s judgement, personality and skills come together to address a frustrating, possibly irrelevant problem. When done well it can be a microcosm of ‘trying them out for size’ done well.
When done poorly, either can be just going through cargo cult motions. Getting someone to develop an app for a doesn’t inherently tell you anything about their ability to rock the boat and think critically (or not, as you may prefer).
RDO
on 05 Jan 12What I have against these type of litmus tests is that they put the prospective employee under more scrutiny, more pressure and more at stake than they would likely be in any real scenario. I’ve heard of people getting not-so-trivial questions about Bayesian probability, combinatorics and really deep, deep low-level programming questions for a job that had little to do with the above.
As a personal anecdote, when I was applying for jobs straight out of college nearly 1/2 decade ago, after 2 rounds of on-site interviews (which I thought went pretty well), I got a company have me take a 4-hour online proficiency test. I declined fully knowing this meant not getting the job. Fortunately, I did get several job offers from other employers that saw the value and potential I had w/o the need to have a proficiency score be the deciding factor.
In short, I agree. If you are a small or growing company, just give your prospective employees a 1-2 week sandbox project. As this article claims, it’ll be rather obvious whether the person belongs to your team or not.
Anton
on 05 Jan 12I agree with your point in looking at the real code they’ve written in the past. But I don’t see any substantiation in your argument of not letting a candidate make some kind of a test. I have a neutral opinion about it, but this post did not convince me to stop doing it.
Babar
on 05 Jan 12Google is worth one gajillion dollars and they use quizzes, so there!
Adam
on 05 Jan 12Anyone who says they’re more interested in thought process than the answer is kidding themselves. Candidates who get your quiz right will impress you more than those who don’t.
And you have no idea how tempted I am to fake like I haven’t heard your before and “work them out” for your approval. What a farce! Stop running your interviews on fear. It’s like a date. I know programmers are bad at dates, but if you relax a little, you will learn a lot more about your candidates.
Kathy Sierra
on 05 Jan 12To those who say “it is not about the answer but the thought process...” the problem is the speed with which they are expected to think. Challenges like this favor intuitive thought over logic, glibness over reason. So, if the candidate’s job involves a lot of quick, clever responses without much deliberate, rational, logical thinking, long-term thinking then hell yes—quiz away.
Granted, true expertise does mean an expert has a large store of patterns to draw upon, and much of that means near-instant recognition and ability to apply, but still… expertise is also HIGHLY context-specific, and these “parlor tricks” don’t often map well to performance in the real world (except for those whose profession IS performing parlor tricks…)
Nick
on 05 Jan 12For designers, the corollary is the interviews where they ask what design books you’ve read, what design blogs you subscribe to, what designers you admire, etc.
Instead of, you know, just looking at the portfolio in front of them and asking questions about the how and why of the actual work you’ve completed.
chris a
on 05 Jan 12i love reading stuff like this for this personal reason:
many years ago, i went to chat with a guy about a project he was working on. this wasn’t even a job interview, we had friends in common, and similar interests. we talked work/lifestyle/dev philosophy for a long while. and at the end, he was like, “well why don’t you take our test”. i thought, “wait. wtf?”. before i knew it i was in front of a laptop, with screen-capture on and camera pointed at me.
i couldn’t deal with the problem, because all i could think was “wait. wtf?” and occasionally: “really?”.
2 minutes later, i got up and gave it the all “it was good talking to you, but i’m not interested in this”.
the next week i got a email along the lines of. “yea, i talked with my partner, and we agree that the test was a little unfair, why don’t you come in for another chat”. and, well, that’s past the end of the story.
i will never do this to anyone. further, that an employer would do this to anyone raises big red flags for me.
so, venting done. time to leave it behind.
cheers.
Amy Cham
on 05 Jan 12In my last developer interview before leaving that sphere, I had one of those whiteboard coding things. Of course, I totally bombed it.
Not only is someone sitting there staring at you, but you’re coding with a marker?? Half the time when I was coding I didn’t even know what I wrote until I finished and read over it…it’s just automatic. Using a whiteboard throws everything off. It’s like cutting your food with the wrong hand.
I definitely understand why they want a live coding challenge, but is there some reason that can’t happen with a keyboard?
Dmitriy Likhten
on 05 Jan 12My favorite interview:
Phone Screening:
Person 1: Background info on company/position. ~5min
Person 2: Basic computer skills. What is a db index, what is it for? Simple java questions. Can you write a simple in-line sort algorithm or something like that. Nothing complex or tricky. ~10min
Interview:
Person 1: Personality test. Tell me about yourself. Then a bit about the company, team, etc. ~20-30min
Person 2: Lets discuss designing a payment gateway api. What are the concerns. How to ensure that when problems happen you don’t charge the client twice, and you do charge the client. Etc. It’s fairly involved. ~30-60min
Person 3: Lets pair program creating an interface to some external api that is buggy. He wrote unit tests, I wrote impl. We discussed problems, lets make the code more ruby-like, this was my intro to Rspec actually. ~2hrs, fun.
The interview got me really excited about the team actually.
David Henner
on 05 Jan 12You beat me to the punch. I was about to write a similar blog post. On the flip side I have had companies only look at code and want to hire me. That was a bad experience because part of an interview is feeling out the employer. I had to say no because I had no way to judge the way they work.
I think the best solution is to look at the person’s code and then let them sit with someone either pair programming or just talking about real problems they solve. Don’t be afraid to show someone your company’s good and bad code.
Vincent Turner
on 05 Jan 12We used to ask a set of questions in interviews to ascertain proficiency in the actual concepts specific to the language/s we needed the people to work in as a sorting mechanism for the ‘everyone is Advanced in Java’
BUT,... we prefaced the questions with the statement
“we don’t expect you to know all of these, they simply represent an opportunity to hear you explain the things you do know to see how you think and communicate”
.. we would then look at a personal project along the lines your article suggested
Doug Johnson
on 05 Jan 12About a year ago I was applying for a job as a Technical Evangelist…got through the phone interviews, made all the preps to fly for the interview across the country, built a presentation to show how I’d do the user group gigs. I met with the CTO and had a great interview and then sat down with my future direct boss who wanted to run through a linked list exercise on a white board to see if I knew how to program. I know multiple languages, have been developing for…ever…but my current daily use language, Delphi, doesn’t use linked lists directly or has better options, so I fell back on OLD knowledge from when I used to teach C. Bad. Never did get to do anything that I had prepared for, including things that I thought would be vital to build the community, and was told I didn’t have the technical chops…as if. Really annoying. Certainly left a bad t
Andrew Jones
on 05 Jan 12I’ve been subjected to this sort of things a few times, and it never seems to work out. An interviewee is already nervous enough without putting them on the spot to work out a solution to some made up problem. Yes, it’s good to get a feel for how someone would approach a problem, but this is not a “real world” situation. I think employers are missing out on real talent if they judge you by what you could spit out under a spotlight.
Also, I’ve been asked on multiple occasions to do “homework” for prospective employers. They’ll give you some scenario, or a photoshop mockup and ask you to build something within a certain amount of time.
First, this is insulting. It assumes you have nothing better to do than to work for free. I have a job, a kid and a long commute. It’s unrealistic. And usually they’ll give you a ton of restrictions that you would never have in the real world. Like, build this prototype with a bunch of ajaxy goodness and btw, you can’t use jquery (even though everything this company builds uses it heavily).
After doing this a couple times, I now flat out refuse to do it. I don’t work on spec.
Kathy sierra
on 05 Jan 12@Nick - I disagree… Asking someone about the books they have read or designers that have inspired them is not a “performance” but an excellent indicator of key attributes like curiosity, passion for the work, and most significantly - the value they place on learning and development. Looking ONLY at their portfolio does not tell you if they are the person who feels they have reached competence and will keep doing what they are doing, vs. someone who continues to push the edges of their abilities. In other words, at the end of ten year’s experience, do they have ten years, or just one year repeated ten times…
But I can imagine that some interviewers would use this simply to look for a “right answer” rather than an indicator of a growth-mindset.
Doug
on 05 Jan 12I have coworkers that give puzzles and hard coding questions and don’t agree with it. But when I am the interviewer, I usually ask the person to write a “Hello World” program in any language. I use that a starting point for a discussion then we expand on that. You would be surprised how many can’t do it. I am still waiting for the ‘print “hello world\n”’. However, people tend to favor C# and Java.
Example questions I ask: -Would that compile? -What would you change to unit test that? -Why do you need ‘public’? -Have you written a web service before? How would you change that into a web service?
Every interview is different depending on how the candidate answers. Interviews for me are to get to know the person better, how they think, how they would fit into a team, and are they self motivated. Can they figure something out? Do they ask questions? Can they say “I don’t know?” It is a great filter.
I also agree in reviewing code they have written in the past. But most can’t bring anything substantial because the code belongs to some other company.
Bill
on 05 Jan 12While I am not Gods gift to programming by any means, I am pretty good. But I suck at those tests. When I was about 35 or so and a good 15 years out of college I interviewed at Amazon and Google. I got a lot of those O(n) problems etc. It had been 15 years since I had seen one of those, and in 15 years of programming in a diverse set of industries, had never really used one.
I am sure I would have done well at either of those jobs, just as I have done really well at the jobs I did get after those interviews.
My last job interview was pretty much a phone only interview but I had finally done enough side work to have something public to show (most of my previous work was IT related and internal facing).
Jon Rosebaugh
on 05 Jan 12You can look at “real code they’ve written” because you hire, what, five people a year? At the scale that Google hires at, it’s impossible to expect every applicant to have an extensive open-source record. (I interview candidates at Google. I don’t think I’ve ever interviewed a candidate who listed open-source work on his/her resume.)
So what Google asks is not a brain teaser, not a math test, but a programming problem. It’s a simplified coding problem, designed (by trial and effort, mostly) to make sure a candidate has the most basic coding skills. Can you write a for loop and traverse an array? Do you know what sort of data structures are appropriate for what sort of situations? Do you recognize the inefficiency problem you might have when you traverse an array inside a loop that is itself traversing the array? Things like that.
So, yeah, these problems might seem simplistic or offensive to you, but we ask them because a large proportion of our applicants simply can’t solve them.
Ja
on 05 Jan 12I completely agree! I’ve recently been interviewed by Microsoft, Amazon and Activision and all three have been full of these unrealistic quiz type questions. I’ve been coding a long time and am completely capable of doing the job I applied for yet these quizzes and brainteasers are standing in my way. The crazy part is that all these companies are desperate for software engineers! All they need to do is take a look at my resume and test me on some relevant material and they’ll see I can do the job. But instead I get asked to implement some kind of modified heap sort over the phone… in the real world I’d Google it in 5 minutes. Bah.
ploogman
on 05 Jan 12@ David hilarious post
I recently was tempted to work with a company that had a take home quiz that was irrelevant, time consuming and in other words, stupid.
I could solve the puzzle project, so could anyone. I could show my work as requested, so could anyone. But it was unpaid, no guarantee, gave me no insight into working with them (at least no positive insight) and overall was pretty obnoxious.
Glad you voiced your opinion on this even if you get flamed. It’s worth it.
Johan Strandell
on 05 Jan 12There’s an interesting summary of a meta-analysis study about recruiting here: Selecting talent: The Upshot from 85 Years of Research
Work sample tests is the highest ranked method. If I understand the linked description correctly, this seems to be pretty close to what 37signals does. (Combined with the “job tryout method”, ranked 7th in the study.)
I think puzzles fail because they test very narrow skills, and unless the job actually involves sending people across bridges with only one lamp and similar situations, they don’t say very much about the skills needed for the job.
My guess is that this is due to cognitive bias: it’s much easier to remember and be impressed by unlikely situations solved by this kind of problem solving (e.g. figuring out why a server can’t send email to places more than 500 miles away), so many people focus on them rather than the normal day-to-day skills that actually are more important.
Matthew
on 05 Jan 12There’s a time and a place for those sort of questions. Usually if you’re hiring someone who’s going to spend most of their time knee deep in algorithms and things of that nature. For people that just need to create API’s, get an interface up and running, do the day to day work, etc I don’t feel it’s a very good gauge of their talent. I personally feel I’m a good programmer but I never really enjoyed those sort of things in school and don’t really do well at them now. But if you want me to do the programming you need done I can probably do that. I guess it depends on what is more important to the employer. Can you write code or can you tell me what train arrives in Chicago first?
Troll
on 05 Jan 12Google is worth one gajillion dollars and they use quizzes, so there!
Pete Nelson
on 05 Jan 12I totally agree – for me it’s all about the bigger picture rather than solving algorithms. Any employee has got to understand how the company actually builds code on a day-to-day basis and see how they fit within that – therefore a trial period is mandatory for anyone that shows promise. Plus they need to understand the business needs and show that they are working towards them rather than necessarily spending years optimising one algorithm.
TD
on 05 Jan 12I would much prefer a puzzle or quiz type question to the standard HR interview type questions “What’s your biggest weakness?” etc. At least it tells you something about thought process.
john Hinnegan
on 05 Jan 12It depends on what the ‘quizzes’ are. In my experience, they’re applying a fundamental CS concept - like SORTING - to a problem. It’s testing if you really understand the basic concept and if you can apply it, which is, IMHO, a pretty good way to evaluate people.
Also, I would be ashamed to work anywhere that employed an engineer who couldn’t sort a list of numbers. Seriously.
stephen
on 05 Jan 12Code tests and teasers in interviews are fucking stupid. Google really poisoned the well with this one. Now everyone thinks you have to do 3 hours of white board before you even consider hiring someone.
This is what code tests show: - the people who are hiring don’t know what they are doing - the ‘i went through it so you will too’ hazing mentality - set’s up a company vs employee dynamic that is counter productive
This is what they don’t show: - how an employee will perform on the job - interpersonal skills ( which are more important than being able to write out quick sort on a board ) - every programmer has their strengths and weaknesses, code tests do nothing to reveal this. - not real working environment problems ( “oh so your business is counting digits in a given number?”)
John Miller
on 05 Jan 12I don’t really agree with David’s post, at least for companies that do significant hiring.
I’ve done a lot of interviewing and being interviewed over the last 20 years. Ultimately I’ve come to accept that big-company interviews have a simple goal: minimize false positives, even at the expense of passing over great people.
In other words, you may be a smart, well educated, great programmer who fails the interview, but it’ll be rare that you lack those attributes and pass the interview.
Does this approach miss great people? Absolutely. But the cost of hiring someone who doesn’t meet the bar is incredibly high in every dimension.
And unfortunately, if you’re a big company letting someone ‘in the door’ for a try-out is fraught with many of the same liabilities as hiring them outright, and some new ones besides.
There’s got to be a better way to interview, but so far I haven’t found it. My approach remains: test basic programming knowledge (whiteboard problems), design skills / problem solving, and iterative design / problem solving skills.
If a candidate isn’t a definite hire, then they’re definitely a no-hire because of the risk. It’s a shame because we miss great people, but at the end of the day we wind up working with great people who have a super-set of the basic required skills for the position. From any company’s perspective that has to be a win.
Brian
on 05 Jan 12The job I have now had me build a project. The requirements were clear, they gave me a week to build it, and they had me deploy it to a production environment as well as send in the code.
The actual interview consisted of me defending my code, various HR related questions but mostly defending my code. I really enjoyed the interview and am glad I got the job. I knew during the interview that this is the kind of company I wanted to work for.
Winfield Peterson
on 05 Jan 12I couldn’t agree more with this article.
At PatientsLikeMe, we do not use puzzles, but real world problems that map closely to how we work:
— Screen
- Code Sample (Github) or Challenge => Phone Screen
- Phone Screen – 15m of light technical questions/personality
— Interview
- Project Presentation – show your best work and passion, take questions about your choices
- Design Challenge – given a loose user story, develop a design and plan of attack to deliver a feature
- Pair Programming – hack on a simple Rails problem to show how you code and collaborate on something straight-forward
By the end of this process we have a strong feeling for the candidate. More importantly, the candidate should have a strong feeling for us – how we work, who we are, and what we value.
— PS: We’re hiring right now.
http://www.patientslikeme.com/about/careers/1-experienced-ruby-on-rails-developers
Joe
on 05 Jan 12It’s very basic: google is hiring engineers, 37 signals is hiring programmers. Building out UI and frontend products has a very different requirement than building out database infrastructure, search algorithms, or sophisticated compiler frameworks.
Dan
on 05 Jan 12Placing emphasis on singular tasks like writing out an algorithm in a specific language reminds me of the Steve Jobs keynote at the 97 WWDC during which Jobs is lambasted for doing away with a particular programming tool (I forget the name). Jobs apologizes but asserts that any technology is merely a tool for getting a job done. You’ve got to start with the problem and let the problem define the solution, and not the other way around. That’s why focusing on a programmer’s expertise in one specific technology is short-sighted. I’d be much more interested in understanding their fundamental design methodology and philosophies.
Any language or technology can be learned quickly, but the underlying philosophies take much longer to develop.
Peter
on 05 Jan 12John Miller, so you just told us that to panicly avoid occasional C-player, you miss all the A-players and hire only B-players. And this has to be win?
Jeff Darcy
on 05 Jan 12The problem with “it’s OK so long as you’re evaluating the thought process rather than the result” is that it’s really hard to do. Interviews are stressful for many people in fairly unique ways. How people respond there has only a weak relationship to how they might respond even in other stressful situations, let alone in more common ones. It takes tons of training not just to overcome the natural reaction to seeing someone flail at a task you’d find easy (if only because you’ve had a lot of practice in less stressful situations) but also to look past the nerves to determine how their interview behavior might translate to on-the-job behavior. Most professional educators or psychologists would laugh at the idea of making such a leap based on such a brief and uncharacteristic encounter. Engineers, who are generally untrained and often socially inept besides, should stick to things that can be determined accurately in a 45-minute interview.
John
on 05 Jan 12A lot of programmers claim then can do the work, they can always say “oh, I use google and I never use whiteboard, we never sort arrays from scratch in real life, etc” , a lot of programmers also know how to use API’s, etc, so, how to select the best ones? make them solve something that requires to use the brain, not only copy and paste, google, etc.
Paul
on 05 Jan 12I agree that the puzzle-solving during interviews is lame.
In fact, I’ve said exactly that same thing to an interviewer once: “Unless you guys spend all your time here re-designing quicksort, why do you want me to implement it on a whiteboard?”
These days, I usually bring a couple tricky differential equations w/ me to interview, and when the time comes when they say: “Do you have any questions for me?”, I respond: “Why yes, in fact I do”.. and I turn over the marker to him, and ask him to solve the following differential equations x, y, z, etc, on the whiteboard. Invariably, they flame out as badly as you might expect.. even for quite simple vanilla differential equations.. “But didn’t you take calculus in college?” I ask… “errr… yeah”... they stumble… “And you can’t solve this?” etc…
Point is, you can turn the tables on them.. or at least, you can do that if you feel the interview isn’t really going all that well anyway.
As far as probing “how people react under stress”, again, in the 12+ years I’ve been doing programming, it isn’t exactly like running a flight control tower or something. If you have a programming job where you’re “under a lot of stress” all the time, you’ve probably got the wrong job. How a person “reacts under stress” is not, I think, necessarily a super important variable in most programming jobs. The only on-the-job stress I’ve encountered are tight deadlines, and again, that’s a different sort of “stress” and “mental process” than what the brain teasers put you through.
Tonio Loewald
on 05 Jan 12In the end, isn’t it like everything? It’s the details and context that count. E.g. if you ask “how does a flush toilet work?” (listed by someone once as their favorite interview question for engineers) and then rejects anyone who cannot provide the exactly correct answer, you’re an idiot. But if what you’re looking for is (a) the sense of curiosity, (b) imagination, (c) knowledge of basic engineering and physics principles involved, and (d) intuition then it’s probably an excellent interview question. But if you’re interviewing someone who has clearly never had indoor plumbing then you’re still an idiot.
Circling back to puzzles and parlor tricks — if you start out with a good general question and then zero in on interesting details as the interviewee answers then you may get very useful results. If you start with a specific question and then trip up the interviewee on trivial details of their answer, you’re an idiot.
As for “trying programmers on for size” — it’s a great way to hire people who don’t have a job, but if you’re trying to hire someone who has a good job and you try some kind of contract-to-hire deal then you won’t get far. Again, Google is doing things like hiring folks away from Apple and Microsoft — they’re not going to be “taken for a spin”.
Adam
on 05 Jan 12@Johan: That study had nothing to do with developers and the job market has changed significantly since 1998. Someone should do a study that a) focuses specifically on our industry and b) quantifies the corrolation between quizzing and the satisfaction of both the employer and employee after the first, say, year of employement.
@john Hinnegan: When’s the last time you really needed to write a sorting algorithm from scratch? Is memory your sole measure of competence?
@John Miller: If someone isn’t doing a good job, why can’t you fire them? How do you know how effective your tests are at weeding out “false positives”? What does that term even mean?
This isn’t just about being nice to people who suck at whiteboard programming. This is about how we—as an industry—hire people and what kind of relationships and attitudes these hiring practices foster between employers, employees and the outside world. What is a company saying about itself when it forces all its programmers through a time-consuming, adversarial gauntlet fraught with suspicious programmers paranoid about “false positives”?
To me it says, “We don’t know how to manage people or run a business, so we’ve built a club of puzzle masters instead.” Is that what we really mean to say about ourselves?
Steven J. Owens
on 05 Jan 12My favorite take on this was Brandon Van Every’s response, in a mailing list discussion several years ago:
John Szeder:
How about the microsoft style questions:
You are in a room with three light switches in the off position and a closed door. In the next room are three light bulbs, each one wired to the switches you are next to. If you can only flick two switches on, and only before opening the door, how can you tell which switch controls each light bulb?
Brandon J. Van Every:
I would ask clarifying questions about who I had to kill on the other side of the door, then attempt to kill them if I got a random guess wrong. Asked what I thought my odds of success would be, I’d say at least I had some training for killing people, if not a lot of gun disarmament work. If I thought it likely I’d end up in such a Dr. No situation, I’d have spent more time training my killing skills. Offer to break the interviewer’s arm if they don’t believe me.
John
on 05 Jan 12@Adam, it’s not about when was the last time I had to sort an array, it’s about knowing the foundations and knowing how to solve problems, knowing how to figure out solutions given a problem, it’s not just about knowing how to google or how to use an API, of course knowing API’s is useful, but almost any developer can do that, the skilled ones can do more than that.
Sara Taillon
on 05 Jan 12My team used to ask programmers to code on the whiteboard during interviews. One guy flubbed the whiteboard coding and we hired him anyway. I’ve worked with that guy for the past 10+ years, since that interview. He’s still, to this day, one of the most solid coders I’ve ever worked with and I’m glad we hired him. I don’t ask for whiteboard coding anymore.
Joseph Misiti
on 05 Jan 12I couldnt agree more. A few months a back I interviewed at a well-known start-up for a data science job (consulting, not full-time) and they put me through 6 hours of whiteboard interviews anyways. I told them before I even came in I had no intentions of working full-time, I simply wanted to help them look at their interesting data set. The funny thing about the whiteboard questions was they had nothing to do with the position I was interviewing for—Not one stats question was asked until hour six, and by then I was so annoyed I didnt even want the job anyways and fucked it up and left. I left that interview never wanting to work for the company, and realizing that I would never want to put someone through that. I founded my own company instead (which is why I only wanted to do consulting in the first place) and we successfully hire employees via the contract-to-hire method. The funny thing is the company would have saved 6 hours of engineering time by simply giving me a computer for 8 hours and giving me a problem to solve. They would also have had a much better idea of what I was capable of.
jeff
on 05 Jan 12Well if I need 9 minutes and only have a 3 and 5 minute timer, I personally would use the 3 minute timer 3 times. But maybe there is something else to that quiz :)
As for a quiz interview, I am with hman, I use it to leave.
I was once asked “Why are manhole covers round?” I said “Because manholes are round, and other shape would be silly for the cover” they did not like that answer.
jeff
on 05 Jan 12Well if I need 9 minutes and only have a 3 and 5 minute timer, I personally would use the 3 minute timer 3 times. But maybe there is something else to that quiz :)
As for a quiz interview, I am with hman, I use it to leave.
I was once asked “Why are manhole covers round?” I said “Because manholes are round, and other shape would be silly for the cover” they did not like that answer.
Brandon
on 05 Jan 12This article is fantastic. I’ve been on the receiving end of these “puzzles” and they do nothing for the candidate and pretty weakly show how the 2 parties would work together.
My response, in the past, has been that it doesn’t truly reflect my ability to solve their real problems and most just move on.
Networking has a tremendous effect on the ability to fill a position and find a good working match. We’re working to make this easier at http://meeteor.com where we match people with employers and industries they’re interested in. We’ve had some great feedback and would love more.
Cheers!
Dade
on 05 Jan 12This is the most important sentence:
“The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written”
I actually had a company ask me to take a “test” before they considered hiring me. I answered them by saying, “You want to test my skills? Look at the 50+ sites I’ve coded, then tell me if I need to take a “test” to prove I code.”
You can’t replicate the ability to code in an interview. The best way is to break down the code a person has written. It will tell you mountains more than answering some quiz on the fly in a board room.
Josh Jones
on 05 Jan 12I had a similar experience with a Microsoft interview involving a puzzle with a horse race track. Had no clue what the answer was. lol
In a later job interview I brought my laptop with all my code and said “Look what I did!”, which got me the job almost on the spot.
John Constantine
on 05 Jan 12I would never hire based on puzzles or quizzes either. But that doesn’t mean there isn’t value in presenting a candidate with a problem they’ve never seen before and working it through with them. When they “know” the answer, you learn almost nothing. It’s when they don’t know the answer that you learn things about them. What’s their approach, willingness to accept guidance, degree of confidence, their intuition, did they find the problem fun, challenging or painful, do they gloss over details, do take the time to understand the problem, will they communicate with you during the process, etc. If you are their partner in solving the problems, you will learn a lot about them. As you say, there are people who are wonderfully talented, a pleasure to work with, and incredibly productive who in the end never arrive a one of the “better” answers to some of the problems. But all of them exhibited plenty of their good attributes during the process.
Ryan
on 05 Jan 12I’ve done one of these quizzes at the start of my freelancing career. By the end of it I was angry at the company for the unpaid “work” and ultimately did not bother completing the last question.
They really wanted me and begged me to complete it but as others have pointed out the task was so stupid and irrelevant for the job. It revealed more about their terrible business practice than anything. It was a very small company just starting up so it was really unnecessary.
Chilly
on 05 Jan 12I have been involved in recruiting for the past 16 years and have done both with test and without. I have found that the coding test is one of the better ways to hire a developer. The company can see the candidate’s code and how that person would solve a simple problem. Some tests have come with issues in it already and some a developer builds from scratch. It is a must to try to follow up on that code they wrote so their time doesn’t feel wasted.
Other “brain teasers” get to how a person may think and it isn’t about the right answer as much as how do you deal with a problem you may have never seen before.
I have seen people hired that failed some portion of the interview process and in a lot of cases cost makes a difference in that decision. The higher the price the more expectation to hit the ground running.
Steko
on 05 Jan 12The same is true of any qualification. Millions of jobs require college degrees that have no relation to the applicants ability to do the work.
However the degree requirement screens out a disproportionate amount of bad applicants. Sucks to be one of the good applicants that get screened along with them but that’s not going to change the fact that the requirement is a net positive.
Programmers are in short supply so we might guess that extra screens like this are counterproductive but companies like Google are extremely high prestige and can afford it.
Tom B. Taker
on 05 Jan 12I once got hired for a programmer job because I was the only one who got a question about and HTML correct on the hiring quiz. And yeah, I thought that was stupid.
gh
on 05 Jan 12If you’re hiring people to develop web apps the ability to write a sort algorithm on a whiteboard is irrelevant to the job they’re going to do. If I asked an applicant for a Rails job to sort an array and saw them writing a sort algorithm I’d have to think that their knowledge of Ruby was so limited that they didn’t even know that Enumerable has a sort method.
Surely it’s easy enough to give the interviewee a machine to work on with a big screen so you can see what they’re doing, and ask them to write something small, even Hello World to start, while watching and asking about their coding decisions as they make them. If the interviewee doesn’t even know how to start the REPL the interview can end fast, but the good applicants can spend ten or twenty minutes showing you what they can do. If what they can do includes looking something up in online documentation (and knowing where to find it quickly) you should probably hire them on the spot.
John
on 05 Jan 12@gh, anybody can use Enumerable, that’s not a good way to know if somebody is a good developer, that doesn’t make the developer think, coding something from scratch is more difficult and requires more skills.
Tim P
on 05 Jan 12In a web dev job interview the two guys (it was a peer interview) asked me what the SQL was to select the manager from an employee table, where the manager was also an employee – you know, referencing the same table. I told them, off the top of my head, without writing it down.
Later, when I started the job they told my they thought I was amazing. I thought they were idiots. As it turns out they were not good developers. Luckily they were not a reflection of the general talent at the company.
Interviewer
on 05 Jan 12There seem to be a lot of people here saying “oh, I suck at those tests but I’m a really good programmer and I’m sure that I would have been perfect for the job”. Newsflash: not everyone can be as good as they think they are. Maybe, just maybe, the tests do actually work and there’s an element of “unskilled and unaware of it” going on..
Jeff Putz
on 05 Jan 12The truth is that the classic computer-sciencey stuff is far less important in a world where we’re using managed and interpreted code. Design patterns still count, particularly in OOP, but knowing everything the gang-of-four wrote only means you’re good at memorization.
Real code speaks to me more than anything you can put on a white board. Descriptions of how you or your team built out a particular architecture lets me see how you think. It shouldn’t surprise anyone that the most successful gigs I’ve had involved interviews like that.
Anonymous Coward
on 05 Jan 12You make todo lists, you don’t need real software engineers
Anonymous Coward
on 05 Jan 12Actually I am curious, What real Software Engineering challenges you guys have?
Al
on 05 Jan 12It’s not about know everything, is about to know where to find the help you need to solve the problem.
Every company have repetitive problems, after a couple months working there you will be able to solve any kind of problem, and them it becomes boring and you want a new challenge.
This is basically the developers world.
mike
on 05 Jan 12what do you think about gild? they’re doing quiz based on knowledge where you can see if someone knows or know a specific field personally I think that you can cheat very easily, just try all the questions, with some luck you’ll go at the top
Brendan Rehman
on 05 Jan 12The answer to hiring a good developer is to sit him/her down in front of the keyboard with full internet access and then ask them to develop a small app, preferably an app that relates to your are of business. Its how I’ve hired developers and 9 times out of time you get a good feel for their level of skill and you get to see their code immediately. I generally inform the developer that the interview process will be 3+ hours of their day and I spend a little time before hand preparing a small project for them to code.
FranG
on 05 Jan 12Another great post, of course, but you’re again being too generous with your assumptions. You have to understand the motives in play here.
Consider your motives at 37signals. You’re a small shop where you will all succeed or fail based on the actions of a single person. The wrong person comes in and writes some crap and they will bring the whole place down. Similarly, you know from personal experience that the right two guys come together and can build an empire.
At a big company that has already won its customers, it pays much better to not look out for future success (as you and I think of it) in your hires. The worthless middle-managers increase their responsibilities (and take-home pay) by hiring more mediocre, unwashed masses. The engineers build their reputations with fake credentials and pretend knowledge like whiteboard puzzles. New coding superstars challenge the established fiefdoms in both cases. There is often a huge pile of cash to split up based on the work of a few lone heroes here and there.
Engineers need to recognize how they’re being taken advantage of by these dynamics.
By the way, I continue to enjoy how your Getting Real philosophy shines through all your posts.
Peter Ashford
on 05 Jan 12I totally disagree. Yeah, ‘puzzles’ might be silly, but coding questions are not. I’ve hired a bunch of people and seeing how people answer questions is very insightful.
I usually start by letting them know that I don’t expect them to get all the answers right and that some of the questions are quite hard, but I ask them to gauge their depth of knowledge and to see whether they can present an appropriately structured response to code issue – not whether it works or not, but if it makes sense and could work with some debugging.
When I was hired for my first job, they asked me some questions in C which I’d not programmed in for ages at that point. They could see that my C programming was rusty but (a) that I had done some C programming and did know roughly how the standard libraries worked and that (b) I could break down a problem into steps and code up a solution.
These are useful things to know and present a very quick sieve for removing poor candidates.
What’s more, I expect engineers to be able to discuss solutions. In practice, we do get asked ‘how would you solve XYZ’ and are expected to be able to answer the question or at least provide a few ideas of possible answers.
Quite frankly, if someone got all sulky because I asked them a coding question, I wouldn’t think they were worth hiring. If you’re a engineer, it’s not unreasonable to be asked engineering questions.
Jamie Flournoy
on 05 Jan 12There’s a pretty strong bias in these comments toward a “world where we’re using managed and interpreted code”, to quote one commenter; this is a world of tying together libraries and services other people wrote, not having to worry about performance or big data, etc. That’s probably the case with 99% of programming jobs, and has been the case for most of my career too. Still, if you’re interviewing for a front end developer job and you can’t code ‘document.write(“hello world”);’ on a whiteboard (or better yet, in a shared doc during a phone screen), that’s a pretty good indicator that you should not get the job.
Coders should be able to code, and there are a LOT of people trying to get programming jobs who can’t write trivial code in any language when confronted with a pencil and a post-it note. To the commenter who said “Look at the 50+ sites I’ve coded”, how do I know you coded them, and it wasn’t your expert subcontractor, the person who sat next to you at your last job, etc.?
I agree that whiteboard coding is not ideal; I prefer pair programming too. The best coding test I ever had to pass was “pair with me and let’s TDD a queue class in Ruby and rspec right now”. It only took a few minutes, and had a side benefit of making me respect the interviewer and want to work with him because in the process of pairing with me he proved to me that he could code too, and that we could communicate well. But however you structure it, it’s just more time-efficient to contrive an easy little (less than 1h) programming task, put the candidate on the spot, and see if they can perform or if they faceplant, than to hire everyone who seems smart and then fire them a few days later when you realize they’re actually incompetent.
A final note: data structures and algorithms are actually relevant to some programming jobs. That language runtime you like, that server you like, that database you like, that distributed version control system you like, that library you like etc. had to be written by someone. Some programming jobs actually involve constraints or challenges that are not easy to solve with off the shelf solutions because they the problem is unusual in some engineering way (runtime, CPU cycles, tiny memory footprint, huge dataset, etc.), and so you may need to be able to dig deeper and write something lower level than you’re used to. You might never encounter that in your career but that doesn’t make it stupid for companies that have this kind of problem all the time to hire for this skill set.
Levi Senft
on 05 Jan 12I told the last interviewer that wanted me to do one of these that I’d just look up a code snippet on Google to solve the problem rather than wasting my time to figuring out how to reinvent the wheel.
I will walk out of any interview where they ask me to do these types of tests. They don’t give the interviewer any insight into how I provide value to their company or how I can solve their business problems.
In my experience interviewing people, the most effective method was to ask to review their code. As part of the code review I have them walk me through the code. This has helped me identify people that brought in good code that they didn’t write themselves.
GeeIWonder
on 05 Jan 12Some things looking at real code and ‘bigger picture issues’ doesn’t tell you a well-designed situation can:
-Truth to power. -Handling not being able to do something, or do it easily, or do it well. -First principles. What can you toss out the window? -An ability to check an ego
Some people are very comfortable being the smartest guy in the room, or the guy who was in Wired a while back, or the most senior guy in the room if that fails. These people are usually called teenagers, and most of them learn one way or another that there are other really smart people around and just dismissing ideas by swearing or by sarcastic comments or swear words does not necessarily invalidate their opinions.
But not all of them do. Lots of places (including and often especially universities) lose/let go of really smart and otherwise valuable people within a few years because they can’t handle or deal with the size of other fish as they’re bowl is expanded. A test to screen them out (as the one you mention apparently did) is well warranted in many contexts.
Shawn Cardwell
on 05 Jan 12I am not sure how many get to the in-person interview, but I’ll share a few of my experiences from my 37signals interview.
I know this post is about hiring programmers, but a large chunk of my time interviewing for support at 37s was in a room sitting in front of a macbook taking some sort of personal assessment/profile/skills test. Pen and paper were needed, as I was handed a Tufte pad to work out the puzzling parts. Afterward I asked 2 people about the test and they had never taken it. Mr. Fried included.
I love 37s, been a customer for years, and worked my ass off to get to that interview. The team was great, the office spectacular, and the job would be a pleasure, but I left Chicago wondering why I flew in to take an online test. A test that I received neither the results nor feedback.
I really mean no harsh criticisms, just honestly identifying the disconnect I had between wanting to engage in the process of the job with people versus sitting for a test. Don’t get me wrong, I did get to talk with the team. But the testing time seemed like a lost opportunity.
I am sure 37s has determined good reasons to give these assessments, and I must have flamed out at some point. They hired someone else. But from the support side (and again I know it is referencing programming) DHH’s post does not represent my interview experience with the company.
Sure, I wish I had another shot. That day at 37s I saw a lot that I could learn and a lot I could speak into and push ahead. Unfortunately they missed out on a great supporter and contributor. Perhaps a puzzle sent me packing. Life goes on.
I am grateful for the chance I had, but I just wish the support process looked more like DHH is advocating here for programming. Maybe apples and oranges. Maybe not.
Hope this can help the discussion.
And… if anyone is looking for a recovering educator that loves helping people and didn’t make the 37s cut, look me up.
Matt B.
on 05 Jan 12I agree with this article. Unfortunately, I fell into the trap of partnering with a dude who shits gold software and polished code, yet has absolutely no personality, is anti-communication and thinks he’s god’s gift to business because he’s a good programmer.
Anyone ever met this type of guy? Don’t ever dare ask the king for help, because it’s not worth his time to help a mere pawn out.
VariousArtist
on 05 Jan 12When I needed to hire a LOT of people in a short time at a mid-sized company, and was lacking time to interview during regular hours, I simply took the candidate out to lunch. I found the relaxed atmosphere a great way to get to know candidates as human beings, and get “under the covers” a bit more. What they did in their lives, and what motivates them to be alive, became just as important as the career points relative to the job requirements. I never regretted a single hire I made this way.
Fast-forward a few years later and I get the chance to be interviewed at a “social networking site” that no-one uses anymore. I came to the interview holding some choice printed documents and online examples of my work and boasting two recent patents. But that was all cast aside and I was asked to write a basic get/set statement on a white board. I thought they were kidding but it was all for real. The interviewer then stuck his feet up on the table and handed me the pen, leaning back on his chair almost enjoying the awkwardness of the moment. I was subsequently interviewed by 3 more people over a span of four hours, They were barely interested in asking about my work, some of the really cool stuff I had done and showcased, personally, at CES. It was just one white board puzzle after another. I was dumbfounded, and for the first interview ever a little perturbed. I went on to other bigger and better things, and I put it behind me although I have to wonder if this kind of culture was at the root of their failing, once proud dot com social site.
Bob Foster
on 05 Jan 12@Jon It must have been a long time since you interviewed at Google. I’ve talked to several people who have. They all went through a couple of levels of phone interview with sample questions like the probability of finding the blue ball or randomly sorting a deck of cards in one pass. In other words, puzzle questions. Granted, a company that gets a few hundred thousand applications a year needs a way to thin out the herd, but it seems clear only the puzzle solvers actually get an in-person interview where they might be asked to demonstrate they can write a simple program. The rap about Google’s hiring practices: they like young puzzle solvers with graduate degrees. Fine, then that’s what you get. But perhaps it explains another widespread observation, that Google doesn’t “get” user interface.
AndrewM
on 05 Jan 12Back in the day Microsoft would use puzzles in their interviewing process. The idea may have started with them, or perhaps somewhere else.
In talking with people who have done hiring there, it sounded as if the reasoning behind the puzzle questions was that they were giving an IQ test—but without the implications of actually giving an IQ test to candidates. IIRC there may be some legal problems with requiring an actual IQ test.
In any case, it turns out that Microsoft found out that high IQs didn’t necessarily imply ‘better’ engineers – so they stopped relying on them, probably around 10 years ago now.
YMMV, that’s just what I picked up from talking with people who retired from there.
Steve
on 05 Jan 12If doing whiteboard puzzles is such a time-saving technique that magically detects the best candidates for employment, why not save time in other areas of endeavor. Why not have pilots answer a single question before allowing them to fly our families around the country? Let’s save time and have SAT tests with a single question. Heck, there are lots of puzzles on the LSAT, maybe they should save everyone a bunch of time and have one question LSAT exams?
The reason Stupid Human Tricks on a whiteboard don’t work is that a single question is not statistically relevant. If you want to test someone’s ability you would need a standardized test with enough questions and participants to be statistically relevant.
In 26 years as a full-time programmer, I have never had occasion to write code in front of a group. Why test someone on a skill that they will never use? Does it follow that because a person can’t pee in front of a group that they are lousy at peeing?
Dave Sailer
on 05 Jan 12Read “First, Break all the Rules: What the world’s greatest managers do differently”, by Buckingham and Coffman.
Check your library. After I flipped through it for five minutes I went straight home and ordered a copy.
Research-based, out of Gallup. Seriously good.
Great people doing great things in great ways does not involve scribbling on a wall. Ever.
And if the comments from “Kathy Sierra” are from THAT Kathy Sierra, someone please tell her thanks for existing.
Matt J.
on 05 Jan 12The only test you need to give nearly anyone is asking them to read a page from a book out loud. Heck, you can probably scrap the rest of the interview.
Anonymous Coward
on 05 Jan 12I recently left an interview for a front end developer position after they asked me to hand write how i’d write a clock in javascript.
bradford
on 05 Jan 12So, you don’t use puzzles or tricks to determine who you are going to hire. Do you use OS choice?
from: http://david.heinemeierhansson.com/arc/000433.html
“I would have a hard time imagining hiring a programmer who was still on Windows for 37signals. If you don’t care enough about your tools to get the best, your burden of proof just got a lot heavier.”
Adam
on 05 Jan 12@Peter Ashford: Maybe you should warn your candidates that you expect to watch them to code on a whiteboard. Apparently, some of them do think it’s unreasonable. It will save you both some time.
@VariousArtist: Your story is reminding me of a time when I was accused of cheating on my coding assignment because I choked at the whiteboard. The truth was simpler: I understood singletons in PHP but didn’t understand why Arrays are bad at being hashes in JavaScript (at the time). Two different questions on two different subjects, but the accuser saw this as a red flag for dishonesty, walked out the door in a huff, leaving the recruiter to apologize and me to put away my résumé (which he hadn’t read) and make the long trip home. Mercifully, the job market is in our favor. It doesn’t pay to be a jerk.
Edgard Castro
on 06 Jan 12Been there, done that. One of my first interviews had some kind of psychology test that had nothing to do with the position.
At that time, Internet was just starting and I was one of the very few guys that had the knowledge that was necessary to the company to start their business.
I said exactly “fuck it” and left the place. A day later the guy that was interested in me going asked what happened and I just said that i wasn’t interested anymore. ...and took a much better job two days later.
Fucking stupid hiring practices that doesn’t understand real life.
Fuck it. Fuck it.
Nicole
on 06 Jan 12Thank goodness I’ve never had an interview like that!
John
on 06 Jan 12@Edgard Castro, don’t they understand real life? google is not real life? facebook is not real life? those companies do those kind of interviews, there must a reason, right?
Joshua
on 06 Jan 12What a silly post. The only reason we use brainteasers is because it’s illegal to use IQ tests directly. For employers who care about selecting from the top X% of the IQ pool, the puzzles are indispensable. For everyone else, they’re not very useful.
tmarthal
on 06 Jan 12How do you convince programmers to work for 20 to 40 hours on a sample project? What if they currently have a job or are transitioning off a contract and can’t find the time to take off 20-40 hours from their “day-job”?
Do you offer to pay them for their half-week’s worth of time? How do you negotiate rate?
Jim Strathmeyer
on 06 Jan 12Looking at the code I’ve written? So you want to see what I’ve done in my spare time (hint: it will be less if I have less of it) instead of judging my actual work product? Do you think they hire surgeons by saying, show us some surgeries you’ve done in your free time?
Edgard Castro
on 06 Jan 12@Joshua Google does NOT interview like this. It’s a myth. Been interviewed by Google and they never asked for tests. First they phone-screen with some basic questions then they have interviews with more in-depth questions with 3-4 engineers. Not a single test.
Edgard Castro
on 06 Jan 12That was @John, actually. :D
tonetheman
on 06 Jan 12unless solving puzzles can magically get me more customers… i think using them for hiring is insane.
James Bach
on 06 Jan 12After many years of hard interviewing as a test manager (in a developer tools shop, for most of that time) I cam reluctantly to this conclusion: I can’t discover what I need to know about a person from an interview. I do them anyway, because I can collect a few nuggets and impressions that are useful.
The only way I can become confident is by working with a candidate over a period of time. Even that is no guarantee. I worked with one guy for YEARS before deciding he was a loser.
Reviewing examples of his work helps. Doing a small project (a “first date” helps). Quiz questions can help, too if they are realistic, and you don’t worry about the right answer so much as the style of interaction and thinking. All these things help, but I’ve learned not to trust any of them.
Peach
on 06 Jan 12Microsoft also had me write a bunch functions during the interview. The worst part is that they want me to write them all on a whiteboard.
IMHO, it just stupid to ask people to write codes on a frikkin white board. We have Google for a reason. It’s all about locating information now, not memorizing all the frikkin functions in the world.
Just my personal experience. I’m glad I didn’t get a job at Microsoft. The failure of the interview got me to take risk and went on an entrepreneurial route. Never look back again since then. :)
GeeIWonder
on 06 Jan 12Strong words, interesting viewpoint. What did the interviewers say when you said this? Did you suggest an alternative, or invite one?
Congrats. It sounds like the test worked for both of you then, as it wouldn’t have been a good fit for you either apparently.
Benjy
on 06 Jan 12I got through three layers of phone interview at Google and blew it at the all-day on-site interview by flaking out when asked to write perl on a whiteboard. Specifically, an implementation of cron. I’d never written code without a text editor before, I couldn’t handle it, and the interviewer wrote me up as (correct me if I’m wrong, googlers, but I think this is the internal term) a “strong negative”.
Now, many years later, I have my own company and I’ve learned from hiring-and-firing experience that the Google method lacks both sensitivity and specificity: it excludes some people who are extremely talented and includes some people who are team wreckers.
This extends to people’s architectural instincts i.e. how their systems relate to other systems, so I’m wasn’t surprised to see Steve Yegge agonising over how weak Google are at platform design.
Robin
on 06 Jan 12Supporting Example:
As a physics student, I’ve had my fair share of math and analytical thinking. I’ve had courses on numerical methods, introduction to c++ and the like, so I know most basic concepts such as O(NlogN) and whatnot. I boldly think that with relatively little effort such as taking a data structures course and practising this kind of questions, I would probably get reasonably good at these interview questions.
However something would still be the same: I can’t code. I don’t know how things are done in the ‘real world’. I’m too inexperienced to use my tools, due to never having written anything more than simple code fragments.
Hence I agree that from what I’ve heard, these kind of interviews test the wrong thing. Unfortunately, there are probably not that many possible frauds like me, else this system surely would be crumbling
Teeves
on 06 Jan 12Haha. I got asked how many street lights I thought were in Edmonton in the first interview I ever had. I did not get the job…...
Dave Hayden
on 06 Jan 12I’ve never used puzzle questions in interviews, myself. I’ve only ever interviewed for coworkers, not underlings, so I only look for how well I’d get on with them, make sure they have a sense of humor and are generally interesting people. To check technical ability we ask for sample code. But on the other side of it, people are discarding puzzle questions as somehow insulting to their dignity and I think that’s missing the (optimistic) point. I went off on a jag and wrote more about that than was worth posting in a comment field:
http://opaque.net/~dave/?p=133
Roland
on 06 Jan 12It’s not about the code that someone actually writes – its about looking at how the programmer thinks and communicates. We don’t mind if someone puts a big box up instead of some real code so long as they say “this bit should do this but I’d need to look it up”. Programming test also show up people with zero programming knowledge trying to blag that job but it also lets those with real ability shine through. If your testers are too dumb to spot these guys then don’t work for them regardless of how you did in their test. But hey, if you don’t like whiteboards, give them a clean computer, access to the internet and an editor and tell them to get on with it – no excuses for not getting it 100% right but does it really show you that much more?
Likewise, I ask analysts and designers to draw something. If they can’t draw on paper or a whiteboard in such a way that I can understand them, how are they ever going to communicate with anyone else?
Alexander Turner
on 06 Jan 12I have had two of these, one good and one bad.
1) A set of questions about sorting and graph searching which I then coded on paper in C++. It was a great job interview and a great job and I used the same idea several times to interview other people. The key was to inspect someone’s reasoning skills much more than their knowledge of a particular algorithm.
2) A set of questions on a web based thing about Java. It had questions on the awt for goodness sake! Who uses the awt enough now to remember it (nowadays)? I would positively not hire someone who knew awt, jdbc by heart at the same time because the chances are, they would be a one trick pony.
My conclusion, if you find a way of testing a persons brain – great. If you are testing their knowledge – bad.
Yo Momma
on 06 Jan 12How many times did you not make it into Microsoft, Google, or Yahoo! (when it didn’t suck) before you convinced yourself that problem solving isn’t important to computer scientists? Of course it is! People that write code for a living aren’t necessarily computer scientists, they’re just using tools that others have made for them. I’m not saying that everybody has to be DHH or write a language (or in assembly or binary), but the world would certainly be better off with a few more Vint Cerfs, Tim Berners-Lees, and Dennis Ritchies (RIP).
Most individuals that do well on these puzzling interviews are those that enjoy (not dread) fun problems. It’s these kind of people that I’d want on my team. Most smart nerds do understand that business or social connections are also a hard problem, and they can even hack stuff together if they want to (I swear), so I see no real reason why one wouldn’t prefer an engineer that can pass these tests you loathe. Those that are over prepared or qualified will be able to hang when they’re suddenly just prepared enough, rather than quit, ignore, or continue to sap money off of your company without doing anything. Additionally, it’s been anecdotally proven (to me, at least), that good engineers can deliver 10-50x better than average to me engineers, at only (let’s say) 2x more salary. Now there’s a deal!
It’s the ability to solve hard problems (like search, maps, good email, social graphs, scale) that make you the gajillion dollars that Facebook and Google have. Just sayin’.
Andy
on 06 Jan 12I agree that this kind of puzzles are a silly – but they can offer a valuable insight into how someone goes about tackling a problem. i.e. they show your thinking process. That’s far more important than having someone just write the correct answer on the whiteboard. I have three personal experiences to share:
1) I was once asked to write some Perl code – on paper – with no reference books or access to the internet. I’m not sure what this was testing, my memory for Perl syntax or my handwriting. I was furious that I was being asked to do this in an interview, and told them so. I didn’t get the job.
2) I was asked to supply some code I’d written to the interviewers beforehand. All of the code I’d worked on at that time was owned by my employers, but I sent a small sample anyway. In the interview I was subjected to a hostile code review with the entire 6-person team pulling the code apart for anything they could dream up, without any appreciation of the constants (especially time pressure) that code was developed in. They even criticised the length of the lines and lack of line breaks – I never intended to print the code out on 80 character lines FFS. ! I didn’t get the job.
3) I was asked to write a small iPhone app based of a 1-page spec (after answering a question paper with such gems as whether Obj-C allocates on the stack or heap (correct answer: totally fucking irrelevant)). Writing the app was quite fun, I was allowed to work as I normally would, and had time to add a few embellishments. I got the job, but didn’t take it.
So from those experiences I learnt that dreamed-up exam paper questions are utterly useless unless you’re testing for memory retention. Instead, working though a problem, and even coding it up there and then in the interview, demonstrates how you work and what you know much better than any amount of questioning.
Jonathan DeMarks
on 06 Jan 12What do you think about a service like DevPlusPlus? It tests on actual skill, and can be assigned out as homework to get a general feeling on how well a person can perform in a variety of languages.
txklgr
on 06 Jan 12Oh cool ! A bunch of copy-paste programmers that complain that they cannot “do parlour tricks”, programming without others doing the thinking for them.
I’m sure you are perfectly capable CRUD programmers. Now let’s see you actually implement a nice new algorithm … obviously without the internet.
You see, interesting problems do not have their solutions on published on the internet. If you depend on that, you cannot solve them. Not now, not ever.
But if you prefer it not to be in riddles, sure. How about difficult datastructure questions :
Fastest sort algorithms : give sort algorithms that are O(1), O(n), O(n log n) and O(n2) and illustrate in each case why you’d use them.
Don’t pout, if you’ve gotten a datastructures course you will have seen the answer to this question. It’s not easy to find on the internet. You still need to hold your own in a discussion about implementing these algorithms.
What’s a SAT solver ? Give at least 3 theoretical problems and present mitigating factors. What have you used it for and why ?
What does rocket stabilization have to do with Linear programming ? Will this idea work in practice ? Why/why not ?
General AI : give the various basic search algorithsm (if you can’t get to 5, don’t bother). Implement A* right here on the blackboard. Make the obvious improvement to A* ... I need an algorithm that finds me good trading strategies based on long lists of datapoints. Essential is that I can explain to the board what those algorithms are doing. What would you advise and why ?
If you’re interviewing for C+. What’s “typename” ? What purpose does it serve ? Give the different places in memory where C+ variables might be allocated (if you only give 2 you’re not including the linker behavior) ?
If you’re interviewing for a Java position … give the different ways template arguments can be specified (how many ways do you get). What is the ternary operator ? Give the different meanings of “final” in all the different cases. Java is such a tiny language that you really can be asked the last tiny little detail, in a way nobody would expect you to know about C++, except perhaps the compiler groups.
Are these questions hard ? Oh golly. Unfairly hard ? Then go and learn about them ! The whole point is not to find programmers that can implement a 3 screen android app, the goal is to find people that can write an android app that can take a picture of a sudoku puzzle, ocr it and then solve it live on screen.
The questions are so very hard because there are not so many jobs in this, and you really can’t lower the complexity requirement all that much … because you’d just hit a wall and never pass it.
Would you prefer questions like this to “parlour tricks” ? Just ask your interviewer. If they’re serious I’m sure they’ll oblige. The whole idea of parlour trick questions is to get into one of the above discussions. The real test is not getting the parlour answer right, the real test is holding your own in a discussion about the above topics (again merely getting the answer right to the above questions is also not good enough, it’s the discussion that matters). Missing one of the parlour tricks usually will result in the interviewer giving you the answer. If you then go mad and insult him thinking you’ve “lost” the interview, yes then you probably did. But not because he had to give you the answer.
And that silicon valley investors have felt the need to hire cs experts to write crud apps … well I had nothing to do with that.
Pradeep Kumar
on 06 Jan 12I am puzzled a program, 22/7 upto to 300 decimal point
code lover
on 06 Jan 12Nonsense post… but comments worth reading..
Artur
on 06 Jan 12I had this kind of experience once, and I still think it was pointless. When I myself had to the opportunity of hiring people, I asked about software engineering fundamentals, using a written test, which I think allows people to develop their thoughts with more freedom and confidence. But these discussions make me think about how we write code and what one really needs to be a good developer. And I still think this type of puzzle-based selection is shallow and empty. Maybe a good practice to feed an image of “coolness” for the kind of company that values this.
Michael Gray
on 06 Jan 12I see these situations in reverse: Companies who demand these idiotic questions are not worth working for, and as soon as one hears such a thing, one should just pack-up and leave. Pronto.
Karl
on 06 Jan 12As in independent consultant/programmer, I’m going to bookmark this so the next time a hiring manager gives me one of those quizzes, I’ll have something handy to send them.
Just Dave
on 06 Jan 12We use Simple white board code questions as a filter. We start with trivial tasks, and add constraints.
Understand that it’s just one tool in the interview. But it is a filter. If you can’t do the basics….
If you cannot nail reversing a string, and then show the changes required to make the code more general (templates, or for other problems, delegates, using interfaces) then you don’t pass the filter.
It does not matter if you remember the exact syntax or function names, that’s what resharper / google is for. But if you start having loops within loops to reverse a string, the interview will be politely short. (this happened!)
Again, it’s one tool. Show us you know the basics, and show us how you deal with addition constraints.
Then we move on. What have you worked on, what part did you play in that, tell us about etc etc etc
I have had “senior engineers” who could not cope with these simple tasks on a white board.
“Trick puzzles” mostly tell you if the candidate has read up on the trick puzzles web site.
CA Reed
on 06 Jan 12I had a job interview which started off with 20 questions. Around question #15, I was not familiar with the topic he asked about, so we continued on. After we finished, he told that I was one of the few that got #15: he made it up. He said he would get all sorts of answers to a question where the premise didn’t even exist.
AJ
on 06 Jan 12I have had my ‘more-than-I-deserve’ share of nightmarish experience related to puzzle solving, and text-book questions in interviews. I have had an interviewer ask me, “In 19xx, a famous computer scientist design a algorithm that changed the world of computing. I want you to write that algorithm on this whiteboard”. WTF? Really?
I have been known for delivering quality products, and always on-time. I don’t memorize algorithms – that’s what Google is for to be honest. And then they teach us in CS-101; “Do not reinvent the wheel”.
I don’t need to solve puzzles that are more mathematic in nature than they are analytic. I don’t mind discussing, and solving a real world problem. In fact in one of the best interviews I have had, the interviewer gave me a laptop with bunch of IDEs, SDKs installed, and with full internet access. He gave me 3 different problems to solve, and just watched me while I was at it. He saw me refer Google for some weirdish APIs that take 11 different parameters, he saw me design the solution on a piece of scratch paper, he saw me write code, and he saw me solve the problems.
Next, he took me to his cube. Showed me some code, told me the problem he is having, and asked me how would I go upon fixing it. As we were talking, he followed what I was saying, and changed the code accordingly. He ran it on the debugger, found some issues, and then we discussed how to fix those issues. Not saying I fixed his bug, but he got a taste of how I work in the real world, he got a feel for whether or not I will fit in his team.
With such an interview, even if you don’t get the job (by the way, I did get it), you don’t feel too bad. You feel you were given a fair chance of demonstrating what you know best.
As opposed to the interviews I have had at Amazon, Microsoft, and other such technical giants who ask weird algorithms, puzzles, and 1970s stuff, I’ll take a”close-to-real-world” interview setup anyday.
Martin
on 06 Jan 12I don’t mind doing a simple quiz but I have been to interviews where they ask you how you would solve a business problem that are currently having. You don’t get the job but a few weeks later you see that they have solve the problem.
LALa LAne
on 06 Jan 12I recently had my first interview in about 14 years. It was for a SysAdmin position at a Game Dev. company.
I am a SysAdmin by trade, and a midnight programmer for myself.
My interview was at 10AM on a Monday. The first thing that happened was I was given a chair in a public hallway (next to the reception desk, where people had to walk around me) and 10 minutes to answer like 10 questions.
Mind twisting, ‘trick’ questions that if you don’t know the trick, you’re not going to get answers in less than 10 minutes.
Anyhow, I bombed. My mind was prepped into ‘people-person’ mode, which takes a lot of effort for a computer geek. :) I wasn’t expecting a (ridiculous) logic quiz for a network/system administrator position.
WHY OH WHY would you put someone in that kind of pressure situation without a least a bit of warning?
In the end, after an interview that involved me explaining how DNS works, followed by being asked way too many questions about my opinion on programming languages, and which software projects I had completed, I’m glad I didn’t get it.
It was apparent they didn’t really know what they were looking for, or at lest they were advertising something different than what they wanted.
Their answers to my basic inquiries about the status of their network and data didn’t instill a feeling of them wanting to do anything different/better: “We keep our data backups in a vault in the basement, and that’s all we need”.
Anonymous Coward
on 06 Jan 12@txklgr: Have to agree with you, except for the part where you “flamed” all the others for being copy-and-pasters ;-)
I don’t think that funny quizzes and tricky questions are needed to get into a discussion that reveals technical expertise. I try to make it sounds like child’s play, because that is what brings nothing.
Its the same with interviewers that try their best to get you to show weaknesses and flaws. Interviews should be about getting to know each other, and i don’t think that can be done by asking funny questions and doing psychological tricks.
Your’e right about the technical discussions, but technical detail discussions don’t require parlor tricks.
Kirk
on 06 Jan 12I’m not a fan of puzzles or brain teasers in interviews. I like the quick preliminary phone interview to see if communication skills are ok and also any red flags for strange personality or attitude issues (mesh factor is important to me). I like to ask a few technical questions in the preliminary phone interview as well but not dive too deep until it is determined they are worthy of in person interview. Once in person then ask a broad range of technical questions (troubleshooting performance issues, language features, database skills, etc) but also get into specific tool issues. I mean we as developers use tools and I expect them to know the difference between pressing F10 vs F11 when debugging in Visual Studio!
Seneca
on 06 Jan 12I’d say it is just cargo cult interviewing. At Google and Microsoft, some of their important hires are for people who will develop the “next big sort algorithm”. Unfortunately, if you interview one programmer that way, I guess they feel they need to do it to all their programmers, so it becomes the company norm. From there, everyone trying to cargo cult success follows the same pattern. While not exactly a programming puzzle, where I work, we do use Codility before we go to the bother of doing a phone or in person interview. It is amazing just how many people apply for a job and can’t program anything.
Enzo
on 06 Jan 12Using these kinds of interview parlor tricks doesn’t produce good candidates or good products. Google hasn’t produced anything of value in years and they hire using this method all the time. I’m more concerned with personality and whether the candidate is a good fit with the team. Is the candidate someone who I want to work with and spend time with? I don’t want to work with anymore anti-social weirdos who can’t hold a conversation. If the candidate is lacking certain skills that doesn’t necessarily concern me. What concerns me is whether or not they can get up to speed fairly quickly and become proficient because the vast majority of software development is really not that hard at all. It’s just not. Most of the time you are not solving anything that hasn’t been done a million times already.
Brian Balke
on 06 Jan 12We do give a proficiency test, but the primary reason is to try to get a sense of skills coverage. In a small shop like ours, where quality is far more important than quantity, everyone has to do some design, documentation and testing, along with coding. We usually propose simple problems, and test breadth rather than depth.
In a oral interview, I would prefer to ask questions that I don’t know the answers to. I like to set it up as a problem-solving session, and put the candidate in the position of being allowed to lead the discussion at the white-board.
DHH
on 06 Jan 12Shawn, the hour you spent was taking the caliper personality assessment. It helps us figure out whether their personality is right for a certain position (the qualities needed for support vs programming are quite different). Caliper then gives us their opinion on whether they think there’s a match based on the job description and we factor that into the decision. But for hiring support people, the overwhelming weight is put on the writing samples (that’s what you’ll do all day) and what the rest of the team thought of you.
tmarthal, when we give people a 30-day trial, we pay them for their time. Generally the pay is 1/12 of what the yearly salary for that position would be. Some trials for things like design are shorter and are paid accordingly.
Tim
on 06 Jan 12The problem with everyone who would just google the answer during their normal work day is that they probably don’t have the skill to evaluate the code they get from google. I have seen plenty of situations where someone pasted in code they found that solves a similar problem, but turned out to be horribly inefficient because of the slight differences between the problem they were trying to solve and the problem from whatever blog post they read. It is also a matter of efficiency. Someone who can sit down and code for 3 hours at a stretch is going to usually get more done than someone who has to alt-tab over to google every 15 minutes find a piece of code and customize it to their situation.
I am not saying that they need to write their own ORM on the board, but if they can’t write a method to sum all of the numbers in an array in the language of their choice, that is a problem.
As for those who say that it is too high pressure to be trying to code with people watching you, that is part of what I am evaluating. If you can’t deal with pressure I want to know that before I hire you and you crack because of a tight project deadline.
If you are such a prima donna that a particular type of question makes you write the company off immediately, my team is probably better off without you. You are being as narrow minded as you are implying the interviewer is.
Phil
on 06 Jan 12This subject came up on Brad Feld’s blog: http://www.feld.com/wp/archives/2011/12/more-on-recruiting-programmers-to-your-startup.html
these guys like doing a 4 hour coding challenge.
Not Dennis Ritchie
on 06 Jan 12One of my favorite interview stories: I was once being interviewed simultaneously by a CTO and a developer (small company). Things were going well. They had even mentioned money which was in my ballpark. Oh, this was a web development job with .NET and SQL Server backend, both of which I had experience with. I think I did well with the technical questions. At some point in the interview JSON came up, I basically said I had heard of it, it sounded interesting, but I never used it. The developer says, “Okay, we’re just starting to use it in some places, I’m not worried about you picking it up.” Conversation continues… Then the CTO points to the whiteboard and asks, show me how you would do X in JSON. I don’t even remember what it was. I thought he was kidding, because we had been joking around at various points in the interview. Well I got up and wrote: www.google.com. Well apparently that didn’t go over too well with the CTO. The developer was smirking though. The interview kind of just ended shortly after that. Afterward the headhunter that sent me on the interview called and said the developer really liked me but the CTO didn’t think I would work out but he couldn’t find out anything specific. Fast forward a year or so, I get a consulting gig at a company and the developer who had interviewed me is working there. He remembers me, especially for what I wrote on the whiteboard. He told me that the CTO wanted his developers to just know everything, “I’m not paying them to google!”. So when I wrote that on the board there was no way I was getting the job. Of course the developers did use google, but never let on to the CTO. In the long run it worked out for the best for me anyway as the company tanked. The developer I met left when they asked him to work for nothing.
Tim Downey
on 06 Jan 12We’ve basically boiled our interviewing down to to easy things. We send out a quick takehome problem that demonstrates basic proficiency with Java (no tricks) followed with a simple onsite to determine if the candidate is a “jerk”. Not being a jerk is more important than proficiency. It’s easier to learn the tech than to stop being a jerk.
If they clear that, I make an offer. We’ve hired a dozen or so people in the last year and have had great success.
John
on 06 Jan 12By the way, I’ve seen that some companies ask very hard questions, they want you to solve lot’s of CS problems, Big O, etc, and when you get the job….surprise, their code is a mess and they just want you to do simple stuff, so, what’s the point of doing that then? I am not against those hard questions if the job (and pay) is according to what they ask in interview.
Stephen
on 06 Jan 12I was once “tested” by a guy who knew squat about programming; he used a new “test” that simply asked questions about how to use certain controls. As I had never used a lot of the controls I of course “failed”. I explained that I had not used these controls, but could certainly learn how. I was later tested by the head hunter that got me the interview; his conclusion was that I knew what I was doing. Thankfully I did not get that job; I could see it would have been an impossible task as the head of the project was not a programmer, and most of the controls I was “tested” on would never have been used in the project. The jobs I have landed all included me supplying code samples and some basic discussions of problem solving and general skill sets. IMOH, these “test” things are a waste of everyone’s time. I want to see what the candidate can do, not theory.
Tim Cull
on 06 Jan 12totally, I hate quizzes.
I interviewed at Google once a couple of years ago and totally bombed. The irony is, if I’d had the exact same interview right out of college I would have rocked it. Does that mean I was a better programmer right out of college than I am now after 13 years of experience? I think not.
When I conduct interviews, I’ve got the process narrowed down to the same 2-5 questions, plus asking about things listed on a resume to see if they really know what they say they know. The questions I ask are all about seeing how this person reasons, how tightly they hang onto bad ideas, and whether they ever think about what’s going on “under the covers” of their chosen technologies. After conducting 100+ interviews over time, I’ve found this process to be extremely predictive of actual success in the job.
Steve Naidamast
on 06 Jan 12I completely agree with the author of this post.
I have been in the IT field 35+ years and still come across the most ridiculous interviews whereby the interviewer asks the most nonsensical technology questions including performing with the “whiteboard”...
There is only one way to gauge the worth of a perspective candidate and that is to find out how much they enjoy doing what they are doing… or want to do… If you can ascertain that than half the interviewing job is completed since most professional developers who enjoy the work actually do it quite well for a company’s requirements.
Interviewers who spend endless hours interrogating technicians about an entire slew of technologies usually are doing it for their own benefit and not that of the company’s. And most often these same people haven’t a clue as to how one builds a solid, well functioning system.
As other posters have noted, most technicians that can successfully pass such interviewing processes are very good at minute details but really can’t program well in a team, oriented environment since they already believe they know everything that they need to know…
John Bannick
on 06 Jan 121. Read some of their code and discuss it with them 2. Have a short design discussion with them 3. Ask a former boss how good they were
I was a hiring manager for years. Now I code again. it’s more fun.
Jon
on 06 Jan 12... could just flip the 3-minute one 3 times. >.>
Robert Buckingham
on 06 Jan 12My first programming problem was for an assembler language program to simply add two numbers. I had had no prior training and it was my first time programming and my opponent was an IBM programmer who was also working on his master’s degree in computer science. His score – 11 instruction. My score – 12 instruction – but, my program had an instruction at the end that took it back to the beginning where is stopped and waited for the next input. – His just executed once and then simply ran off the end into no man’s land.
As for the puzzles, they should allow you the same resouces they allow their programmers in their environment – they should allow you access to a computer complete with the necessary development environment and internet access.
Robert
on 06 Jan 12Once I did not get hired for a Perl position because I could not remember off the top of my head how to use hashes. I worked a lot on the reporting side of Perl (converting COBOL programs to Perl) and most of the focus was either greenbar reports or HTML/PDF/MS Office outputs. I never really used in day to day coding and it took a few for me to remember how they worked. Before you start flaming me on a “basic” perl skill, I live in the work of physical temp files and arrays. Just my style. Fortunately I was hired for another position at the same company a couple of weeks later. I see those people every so often and it doesn’t look like they are too happy to see me around.
Robert
on 06 Jan 12On a side note, I think personalty tests are a waste. I have both taken and see the results of tests others have taken. I was once on an interview team and we all recommend not to hire a candidate. But the test said otherwise. As a result, we had to deal with a person whom was a very bad fit. I thought the guy was an ok person, but just did not fit and I knew he was going to hit the road within six month…and he did. My boss just couldn’t understand why he decided to give his notice and I told him he wasn’t going to fit and those tests are put together by marketing people who think most organizations work a certain way. Organizations are like snowflakes…NO two are the same.
Shawn Cardwell
on 06 Jan 12@dhh Thanks for the response. All the best.
doctor_shim
on 06 Jan 12Unless the job requires applying defined functions to software, or generally being versed in advanced mathematics – asking someone to write code on a whiteboard is more of a indication of the quality of the employer.
I have had potential employers quiz me on extremely simple sorting algorithms, demanding I implement them in “the language of my choice” on the whiteboard. One of them was for a low-paying JavaScript position. Unfortunately for me, their whiteboard did not interpret or compile most of the languages I work with, and UML apparently was not an option for them!
Analgesic
on 06 Jan 12I’ve noticed that HR people tend to be very lazy (more so than programmers) and the point of the whiteboard questions or the silly puzzles are to give a quick (if inaccurate) metric of that person’s skills.
They don’t particularly know, or care, about programming or the big picture. To them it’s just about throwing out the “gotcha” question and seeing the response.
Teresa
on 06 Jan 12I don’t like quizzes and write your code on white board thing as well. And the fact that I look up on the internet for code samples, explanations or articles doesn’t imply that I can’t write it from scratch or that I can’t judge the quality of what I find. The same way if I have to read a couple of man pages to remember the particular syntax of a shell command doesn’t tell that I’m not a good programmer. One thing that I like to do in interviews is for example some code review, debugging, or even participating in tech support calls.
Me
on 07 Jan 12There’s no such thing as a great interview; pretty good interviews are about the best you can hope for. Practically every interview style can end up letting in poor candidates and/or filtering out good candidates.
When I was contracting, I interviewed in many different places, and were all sorts of interview styles under the sun: CS concept-based, puzzle-based, API memorization, behavioral, personality-based, etc. A lot of people would google websites with sample interview questions and ask the same questions that I’d heard a dozen different times because they couldn’t bother to create their own. There were some interviews I thought were terrible, but I respect their right to do whatever they want and fail to get good candidates if that’s what they truly wanted.
The interviewer’s main job is never doing interviews; it’s doing something else in the business, and interviews are generally side tasks that they do because they have to. Even if I think they do bad interviews, I never complain; it’s their business to succeed or fail in, not mine.
Matt tate
on 07 Jan 12All this talk of interviews is making me stressed.
Matt Grdinic
on 07 Jan 12Interesting take but not convinced with the core argument.
I believe their truly is a divide in the quality of applicants that “trick” tests will expose, and that such tests are invaluable provided the position matches the test at hand.
So let’s say you take an average programmer like me and ask me to whiteboard some code.
Now ask, oh I don’t know, this guy to do the same:
http://www.adobe.com/technology/people/seattle/agarwala.html
Guess who’s going to do better? And you know what, that guys not just smarter, but smarter to the point of making me look stupid in any test situation regardless of content and question.
But oh I should say, I run my own (successful) software company so I MUST be just as qualified and worthy a candidate…but I’m not. I say that not out of some false sense of humility but because it’s true : )
The world’s full of super smart people and I’m not one of them—whiteboard tests and the like would prove it.
Thankfully I’m also full of good ideas for my company so it continues to thrive—though I would argue not so much because of my coding skills, but because I dictate the pace of innovation and development. If something’s too hard I have the luxury of not implementing that feature.
I have, in other words, the luxury of time. I’m on my own dime, a luxury working stiffs and those they hire do not have.
Maxx Daymon
on 07 Jan 12Microsoft learned a long time ago that people who were good at brain teasers were good at brain teasers, not necessarily business or code. After Google and Facebook grow up some more, they may relearn the same lesson. We seem to spend a tremendous amount of time in this industry relearning the same old lessons over and over. Perhaps a massive case of NIH syndrome.
A lot of good information in the comments as well:
http://www.sellsbrothers.com/interview
Another anonymous coward
on 07 Jan 12What would you call a good programmer? You don’t explain this at all, so it’s difficult to judge whether math problems, or API quizzes are relevant for you when hiring.
If someone isn’t able to solve math problems and has no idea of the APIs available in a language he pretends to know, is completely oblivious to data structures and algorithms, and isn’t able to write even a sketch of some real code on paper (not necessarily compilable code), IMO there’s a very dim chance that he’s able to write usable code at all.
IME, you get a high speed team churning out very clean and well structured code when a few conditions are met: the people like working together, the team’s structure has been stable of at least half a year or so, their programming skills and knowledge are approximately at the same level, and they are quite skilled. Essential skills included: design patterns, the habit of proper naming and writing short methods, algorithms and data structures, TDD, or at least unit testing most of the code, scripted testing, using a continuous integration system, continuous refactoring, and, very important constant and effective communication with all project stakeholders.
Now, it’s pretty much impossible to hire somebody who already has all these skills – such people already have a job they won’t leave, and their employer would probably chain them to the desk if they’d want to leave. So what you’re looking in interviews is not what somebody knows, but what his/her potential to learn is. Problem solving, attention to details, the ability to cope with unpleasant situations, decent communication skills and (you may laugh) a good sense of humor are definitely qualities which you absolutely need to test.
Just my opinion. Based on about 15 years of experience in the industry.
bezboznik
on 07 Jan 12my response to when it happened to me:
“you’re kidding, right? you expect me to me memorize the fucking dictionary of C++ and regurgitate it back to you like some fucking asshole programming 101 class exam in college? fuck you.”
and i walked out.
Dejan Marjanovic
on 07 Jan 12When I interview engineers, just ask them to describe me something they listed in their CV. Usually, personality is all I care for. EVERYTHING comes from it.
Being good well structured coder is just the reflection of inner state of mind – people without peace of mind can never have such ability. Those who have it even do not notice they are good in coding because of it.
Having ability to code well necessarily leads to design patterns, meaning engineer will be able to code a pattern without knowing for it from before.
Communication skills are mainly influenced by ability to respect other people and to have an urge to express creativity in positive way. Again, character.
In order to notice all those things, positive atmosphere is necessary, meaning no feeling of testing, judging, etc, so that candidate can become as relaxed as possible. And after some experience, it becomes quite obvious what kind of person each of candidate is, and what can be expected from each.
It doesn’t matter if someone knows this or that algorithm. It matters is he/she able to learn it without effort and adjust it to given circumstances. For that, you need highly energized minds. Again, character.
Daniel R.
on 07 Jan 12It’s funny you’re writing on this while Facebook is about to launch this year’s Facebook Hacker Cup, which is a worldwide scale riddle competition used to hire new people.
While I totally agree with the fact that your ability to solve a mathematical riddle has nothing to do with your actual coding and work skills (your ability to write clean code, to document and do unit testing, work with others, ect.), it is still a fun way to get people (or at least the geek ones) interested in your company and talk about it. It’s always easier to recruit people when they’re coming by themselves instead of having to go get them.
Besides, this is good for the image of tech companies since the ability to solve this kind of problems is associated with the idea of a superior intelligence. This is a big misconception but anyway that’s what people think. And in the end, since thousand of people sign up for this kind of events, companies can still find some really skilled people that happen to also be good at riddles.
To find new talents and get people talk about you, Facebook has the Hacker Cup, Google has the Google summer of code, and you guys have Ruby on Rails.
Cheers.
MikeFM
on 08 Jan 12A couple minutes talking to somebody about their experience is usually enough to feel out how good they can be. It’s mostly not about what they know. It’s about being engaged enough to teach themselves something complex and work through real problems. In computers everything is changing rapidly and we are often asked to do things that we’ve never even heard of before. Sometimes we even get to figure out the question as well as coming up with the answer.
Being able to figure out puzzles isn’t a bad indicator BUT it’s not nearly as good as a good portfolio. Look at a company like Google.. sure they hire people with puzzles but people with good portfolios get their product bought.. largely to get the programming team. Call it a few million dollars as a signing bonus.
itoctopus
on 08 Jan 12Amazon, Google, Microsoft, and the likes have all these crazy puzzles that they ask you during the interview. It’s either you know beforehand the answer to that puzzle or not. Anyone can invent a puzzle that nobody will be able to guess, but once other people are TOLD how to solve the puzzle, then it will seem very simple and logical. The problem is that these interviews amount to nothing when it comes to evaluating new staff.
The 4 main jobs I had my life working for others (not for itoctopus) I was asked to do a project, and everytime I did the project and I exceeded the expectations. Everytime I was in a hiring position I gave a project to the candidates. Some candidates used to have other people do the projects for them, but they were easily caught during the interview process.
Doug
on 08 Jan 12It seems to me that things like quizzes is easier for the interviewer than the interviewee. Ultimately, big companies that have hundreds or thousands of applicants need to be weeded out. These types of games makes it easy to eliminate candidates without having to (a) spend the time to really get to know each candidate (perhaps prudent) or (b) be forced to use one’s judgement – a means of attracting scrutiny from co-workers with different opinions – to select a winner from mostly evenly-matched candidates. My theory is that hiring at random would be about as effective after you figure out if the guy isn’t a crook, is decent to communicate with, or is otherwise a fraud.
Assaf
on 08 Jan 12Some of the quiz-mentality is a result of organizational pressure – interviewers are expected to reduce risk of new hires not knowing their stuff. Questions like “why didn’t we give this guy a C++ test?” take discipline not to ask when it turns out the guy can’t pick up the language fast enough.
But test-driving candidates by hiring them to work on small projects is not a good solution. That’s why we do the “GitHub Job Interview”: http://blog.gigantt.com/2011/12/github-job-interview.html
Stefan Kühn
on 08 Jan 12In a coment up here (3rd coment) someone (nic) told us, that large companies like Amazon and Google must hire using theese bizarre techniques just because of their size. I dissagre on that. The problem is that many large companies don’t trust their “bosses on the lowest level”. Most large companies have a central department that does all the hiering. To manage, they have to automate (!) the hiering prosess using this bizarre techniques. It would be better to delegate the evaluation of the new employee to the “boss on the lowes lewel (say “group-boss”)”. This boss mostly has the responsibility for 5 to 10 employees, and also is the one that hast to work with the new employee for the next years!!! This way the group-boss hast to manage one or two applications per year, wich would be no problem… The trend towards centrilized processes shwows imo the lack of confidence in the own management…
Paul Kelly
on 09 Jan 12We want to hire programmers. We want to know that they can program. We ask them to do some “whiteboard programming”. We also say that we don’t expect them to get things syntactically perfect on a whiteboard, and that we don’t expect them to have APIs committed to memory, or to know the best algorithms without access to reference information.
None the less, we expect candidates to be able to write some code, to be able to explain it to us, and to have some idea of how it would perform. I don’t understand why so many posters (or the writer of this blog) are offended by the idea that you would ask a candidate for a programming job to demonstrate that they know some programming. “Show me some of your code” fails for two reasons – it discriminates against those straight out of university, and – guess what – people don’t always tell the truth on their job applications. Hiring the wrong person is always expensive, even though you can let them go during a probationary period. Hiring the wrong person isn’t fair to them either – it does no-one any good to be hired for a job they can’t do.
Although occasionally we might have passed on some individuals who probably could do the job, it has certainly filtered out lots who would have absolutely no chance of doing it, and those who have passed it usually work out pretty well.
Richard
on 09 Jan 12Using any public code repository in order to determine a programmers skills is plain wrong. Of course it can be part of the conversation but it should never be used to exclusively reject a candidate [http://wp.me/p1C1sQ-9I]
Mike Bethany
on 09 Jan 12In one interview, for a job I didn’t get, I was explaining how I built and trained an Artificial Neural Network to identify and OCR license plates in images from different cameras at different angles and light conditions and the guy asked me, “Can you tell me what a ‘case’ is?”
My brain was on such a high level concept wise it threw me. I asked, “You mean in SQL?” He said, “Uh, sure.” So I racked my brain trying to remember what a case statement in SQL was but couldn’t for the life of me.
I asked him what it was and he gave the worst explanation for a switch/case statement I’ve ever heard. I had no idea what he was talking and didn’t figure ott what he meant until I got home. Palm -> forehead -> repeat.
Lesson learned: When you’re being interviewed by a guy with 20 gold necklaces and a shirt unbuttoned to his nipples keep it simple.
Developer Art
on 09 Jan 12Puzzles is one of the hiring strategies which simply don’t work. Sadly those who employ them couldn’t care less. For them it’s a game.
What they don’t realize is that the programming community is actively developing a litmus test of bad companies you should avoid working for. As the first impression starts with the interview, the companies get filtered out just as swiftly based on their barbaric interviewing techniques. They don’t get people and consequently ruin their reputation as the word spreads out like the wildfire.
I strongly believe that the hiring of programmers should be based on choosing the right personality because everything comes from it. Diplomas, code samples, references, they all mean nothing without the right personality. The most skilled programmer could be a complete jerk who can completely ruin the team spirit in 2-3 weeks. I’ve seen this happen. Test the personality before everything and only then consider additional criteria if you must.
Here’s my opinion on the subject:
The technocultural fit
Justin
on 10 Jan 12I think this depends upon the work which recruited employee is going to do. If it is daily changing requirements environment or R & D work, then it makes sense to check the logical ability of the person. Again standard puzzles will not do in this case. Interviewer need to ask the purely logical puzzles in this case. Here at there are such puzzles.
Justin
on 10 Jan 12I think this depends upon the work which recruited employee is going to do. If it is daily changing requirements environment or R & D work, then it makes sense to check the logical ability of the person. Again standard puzzles will not do in this case. Interviewer need to ask the purely logical puzzles in this case. Here at http://www.wpcStylePuzzles.com there are such puzzles.
Nobody
on 10 Jan 12I totally agree with this article, the first paragraph describes exactly what happened to me recently in an interview. I had to write ridiculous Javascript expressions, or “puzzles”, like +!{}[‘foo’]. Nothing to do with the actual complexity software development.
I consider this practices a missed opportunity, for both the interviewers to pick the right person, and for the interviewed, to know his/her actual level or value in the job market.
Tyler
on 10 Jan 12Whoever said showing example code is discriminatory to people out of university, that’s an even better example. You can tell if someone is curious and intelligent if they have their own projects and work. Ask if they have any home-projects.
Coding done by people at home will tell you a lot more about the person than any stupid quiz ever will.
Also, by using a whiteboard, you aren’t only testing their coding but ability to function in unique environments. I consider this bad for most companies. A great programmer will fail just because it’s not a familiar vehicle for their coding.
Puzzles aren’t indicative of anything apart from logic and quick-thinking. Both can be useful in programming, especially for a large company solving unique problems, but DON’T REINVENT THE WHEEL. The majority of programming is just going to be implementing a variety of simple solutions together in one problem and the majority of them, if not all, have already been discovered by someone else.
One interesting example is a young-animator posted a parody of Bambi (He called it Bambee) on Newgrounds. It was violent and crude but also hilarious and very well animated. It was quite an interesting shock to the community when Dreamworks approached him offering him an entry-level animation job.
Also, Companies should give basic but interesting jobs to programmers to do for them. If the programmer is interested in the position and would enjoy the job, it wouldn’t be the end of the world to do it for them. If it’s something that takes around 6 hours, that could easily be done in a single night or two, on-top of working.
Questions like “The number of street-lights in Edmonton” is stupid. Asking to see their personal projects or reviewing code they have done with them is best. It’ll weed out bad programmers who don’t have curiosity or self-motivation (Lack of Home Projects) and lack fundamental understanding and skill (Can’t explain their own code).
This discussion is closed.