Homework during the hiring process…no thanks!

Not too long ago, I was on the job market looking for work as an applied statistician or data scientist within the the online marketing industry. One thing I’ve come to expect with almost every company is some sort of homework assignment or challenge where a spreadsheet would be presented along with some guidelines on what type of analysis they would like. Sometimes it’s very open ended and at other times, there are specific tasks and questions which are put forth. Initially, I saw these assignments as something fun where I could showcase my skill set. However, since last month, I’ve come to see them as a nuisance which can’t possible be a good indicator of whether someone is ‘worth hiring’ or not. I get it, companies often get inundated with resumes and they need effective processes to sift through them. And I get the value of getting some document which outlines how an applicant thought about a problem and generated some valuable insights.

With all that said, do we seriously think that homework assignments and challenges during the hiring process are the most effective way of getting the “best candidate” (whatever that means). I don’t have any data to suggest either way, but am inclined to believe that companies and analytics hiring managers need to develop better ways of assessing the quality of candidates. It these assignments are really about assessing who is most serious about a role to spend a few hours of their free time answering some ‘simple’ questions and putting together some basic lines of R or Python code, then so be it. But I think a better process can be put forth that allows companies to find the right candidate.

I’ve been part of the hiring process and I’ve also gone through months of looking for employment. Based on my experiences on both sides of the table, here’s my view of what is most effective when looking for analytics professionals, applied statistician, or data scientists. Ultimately, my feeling is that the only way to assess whether a candidate is worth hiring is by effectively testing prospective candidates in a more formal manner. The key is to have the applicant complete this stuff during the interview as that would remove the task from being characterized as a take home homework assignment.

Part 1: Quantitative Skills
To assess a candidates quantitative proficiency, here are some techniques that work well based on my previous experience.
a. Put together a document with an existing business problem and some of the analysis that’s been put together to answer them. Ask the applicant for suggestions on the limitations of the current approach and what they’d do if that project was handed to them.
b. Put together a basic statistics test which inquires about simple probability theory and inferential statistical principles. Ask the candidate to answer those questions in an informal setting to ascertain what they know and how work through problems when they don’t know the answer.
c. Ask the applicant to read a statistically demanding document and then request a summary plus feedback from the candidate. This should also tell us something about what the candidate knows about statistics and whether they can summarize the relevant parts in a satisfactory manner.

Part 2: Technical Skills
To assess a candidates technical proficiency, here are some techniques that work well based on my previous experience.
a. Show an applicant some imperfect code that is unnecessarily long or could be improved. Ask them to look it over and provide their suggestions on how’d they do things differently.
b.Put together several small code snippets in various programming languages that the candidate may or may not know. Ask them to go through the code, identify what is happening at each step, and explain the final result.
c. Have the applicant share their work on some interesting work or non work related project that they did recently. They can talk about specific aspects of their code and consider if there is anything they’d do differently now.

The possibilities are endless, but there has to be better ways to assess the quality of candidates to analytics roles than the ‘homework assignment.’ In any case, I’ll be refusing to do any more assignments as a part of the hiring process.


PS: In time, I finally opened up to the notion of ‘technical assignments.’ Don’t get me wrong, I have and will never use them when I am in a hiring capacity, but I finally accepted that I’d have to do a few homework assignments here and there.

PPS: Oddly enough, I’ve received a few rough/rude emails from hiring managers regarding this post. If you don’t like my perspective, feel free to write your own blog post about it. Personally, I don’t see the need for someone to send a mouthy email to me just because they have a different perspective.




16 thoughts on “Homework during the hiring process…no thanks!

  1. I agree with your view that homework is out of bounds but your solution seems to be a new kind of homework. For older candidates you could ask for samples of publications and/or a sample of past work. Besides the things you mentioned, homework could represent a significant investment of time towards a job which may be pie-in-the-sky. I recently completed a “homework assignment” only to be told that the job had been mistakenly posted! I was told it was only going to be filled internally. Perhaps it was a lie and they just didn’t want me but, from my perspective, it was a waste of time which also left me with an embarrassing feeling of “reveal all”, to no avail.


  2. A sample would be great also. That or an “informal test” during the interview is what I’m proposing as an alternative to a take home homework assignment.


  3. I know a lot of people that “shut down” on interviews because of the pressure of being watched. They prefer a lot to go through the homework at home than having to answer some questions at the time of interview. Plus, I think the homework style better portraits the real work scenario, where most of the time you could have time to think and concentrate. I do like your suggestions on analyzing a specific scenario, but that could be given as homework as well.


  4. Robert Young, most of the homeworks I’ve seen covered hypothetical situations, or problems that were already solved inside the company. Why would I take the time to review a bunch of guys code to include in my application if I can do that in much less time? It would take more time to actually review the solution than to implement by myself.


  5. technique “a” of part 1 can be a conveniently dishonest way for an employer to get either ideas or even work from the candidate for free! I have seen that in practice many times happening, at which point I say bye bye to them. Especially for start-ups and for data science positions, I would advise candidates to be extra careful when they get these “homeworks”.

    Also, these homeworks should only be given to candidates without an educational or professional background in data science relevant fields, such as statistical science, machine learning etc.

    If the only way for an employer to weed out candidates is through these type of homeworks, then the employer is simply not worth it.


  6. SInce even Google Inc. has said there’s pretty much zero correlation between any interviewing technique or rating system and subsequent quality of work performed by new hires, I rather doubt that your untested approach will be any good either.


  7. I’m struggling to understand what your main objection is. If we take software engineers as a similar field (with which I’m more familiar) then doing hands-on small assignments is the best and pretty much the only reliable way I know to assess their skills. Then the choice is only between doing them a) during the interview, b) as a homework and c) as a trial run within the hiring company and the target team for a day or two. Option C is the best, but depending on the market and on their current employment, applicants might not agree to that. Plus, it’s a distraction for the target team. Option A works, but only for small tasks due to time limitations. You can extend interviews to 6-8 hours as in Google, but not all companies practice that. So what is your real gripe with Option B?


  8. Besides the fact that they are an unnecessary waste of time for the applicant, my main complaint is that these “homework assignments” are not a good indicator of whether someone will succeed in a position. We all preach about data and how useful it is. Well, if the “homework assignment” approach doesn’t help us predict whether someone will do well when hired, let’s find a better metric or measurement tool. I’m annoyed with the status quo approach.


  9. > Besides the fact that they are an unnecessary waste of time for the applicant?

    Really?!? Is applicant trying to get a *job* or what? If an applicant doesn’t want to spend a couple of hours on homework then why should hiring managers and employees spend hours and hours interviewing people?

    > Let’s find a better metric or measurement tool

    Sure, but what are the other options than hands-on assignments?

    Knowledge-based questions are not a good indicator, because you don’t have to remember all the details of Framework X or Tool Y to be a good developer. IQ puzzles a-la Microsoft and Google are off-topic and often criticized. Again, you don’t have to be a good puzzle solver to be a good developer.

    You had some useful suggestions in Part 2, but I feel that those aren’t going to be sufficient for me as an interviewer of software developers without a hands-on assignment.

    A simplest hands-on assignment can in an hour or two weed out applicants who can’t program. Which in our market is about 14 out of 15 applicants. It’s a well known problem in industry: see “Everyone thinks they’re hiring the top 1%.” http://www.joelonsoftware.com/items/2005/01/27.html

    The only question to me is how to structure hands-on assignments (see the previous comment).


  10. You keep saying drawing parallels to software developers. My post is regarding statisticians, analysts, data scientists, etc. For our roles, programming is secondary and a means to an end. Therefore, I’m more concerned about your knowledge of statistical theory than anything else.


  11. — why should hiring managers and employees spend hours and hours interviewing people?

    Umm… they need a warm body to fill up a chair so they can claim more reports than other managers??? Hiring is a two way street, really, although the current over-population situation allows many employers to view hires as just cogs in the corporate machinery. Hirees tend to believe it too, so there’s seldom any attempt at honest dialog during the process. That Charley Chaplin movie comes to mind. As my Pappy used to say, “Marry in haste, repent at leisure”.

    All too often, hiring agents don’t know exactly what hires will be doing, beyond gross generalities, “big data”, “glms”, “Bayesian baseball analytics”, and such. Such hirers tend to take the position, “I don’t know who I need, but I’ll know him when I see him” (and normally a him, fur shur). Most application development fails, and so does most hiring. Largely because no one takes the time and effort to think through what needs be done.

    Agile, the au courant buzzword, makes this all worse, since its premise is “I’ve no idea what I’m doing, but if I bang on it long enough, I’ll figure it out”.


  12. > Hiring is a two way street

    This is exactly my point. I was not arguing that hiring is so unimportant that hiring managers should avoid it. My argument is that it is two-way street and the applicant should be expected to spend significant time and effort as well. Even more so, because for an unemployed person it is a full-time job to try to get a job whereas hiring managers and employees are busy full-time with their work.


  13. > You keep saying drawing parallels to software developers. My post is regarding statisticians, analysts, data scientists, etc. For our roles, programming is secondary and a means to an end.

    Sure. I had a disclaimer about that. I do interview people in adjacent fields though: QA, requirements, data architects, etc. So some lessons do apply across, don’t they? Especially your “Part 2: Technical Skills” would apply, which talks 100% about the code.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s