* Either they’re strictly timed, introducing an artificial sort of pressure that’s rarely present on the job, or one ends up competing against people who spend 3 or 4 times the suggested amount of time. I don’t know about you, but I can do a way better job on most of these sorts of tasks if I spend a large multiple of the expected time on it, too.

* The tasks are often poorly specified, sometimes with the justification that it tests the ability to clarify requirements. I don’t think that argument holds water, because all “clarifying requirements” does when one doesn’t have ready access to folks who can answer those kinds of questions is introduce insane amounts of latency and frustration into the process.

* Even when the tasks are reasonably well specified, the grading rubric is often kept from the candidate. At work, if I don’t know if I or my team is building the right thing or doing it in the right way, I can go ask people (PM, tech lead, EM, another team member, etc.)

* Finally, most of these “take home assignments” I’ve seen that are supposed to be in the 2-4 hour range to finish don’t get me as a candidate out of 2-4 hours of giving impromptu algorithms lectures at the whiteboard. If the prize for putting in 2-6x as much time as I would to get to an onsite interview with another company is just to come onsite with your company and spend all day doing the exact same thing, then, no thank you.

I realize that few or none of these things apply to the process at Matasano, but there is another one that does: I can already get a job that pays reasonably well, doing things I already mostly know how to do, without spending hours studying a field I know little about.

That is arguably a feature and not a bug, so, I don’t mention it as a criticism, but rather as a reason why I probably wouldn’t do Matasano’s work sample test. For the type of people they’re looking for, that’s probably fine. It’s just that I’m not one of those people.

I had one of these tests say they wanted the project delivered as a git repo and would use the timestamps on the commit history to figure out how much time was spent. I just laughed at that and figured the people who knew how to forge the timestamps would do that to make themselves look good.

The only alternative I can think of to explicitly tracking time is to just not give a time expectation and ask how much time they spent. That way, answers aren’t biased by you anchoring a range in their mind. This has other obvious disadvantages, but it would take care of the “candidate spent 2 days on a 2 hour project” problem.

Meanwhile, if you commit to relying on challenges, you have a pretty major incentive to put time in to help candidates through them (if everyone bombs out of them, you don't hire anyone; if you're not rigorous, you're hiring randomly and you know it). We'll spend $100 and several hours for blind candidates (or at least: for every candidate we'll actually let into our hiring process) because we need the process to work, and to actually highlight good fits, in order to staff the company.

As always, a qualification process run haphazardly won't work and will alienate people.

Again, my suspicion is that most companies that do take-home problems are cargo-culting them, and not actually hiring based on their results. Rather, they're relying on the same interviews everyone else does, and using the challenge as a very elaborate form of FizzBuzz.

They simply tell you "this should take you 4-6 hours" and then you're left trying to figure out the tradeoffs to make. Usually that 4-6 hour exercise can EASILY take double the time, between you trying to get the best possible solution, handle all the edge cases, going back and forth with the interviewer on questions about the exercise, overthinking what the person on the reviewing end of the exercise is going to see as some irredeemable element of taste that they do not share with you etc

Then after putting in all that work, you possibly get to go through an actual interview process, get a form rejection email or even better never hear back from the company at all ... after going through that 2-3 times you learn that the upside for you is low, and you could probably make more progress with other firms in that same time.

What I personally have arrived at is that if a company isn't paying me for my time and the exercise will take over 2 hours, I simply won't participate. We'll see how it works out.

TLDR; Companies need to properly limit the scope of their take home assignments.

Suppose the challenge is to scrape some movie reviews and cluster them. On the surface, this is fairly reasonable. If you’re familiar with some relevant concepts and libraries, it isn’t too hard to get something up and running. On the other hand, measuring document similarity is still an open research problem. Should I do something fancier than a bag-of-words representation? What about the clustering algorithm? Evaluation?

I should show some kind of result. Is a simple tSNE plot enough to cover “visualization experience required”, or do I need a GUI that lets people tweak hyperparameters?

Maybe this is a sanity check to make sure I can code, so I should focus on documentation and tests. Or maybe I should make the scaler incredibly robust and scalable, since they kept talking about Big Data?

This isn’t an insoluble problem. Companies just need to be very specific about what they expect and how it’s going to be evaluated. If the point is code quality or ML chops, come right out and say so.