There used to be a fairly simple rule for hiring exercises: don’t ask candidates to work on your own product or a core business problem. Use something adjacent enough to test their judgement, but unrelated enough that you aren’t getting free consulting. The exercise should give somebody enough material to make decisions without pretending that a few hours of work can reproduce the context of a team already living inside the problem.

That said, I’ve recently seen exercises as absurd as “turn $10m of catalogue into $300m,” and asked to produce turnaround plans for products while being provided deliberately incomplete data. A year ago, somebody asked me to sign an NDA before they would even send the exercise over, which seemed like a useful indication that whatever followed had already crossed from hypothetical assessment into unpaid consulting. I declined.

I understand why companies use their own products. They know the context, the conversation feels relevant, and interviewers can challenge the answer using years of information they already have. The candidate, meanwhile, has an attachment and somewhere between two and four hours. This imbalance would be fine if the purpose were to understand how somebody frames uncertainty, but increasingly the output is judged as though it were a genuine strategy: comprehensive, confident and attached to an outcome.

You can point out the missing data and obvious blunders, like having fixed trading hours or requesting a profile picture before a cent has been deposited. Those are useful observations, but there is a universe of possible causes that only proper measurement and observed behaviour can narrow down. Perhaps the profile-picture request is causing people to leave. Perhaps the people reaching it never intended to deposit. Perhaps they arrived through the wrong acquisition channel, cannot use their preferred payment method, do not trust the product, or simply do not understand why it exists.

You can identify a problem, but it might not be the problem.

The usual defence is that product work always happens with incomplete information, which is true. The difference is that, in a working environment, you can do something about it. You can talk to people, inspect the funnel, improve the instrumentation, release something small and observe what changes. The output is rarely fixed: it is a vehicle for discussion and critique. Nobody expects the first answer to be right. If it was, you wouldn't need to hire at all.

A take-home exercise removes most of that process, freezes the first answer and then compares it with somebody else’s. The candidate who is honest about the uncertainty risks appearing less comprehensive than the candidate willing to fill every gap with an assumption. A careful diagnosis becomes indecision, while a neat deck containing a confident line from the bottom left to the top right becomes strategy.

At that point, the exercise is testing who can make the most brazen assumptions, package them into the neatest story and pitch the most convincing solution to win the job. That is a real skill, and product people obviously need to explain their decisions and bring others with them. But persuasion supported by evidence is different from confidence used in place of it. One is part of product work. The other is sales.

There are straightforward alternatives. Use an adjacent problem with enough information to support a decision. If the problem is real and the company might use the output, pay for the work. If the goal is to test discovery, judge the questions, assumptions and proposed experiments rather than the apparent completeness of the remedy. Better still, make it a conversation: introduce new information, challenge the first answer and see whether the candidate can change their mind without treating that change as failure.

When candidates are denied the information and time needed to diagnose a problem, then judged on the comprehensiveness of their solution and the confidence they project in its outcome, the exercise tests sales more than product, design or engineering. Sell me this pen.