The safe pilot
The safe pilot
A managing partner has signed off a pilot, and the firm now wants to play it safe. "We will use mock data. We will set up a few role-based scenarios, walk the tool through them, and see how it performs. No real client files, nothing sensitive, no exposure to a vendor nobody has vetted yet." — this is the version I hear most often.
A lawyer's whole training points this way: when there is a risk, you minimise it as much as possible. Keep control, reduce the downside, prove the thing on invented material before anything real touches it. This safety net also quietly guarantees the outcome the firm is trying to avoid.
Mock data is great. It is built to be clean, complete and well-formed, which is the opposite of what arrives on a real matter. Role-based scenarios run well too, because the person running them wants the session to go well.
The real work does not cooperate. It comes in late and badly formatted, with the one clause nobody anticipated, handled by someone under deadline pressure who reaches for the tools they already use rather than the tool being piloted. None of that turns up in a rehearsal, but all of it decides whether the tool holds up in practice.
So the firm runs a clean exercise on invented material, watches the tool do well under clear-sky conditions, and reasonably concludes it works. The pilot converts to a subscription. And the real conditions — the messy file, the edge case, the busy associate who never logged in — arrive for the first time in month four, with actual clients in the matter and the licence already paid. The risk was never removed. It was postponed to the most expensive moment available.
The same logic produces a second distortion, and it runs on the same mechanism. Firms tend to ask for shorter, smaller pilots, because they feel like a smaller commitment. But the short window hollows things out the same way. In the first weeks nothing is properly configured. By the time it is, the core users — the ones with no spare attention — have not opened it once. The clock runs out before anyone under real pressure has changed how they work.
Both instincts come from treating the pilot as a risk that needs to be minimised. And a pilot kept small enough to feel safe is too small to tell you anything. You have removed the data that misbehaves, the timeline that reaches core users, and the stakes that change how people act. What remains is a demonstration. A demonstration that was already available, for free, from the vendor.
A pilot is the firm's instrument for answering one question: does this tool hold up here, in this practice, with these people and this work? You cannot answer that question with the real work kept out of the test. The conditions that feel like a risk are usually the conditions that carry the answer.
There is always a level of controlled exposure a firm can choose to accept. That decision belongs to the firm. It is not something the vendor can set up for you and a short synthetic pilot will never force you to make it.
In every pilot I have been part of, the difference comes down to a decision made before the start date. The successful pilots were run by teams that had worked out, in advance, how much real exposure they were prepared to accept in order to learn something real, and had governed it. Others defaulted to zero, which felt careful, and found out later that careful had been expensive.
The safe pilot may feel like the cautious option. But the exposure it spares you in the first month is the one thing that could tell you whether it's worth your investment.