Before a big AI commitment, run a small test that produces facts

When a board or a CEO asks for an AI plan, the natural response is a big one: a strategy exercise that maps every function, a platform commitment, a program with its own budget line. It feels responsible. The stakes are high, and nobody wants to back the wrong tool or the wrong use case.

Yet the questions that decide whether a tool helps your business are questions about your business. Will the people doing the work use it? Is the output good enough for your customers? What does it cost once it runs every day? A planning document can list those questions. Answering them takes a small test, run on real work, with a date to decide.

In 2020, while running my own companies, I wrote down a rule for decisions that won't come clear: when a decision isn't clear, act to get the facts. This post covers what I wrote, the limits I put on acting fast, and how to apply both to a first AI pilot: how small to make it, how long to run it, what result should decide it and what to settle before it starts.

DisclosureJake Jorgovan co-founded Alpha Apex Group, which this essay links to.

When the path isn't clear, the facts are missing

In May 2020, a few weeks into the pandemic, I had been pulled back into heavy work on my businesses after seeing how fragile they were. I wrote an essay about the rule I kept coming back to: "When the path isn't clear, it's because you lack the facts to make the right decision."

By facts I meant information that would help you make a better decision: "feedback from the market, from your customers, from prospects, etc." My example was Content Allies, the podcast company I founded. I was debating a change to its positioning and struggling with it, because "I was thinking in hypotheticals, and what I really needed was more conversations with these new prospective customers."

The rule I took from that period was short: "Action leads to facts. Overthinking it doesn't help."

Acting didn't always mean being right. My first pivot of Content Allies' niche "was a flop", I wrote, "but I only learned that by shifting and trying something new." The facts from an action "will tell you if you made the right decision, or if you need to course correct." I wrote it during the pandemic, but I argued that business is always uncertain and that the framework "can hold true through any times."

An AI decision can have the same shape. For a pilot, the missing facts may be how the tool performs in your work. Planning can define the questions; a test supplies evidence about adoption, quality and cost.

The limits I put on acting fast

"Don't overthink it" came with limits, and I wrote them down as well.

Planning keeps a place, measured in weeks. The Daily, the series of one-lesson posts I wrote from 2018 to 2020, made the point twice. Before starting Lead Cookie, the LinkedIn outreach company I founded and later sold, I spent roughly three weeks planning which opportunity to pursue, and I credited those weeks with setting the course for what was then my most successful business. "Note that I said 3 weeks of planning, and not 3 months," I wrote in November 2018.

Seven months later I described the opposite trap, spending "all my time thinking about what I was going to do and not enough time actually doing it," and still kept a place for planning: "Sure, there needs to be a healthy amount of strategic planning in the mix. But beyond the high level planning, you don't need to spend the rest of your life in thought."

New ideas get a wait. I wrote in a 2023 essay on taking new businesses to market that I had learned the hard way not to start everything right away. My advisor's rule helped: "If it's a good idea today, it will be a good idea 30 days from now. Give yourself 30 days before jumping into anything." I applied it to "totally new ideas out of left field", and I did not count "waffling on an idea for a long time" as part of the 30 days. Once I decided to commit, I committed.

Bets get chosen for how they will reach people. The same essay made a second point: whenever I selected an opportunity, I tried to think through how it would reach its customers at the moment I picked it, because a great idea goes nowhere without a good way to reach the people it is for.

The last limit is stopping. In a June 2020 podcast episode I described launching several new businesses with partners at once. Because they ran on the teams I already had, a concept could go from idea to launched business in about two weeks. I said I would be selective about which opportunities to back, and that I knew some would fail. One already had. I had pitched a content agency mastermind to "like 15 people in my network. No one signed up. Was like, kill it." I dropped it after a few weeks, because it did not get the early engagement I had expected.

Taken together: plan in weeks, wait on ideas that are genuinely new, choose the bets, give each one a short horizon and stop what doesn't land.

Applying the rules to an AI pilot

I wrote these rules about running and launching companies between 2018 and 2023. Applied to a first AI pilot, they come down to five steps.

1. 1

   Name the decision the pilot will inform

   Before it starts, write down what will be decided at the end: extend the tool to more teams, change how it is used, or stop. Then write the result that would justify each. Record how the work performs today on the same measures (quality, time and cost), and count the time people will spend checking and correcting the tool's output. When Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept (opens in a new tab) by the end of 2025, "unclear business value" was one of the four reasons it gave. A result named in advance gives the pilot a business value to test.

2. 2

   Keep it small and use what you already have

   Choose one team, one workflow and a tool you can switch on quickly within the data and security rules you already have. My 2020 tests could launch in about two weeks because they ran on teams that already existed. A pilot inside an existing team's work has the same advantage: the people already know what good output looks like, and you can compare the tool's work with theirs.

3. 3

   Decide who will use it, and how it reaches them

   The 2023 point about reaching customers applies inside a company too. Name the people who will use the output and where it enters their day. If nobody uses it, find out why: that is a fact about adoption, even though it leaves the quality of the output untested.

4. 4

   Put an end date on it

   Fix the review date before the pilot starts, so the test ends with a decision instead of drifting on. Choose a period long enough for the work to repeat a useful number of times; for a task that comes up rarely, set a number of completed cases as well as a date.

5. 5

   Judge it on facts from the people who use it

   In 2020 I meant feedback from the market, customers and prospects. For a pilot, the facts come from the people using the tool, the people checking its output, the customers who see the result and the cost of running it. At the review date, make the call my essay described: the facts "will validate your decision or give you data to course correct." If no one is using it, record why, then decide whether to change how it enters the work or stop it, as I stopped the mastermind.

Image

Zoom

The approach described in this article, shown as a visual guide.

What to settle before the first test

Acting fast makes sense when a wrong step is cheap and easy to undo. A test that touches customer data, customers or job candidates may be neither, so a few things belong in the planning weeks.

The 30-day rule was a wait before committing to a totally new idea, and long waffling did not count toward it. It did not set the length of a test. That was a rule for committing to a new business idea. For an AI pilot, keep the small test you can undo separate from the larger commitment it is meant to inform.

Besides unclear business value, Gartner's 2024 prediction gave three reasons for abandoned projects: poor data quality, inadequate risk controls and escalating costs. Each points to a decision to make before the pilot starts:

• Data: what the pilot may use, and what it must not touch.
• Review: who checks the output before it reaches a customer, a candidate or a decision.
• Cost: a ceiling for the test, and an estimate of what it would cost to run at full scale.
• The law where you operate: a hiring pilot shows why. New York City's Local Law 144 of 2021 prohibits employers and employment agencies from using an automated employment decision tool (opens in a new tab) unless it has had a bias audit within one year of its use, information about the audit is publicly available, and certain notices have gone to employees or job candidates. Whether a given tool falls under the law depends on what it does and where it is used, so check what applies before a tool touches a hiring decision.

For a fuller structure for managing AI risk, NIST publishes an AI Risk Management Framework (opens in a new tab), released in January 2023 and intended for voluntary use, with a profile for generative AI added in July 2024.

Image

Zoom

NIST’s AI RMF 1.0 places governance across three ongoing functions: map, measure, and manage. The framework is voluntary, and these functions are iterative rather than a one-time checklist.airc.nist.gov (opens in a new tab)

These checks belong before the pilot starts. Aim to settle them in weeks; if that is unrealistic, narrow the scope or delay the start until the checks are complete.

The 2020 essay asked a question that fits any AI plan that has been in review for longer than the test would take: what decision are you overthinking?

((The two posts from The Daily quoted here, 017 (November 9, 2018) and 165 (June 17, 2019), are no longer online; the quotations are from the site's archive of September 24, 2026.))