How to pick an AI agency: 8 questions that reveal fit

How to pick an AI agency

To pick an AI agency well, you need a way to separate the firms that have built working systems from the ones selling confidence they cannot yet back with evidence. Most buyers discover the difference after three months and $15,000, still waiting on a roadmap presentation. The eight questions below surface that gap inside a single 45-minute first call. Some agencies welcome them. Some hedge every answer. The pattern across the answers tells you almost everything you need before you commit a budget to an AI agency.

1. Can you show me a working system built in the last 90 days?

The 90-day filter matters because the field moves fast enough that anything older starts to look theoretical. A good agency can pull up a Loom, a dashboard, or a running product and walk you through it without notice. The system does not have to sit in your industry, but it has to be real, in production, and generating measurable output for an actual client. A case study PDF, a press release, or talk of confidential clients they cannot name is not the same thing. Operators who have built things show the tool itself, doing the job it was built to do, without a legal review first.

2. Who actually does the work, your team or a white-label?

This question makes some agencies uncomfortable, which is itself a signal. A substantial portion of AI agencies resell white-label tools or push delivery to contractors they manage loosely. None of that is illegal, but it changes what you are buying. The honest answer is clear: these are our full-time people, here are their backgrounds, here is how we staff delivery. If the reply mentions a network of specialists or partnerships with vendors, ask who will be in your Slack channel and writing the code.

3. What is the smallest first deliverable I can see in 30 days?

Agencies that build tend to front-load delivery, because trust comes from showing progress, not from presenting another slide. A 30-day deliverable is not the finished product. It is a working piece of the system, a pilot that processes real data, or a dashboard wired into your actual tools. If the answer is a discovery phase or a technical audit, you are paying for preparation rather than output. Ask exactly what you will be able to touch, test, and reject at the end of week four. If that answer stays abstract, the engagement will too.

4. How do you price, by deliverable or by retainer for strategy?

Retainers tied to vague strategy outputs are the oldest money sink in the agency business, and they apply to AI agencies exactly as they applied to digital marketing agencies a decade ago. The retainer model is not automatically bad. It works when scope is clear, deliverables are defined, and output is measurable. What fails is the arrangement where you pay a monthly fee and get access to a team. Operators who have watched bad retainers burn clients offer deliverable-based pricing, or at minimum a retainer with a defined output list each month. If an agency cannot tell you what you receive for the money, that is information to hold before you sign.

5. Can I talk to a current client without you on the call?

Reference calls with the agency present are structured conversations. The agency has briefed the client, the client wants to be helpful, and the call confirms what you were already told. What you want instead is an unmediated conversation with someone who paid money and received work. Ask whether any current client would take a 20-minute call with you directly. Good agencies have clients who are glad to, because the work speaks for itself. The ones who hedge tend to have relationships that are fragile, very recent, or dependent on managing the story.

6. What is your stop-loss, when do you tell me to stop spending?

Most agencies do not have a stop-loss. They have a renewal incentive. An agency built around operator outcomes will tell you when to stop: when a system is not working, when the problem is not solvable with AI, or when your budget is better deployed elsewhere. This is not charity, it is economics. A client who pauses a failing project and returns later is worth more than one who churns angry after eight months. Ask directly: under what circumstances would you tell us to reduce or stop spending? The reply will be specific or evasive, and either way you learn something.

7. Do you stay embedded or hand off and disappear?

Some agencies are better at building the initial system and would rather not own ongoing support. That is a legitimate model as long as you know it going in. What is not legitimate is a hand-off framed as your team owning the system when the documentation is thin and the first person who changes something breaks it. Ask where maintenance sits and what happened on their last three hand-offs. The honest answer to that question is almost always the most illuminating thing on the call, because it shows how they behave once the invoice is paid.

8. What is the smallest red flag in our brief that you would push back on?

This question cuts through the sales dynamic faster than any other. An agency that sees nothing to push back on is either junior, not paying attention, or telling you what you want to hear. Every brief carries a constraint or an assumption that needs testing. The agencies worth hiring name at least one thing that gives them pause: your data quality, your capacity to absorb change, or your definition of success. Sellers tell you it all sounds great and that they have done exactly this before.

What good answers look like versus bad ones

An operator-run agency opens a browser tab for question one, names the people on your account for question two, and gives a specific deliverable and date for question three. It prices by output, has clients who speak directly, has ended bad engagements before, maintains systems after launch, and pushes back on your brief because it read it. A seller-run agency answers with a PDF, talk of partnerships, and a discovery phase, prices by monthly access to a team you cannot measure, and tells you your brief sounds exactly like its specialism. The distinction is rarely clean on a first call, but a pattern of evasion across several questions eliminates an agency faster than any deck can rebuild its credibility.

For more on what to watch for before you commit, the AI agency red flags post covers 11 specific patterns worth walking away from. If you are still deciding whether you need an agency at all, our pillar guide on what an AI consultant actually does breaks that decision down in detail.

How twohundred would run this

If we were on the other side of that call, we would expect every one of these eight questions. We would open a system built in the last 90 days, name the person writing your code, and commit to a first deliverable you can test inside 30 days rather than a discovery phase. That is the model behind our AI implementation services: build something real early, prove it on your data, and earn the next phase rather than assume it. Use the eight questions as a filter on any firm, including us.

Frequently asked questions

How many questions should I ask an AI agency before hiring them?

Eight is a working set for a first call. The point is not to march through a checklist but to listen for the quality of each answer. Two or three pointed questions answered with real specificity tell you more than eight answered with hedging. The question about red flags in your brief is the highest signal, because it tests whether the agency engaged with your situation or slotted you into a standard pitch.

What is the fastest way to tell a good AI agency from a bad one?

Ask to see a working system built in the last 90 days, with no preparation time. Good agencies pull this up immediately. Agencies that struggle either have not built much or have built things they cannot easily demonstrate. Pair it with the question about red flags in your brief and you usually have a clear read inside the first 15 minutes.

Is retainer pricing always a red flag for an AI agency?

No. Retainer pricing fits when the work involves ongoing development, iteration, or long-term support. The problem is a retainer attached to vague scope and to success metrics the agency controls. Ask exactly what you receive each month for the fee. If the answer is access to a team rather than a named output, that is where the risk sits.

What should I do if an AI agency refuses to share reference contacts?

Treat it as a data point worth noting. Refusals are sometimes explained by NDAs, but most clients who are happy with an agency will take a short direct call. If a firm cannot facilitate a single reference conversation, ask why, and weigh the explanation against everything else you saw. A pattern of evasion across reference checks and the other questions matters more than any one refusal.

Ready to put a firm through these eight questions? Start with our AI implementation services and judge the answers yourself.

---

Related Services

For businesses that want strategic guidance on where AI fits before engaging a vendor, AI consulting services explains the advisory approach. Hands-on deployment is covered in AI implementation services.