AI Vendor Evaluation Checklist

Score any AI agency or vendor across 15 make-or-break criteria before you sign. Mark each Yes, Partial, or No — the tool tallies your score and flags the deal-breakers.

Data & privacy

They will sign a data processing agreement (DPA).
A vendor who refuses a DPA is a hard no if they touch customer data.
They confirm in writing your data is NOT used to train their models.
Get it in writing, not just a verbal “we don’t do that.”
They disclose which model providers / sub-processors sit underneath.
OpenAI, Anthropic, AWS, etc. — you should know where your data flows.
They can tell you how long your data and prompts are retained.
Vague answers here usually mean “indefinitely.”

Capability & fit

They provide references or case studies for similar work.
Ideally in your industry or a comparable business size.
They scope the work realistically instead of promising everything.
Beware anyone who says yes to every requirement instantly.
They can explain how the solution works in plain language.
A refusal to explain (“it’s proprietary”) is a black-box warning.
You will own the deliverables, prompts, and workflows they build.
Confirm IP ownership transfers to you, not just a license.

Commercial

Their pricing model is clear (hourly, fixed, or retainer).
You should understand exactly what you’re paying for.
They are upfront about ongoing API / usage costs you’ll bear.
Model usage is a real monthly cost — it should not be a surprise.
They do NOT demand the full fee upfront.
Milestone-based payment protects you if things go wrong.
You can leave without losing access to your own systems/data.
Ask what happens on day one after you cancel.

Support & reliability

There is a clear plan for what happens when something breaks.
Response times, who to call, and how fixes are handled.
The design keeps a human in the loop for high-stakes actions.
Fully autonomous AI on money or legal decisions is risky.
They have a plan for handling AI errors and hallucinations.
Every AI system gets things wrong — the plan matters.
0 / 30

Answer the questions to score your vendor

How to use this checklist

Run every AI vendor or agency you’re considering through the same 15 questions and you’ll compare them on evidence instead of on how good the sales call felt. Any answer of “No” on a red-flag row (they won’t sign a DPA, won’t explain how it works, want all the money upfront) is worth resolving before you go further — those are the issues that turn into expensive problems later.

Pair the scorecard with our free AI data processing agreement template and the how to hire an AI agency guide, which walks through each question in more depth.

Not sure how to read the answers?

Send us the vendor’s proposal and we’ll give you an honest, no-obligation read on whether it’s a fair deal and where the risks are.

Book a Consultation

Frequently Asked Questions

  • Score them on four things: how they handle your data, whether they can actually do the work (with references), whether their commercial terms are fair, and how they support you after launch. This checklist turns each into a concrete question so you can compare vendors on the same scale instead of going on gut feel.
  • The highest-leverage questions are: Will you sign a DPA? Is our data used to train your models? Who are your sub-processors? Do we own what you build? How do you handle errors and downtime? And how exactly are we billed, including ongoing API costs? Each is a row in the scorecard above.
  • Refusing a DPA, refusing to explain how the system works, guaranteeing specific results, demanding the full fee upfront, having no references, and being vague about where your data goes. The scorecard flags these automatically when you mark them “No.”
  • Treat the score as a comparison and risk tool, not a pass/fail. A vendor scoring in the top band with no red flags is worth advancing. Anything in the bottom band, or any red flag at all, deserves a direct conversation before you sign. Use it alongside our free AI data processing agreement template.

This tool provides general guidance to help you evaluate vendors and is not legal advice. Always review contracts and data agreements with a qualified professional before signing.