How to Evaluate an AI Vendor

Evaluating an AI vendor is a procurement and operating decision, not a contest between model demos. The right question is whether a provider can deliver a defined capability, with acceptable quality, security, economics, and exit options, inside the workflows and controls your organization can operate. This guide presents a vendor evaluation framework for model providers, API companies, software vendors with embedded AI, data suppliers, and implementation partners. It does not publish vendor rankings, invented scores, or universal best choices.

A vendor that looks impressive in a controlled demo may still fail on contract terms, rate limits, data residency, support, reproducibility, or total cost. Conversely, a less glamorous provider may fit a narrow workflow better because it offers stable interfaces, transparent evidence, and a credible migration path. Evaluate the complete service and the decision it supports.

Start with the decision, not the vendor list—especially when comparing China-market and export stacks

Write a one-page use-case brief before contacting suppliers. Define the user, workflow, input data, expected output, decision impact, volume, latency target, regions, retention needs, and fallback process. State what the system must never do. “We need generative AI” is not a requirement; “classify incoming claims, return a reason code within two minutes, and route uncertain cases to a reviewer” is testable.

Separate capability requirements from preference requirements. A capability requirement might be structured output, private networking, batch processing, or a supported language. A preference might be a particular dashboard or model family. This distinction prevents a polished feature from outweighing a missing control.

Map the workflow around the model. Identify source systems, retrieval stores, human review, downstream writes, monitoring, and support ownership. An AI vendor may supply only inference while your team owns prompts, evaluation, identity, orchestration, and incident response. Procurement should price and assess the whole path.

Define evaluation dimensions and evidence

Use a scorecard as a structure for evidence, not as a fake precision machine. Each criterion needs a definition, a weight justified by the use case, an evidence request, and a pass condition. Avoid awarding points merely because a vendor says “enterprise-ready.” Ask what was tested, under which assumptions, and whether the evidence applies to your data and traffic.

Record unknowns separately from failures. An unanswered question is not proof of weakness, but it is procurement risk. Assign an owner and deadline for every open item, and make unresolved high-impact items approval conditions rather than burying them in an average score.

Test capability with representative work

Build a task set from real production-like cases, with sensitive values removed or handled under an approved test arrangement. Include common, difficult, ambiguous, multilingual, long-context, and malformed inputs. Include negative cases where the correct behavior is to abstain, ask for clarification, or route to a human.

Use blind evaluation where possible. Hide vendor identity from reviewers, randomize order, and define a rubric before seeing results. Measure task success, factuality or extraction correctness, refusal and abstention behavior, structured-output validity, reviewer effort, and downstream rework. For classification, inspect confusion matrices and costly slices instead of relying on aggregate accuracy.

Test the application contract, not just a chat transcript. Does the API preserve fields, return stable error codes, support retries safely, and handle partial failures? Does a streamed response terminate correctly? Can the client distinguish a provider refusal from a timeout? A model can produce good text while the service remains unsuitable for production integration.

Repeat tests across realistic settings: temperature or equivalent controls, context sizes, concurrency, retries, tool calls, and version pins. Ask the vendor to identify nondeterminism and silent updates. Preserve prompts, inputs, outputs, model identifiers, configuration, timestamps, and evaluator notes so results are reproducible.

Inspect API, product, and integration contracts

Require a written interface contract. It should describe authentication, request and response schemas, maximum input and output sizes, supported content types, streaming semantics, pagination, idempotency, error codes, retry guidance, and version lifecycle. “OpenAI-compatible” or “REST API” is not enough; compatibility details determine migration cost.

Clarify identity and authorization. Determine whether the service supports SSO, service accounts, scoped keys, role separation, private connectivity, tenant isolation, and audit logs. Never let a shared personal API key become the production identity. Establish how keys are rotated, revoked, and attributed to teams or workloads.

Examine operational boundaries. What happens at quota, timeout, regional outage, malformed input, provider maintenance, or model retirement? Can traffic be routed to a fallback? Are retries safe for requests that trigger tools or billable work? Contractual uptime without usable error semantics does not protect the product.

For embedded AI software, inspect permissions inherited from the host product. Determine whether the feature can read or write records, whether administrators can scope it by group or region, and how generated actions are reviewed. A vendor’s model quality cannot compensate for an integration that silently expands access.

Evaluate data handling and privacy boundaries—and vendor Responsible AI claims

Draw the data flow from collection to deletion. Mark prompts, files, retrieved context, embeddings, outputs, feedback, logs, support tickets, backups, and evaluation copies. For each item, ask where it is stored, who can access it, how long it remains, whether it is used for training or service improvement, and how deletion propagates.

Contract language should distinguish customer content from provider telemetry. “We do not train on your data” may still leave retention for abuse monitoring, human review, support access, or derived metadata. Ask whether opt-outs apply by endpoint, account, region, and subprocessor. Require notification before material changes.

Classify the use case before selecting a data path. Public material, internal confidential documents, personal data, financial records, and health information require different controls. If the vendor cannot support the required region, retention, encryption, access review, or deletion evidence, remove the data from scope or remove the vendor from consideration.

Test deletion and export rather than accepting a policy statement. Ask for a sample process and expected evidence. Determine whether vectors, caches, fine-tuning artifacts, backups, and support copies are included. Portability is a privacy control as well as a commercial one.

Assess security and resilience

Request a concise security architecture: identity plane, tenant boundary, network path, storage, key management, logging, administrative access, and incident response. Map the vendor’s controls to your own threat model. Pay particular attention to prompt and file isolation, retrieval connectors, tool execution, model supply chain, and support personnel access.

Ask how the provider handles vulnerabilities in models, containers, dependencies, and data connectors. Ask whether artifacts are signed, whether releases are pinned, how changes are communicated, and how quickly a compromised endpoint or key can be disabled. For agentic features, identify the exact actions the vendor can take and where deterministic authorization is enforced.

Resilience includes capacity and concentration risk. Review regional architecture, dependency providers, disaster recovery objectives, maintenance windows, status transparency, and historical incidents. Do not infer availability from a large brand name. A service can be reliable globally while failing your region, quota tier, or critical time window.

Require an incident process that names notification windows, evidence preservation, customer cooperation, remediation, and service credits where relevant. Credits do not restore a lost decision or exposed record; they are not a substitute for containment and communication obligations.

Understand quality, change, and evaluation drift

AI services change in ways ordinary software users may not notice. A model update, tokenizer change, safety filter, retrieval index, or system prompt can shift outputs while the endpoint remains technically compatible. Ask what can change without consent, what can be pinned, how long versions remain available, and whether old evaluations can be rerun.

Create a vendor regression harness under your control. Run representative tests before onboarding, after material changes, and on a schedule appropriate to risk. Compare quality, latency, refusals, structured-output validity, cost, and slice behavior. Keep a known-good fallback and document the threshold that triggers rollback or re-evaluation.

Do not outsource acceptance criteria to benchmark scores. Public benchmarks can inform discovery, but they do not represent your prompts, languages, documents, workflow, or error costs. A procurement decision should cite your evidence, the test conditions, and the remaining uncertainty.

Model economics beyond list price—and funding signals that are not product proof

Build a unit-cost model from actual workload assumptions. Include input and output tokens or units, context growth, retries, failed calls, batch discounts, storage, retrieval, egress, human review, observability, implementation, support, and fallback traffic. Calculate ordinary, peak, and adverse scenarios. A low per-call price can be expensive when it increases review or retry volume.

Read pricing definitions carefully. Determine whether cached input, tool calls, multimodal content, minimum commitments, reserved capacity, and overages are charged differently. Ask what happens when quotas are reached and whether the vendor can change prices or meter definitions during the term.

Compare providers on the same completed workflow, not the same model call. If one vendor returns a valid structured result on the first attempt and another needs multiple repair calls, the invoice and latency differ. Include engineering effort to normalize APIs and maintain prompt or adapter compatibility.

Contract for ownership, portability, and exit

Clarify ownership of inputs, outputs, prompts, fine-tuning data, adapters, evaluations, and custom configurations. The contract should address provider rights to use content, confidentiality, IP claims, indemnities where offered, and responsibility for third-party components. Legal review should be specific to the data and action risk of the use case.

Design the exit before signing. Identify export formats, retained identifiers, replacement architecture, dual-run period, notice obligations, deletion certificates, key revocation, and migration staffing. Avoid proprietary wrappers around every application boundary. Keep prompts, schemas, eval sets, routing logic, and observability in your control when feasible.

Portability does not mean every model is interchangeable. It means the organization understands which pieces are provider-specific and has a credible plan for the rest. Document the cost and quality loss of migration rather than making an abstract promise of freedom.

Run a disciplined procurement process

Use the same brief, test set, contract questions, and timeline for shortlisted vendors. Separate discovery from selection: first learn what options exist, then compare evidence under controlled conditions. Give vendors a written clarification window and log any answer that changes the evaluation.

Include engineering, security, privacy, legal, finance, procurement, operations, and the domain owner. Each function should approve the dimension it can evaluate. The business sponsor owns the outcome and residual risk; procurement owns commercial discipline; no spreadsheet average should override a critical security or privacy failure.

Make the decision reversible where possible. Start with a bounded pilot, synthetic or minimized data, spending limits, and a defined stop condition. A pilot should test integration, user behavior, support responsiveness, and operational evidence, not merely create a demo. End it deliberately: adopt with controls, extend with named gaps, or close and delete data.

Common vendor evaluation mistakes

The demo trap rewards theatrical prompts. The benchmark trap substitutes public scores for task evidence. The price trap compares per-token rates without workflow cost. The contract trap accepts vague training and retention language. The lock-in trap lets a vendor-specific abstraction enter every product surface. The consensus trap averages opinions until a critical objection disappears.

Other failures include testing only happy paths, allowing the vendor to choose the test set, ignoring support during the pilot, treating an assurance report as a current architecture review, and assuming a large provider will automatically meet a local data or latency requirement. The remedy is documented evidence tied to a decision and a named owner.

A practical decision record

At selection time, record the use case—do not substitute a top companies list, alternatives considered, evaluation version, data classes, architecture, scores with evidence links, critical gaps, contract exceptions, expected unit economics, operational owner, fallback, exit plan, approval authority, and review date. State what was not tested. Record whether the decision is for a limited pilot or a production commitment.

Re-evaluate when the provider changes the model, price, data terms, region, tool capabilities, or support tier; when your workload changes; or when an incident reveals a new risk. Vendor evaluation is a lifecycle practice. A supplier that fit a low-volume drafting pilot may not fit a high-volume workflow with regulated data and automated writes.

Closing

Good AI procurement turns enthusiasm into an accountable operating choice. Define the decision, test representative work, inspect contracts and data paths, measure complete-workflow economics, and preserve a credible exit. Evaluate evidence rather than vendor reputation, and keep critical controls under organizational ownership.