AI vendor comparison scorecard
Choosing an AI vendor on vibes is how organisations end up locked into a contract that fails the security review six months later. This scorecard makes the choice explicit: you declare what matters and how much, score each vendor on those criteria, and the tool ranks them by a weighted total. The output is a defensible artifact you can drop straight into a procurement deck.
How it works
You start with common enterprise criteria — security & compliance (SOC 2 / ISO 27001), data residency, uptime SLA, context window, price, support, and ecosystem maturity — each with an importance weight from 1 to 5. Then you score each vendor 1–5 on every criterion. For each vendor the tool computes a weighted sum (score × weight, summed across criteria) and normalises it to a percentage of the maximum achievable, then ranks vendors highest-first. Add or remove rows to match your real shortlist and RFP.
How the weighted scoring works
For a concrete example: suppose you are evaluating three vendors on five criteria with the following weights (declared before you look at the vendors):
| Criterion | Weight | Why this weight |
|---|---|---|
| SOC 2 / ISO 27001 | 5 | Dealbreaker — legal cannot proceed without it |
| Data residency (EU) | 5 | GDPR requirement, non-negotiable |
| Uptime SLA | 4 | Critical path for production use |
| Context window | 3 | Important for our use case but workarounds exist |
| Price per 1M tokens | 3 | Significant at scale, not a dealbreaker |
| Developer ecosystem | 2 | Nice to have, not essential |
A vendor scoring 4/5 on SOC 2 (weight 5) contributes 20 points; a vendor scoring 5/5 on developer ecosystem (weight 2) contributes 10 points. The security-and-compliance dimension correctly outweighs the ecosystem score two-to-one. Setting weights first is what prevents a polished demo from winning over a security-reviewed runner-up.
Criteria to consider beyond the defaults
The pre-loaded criteria cover the most common enterprise requirements. Depending on your use case, you may want to add:
- Fine-tuning support — essential if you plan to customise the model
- Structured output / function calling — required for agentic workflows
- Latency (P50/P95) — critical for real-time or interactive features
- Batch API availability — important for high-volume, non-latency-sensitive workloads where per-token cost matters most
- Model versioning and deprecation policy — often overlooked until a pinned model version is deprecated and breaks production
- Audit logging and exportability — required for some compliance frameworks to demonstrate what was processed and when
Add these as custom criteria and weight them to your actual needs.
Tips and notes
Set the weights before you score the vendors — deciding priorities while staring at the scores invites bias toward a favourite. Reserve a weight of 5 for true dealbreakers (a missing SOC 2 report should sink a vendor regardless of how good the demo was). Score from evidence, not marketing: trial the models, read the DPA, and ask for the actual SLA credit terms. When two vendors finish within a few points, the ranking is a tie — break it on a hands-on pilot or contract flexibility, not by nudging a score. Everything stays in your browser, so you can iterate freely.