A company chose its AI vendor after an impressive demo and competitive pricing.
The AI performed exactly as shown in the demo. Features matched the sales pitch. Implementation went smoothly.
Eighteen months later, regulators asked the company to explain how the AI makes decisions affecting customers. The company called the vendor. The vendor said that information is proprietary. The contract didn’t require them to disclose it.
The company couldn’t explain their own AI to regulators because they never asked the right questions before signing.
Training Data and Bias Testing
The most important question most companies never ask is what data the AI was trained on.
Training data determines what the AI learned and what biases it carries. Vendors who can’t explain their training data sources or what bias testing occurred are selling AI they don’t fully understand.
Ask specifically how bias testing was conducted, what protected characteristics were tested, and what results were found. Vague answers about “rigorous testing” without specifics indicate inadequate testing. Request documentation rather than accepting verbal assurances – vendors with genuine testing processes can produce it.
Explainability
Before deploying AI, companies need to understand what they’ll be able to explain to regulators and courts when challenged.
Ask what level of explainability the AI provides for individual decisions. Can the vendor show why a specific customer was rejected or flagged? Can they provide this in plain language rather than technical terms?
If you can’t explain AI decisions when regulators ask, that’s a governance failure. Evaluate vendors on whether their AI can be explained before you’re required to explain it.
Performance Guarantees That Actually Mean Something
Most vendor contracts guarantee uptime and response time. These don’t address what matters most for AI governance.
Negotiate performance standards covering accuracy across different demographic groups, not just overall accuracy. AI that performs well on average but poorly for specific populations creates discrimination liability regardless of overall metrics.
Establish requirements for vendors to notify you when model performance changes materially. Define what happens when AI produces harmful outcomes. Contracts should specify vendor obligations to investigate and remediate when AI failures cause demonstrable harm.
Data Rights
Understand exactly what the vendor does with your data before signing anything.
Many AI vendors use customer data to train their models. This creates competitive intelligence risks if your data improves models serving competitors, and privacy risks if customer data gets incorporated into vendor AI without proper consent.
Contracts should specify that your data cannot be used to train vendor models without explicit permission, that you can export all data in usable formats at any time, and that vendor data rights terminate completely when the contract ends.
Exit Rights
Every AI vendor evaluation should include analysis of how the relationship ends.
Standard contracts favor vendors with long terms and limited termination rights. AI-specific exit provisions should allow termination if the AI produces harmful outcomes, if the vendor can’t provide required explainability, or if the vendor materially changes the AI in ways affecting your compliance obligations.
Vendors who resist reasonable exit rights are betting you won’t want to leave once you’re dependent. They’re usually right unless you negotiate exit terms before signing.
The Right Standard
The right standard for AI vendor evaluation isn’t which vendor has the best features at the best price.
It’s which vendor can answer hard questions about their AI, provide documentation supporting those answers, share liability when their AI causes harm, and give you genuine freedom to leave if things go wrong.
Most vendors can’t meet this standard. That tells you something important before you sign anything.
Brian Alexander
Founding Partner, VisionAI+ Consulting Group
Former General Counsel, AI Engineering Firm
Certified ISO 42001 Lead Auditor