An AI rejected a job candidate.
The candidate asks why.
The company asks their data science team.
They say “the model weighted 47 factors.”
Which factors mattered most for this specific decision?
“We can’t tell you – it’s a neural network.”
The company is about to get sued. And they can’t defend the decision because they can’t explain it.
The Explainability Problem
Most companies use AI they can’t explain.
Not because they’re hiding something. Because the AI itself is a black box.
Deep learning models. Complex neural networks. Vendor AI with proprietary algorithms.
They work. They produce decisions. But nobody can explain why they made a specific decision for a specific person.
And that’s becoming a legal problem.
Why Explainability Is Now a Legal Requirement
Multiple regulations now require AI explainability.
EU AI Act: High-risk AI systems must provide “appropriate information” about how the system works and the logic behind decisions.
GDPR Article 22: Individuals have the right to obtain an explanation of automated decisions that significantly affect them.
Equal Credit Opportunity Act: Lenders must provide specific reasons for credit denials, even when AI makes the decision.
State AI laws: Colorado, California, and other states require explanations for AI decisions in specific contexts.
Employment law: EEOC guidance requires employers to explain why candidates were rejected, including AI-driven decisions.
“The algorithm decided” isn’t legally sufficient.
Companies need to explain why the algorithm decided that way for this specific person.
What Most Companies Actually Have
When asked “Can you explain why your AI made specific decisions?” here’s what companies say:
“We can show you the model architecture.” That’s not an explanation. That’s a technical diagram. It doesn’t explain why the AI rejected this candidate vs. accepted that one.
“We can show you what factors the AI considers.” Listing 47 factors doesn’t tell you which factors mattered for this specific decision.
“We can show you the model’s overall accuracy.” “The model is 85% accurate” doesn’t tell the rejected candidate why they were in the 15%.
“The vendor says their AI is explainable.” Can they provide case-level explanations? Or just general descriptions?
“We use interpretable machine learning techniques.” Can companies actually produce explanations when someone asks? Or is it theoretical?
Most companies have general information about their AI. They don’t have specific explanations for specific decisions.
The Two Types of Explainability
There are two types.
Global Explainability: “Here’s generally how our AI works. It considers these factors. Here’s the logic.”
This satisfies curiosity. It doesn’t satisfy legal requirements.
Local Explainability: “Here’s why the AI made this specific decision for this specific individual. These three factors were most important. Here’s how they were weighted.”
This is what regulators and courts want. This is what most companies can’t provide.
When someone files an EEOC complaint or a discrimination lawsuit, they don’t want to know how the AI generally works.
They want to know why it rejected them specifically.
Why Black Box AI Can’t Be Explained
Some AI systems are inherently unexplainable.
Deep neural networks learn patterns through millions of parameters across dozens of layers. You can’t trace why a specific input produced a specific output.
Ensemble models combine multiple models. The final decision is a weighted combination. You can’t point to a single factor as “the reason.”
Vendor proprietary algorithms won’t be shared. Vendors claim it’s a trade secret. Companies can’t explain what they don’t know.
Complex feature interactions mean the AI learns that certain combinations of factors matter. Not individual factors alone. You can’t reduce it to “you were rejected because of X.”
If AI falls into any of these categories, companies probably can’t provide the explanations regulators and courts require.
What Happens in Litigation
The pattern in multiple cases:
Plaintiff: “I was qualified for the job. Why was I rejected?”
Company: “Our AI screening system determined you weren’t a good fit.”
Plaintiff: “Why? What about my application led to that decision?”
Company: “The AI analyzed multiple factors.”
Plaintiff: “Which factors? How were they weighted?”
Company: “We can’t provide that level of detail. The model is complex.”
Plaintiff’s lawyer: “So you’re telling the court you made an employment decision, but you can’t explain why?”
Judge: “That’s not acceptable. If you can’t explain the decision, you can’t prove it wasn’t discriminatory.”
When companies can’t explain decisions, courts infer discrimination.
The inability to explain becomes evidence against them.
The Vendor Excuse Doesn’t Work
Companies try to shift responsibility to vendors.
“We use a third-party AI system. The vendor is responsible for explainability.”
This doesn’t protect anyone.
Companies deployed the AI. They’re responsible for its decisions. “The vendor couldn’t explain it” doesn’t absolve them.
They made the decision. The AI was their tool. They’re liable for outcomes.
Courts hold companies accountable. “We trusted the vendor” isn’t a defense. Companies have a duty to ensure they can explain and defend their decisions.
If companies can’t explain their AI’s decisions, they shouldn’t deploy it. Even if the vendor says it’s “state of the art.”
Real Examples
Hiring AI: A company deployed AI to screen resumes. A rejected candidate asked why. The company said “The AI determined you weren’t qualified.” The candidate met every listed requirement. The company couldn’t explain what disqualified them. Result: EEOC complaint. Settlement. Complete rebuild of hiring process.
Credit Scoring: A lender used AI for credit decisions. An applicant with a 750 credit score, high income, and low debt ratio was denied. The lender couldn’t explain why. Result: ECOA violation. Regulatory penalty. Consent order requiring explainable credit decisions.
Insurance Underwriting: An insurer used AI for risk assessment. A customer was charged higher premiums and asked why. The insurer couldn’t explain which factors made them high-risk. Result: State regulator investigation. Had to switch to explainable models.
The Questions That Need Answers
If someone challenges an AI decision, companies need to answer:
“Why was I rejected/denied/flagged?” Not “the AI decided.” Specific reasons tied to specific facts about this person.
“What could I have done differently to get a different outcome?” If companies can’t tell someone what would change the AI’s decision, they can’t prove the decision was based on legitimate factors.
“Did the AI consider factors it shouldn’t have?” Protected characteristics. Proxy variables. Irrelevant information.
“Is the AI treating people consistently?” If two similar people get different outcomes, companies need to explain why. “The AI is complex” doesn’t cut it.
Can AI answer these questions for every decision? If not, there’s an explainability problem.
What Explainability Actually Requires
Case-level explanations. For each decision, identify the top 3-5 factors that drove the outcome and show how they were weighted.
Human-understandable reasons. Not “parameter X had a coefficient of 0.47.” Plain language: “You were rejected because your experience didn’t match the role requirements.”
Documentation that can be produced. Not theoretical explainability. Actual records to show regulators, lawyers, and individuals.
Consistent methodology. Proof of applying the same logic across all decisions. No unexplained variations.
Validation of prohibited factors. Proof the AI didn’t consider race, gender, age, or proxy variables.
This means either using inherently interpretable AI models or implementing post-hoc explainability tools that actually work.
When to Avoid Black Box AI Entirely
For some use cases, black box AI is too risky.
Employment decisions. Hiring, promotion, termination. If companies can’t explain why someone was rejected, don’t use AI.
Credit decisions. ECOA requires specific reasons. If AI can’t provide them, it’s non-compliant.
Healthcare decisions. Medical AI that can’t explain its reasoning creates liability and erodes trust.
High-stakes decisions affecting individuals. Anything where someone might challenge the decision.
The performance gain from complex black box models isn’t worth it if companies can’t explain decisions when challenged.
What to Ask Vendors
If buying AI systems, ask these questions:
“Can your AI provide case-level explanations for individual decisions?” Not “is it explainable?” Specifically: can it explain individual decisions?
“What do those explanations look like?” Ask for examples. Real explanations for real decisions. Not marketing claims.
“Can we access explanations in real-time when someone asks?” Or is there a delay?
“What happens if we’re in litigation and need to explain a decision from six months ago?” Can vendors produce historical explanations?
“Have your explanations been tested in regulatory proceedings or litigation?” If not, you’re the guinea pig.
If vendors can’t give clear, confident answers, their AI isn’t as explainable as they claim.
Ask Yourself This
Pick the highest-risk AI system. The one making decisions about people.
Now imagine someone files a discrimination complaint about a decision from last month.
Can the company produce a specific, defensible explanation for why the AI made that decision?
If the answer is no, there’s an explainability problem. And it’s going to cost money.
Schedule a 30-minute AI Explainability Assessment where we’ll:
- Review AI systems for explainability gaps
- Show what legally sufficient explanations look like
- Identify which AI systems create the most explainability risk
- Give options for implementing explainability
We’ve helped dozens of companies move from black box AI to explainable AI systems that can withstand regulatory scrutiny and litigation.
Because “the algorithm decided” stops being an acceptable answer the moment someone challenges the decision.
Make sure there’s a better one.
Brian Alexander
Founding Partner, VisionAI+ Consulting Group
Former General Counsel, AI Engineering Firm
Certified ISO 42001 Lead Auditor