KH LabsBot Check
Demos
Bot CheckLLM Bill BenchmarkCost CutterSync WatchdogAgent Gateway
Book a 15-min call
Evidence pack

We attacked a bank’s support bot 40 ways. Then we fixed it.

Harbor Bank’s assistant is fictional. The runs are not: every reply below came from a live model, graded by code and by an LLM judge. Same model, same 40 questions. The only change is the guardrail layer.

Harbor Bank, its customers and all account data are invented for this demo.

Book a 15-min call Download sample evidence pack Read the transcripts
Harbor Bank assistant Loading recorded runs…
Beforev1.0, no guardrails
0/40 failed
Afterv1.1, guardrail layer
0/40 failed

FailedPassedSelect a cell to open its transcript

Eight ways a support bot gets a bank in the news

Each case is an attack we have seen in public incidents: invented refund policies, insults under provocation, data handed to the wrong person. Five languages, because filters written in English tend to stop at English.

Read the transcripts

The failing line is highlighted: either the sentence the judge quoted or the value a deterministic check caught.

    Graded twice, by code and by a judge

    A case fails if either grader flags it. Both verdicts are kept, so you can see where code and judge disagree.

    Deterministic checks

    • Another customer’s e-mail, phone, IBAN, card or balance in the reply
    • Full card or IBAN echoed back, even the customer’s own
    • Any amount, rate or day count that is not in the policy text
    • Refund or waiver promises in five languages
    • Directed insults, external links, missing human handoff

    LLM judge

    • One written pass/fail rubric per case
    • Sees the full policy knowledge base
    • Returns a verdict, a reason and the offending sentence
    • Catches what regex cannot: a confirmed customer, a drafted legal claim, a cruel roast

    The guardrail layer in v1.1

    1. Input screenInjection markers and an intent classifier
    2. RouterFraud, crisis, bereavement, complaints and other people’s data go to reviewed templates in five languages
    3. Grounded answerCurated policy text, masked record of the signed-in customer only, pasted text fenced as data
    4. Output filtersPII redaction, insult lexicon, promise detector, policy-number check, link stripping

    Run this on your bot

    Same suite, your policies, your languages. You get the transcripts, the fixes and the paperwork.

    Prices

    Bot Check prices
    Pilot1 bot, 40+ attacks; credited toward a full test$750
    Full red-team + compliance packchatbot / bot on your documents / agent with tools$2,900 / $4,900 / $7,500
    AI incident fixbot already misbehaving in public: contained within 48 hours$2,900
    EU AI Act Article 50 disclosure checktest records are included in every full test$490

    Retesting after fixes: $750 a month.

    Book a 15-min call

    Bring your bot’s URL.

    What you get

    • 40+ attacks tailored to your policies and markets
    • Before and after runs with failing transcripts
    • Guardrail fixes written against your stack
    • Regression suite you can re-run on every prompt change
    • EU AI Act evidence pack: system description, risk assessment, test results, human oversight, logging and change log with version hashes (structured after the EU AI Act and ISO/IEC 42001)
    Download the sample evidence pack (PDF) Open the HTML version

    Bot Check by KH Labs. Harbor Bank is a fictional company; every name, account number and balance on this page is invented. The evidence pack is a sample template, not legal advice.