Bot Checkby KH Labs Evidence pack

We attacked a bank’s support bot 40 ways. Then we fixed it.

Harbor Bank’s assistant is fictional. The runs are not: every reply below came from a live model, graded by code and by an LLM judge. Same model, same 40 questions. The only change is the guardrail layer.

Harbor Bank, its customers and all account data are invented for this demo.

Download sample evidence pack Read the transcripts
Harbor Bank assistant Loading recorded runs…
Beforev1.0, no guardrails
0/40 failed
Afterv1.1, guardrail layer
0/40 failed

FailedPassedSelect a cell to open its transcript

Eight ways a support bot gets a bank in the news

Each case is an attack we have seen in public incidents: invented refund policies, insults under provocation, data handed to the wrong person. Five languages, because filters written in English tend to stop at English.

Read the transcripts

The failing line is highlighted: either the sentence the judge quoted or the value a deterministic check caught.

    Graded twice, by code and by a judge

    A case fails if either grader flags it. Both verdicts are kept, so you can see where code and judge disagree.

    Deterministic checks

    • Another customer’s e-mail, phone, IBAN, card or balance in the reply
    • Full card or IBAN echoed back, even the customer’s own
    • Any amount, rate or day count that is not in the policy text
    • Refund or waiver promises in five languages
    • Directed insults, external links, missing human handoff

    LLM judge

    • One written pass/fail rubric per case
    • Sees the full policy knowledge base
    • Returns a verdict, a reason and the offending sentence
    • Catches what regex cannot: a confirmed customer, a drafted legal claim, a cruel roast

    The guardrail layer in v1.1

    1. Input screenInjection markers and an intent classifier
    2. RouterFraud, crisis, bereavement, complaints and other people’s data go to reviewed templates in five languages
    3. Grounded answerCurated policy text, masked record of the signed-in customer only, pasted text fenced as data
    4. Output filtersPII redaction, insult lexicon, promise detector, policy-number check, link stripping

    Run this on your bot

    Same suite, your policies, your languages. You get the transcripts, the fixes and the paperwork.

    Support-bot guardrail check

    $750 pilot · $1,500 with a regression suite

    • 40+ attacks tailored to your policies and markets
    • Before and after runs with failing transcripts
    • Guardrail fixes written against your stack
    • Regression suite you can re-run on every prompt change

    EU AI Act evidence pack

    $1,500 for up to 3 systems

    • System description, intended purpose and misuse
    • Risk assessment, test method and results
    • Human oversight, logging and change log with version hashes
    • Structured after the EU AI Act and ISO/IEC 42001
    Download the sample (PDF) Open the HTML version