Understanding Confidence Scores: What They Mean and How to Act on Them

What is a confidence score?

Every answer from 0xDoktor includes a confidence score between 0.0 and 1.0. This score reflects how semantically close the retrieved knowledge base passages were to the user's question.

Think of it as the system's self-reported certainty: "How well does what I found match what was asked?"

A score of 0.9 means the knowledge base contained highly relevant content. A score of 0.2 means the system found something loosely related but is not confident it actually answers the question.

How confidence is calculated

The score is derived from the cosine similarity between:

1. The embedding of the user's question
2. The embeddings of the retrieved knowledge base chunks

The similarity values from the top-k retrieved chunks are aggregated (weighted by retrieval rank) to produce the final score. A high score means the retrieved chunks were semantically close to the question.

Note: confidence measures retrieval quality, not factual accuracy. An answer drawn from incorrect documentation can have a high confidence score.

Recommended thresholds

| Use case | Recommended threshold | |----------|----------------------| | General customer support | 0.60 | | Technical documentation | 0.65 | | Compliance or legal information | 0.75 | | Medical or safety-critical | 0.80+ |

These are starting points. After 2–4 weeks of real traffic, review your confidence distribution and adjust based on your false-positive (low-confidence but correct answer) and false-negative (high-confidence but wrong answer) rates.

What happens below threshold?

When a response falls below your configured threshold, 0xDoktor:

1. Returns the answer with a warnings: ["low_confidence"] flag in the API response.
2. Optionally triggers a fallback message in the widget: "I'm not entirely sure about this — you may want to check with our team."
3. Flags the conversation for operator review in the analytics dashboard.

You can configure the fallback behavior (message text, contact link, escalation path) in the admin dashboard under Widget Settings → Escalation.

Using confidence data to improve your KB

Low-confidence conversations are your most valuable feedback signal:

Weekly KB gap review

1. In the admin dashboard, filter conversations by confidence < 0.5.
2. Read through the questions. Group similar questions by topic.
3. For each topic group, ask: "Do we have documentation that answers this?"
4. If not: write it, ingest it, re-test.

Trend monitoring

Track your average confidence score over time. A declining trend often precedes a support ticket increase — users are asking things your documentation doesn't cover. Catching this early lets you fill the gap before it becomes a volume problem.

What to do when confidence is persistently low for a covered topic

If a question about a topic you know is documented consistently returns low confidence:

1. Check that the relevant document was successfully ingested (chunk count > 0 in the KB browser).
2. Review the document structure — is the answer in a self-contained section?
3. Add a dedicated FAQ entry for this specific question with the exact phrasing users are using.
4. Re-test after re-ingestion.

Confidence vs. resolution rate

Confidence is a leading indicator (how well the system found relevant content) while resolution rate is a lagging indicator (whether users actually got their answer resolved). Use both:

• High confidence + low resolution rate: the retrieval is good, but the documentation may be inaccurate or incomplete.
• Low confidence + high resolution rate: your users are getting answers despite fuzzy retrieval — the LLM may be doing a good job extrapolating, but this is risky territory.

Aim for high confidence AND high resolution rate as the steady-state target.