Why Your AI Chatbot Sucks (And How to Fix It in a Week)

I built the AI system that handles 700,000+ monthly customer interactions at Wizz Air. Before I rebuilt it, the previous version had an 18% escalation rate: nearly one in five conversations ended with a customer demanding a human agent. After the rebuild, that rate dropped to under 4%. The core feature did not change. The underlying model did not change. What changed was fixing the six problems below.

Problem 1: Your system prompt is too vague

Most system prompts look like this: "You are a helpful customer support assistant for [Company]. Be friendly and professional. Help users with their questions."

That is not a specification. That is a sentence. An LLM receiving that prompt will fill in all the blanks with its training data, which means it will behave inconsistently, answer questions outside your intended scope, and occasionally invent company policies it thinks sound plausible.

The fix

A production system prompt is 300 to 800 words. It specifies the persona, the explicit scope of topics the chatbot handles, topics it must refuse to engage with, the exact format and length of responses, how to handle uncertainty, and what to say when escalating. Include real examples of good and bad responses. The more specific the prompt, the more predictable the output.

The second version takes ten minutes to write and eliminates an entire category of bad responses. Spend one day auditing your worst chatbot interactions and rewriting the system prompt to address each one explicitly.

Problem 2: No user context

Your chatbot greets every user the same way, asks them to re-explain their problem from scratch on every session, and cannot personalize a single response. This is not an AI limitation. It is an architecture choice you made when you decided not to inject user context into the prompt.

An LLM is stateless by default. It knows only what is in the current context window. If you want it to know that the user has a premium subscription, that they contacted support three times this week, or that their last order was a specific product, you have to tell it. The model cannot infer this from a user ID.

The fix

Build a context injection layer. Before each API call, fetch a compact user context object from your database: account tier, account age, relevant history, open issues, and any other signals that should change how the chatbot responds. Inject this as a structured block in the system prompt.

The Wizz Air system injects a 150-token user context block on every request: booking status, loyalty tier, recent interaction history, and current flight status if applicable. That single change reduced the number of turns required to resolve a query from an average of 4.2 to 2.7, because the chatbot stopped asking for information it already had access to.

Problem 3: You are treating hallucinations as an AI problem

When your chatbot invents facts, most teams blame the model. They upgrade to a more expensive model. The hallucinations continue. They blame the model again.

In most production chatbots, hallucinations are a retrieval problem, not a model problem. The model is not inventing information because it is stupid. It is inventing information because the context you gave it did not contain the right answer, and it filled the gap with something plausible from its training data. The fix is not a smarter model. It is better context.

The fix

Audit your last 50 hallucinated responses. For each one, ask: was the correct information available in the context we provided? In most cases, the answer is no. The right information was not retrieved, or it was not in your knowledge base at all.

Fix 1: Add the missing information to your knowledge base. Fix 2: Add explicit instructions to the system prompt: "If you do not have specific information to answer this question from the provided context, say so explicitly. Do not estimate or infer policy details." A chatbot that says "I do not have that information, but here is where you can find it" is infinitely better than one that confidently states incorrect facts.

Problem 4: Your retrieval pipeline has never been evaluated

If your chatbot uses a knowledge base, your retrieval quality is almost certainly worse than you think. Teams stand up a vector store, embed their documents, and assume the retrieval works because the demo looked fine. Then they are surprised when the chatbot answers questions about topic A with documents about topic B.

This is exactly what happened with the Sanofi compliance AI before I rebuilt it. The retrieval was returning semantically similar documents that were not actually relevant to the compliance query. The model, receiving wrong context, generated answers that sounded authoritative and were 28% incorrect. That is not a model failing. That is a retrieval pipeline that was never tested.

The fix

Build a retrieval evaluation suite. Take 30 representative questions and manually identify which documents should be retrieved for each. Then measure recall@5: for each question, does the correct document appear in the top 5 retrieved results? If your recall@5 is below 80%, your chatbot is routinely working with the wrong context.

Common retrieval improvements: reduce your chunk size (256 to 512 tokens with overlap beats large 2,000-token chunks for most tasks), add BM25 keyword search alongside vector search, and add a cross-encoder reranker on the top 20 candidates. Each of these can add 10 to 20 percentage points to recall without changing your model.

Problem 5: Wrong model for the task

Model selection is over-discussed in the abstract and under-discussed for specific tasks. The instinct is to use the most capable model available. That instinct is expensive and often wrong.

Frontier models like GPT-4o and Claude 3.5 Sonnet are excellent for complex reasoning, nuanced judgment, and tasks with high ambiguity. They are unnecessary for tasks like classifying a support ticket into one of ten categories, extracting a structured object from a user message, or routing a query to the right handler. For those tasks, a smaller, faster, cheaper model performs comparably and costs 10 to 30 times less per token.

The fix

Map your chatbot's tasks and match model tier to task complexity. Use a small, fast model (GPT-4o-mini, Claude Haiku, Gemini Flash) for classification, routing, and structured extraction. Use a frontier model only for the generation step where nuance and coherence matter. This tiered architecture typically reduces per-interaction cost by 60 to 80% with no user-facing quality loss.

At 700,000 monthly interactions, that cost difference is not hypothetical. It is the difference between an AI feature that is economically viable at scale and one that quietly erodes your margins every month.

Problem 6: You have no idea how often it gives bad answers

This is the most common and most dangerous problem. Your chatbot is live. Users are talking to it. You check the uptime dashboard: green. Error rate: 0.3%. You assume everything is fine.

Uptime and error rate measure whether the API call succeeded. They tell you nothing about whether the answer was correct, helpful, or accurate. A chatbot that confidently answers every question incorrectly has perfect uptime metrics.

The fix

Instrument your chatbot for quality, not just availability. Track these four metrics daily:

• Escalation rate: what percentage of conversations end with a request for a human. Trending upward is a quality signal.
• Thumbs down rate: if you have any feedback UI, what percentage of responses are rated negatively.
• Repeat question rate: users who ask the same question twice in a conversation got an answer that did not help them.
• Conversation length vs. resolution: conversations that run more than 6 turns without resolution usually indicate the chatbot is failing to answer effectively.

Set up a weekly review of 20 randomly sampled conversations. Read them. You will find patterns in the failures within three weeks that no metric will surface on its own. This is the single highest-impact improvement you can make to your chatbot quality right now, and it costs nothing but an hour of your time per week.

A realistic timeline

Day 1: Audit 50 bad conversations. Document the failure patterns. Day 2: Rewrite your system prompt to address the top five failure patterns. Add user context injection. Day 3: Build a 30-question retrieval evaluation suite and measure recall@5. Day 4: Fix the worst retrieval issues (chunk size, hybrid search, or missing knowledge base content). Day 5: Set up your four quality metrics and a weekly conversation review process. Day 6: Instrument model tier routing so classification tasks use a smaller model. Day 7: Observe, measure, and iterate.

None of these fixes require switching models. None require a rewrite. They require honest diagnosis and disciplined execution. The chatbot that was embarrassing your product last week can be substantially better by Friday.