Support Assistant Grounded in Product Documentation
A retrieval-based assistant answering customer questions from the vendor’s own documentation, with citations.
View live demo Opens a working demo with sample content in a new tab.
On location
Man at a laptop in an officePhoto: Bench Accounting benchaccounting (CC0)
Businessman working and writing notes in officePhoto: JESHOOTS.COM jeshoots (CC0)
Woman working on her laptop. External hard drive plugged into the laptopPhoto: Shixart1985 (CC BY 2.0)
Working on a laptop outdoorsPhoto: Alejandro Escamilla alejandroescamilla (CC0)
Man working on laptop outsidePhoto: rawpixel.com rawpixel (CC0)
About the Project
This project is a RAG AI support assistant for a B2B software vendor: an assistant that answers customer questions using only the vendor’s own documentation, release notes and security FAQ, and shows the sources beneath every reply. It was built after an earlier off-the-shelf chatbot trial went wrong by confidently describing features that did not exist.
Retrieval-augmented generation changes how an AI customer support chatbot behaves. Instead of answering from general training data, the assistant first retrieves the most relevant passages from the vendor’s documentation, then answers from those passages alone. When it cannot find a good answer, it says so and hands the conversation to a person.
About the Client
The client is a B2B software vendor whose customers configure and integrate its product in many different environments. The support team handled the same configuration questions again and again, even though the answers already existed in a comprehensive documentation site that customers rarely searched successfully.
Trust was the constraint. After the previous chatbot invented features, the team needed an assistant that would be accurate, transparent about where answers come from and honest when it did not know.
Project Goals
Answer routine configuration questions accurately from existing documentation.
Show the source pages for every answer so customers can verify it.
Admit uncertainty and hand over to a human, with the conversation attached.
Measure answer quality before launch and after every change.
Keep running costs predictable with budgets and caching.
Project Overview
The Challenge and Our Solution
The Challenge
Support handled the same configuration questions repeatedly, with answers already written in a documentation site that customers rarely searched successfully. An off-the-shelf chatbot trialled earlier had invented features that did not exist, which cost the team more time than it saved and lost customer confidence.
Our Solution
We built a retrieval-augmented assistant rather than a general chatbot. The documentation, release notes and security FAQ are chunked, embedded and stored in a vector database; a question retrieves the most relevant passages and the model answers using only those, with the sources shown beneath every reply. Where confidence is low the assistant says so and offers a handover with the conversation attached. Before launch we built an evaluation set of 180 real support questions and measured answer quality against it, and we set per-conversation token budgets with caching for repeated questions.
How Retrieval-Augmented Generation Grounds Every Answer
The documentation, release notes and security FAQ are split into passages, converted into embeddings and stored in a vector database. When a customer asks a question, the system retrieves the most relevant passages and passes only those to the language model, with instructions to answer from that material and nothing else.
This is what makes a documentation chatbot trustworthy. The model is used for what it does well, understanding the question and writing a clear answer, while the facts come from content the vendor controls. When the documentation changes, it is re-indexed and the assistant’s answers change with it, with no retraining required.
Documentation, release notes and security FAQ indexed as passages
Answers generated only from retrieved passages
Re-indexing keeps answers current as the docs change
Citations and Honest Uncertainty
Every reply shows the documentation pages it came from beneath the answer. Customers can click through to read more, and support staff can see immediately whether an answer was well-founded. Citations turn the assistant from a black box into a guide to the vendor’s own material.
Where retrieval confidence is low, the assistant says it is not sure rather than guessing, and offers a handover with the full conversation attached. The customer does not have to repeat themselves, and the support agent starts with context instead of a blank ticket.
Evaluating the Assistant Before Customers See It
Before launch we built an evaluation set of 180 real support questions drawn from past tickets, with expected answers. The assistant was scored against it, and the evaluation harness runs again after every prompt, model or retrieval change. This turns “it seems better” into measurable LLM evaluation and catches regressions before customers ever meet them.
Production Architecture and Cost Control
The assistant runs as a PythonFastAPI service with PostgreSQL for conversations and handovers and a Redis cache for repeated questions. Per-conversation token budgets cap spending, and the model is chosen per task so that simple steps use smaller, cheaper models while answer writing uses a more capable one.
The outcome is a support assistant that resolves a substantial share of routine configuration questions without a ticket, while every answer remains traceable to the documentation it came from.
Our Process
How We Delivered It
1
Ticket and Content Review
We analysed past support tickets and the documentation to find the questions worth automating and the gaps in the existing content.
2
Retrieval Pipeline
Chunking, embeddings and a vector database built and tuned so the right passages are retrieved for real customer questions.
3
Evaluation Set
180 real questions with expected answers assembled into a repeatable evaluation harness before the assistant went near customers.
4
Assistant and Handover Build
The FastAPI service, citation display, confidence handling and human handover with the conversation attached.
5
Launch and Continuous Measurement
Released to customers with token budgets and caching in place, and re-evaluated after every prompt, model or content change.
Key Features
What We Delivered
Retrieval with citations
Answers drawn from indexed documentation passages, with sources shown to the customer.
Honest uncertainty
Says it does not know and hands over rather than inventing an answer.
Evaluation set
180 real support questions scored before launch and after every prompt or model change.
Cost control
Token budgets, caching of repeated questions and model selection per task.
Outcome
Results and Impact
A substantial share of routine configuration questions resolved without a ticket.
Every answer shows the documentation pages it came from.
Low-confidence questions hand over to a human with full context attached.
Measured against 180 real questions before launch, and re-run after each change.
Client Feedback
What the Client Says
★★★★★
What sold us was that it says when it does not know.
Support LeadB2B software vendor
Services
Services Behind This Project
The Python, Django & FastAPI, AI Integration & LLM, Database expertise we used to build this ai integration — and can bring to yours.
Practical AI in your product — assistants, document understanding, search over your own content, and OCR or classification pipelines built with Python and modern LLM APIs.
Schema design, query and index tuning, migrations and search infrastructure across MySQL, PostgreSQL, MongoDB, Redis and Elasticsearch.
Learn more
FAQ
Frequently Asked Questions
What is a RAG AI support assistant?
RAG stands for retrieval-augmented generation. The assistant first searches your own content, such as documentation and FAQs, for passages relevant to the question, then uses a language model to write an answer from those passages only. It keeps answers grounded in material you control rather than the model’s general knowledge.
How does it avoid making up answers?
Answers are generated only from retrieved passages, sources are shown with every reply, and low-confidence questions are handed to a person instead of guessed. An evaluation set of real questions is re-run after every change, so any drop in accuracy is caught before it reaches customers.
What content can an AI support assistant use?
Typically product documentation, help-centre articles, release notes, FAQs, policies and internal knowledge bases. The content is indexed and re-indexed as it changes. Content quality matters, so during discovery we often identify documentation gaps worth filling, which also helps customers who search the docs directly.
What affects the cost of building an AI chatbot like this?
The main factors are the volume and variety of source content, integrations with helpdesk or CRM systems, handover requirements, the evaluation work needed for your accuracy targets, and expected usage. Ongoing model costs are managed with token budgets, caching and choosing the right model for each task.
Can the assistant hand conversations over to our support team?
Yes. When the assistant is unsure or a customer asks for a person, the conversation is handed over with the full transcript attached. We can connect handover to your existing helpdesk so it arrives as a ticket in the tools your team already uses.
How is the assistant maintained after launch?
Content is re-indexed as documentation changes, the evaluation set is extended with new real questions, and results are re-checked whenever prompts or models are updated. We also review costs and handover patterns regularly, which often highlights where documentation or the product itself could be clearer.
Similar Work
Related Projects
Case studies built with the same technologies or for similar needs.
