What this automation does
Visitors ask a question in a chat window on your site and get a short answer with links to the page it came from.
- Ingest. A sync job pulls your help center, selected web pages, PDFs and policy documents, strips navigation and boilerplate, and splits each document into passages of a few hundred words with its title and URL attached.
- Index. Each passage is embedded and stored in a vector database, alongside a keyword index for exact terms such as product names, SKUs and form numbers.
- Retrieve. For each question, the bot runs both searches, merges the results and keeps the top five to eight passages above a relevance threshold.
- Answer. A model writes a short answer using only those passages and cites them as links. If none pass the threshold, it says it does not have that information.
- Hand off. On “I don’t know”, on a request for a person, or on buying-intent questions, it offers a contact form or live chat and passes the transcript along.
- Learn. Every unanswered question and every thumbs-down is logged to a review list, which becomes your content backlog.
When it is worth building
Build one when the same questions keep arriving and the answers already exist in writing: hours and locations, shipping and return policies, program requirements, product specs, setup steps. The bot’s main job is to get the right paragraph in front of a visitor who would not have found it.
Do not build one to cover for missing documentation. If half the questions have no written answer, the bot will spend half its time saying “I don’t know.” Write the content first. And if answers depend on who the customer is, you need a full support agent with account lookups. For an internal version aimed at staff, see the RAG knowledge assistant guide.
Architecture
| Layer | What we use | Why |
|---|---|---|
| Content sources | Help center, website, Google Drive or Notion folders | Use the content your team already maintains, no second copy to keep in sync |
| Ingestion | A scheduled job plus publish webhooks where available | Changes reach the bot within minutes for webhook sources and within a day for crawled ones |
| Chunking | Heading-aware splitting with title and URL on every chunk | Passages keep their context, and every answer can link to a real page |
| Vector store | Postgres with pgvector on Supabase, or Pinecone for very large corpora | pgvector is simpler and cheaper at FAQ scale; Pinecone suits large or fast-growing indexes |
| Search | Vector plus keyword search, merged | Embeddings miss exact codes and model numbers that keyword search catches |
| Answer model | Claude Sonnet 5.5 or Claude Haiku 5.5, or a comparable OpenAI GPT model | Haiku-class models are fast and cheap for short answers; step up when questions need more reasoning |
| Widget | A small embeddable script on Webflow, WordPress or your app | Your branding, your data, and no per-seat chat tool required |
| Review | A dashboard of unanswered and down-voted questions | Turns the bot into a list of documentation gaps |
How to build it
- Collect 100 real questions from support email, chat logs and sales calls. Write the correct answer and source page for each. This is the test set.
- Choose and clean sources. Remove outdated PDFs and duplicate pages. Two versions of the same policy is the most common cause of wrong answers.
- Build ingestion with metadata. Store title, URL, last-updated date and a source type on every chunk, so you can filter or boost the newest policy pages.
- Set up hybrid search and tune the relevance threshold on your test questions. Too low and the bot answers from weak matches. Too high and it refuses easy questions.
- Write a narrow system prompt. Say what the bot covers, that it must answer only from provided passages, how to cite, and the exact wording to use when it does not know.
- Run the 100 questions and grade each answer as correct, partly correct, wrong or correctly refused. Fix content and retrieval before touching the prompt.
- Add abuse limits: per-session rate limits, message length caps, and a provider spend alert.
- Launch on a few pages first, such as the help center and contact page, then expand once the review list shows steady quality.
What breaks in production
- Conflicting documents. The 2023 PDF says 30-day returns and the current page says 45. Retrieval finds both and the model picks one. Retire old files and boost recently updated sources.
- Tables flattened into nonsense. Pricing grids, rate sheets and spec tables lose their structure when split into text. Convert them to clean structured text or answer those questions from a small lookup table instead.
- Questions phrased in customer language. Customers say “cancel my plan” while your docs say “subscription termination.” Hybrid search and a short synonym list help more than prompt changes.
- Questions you should not answer. Medical, legal or safety-critical questions need a fixed response that points to a person, defined up front.
- Re-indexing gaps. A crawled page changes, but the nightly sync fails quietly. Alert on sync failures and show the last indexed date on the review dashboard.
- Answer length creep. Answers grow into essays as the prompt accumulates rules. Cap answers at a few sentences plus links, and test after every prompt change.
Timeline and cost
A chatbot on one website with up to a few hundred pages of content typically takes two to four weeks. Lumeor builds docs-based FAQ chatbots in the range of $6,000 to $18,000. The low end is one content source, the widget and basic review logging. The high end adds several sources with live sync, lead capture into your CRM, multiple languages and handoff into a helpdesk. Running costs depend on traffic, and for a typical SMB site they are usually modest. See what an AI chatbot costs for a fuller breakdown.
How to measure success
- Correct-answer rate on the 100-question test set, re-run after every content or prompt change
- Refusal rate, split into correct refusals (no content exists) and missed answers (content exists but retrieval failed)
- Thumbs-up share on answered questions, reviewed alongside the thumbs-down transcripts
- Repeat questions arriving by email or phone for topics the bot covers, compared month over month
- Content gaps closed: number of new or rewritten pages created from the unanswered-question list