What this automation does
Staff ask a question in Slack, Teams or a web page and get a short answer with links to the exact documents and sections it came from, drawn only from material they are allowed to open.
- Sync. Connectors pull documents from SharePoint, Google Drive, Notion or a wiki on a schedule, along with each document’s access list, owner and last-modified date.
- Parse and chunk. Each document is converted to text, split along its headings into sections of a few hundred words, and tagged with title, path, section heading and permissions.
- Index. Each chunk is stored with a vector embedding for meaning-based search and in a keyword index for exact terms like part numbers, policy codes and acronyms.
- Retrieve. When a user asks, the system looks up their groups from single sign-on, runs both searches filtered to what they can see, merges the results and reranks the top candidates.
- Answer. The model gets the question and the top chunks, answers only from them, and cites each claim. If the chunks do not answer the question, it says so and suggests who owns the topic.
- Learn. Thumbs up or down, plus the question, retrieved chunks and answer, are logged for review and for growing the test set.
When it is worth building
Build it when people repeatedly ask colleagues questions that are already answered somewhere in your documents, and the somewhere is spread across several systems. Sales engineers hunting for security questionnaire answers, field staff looking up procedures, and new hires learning policy are typical cases.
Do not build it if your documentation is thin or badly out of date. RAG returns what you have, so a weak knowledge base produces confident answers from stale pages. Fix the top 50 documents first. If your content lives in one place and the vendor’s built-in assistant passes your test questions, buy rather than build. For narrower scopes, an SOP assistant or an HR policy assistant is a faster first project.
Architecture
| Layer | What we use | Why |
|---|---|---|
| Connectors | SharePoint and Google Drive APIs via Microsoft Graph and Google’s Drive API, Notion API | Graph and the Drive API return both content and permissions, which most file exports lose. Notion’s API only sees pages shared with the integration and does not list who can view each page, so map Notion access separately |
| Parsing | Format-aware parsers for DOCX, PDF, HTML and slides | Heading structure drives good chunk boundaries |
| Store | Postgres with pgvector and full-text search, or Pinecone plus a keyword index | Hybrid search catches both paraphrased questions and exact codes |
| Permissions | Access lists stored per chunk, user groups resolved from SSO at query time | Filtering happens before the model sees anything |
| Reranking | A reranker model or an LLM scoring pass over the top 30–50 hits | Raises answer quality more cheaply than a bigger generator model |
| Generation | Claude Sonnet 5.5 or OpenAI’s GPT models via API | Follows citation formats well and says “not found” when instructed |
| Interface | Slack or Teams app, plus a web page | Meet people where they already ask questions |
| Evaluation | A versioned question set and a scoring script | Lets you change parts without guessing whether it got worse |
How to build it
- Collect real questions first. Pull 100 or more questions from Slack history, helpdesk tickets and onboarding notes. These shape scope and become your test set.
- Inventory sources and permissions. For each source, confirm the API returns access lists, how groups map to your identity provider, and how quickly permission changes can be picked up.
- Build ingestion with deletes. Sync additions, edits and deletions. A removed document must leave the index within hours, not linger until the next full rebuild.
- Chunk by structure. Split on headings, keep the heading path in each chunk, and keep tables intact. Test a few chunk sizes against your questions rather than picking one by default.
- Enforce permissions in the query. Pass the user’s group IDs as a filter to the search itself. Post-filtering after retrieval quietly drops good answers and is easier to get wrong.
- Write the answer prompt. Require a citation per claim, forbid outside knowledge for company facts, and define the “I could not find this” response. Mark retrieved text clearly as untrusted reference material.
- Harden against injected instructions. Strip hidden text and comments during parsing, do not render external images or links the documents supply, and give the assistant no write actions. If it later needs tools, see connecting an LLM to internal data safely.
- Evaluate, then pilot. Score retrieval hit rate and answer support on the test set, fix the worst failures, then pilot with one team for two weeks before a wider launch.
What breaks in production
- Permission drift. Someone is removed from a confidential folder, but the index still holds the old access list. Re-sync permissions more often than content, and test with a restricted account every release.
- Stale and duplicate versions. Five copies of the travel policy, three outdated. Prefer the newest version, down-rank folders named “archive” or “old”, and let document owners mark the canonical copy.
- Exact-term misses. Vector search alone fails on SKU codes, form numbers and acronyms. This is why hybrid search is the default, not an upgrade.
- Instructions planted in documents. A shared doc or uploaded vendor PDF contains text aimed at the model. With no tools and no external links rendered, the worst outcome is a wrong answer, which the citation exposes.
- Scanned PDFs and slide decks. Image-only pages index as empty. Detect low text density and run OCR or a vision model during ingestion.
- Answers without support. The model blends two chunks into a claim neither makes. Sample answers weekly and check each citation actually supports its sentence.
Timeline and cost
A first production assistant over two or three sources with permissions-aware retrieval, a Slack or Teams interface and an evaluation set typically takes four to eight weeks. Lumeor builds RAG knowledge assistants in the range of $15,000 to $40,000, depending on the number of sources, the volume and messiness of the documents, permission complexity, and the interfaces required. Running costs are embeddings, model calls and database hosting. For a full breakdown, see what a RAG knowledge assistant costs.
How to measure success
- Retrieval hit rate on the test set: the share of questions where a correct source appears in the top results
- Supported-answer rate: the share of answers where every citation backs its claim, from weekly sampling
- “Not found” rate, split into true gaps in documentation and retrieval misses
- Repeat questions in team channels and tickets for topics the assistant covers
- Permission test pass rate for restricted-account checks (must stay at 100 percent)