Finding information about test coverage today means opening TestRail to search cases, digging through Confluence for spec docs, asking teammates who might remember. It's slow and inconsistent. The RAG Chat tab turns your entire QA knowledge base into a single queryable interface, answered in seconds, grounded in your actual documentation.
The Problem It Solves
Ask any QA engineer what happens when someone on the team asks "do we have test coverage for the 2FA login flow?" The answer is a combination of: opening TestRail and typing in the search box, opening Confluence and hunting through nested pages, pinging someone in Slack who might know, and hoping the answer is right. This is not a knowledge problem; the documentation exists. It's an access problem. The information is scattered across three systems and a human's memory.
The Talk to Tests chat interface solves this by indexing all of that documentation (TestRail cases, Confluence pages and uploaded test-case exports) into a vector database, and letting engineers query all of it through a single conversational interface. Ask in plain English. Get an answer in seconds. Traceable to specific documents.
What Powers It: The Knowledge Base
The knowledge base is built from three sources synced via the Admin panel:
- TestRail sync: every test case across all connected projects is pulled, chunked by section, and embedded into ChromaDB. When you ask "what test cases do we have for card payments?", the system retrieves the actual TestRail cases.
- Confluence sync: product spec pages, runbooks, onboarding guides, and design docs are all indexed. The AI can reference your actual product documentation when answering.
- Test-case uploads: CSV or Excel test-case exports, for suites that live outside TestRail. Each upload is validated, chunked and embedded automatically.
All embeddings are stored locally in ChromaDB, no test case content or documentation leaves your infrastructure when using a local LLM. The vector database is persistent across restarts and only updated when you trigger a sync or upload a new document.
How RAG Works Here
RAG, Retrieval-Augmented Generation, means the LLM generates its answer from retrieved context, not from its training data. Here's the retrieval pipeline for every question asked in Talk to Tests:
The engineer's question is converted into a vector embedding using the same model used during document indexing. The question and the documents now live in the same vector space, making similarity comparison possible.
A hybrid search runs keyword (BM25) and vector similarity side by side across all stored chunks, expands the query with related terms, and reranks the candidates so the most relevant pieces reach the model. These chunks might come from TestRail cases, Confluence pages or uploaded test cases, all searched simultaneously. Repeat questions and embeddings are cached.
The retrieved chunks are injected into the LLM prompt as context, along with the original question. The prompt makes that context the source of truth: the model may tidy wording and structure, but must not invent product rules, screens or API fields the context doesn't support, and must say what is missing when the context falls short.
The LLM writes its answer from the retrieved context, using general QA knowledge only for clarity and structure. This is what keeps hallucination down, and because every answer lists its sources, each claim can be checked against a specific document or TestRail case.
Why "grounded" matters
A general-purpose LLM asked "what are our card payment test cases?" will either refuse or hallucinate plausible-sounding test cases that don't exist. A RAG system asked the same question retrieves the actual cases from your TestRail instance and quotes them directly. The difference between a useful answer and a dangerous one is grounding.
Using the Chat
The chat lives on the Talk to your Tests page. Engineers can ask questions in plain English; the system handles the retrieval and generation. Common use cases:
- Coverage discovery: "Do we have test cases for 2FA at login?"
- Onboarding: "Give me step-by-step details on how to onboard a new HK business"
- Scenario lookup: "What are the critical test cases for Card Payments in SG?"
- Gap finding: "What edge cases are missing from our payment flow tests?"
Every answer shows its sources, how long it took and what it cost, so the team can see both where an answer came from and what the feature spends. Individual test cases in an answer can be copied straight out.
A coverage question answered: the matching TestRail cases with priority, platform, steps and expected results, plus the response time, cost and tokens used.
Multi-LLM Support
The chat backend supports three LLM providers, configurable via environment variable. ChromaDB remains local regardless of which LLM is chosen; only the generation step changes.
Switch provider with LLM_PROVIDER=openai|gemini|ollama, from the admin portal's Studio Settings or the environment config, and restart the app. Switching to or from OpenAI also changes the embedding model, so the knowledge base needs a rebuild. When using Ollama, the LLM runs on your machine; no data leaves your network at any stage of the pipeline. This is important for teams with strict data residency requirements.
On-premise option
Teams that cannot send test case content or product documentation to cloud LLMs can run the full stack locally: Ollama for generation, ChromaDB for vector storage, Flask for the API. Zero data egress. The trade-off is response quality: the cloud models produce notably more coherent answers for complex multi-document questions.
You've seen the full system
Six features, three repositories, one continuous loop: generation, authoring, triaging, healing, adaptation, and now the knowledge layer that ties it all together. Read the full architecture overview or explore another feature.