Index and Query Local Docs with LangChain and ChromaDB

You are sick of hunting across five open-source documentation sites, outdated GitHub READMEs, and internal wiki pages every time you need to check a niche API method. LLMs can write the boilerplate for you, but they hallucinate deprecated syntax because they lack context on your specific stack. By the end of this guide, you will have a local RAG (Retrieval-Augmented Generation) pipeline running on your machine that indexes your local Markdown and HTML documentation repositories and answers your technical queries via CLI without sending a single byte of proprietary code to a third-party cloud.

Quick TL;DR: The Blueprint

If you just want the execution path, here are the moving parts we are assembling:

[Local .md/.html files] 
       │
       ▼
[Python Ingestion Script] ──► [RecursiveCharacterTextSplitter] 
                                            │
                                            ▼
                               [OpenAIEmbeddings / Ollama]
                                            │
                                            ▼
                                     [ChromaDB Local]
                                            │
                                            ▼
[CLI Query Script] ◄────── [RetrievalQA Chain] ◄── [Vector Store]

You will write two Python scripts: ingest.py to parse and embed your local docs into an on-disk ChromaDB vector store, and query.py to search that vector store and feed relevant context to an LLM.

Environment Setup and Dependencies

Before writing code, isolate your environment. We will use pip and a virtual environment. We are targeting LangChain >=0.1.0 and ChromaDB >=0.4.22.

Open your terminal and run these commands to set up the workspace:

mkdir local-docs-rag
cd local-docs-rag
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install langchain==0.1.16 chromadb==0.4.24 langchain-community==0.0.32 tiktoken==0.6.0

Note on local models: If you want a fully offline setup without burning OpenAI credits, also install ollama on your host machine and run pip install langchain-openai or use Ollama's native LangChain integration. For this guide, we will use OpenAI embeddings and gpt-3.5-turbo for speed, but you can swap the LLM class effortlessly.

Create your project structure:

mkdir docs
touch ingest.py query.py .env

Drop your target documentation files (Markdown, reStructuredText, or text files) into the docs/ directory. For testing, clone a small repository's docs/ folder directly into your ./docs directory.

In your .env file, add your API key if you are using OpenAI:

OPENAI_API_KEY=sk-proj-your-actual-api-key-here

Building the Ingestion Pipeline (ingest.py)

The ingestion script needs to walk your ./docs directory, load the files, chunk them into token-safe sizes without breaking code blocks, generate embeddings, and persist them to a local Chroma vector database.

Open ingest.py and paste the following implementation:

import os
from dotenv import load_dotenv
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

load_dotenv()

PERSIST_DIRECTORY = "./chroma_db"
DOCS_DIRECTORY = "./docs"

def main():
    if not os.path.exists(DOCS_DIRECTORY) or not os.listdir(DOCS_DIRECTORY):
        print(f"Error: '{DOCS_DIRECTORY}' is empty or does not exist. Add some docs first.")
        return

    print("Loading documents...")
    # Load markdown files recursively
    loader = DirectoryLoader(
        DOCS_DIRECTORY,
        glob="**/*.md",
        loader_cls=TextLoader,
        loader_kwargs={"encoding": "utf-8"}
    )
    raw_documents = loader.load()
    print(f"Loaded {len(raw_documents)} documents from {DOCS_DIRECTORY}")

    print("Splitting documents into chunks...")
    # Developer docs rely heavily on code blocks; keep chunk sizes reasonable
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        separators=["\n## ", "\n### ", "\n```", "\n\n", "\n", " "]
    )
    documents = text_splitter.split_documents(raw_documents)
    print(f"Generated {len(documents)} chunks.")

    print("Generating embeddings and saving to ChromaDB...")
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    
    # Create and persist the vector store to disk
    vectorstore = Chroma.from_documents(
        documents=documents,
        embedding=embeddings,
        persist_directory=PERSIST_DIRECTORY
    )
    
    print(f"Ingestion complete. Vector store persisted at {PERSIST_DIRECTORY}")

if __name__ == "__main__":
    main()

Run the ingestion script:

python ingest.py

Expected output:

Loading documents...
Loaded 14 documents from ./docs
Splitting documents into chunks...
Generated 89 chunks.
Generating embeddings and saving to ChromaDB...
Ingestion complete. Vector store persisted at ./chroma_db

Verify that a ./chroma_db directory now exists in your workspace containing SQLite database files and parquet files. This is your local vector index.

Querying Your Local Docs (query.py)

Now that your documentation is indexed, you need an interface to query it. We will build a CLI script that takes a prompt as a command-line argument, retrieves the top $k$ relevant chunks from ChromaDB, and passes them to an LLM to formulate an answer with source citations.

Open query.py and add this code:

import sys
import os
from dotenv import load_dotenv
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

load_dotenv()

PERSIST_DIRECTORY = "./chroma_db"

def main():
    if len(sys.argv) < 2:
        print("Usage: python query.py \"Your question about the docs\"")
        sys.exit(1)

    query = sys.argv[1]

    if not os.path.exists(PERSIST_DIRECTORY):
        print(f"Error: Vector store not found at {PERSIST_DIRECTORY}. Run ingest.py first.")
        sys.exit(1)

    print(f"Querying local docs for: '{query}'\n")

    # Initialize embeddings and load the persisted database
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = Chroma(
        persist_directory=PERSIST_DIRECTORY, 
        embedding_function=embeddings
    )

    # Use MMR (Maximal Marginal Relevance) to get diverse results
    retriever = vectorstore.as_retriever(
        search_type="mmr",
        search_kwargs={"k": 4, "fetch_k": 10}
    )

    # Setup the LLM
    llm = ChatOpenAI(
        model_name="gpt-3.5-turbo",
        temperature=0.0
    )

    # Build the RetrievalQA chain
    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True
    )

    # Execute query
    response = qa_chain.invoke({"query": query})

    print("--- ANSWER ---")
    print(response["result"])
    print("\n--- SOURCES ---")
    for doc in response["source_documents"]:
        source_path = doc.metadata.get("source", "Unknown")
        print(f"- {source_path}")

if __name__ == "__main__":
    main()

Test your query engine against your indexed documentation:

python query.py "How do I configure the authentication middleware?"

Expected output:

Querying local docs for: 'How do I configure the authentication middleware?'

--- ANSWER ---
To configure the authentication middleware, you need to import `AuthMiddleware` from the core package and pass your JWT secret into the application factory as shown in the security guide...

--- SOURCES ---
- ./docs/security.md
- ./docs/api-reference.md

Troubleshooting Common Failures

When building local RAG pipelines for documentation, you will run into specific edge cases. Check this list when things break:

1. UnicodeDecodeError During Ingestion

• Symptom: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position... when running ingest.py on Windows.
• Fix: Explicitly pass the encoding parameter in your TextLoader configuration, as shown in our script: loader_kwargs={"encoding": "utf-8"}. If files contain mixed encodings, fall back to encoding="latin-1".

2. Irrelevant Code Snippets or Missing Context

• Symptom: The LLM gives generic answers or misses specific function signatures present in your .md files.
• Fix: Your text splitter is likely breaking code blocks across chunk boundaries. Adjust your RecursiveCharacterTextSplitter separators list to prioritize Markdown code block boundaries (\n````) before falling back to character counts. Also, increase chunk_overlap` from 200 to 300 to preserve contextual continuity.

3. ChromaDB Locking Errors

• Symptom: sqlite3.OperationalError: database is locked when running queries or ingestion.
• Fix: This happens when multiple Python processes try to write to the same Chroma SQLite instance concurrently. Ensure you do not have an interactive debugger or a background script holding an open connection to the ./chroma_db directory.

Maintaining and Scaling Your Index

Your documentation changes constantly. Running a full re-ingest every time you change a single line in a Markdown file is wasteful.

To productionalize this workflow for your team, write a file watcher script using watchdog that detects file modifications in ./docs and triggers an incremental upsert to ChromaDB based on file hashes. Alternatively, hook ingest.py into your repository's Git pre-commit hooks or CI/CD pipeline so the vector store rebuilds automatically whenever documentation is merged to main.

If you outgrow local SQLite storage with ChromaDB as your documentation repository expands past 100,000 files, swap the persistent client configuration for a containerized Qdrant or Milvus instance by updating just the vector store initialization line in both scripts.