How to Build a RAG Pipeline with LangChain and OpenAI

Introduction

Large Language Models are powerful, but they have a fundamental limitation: they only know what was in their training data. If you want your AI application to answer questions about your company's internal documentation, your personal knowledge base, or any private data that OpenAI or Anthropic never saw, you need Retrieval-Augmented Generation — RAG.

RAG is the technique of combining a language model with an external knowledge source. Instead of asking the LLM to answer from memory, you first retrieve relevant documents from your own data, then pass those documents as context to the LLM. The result is an AI system that can answer questions about your data with accurate, grounded responses.

In this guide, you will build a complete RAG pipeline using LangChain and OpenAI, step by step.

What is RAG?

RAG works in three stages:

1. Indexing: You split your documents into chunks, convert them into vector embeddings, and store them in a vector database.
2. Retrieval: When a user asks a question, you convert the question into an embedding and search the vector database for the most relevant chunks.
3. Generation: You pass the retrieved chunks plus the user's question to an LLM, which generates an answer grounded in your data.

The key insight is that the LLM does not need to memorize your data. It just needs to read the relevant chunks you provide and synthesize an answer. This is the same way a human expert works: they do not memorize every document, but they know how to find and synthesize information quickly.

Prerequisites

Before you start, make sure you have:

• Python 3.10+ installed
• An OpenAI API key (set as OPENAI_API_KEY environment variable)
• Basic familiarity with Python and the command line

Install the required packages:

pip install langchain langchain-openai langchain-community chromadb unstructured

Step 1: Load Your Documents

LangChain provides loaders for many document formats: PDFs, Word docs, HTML pages, and more. For this example, we will use a directory of text files.

from langchain_community.document_loaders import DirectoryLoader, TextLoader

loader = DirectoryLoader(
    "docs/",
    glob="**/*.txt",
    loader_cls=TextLoader,
    show_progress=True,
)

documents = loader.load()
print(f"Loaded {len(documents)} documents")

If you have PDFs, swap the loader:

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("docs/report.pdf")
documents = loader.load()

Step 2: Split Documents into Chunks

LLMs have context windows. If you pass an entire 50-page document, you will exceed the limit or dilute the relevant information. Splitting into chunks ensures each chunk is focused and manageable.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " "],
)

chunks = splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")

The chunk_overlap parameter ensures that sentences split across chunk boundaries are not lost. A 200-character overlap is a good default.

Step 3: Create Embeddings and Store in a Vector Database

Each chunk needs to be converted into a vector embedding. OpenAI provides the text-embedding-3-small model, which is fast and cost-effective.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="chroma_db",
    collection_name="my_docs",
)

print(f"Stored {vectorstore._collection.count()} chunks in ChromaDB")

ChromaDB is a lightweight, local vector database. For production, you might use Pinecone, Weaviate, or Qdrant instead.

Step 4: Build the RAG Chain

Now combine retrieval and generation into a single chain.

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

llm = ChatOpenAI(model="gpt-4o", temperature=0)

prompt_template = PromptTemplate(
    input_variables=["context", "question"],
    template="""Use the following context to answer the question. If the answer is not in the context, say "I don't have enough information to answer that."

Context:
{context}

Question: {question}

Answer:""",
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    chain_type_kwargs={"prompt": prompt_template},
    return_source_documents=True,
)

Step 5: Query Your Data

result = qa_chain.invoke({"query": "What is the company's vacation policy?"})

print("Answer:", result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    print(f"  - {doc.metadata.get('source', 'unknown')} (page {doc.metadata.get('page', '?')})")

The LLM will answer based on the retrieved chunks, and you will see exactly which source documents contributed to the answer.

Step 6: Add a Simple Web Interface

For a quick demo, use Streamlit:

pip install streamlit

import streamlit as st

st.title("RAG Q&A")
query = st.text_input("Ask a question:")

if query:
    result = qa_chain.invoke({"query": query})
    st.write("**Answer:**", result["result"])

    with st.expander("Sources"):
        for doc in result["source_documents"]:
            st.write(doc.metadata.get("source", "unknown"))
            st.write(doc.page_content[:200] + "...")

Run it with:

streamlit run app.py

Production Tips

• Chunk size: 500-1500 tokens works well for most use cases. Smaller chunks are more precise; larger chunks provide more context.
• Retrieval count: Start with k=3 and experiment. Too many chunks dilute the answer; too few miss important context.
• Hybrid search: Combine vector search with keyword search (BM25) for better recall.
• Metadata filtering: Store category, date, or source metadata with each chunk so you can filter before retrieval.
• Evaluation: Use RAGAS or similar frameworks to measure faithfulness and answer relevance.

Conclusion

RAG transforms an LLM from a general-purpose chatbot into a knowledgeable assistant that understands your specific data. With LangChain and OpenAI, you can build a working RAG pipeline in under 100 lines of Python. The key steps are simple: load documents, split them into chunks, embed and store them, then retrieve and generate. Start with ChromaDB for local development, and upgrade to a managed vector database when you go to production.