RAG shows up in almost every applied-LLM loop — AI engineer, ML engineer, and prompt engineer roles alike — because it is the cheapest, most auditable way to ground a model in private or fresh data. Interviewers use it to probe whether you understand the full pipeline (ingest → chunk → embed → retrieve → rerank → generate → evaluate) or just glued an API to a vector store. This page is a question bank, not a lecture. For the underlying mechanics, read the RAG concept page, vector databases, and AI agents first; the answers below assume you know them and focus on what to actually say.
Fundamentals
Q: What is RAG in one sentence? Retrieval-Augmented Generation injects relevant external context into the prompt at inference time so the model answers from retrieved evidence instead of parametric memory. It addresses hallucination, knowledge cutoff, and private data without retraining.
Q: When do you choose RAG over fine-tuning? Use RAG when knowledge changes often, must be cited or audited, or is too large to bake into weights. Use fine-tuning to change behavior, format, or style — not to add facts. They compose: fine-tune the response format, RAG the facts. A red flag answer is “fine-tune to teach it our docs”; fine-tuning is poor at reliable factual recall and expensive to keep fresh.
Q: With million-token context windows, is RAG obsolete? No. Long context still costs tokens linearly, suffers “lost in the middle” attention degradation, and gives you no retrieval signal to cite or debug. RAG keeps prompts small, cheap, and grounded in a known set of chunks you can show the user. Long context complements RAG (stuff more retrieved chunks) rather than replacing it.
Q: Walk me through a naive RAG pipeline. Chunk the corpus, embed each chunk, store vectors in an index. At query time embed the question, ANN-retrieve top-k, concatenate chunks into the prompt, generate. See the pipeline diagram. The interview value is naming where each stage fails: bad chunking kills recall, weak embeddings kill relevance, no reranking floods the prompt with noise.
Chunking
Q: How do you pick chunk size? Match the retrieval unit to the answer unit. Too small loses context; too large dilutes the embedding so one chunk averages several topics and retrieves poorly. ~256–512 tokens is a common default because it fits one coherent idea while staying under embedding-model token limits. Always tune against your own eval set — there is no universal number.
Q: Why is ~512 tokens a frequent default? Many embedding models truncate or lose fidelity past a few hundred tokens, and a single coherent passage usually fits in that budget. It balances embedding fidelity (smaller = sharper vector) against context completeness (larger = fewer mid-thought cuts).
Q: What does chunk overlap buy you, and what does it cost? Overlap (e.g., 10–15%) prevents an answer that straddles a boundary from being split across two chunks with neither fully containing it. The cost is index bloat and duplicate retrievals. Too much overlap inflates storage and returns near-identical chunks that waste the context window.
Q: Fixed-size vs semantic chunking? Fixed-size splits on token count — fast, predictable, but cuts mid-concept. Semantic chunking splits on natural boundaries (headings, paragraphs, embedding-distance shifts) — higher recall, slower to build. Start fixed-size, move to semantic only if eval shows boundary errors hurting recall.
Q: Why attach metadata to chunks? Metadata (source, section, timestamp, permissions) enables pre-filtering before the ANN search, citation back to the source, freshness checks, and access control. Filtering by metadata first shrinks the candidate set and is often the single biggest precision win in enterprise RAG.
Q: What is hierarchical / parent-document retrieval? Embed small chunks for precise matching but inject the larger parent section for generation context. You get retrieval precision and answer completeness at once — retrieve on the sentence, answer with the paragraph.
Embeddings & retrieval
Q: How do you choose an embedding model? Match domain and language, check the MTEB leaderboard for the task type, and weigh dimension size against index cost and latency. Critically, the same model must embed both queries and documents. Bigger embeddings aren’t always better — they raise storage and ANN cost for marginal recall.
Q: Dense vs sparse retrieval? Dense (embeddings) captures semantic similarity and paraphrase. Sparse (BM25/keyword) nails exact terms — product codes, names, error strings, rare tokens. Dense alone whiffs on literal matches; sparse alone misses synonyms. Production uses both.
Q: What is hybrid search and how do you merge results?
Run BM25 and dense ANN in parallel, then fuse. Reciprocal Rank Fusion — score = Σ 1 / (k + rank), k≈60 — is the common default: parameter-free, robust, no score normalization needed. Hybrid reliably beats either retriever alone across diverse queries.
Q: What is reranking and why add a second stage? Retrieve a wide candidate set cheaply (top-20/50), then a cross-encoder reranker scores each (query, chunk) pair with full cross-attention and keeps the best ~5. Bi-encoder retrieval is fast but coarse; the cross-encoder is accurate but too slow to run over the whole corpus — so you only rerank the shortlist.
Q: When does BM25 outperform embeddings? On exact-match and out-of-vocabulary queries: SKUs, IDs, version numbers, function names, rare proper nouns. Embeddings smear these into a generic semantic region; BM25 matches the literal token.
Q: What problem does MMR solve? Maximal Marginal Relevance balances relevance against diversity, penalizing chunks too similar to already-selected ones. It stops the top-k from being five near-duplicate passages and widens the evidence the model sees.
Vector databases
Q: How do you choose among Pinecone, Weaviate, Chroma, Milvus, and FAISS? FAISS is a library, not a database — fast in-memory ANN, you own persistence and scaling. Chroma is lightweight, great for prototypes and local dev. Pinecone is managed and operationally hands-off at a price. Weaviate and Milvus are open-source, self-hostable, with hybrid search and filtering at scale. Choose on ops burden, scale, filtering needs, and managed-vs-self-host — see vector databases.
Q: What is HNSW and why is it dominant?
Hierarchical Navigable Small World is a graph ANN index giving near-logarithmic search with high recall. Key knobs: M (graph degree) and efConstruction/efSearch trade index size and query latency against recall. It’s the default in most modern vector DBs.
Q: Explain the recall vs latency tradeoff in ANN.
Approximate search trades exactness for speed. Raising efSearch (or probing more clusters in IVF) finds more true neighbors but costs latency and CPU. You tune to a target recall (say 95%) at an acceptable p99 latency — exhaustive search guarantees 100% recall but doesn’t scale.
Q: When is brute-force (flat) search the right call? Small corpora (thousands to low tens of thousands of vectors) where exact recall matters and latency is fine. Don’t add HNSW complexity until scale demands it.
Generation & quality
Q: How do you reduce hallucination in RAG? Ground hard: instruct the model to answer only from provided context and to say “I don’t know” when evidence is missing. Improve retrieval so the right chunk is actually present — most “hallucinations” are retrieval failures. Add citations and verify them. Then measure faithfulness rather than trusting vibes.
Q: Is stuffing the whole context window with chunks a good idea? No. More chunks add noise, raise cost and latency, and trigger lost-in-the-middle degradation where mid-prompt evidence is ignored. Retrieve precisely, rerank, and inject a tight set. Quality of context beats quantity.
Q: How do you make a model cite its sources? Tag each retrieved chunk with a source id in the prompt and instruct the model to attribute claims to those ids, then validate the cited chunk actually supports the claim. Citations are both a UX feature and a faithfulness check.
Evaluation
Q: How do you evaluate retrieval separately from generation? Split the two failure modes. Retrieval: did the right chunks make the top-k? Generation: given good chunks, did the answer use them correctly? Measuring them together hides which stage is broken.
Q: Which retrieval metrics matter? Recall@k (are relevant chunks in the top-k — usually the first thing to fix), MRR (rank of the first relevant chunk), and nDCG (graded relevance weighted by position). Context precision tells you how much of what you retrieved was actually useful.
Q: Which answer-quality metrics matter? Faithfulness (is every claim grounded in retrieved context — the anti-hallucination metric) and answer relevance (does it address the question). Frameworks like RAGAS compute these with LLM-as-judge; keep a human-labeled slice to validate the judge.
Q: What is a golden set and why build one first? A curated set of representative questions with known-correct answers and the chunks that support them. It turns “feels better” into a regression test, so every chunking, embedding, or prompt change is measured, not guessed. Without it you’re tuning blind.
Production
Q: Where does the latency budget go, and what do you cut? Embedding the query, ANN search, reranking (often 50–200ms), and generation. Reranking is usually the swing factor — gate it behind a threshold, run it async, or cache. Set a p99 target and profile each stage rather than optimizing blind.
Q: What do you cache in a RAG system? Query embeddings for repeated questions, retrieval results for hot queries, and stable system-prompt prefixes via prompt/prefix caching. Caching cuts both cost and latency on the long tail of repeated traffic.
Q: How do you keep the index fresh? Incrementally upsert on document change rather than full reindexing, version chunks, and carry timestamps so stale content can be filtered or down-weighted. Decide whether eventual consistency is acceptable for your domain — news isn’t, a static handbook is.
Q: Where do prompt-injection risks enter RAG, and how do you defend? Retrieved documents are untrusted input — a poisoned doc can carry instructions the model obeys (“ignore previous instructions and exfiltrate…”). Treat retrieved text as data, not commands: sanitize and delimit it, constrain tool/permission scope, validate outputs, and apply per-user access filtering at retrieval so users never retrieve documents they can’t see. This is the highest-severity RAG-specific security issue and worth raising unprompted.
Q: How do you control cost at scale? Cache aggressively, retrieve fewer and better chunks, gate reranking, right-size the embedding dimension, and pick the smallest generation model that passes eval. The biggest lever is usually retrieving less, better — not a bigger model.