Enterprise RAG pipelines have a recall stage and a precision stage. The retriever handles recall. The reranker handles precision. Skipping the reranker, or misplacing security controls around it, is where most accuracy and data exposure problems begin. The retriever’s job is to pull back every document that might be relevant. The reranker’s job is to find, from that candidate set, the documents that actually answer the question. These are different tasks, and treating the retriever as sufficient for both is one of the most common reasons enterprise RAG systems return plausible-looking but wrong answers.
What the Retriever Is Actually Doing
In a standard RAG pipeline, the retriever works against a pre-built vector index. When a user submits a query, the system embeds that query into a vector and computes similarity scores against stored document vectors using cosine distance. This approach (called a bi-encoder) is fast because the document vectors are computed once at ingestion and stored. At query time, only the query needs to be embedded. The comparison is a single mathematical operation across the index.
But speed comes with a trade-off. The query and each document are embedded separately, then compared. The embedding model never sees both together. It compresses the meaning of a document into a fixed-length vector before knowing what any user will ask. Information gets lost in that compression.
Cosine similarity works well when you need conceptual closeness. A document about “customer attrition” surfaces for a query about “churn” even though the words differ. That semantic matching is exactly the value of vector retrieval. Where it breaks down is precision. The top-ranked document by cosine score is the one whose vector sits closest to the query vector, not necessarily the one that best answers the specific question. In enterprise contexts where documents share vocabulary but contain different answers, this distinction matters.
What a Reranker Does Differently
Rerankers use different architectures. The most widely used is the cross-encoder, though LLM rerankers and late-interaction models (such as ColBERT) are also used in production systems. What they share is the ability to evaluate a query and a candidate document together rather than as separate vectors.
In a cross-encoder, the query and document are run through the model simultaneously. The model attends to the relationship between query tokens and document tokens, producing a score that reflects actual relevance rather than vector proximity. This is more accurate because the model can see how specific phrases in the document relate to specific terms in the query. It can tell the difference between a document that mentions the right topic and a document that actually answers the right question. The trade-off is compute. Rerankers cannot be applied to a million-document corpus at query time. That is why they operate on a candidate set: the retriever narrows the search space, then the reranker re-scores within it.
The two-stage pipeline exists because each stage solves a different problem:
| Stage | Method | Optimized For | Applied To |
| Retriever | Bi-encoder + cosine similarity | Speed + recall | Full corpus (millions of docs) |
| Reranker | Cross-encoder, LLM reranker, or late-interaction model | Precision + relevance | Candidate set (top 50–100 docs) |
Neither stage alone is sufficient. A retriever without a reranker returns candidates, not answers. A reranker without a retriever has no practical way to operate at enterprise scale.
Where the Security Problem Enters
Adding a reranker improves answer quality. It also creates a security risk that most teams do not anticipate.
A reranker scores relevance. It does not evaluate access. If a user with limited permissions submits a query, the retriever may surface candidates that include restricted documents. The reranker then re-scores those candidates and pushes the most relevant ones to the top, regardless of who was allowed to see them. An unauthorized document does not fail to surface because it is unauthorized. It surfaces at the top because it is relevant. This is the same over-retrieval problem that RAG pipelines create at the retrieval layer, but the reranker can amplify it. Reranking is explicitly designed to surface the highest-signal document for a given query. If that high-signal document contains sensitive data the user should not see, the reranker promotes it.
Access control cannot be applied after reranking. By the time the reranker has finished scoring, the documents are already ranked for insertion into the context window. Access enforcement at the output layer (filtering responses) leaves sensitive content inside the context window even if it does not appear in the final answer. The correct architecture places access control between the retriever and the reranker:
CBAC enforces access at inference time, not at ingestion, where static permissions fail to account for the identity and purpose of each request. The reranker then operates on a candidate set that has already been filtered, so relevance ranking cannot promote unauthorized content.
How Protecto Secure RAG Handles Both Stages
Secure RAG is built for the full two-stage pipeline, with security controls positioned where they actually prevent exposure. Before embedding, Secure RAG scans incoming documents for PII, PHI, PCI, and context-sensitive identifiers, catching sensitive entities that pattern-matching approaches miss. Privacy Vault then masks sensitive values before chunks reach the vector store, so the reranker scores masked text, not raw personal data.
Between retrieval and reranking, CBAC (Privacy Vault’s context-based access control capability) evaluates identity, purpose, and context per request to determine which candidates from the retriever are authorized for this specific user or agent before the reranker scores them. Every retrieval event is logged: what was retrieved, what was filtered, what reached the context window. This gives compliance teams the audit trail that most RAG systems cannot produce.
FAQ
What is the difference between a retriever and a reranker in RAG?
A retriever uses a bi-encoder to find semantically relevant documents quickly across a large index using cosine similarity, optimizing for speed and recall. A reranker then re-scores the candidate set by analyzing each document’s relevance to the specific question being asked, optimizing for precision. Common reranker architectures include cross-encoders, LLM rerankers, and late-interaction models such as ColBERT. Retrievers narrow the search space; rerankers find the best answer within it.
Why can’t a reranker replace the retriever entirely?
Rerankers score each query-document pair by analyzing them together, which provides better accuracy but is computationally expensive. No reranker architecture can run against millions of documents at query time within an acceptable latency budget. The retriever handles scale; the reranker handles precision. The two-stage design exists because each stage solves what the other cannot.
Where should access control be applied in a two-stage RAG pipeline?
Access control must be applied between the retriever and the reranker: after the retriever narrows the candidate set, and before the reranker scores it. Applying access control after reranking is too late because sensitive documents have already been scored and ranked for the context window. CBAC enforces per-request access decisions at this point, so the reranker only scores what the user is authorized to receive.
Does masking data before embedding affect reranker performance?
Context-preserving masking maintains the semantic structure of the text while replacing sensitive values with consistent masked representations. Because the same masked value maps to the same original value consistently across the corpus, the reranker can still score document relevance accurately. Privacy Vault is designed to preserve AI usability through the full retrieval pipeline (retriever, reranker, and LLM context) without degrading answer quality.
