Why Your RAG Shouldn't Upload Your Documents to the Cloud

Retrieval-Augmented Generation — RAG — is the engine behind nearly every “chat with your documents” product. The idea is sound: retrieve the passages relevant to a question, then let a language model answer from those passages instead of from memory. The standard implementation, though, has a built-in assumption nobody questions: your documents are uploaded to someone else’s infrastructure to make it work.

That assumption is increasingly a problem. Privacy regulations, internal data-handling policies, and plain old caution are pushing teams toward a different model — private RAG, with no upload. The whole pipeline runs on the device. This article explains how that works, where the standard architecture leaks data, and when each approach is the right one.

The standard RAG data flow (and where your data leaks)

A typical cloud RAG setup looks like this:

1. Ingest. Documents are uploaded and stored in the vendor’s object storage.
2. Embed. A model on the vendor’s GPUs turns each chunk into a vector.
3. Index. Vectors are written to a managed vector database — another server, another copy.
4. Retrieve and generate. Each question is answered by a hosted LLM that reads the retrieved passages.

At every one of those steps, your content exists on infrastructure you do not control. Steps 1 and 3 are the obvious leaks — a stored copy of the document and a stored copy of its meaning. But step 4 matters too: even if you delete the document, the model provider may have logged the prompt and the retrieved context.

What “private RAG, no upload” actually means

An on-device RAG pipeline moves every stage onto the client. The embedding model, the vector index, and the generator all run in the browser via Transformers.js on WebGPU (with a WebAssembly fallback). Concretely:

• Chunking happens in page memory — the file is read locally, never transmitted.
• Embedding uses a compact model such as all-MiniLM-L6-v2, producing 384-dimensional vectors on your hardware.
• The vector index is just two arrays in memory (chunk texts and their vectors). No database, no server.
• Retrieval and generation run locally, so the question and the answer never leave the device.

The practical consequence is the one that compliance teams care about: because nothing is uploaded, there is nothing to retain or delete. There is no copy on a third-party server to expire, redact, or subpoena. The private RAG comparison page shows this side-by-side with the major cloud platforms.

Why it matters: GDPR, air-gaps, and the per-token bill

Three reasons teams move to no-upload RAG:

• Compliance. Under GDPR and similar regimes, sending personal or confidential data to a processor triggers transfer, retention, and deletion obligations. If the data never leaves the device, a large class of those obligations simply does not arise.
• Air-gapped and locked-down networks. Once the models are cached, on-device retrieval and generation make no external calls. That means it runs on networks where cloud egress is forbidden — exactly where cloud RAG providers cannot.
• No per-token cloud bill. Inference runs locally, so there is no metered API cost per question. For high-volume internal Q&A, that difference is significant.

Cloud RAG platforms, compared

The leading hosted options — OpenAI Assistants, Google Vertex AI RAG, Notion AI — are strong on scale and team features. They are also, by design, cloud tools: they require an account and they store your documents. An on-device tool trades some of that scale for zero-upload privacy and no per-token fee. The private RAG page compares processing location, retention, and cost across all of them.

The honest trade-offs

On-device RAG is not a universal replacement, and pretending otherwise would be dishonest. Cloud platforms win on:

• Scale. Millions of documents and billions of vectors are a managed service’s job, not a browser tab’s.
• Collaboration. Shared corpora, permissions, and audit logs are built into hosted products.
• Model choice. A cloud provider can swap in a frontier model; an on-device tool is limited by what runs in the browser.

The right architecture is therefore a decision, not a default: upload when the document is public or already in that ecosystem and you need scale; keep it on-device when the document is confidential, regulated, or simply yours.

Choosing between cloud and on-device RAG

A quick rule of thumb:

• Public docs, marketing copy, open research → cloud RAG is fine and often faster to stand up.
• Contracts, HR, medical, unreleased product, anything under NDA → on-device, no-upload RAG.
• Anything that must run on a locked-down or air-gapped network → on-device by necessity.

Start with an on-device RAG tool

You do not have to choose between “powerful” and “private.” The same on-device pipeline powers a ChatPDF alternative, a NotebookLM alternative, and a general private RAG tool— none of which upload a byte. Open the document RAG search and point it at a document you have already read; confirm the answers are cited to the right pages, then decide where the rest of your corpus belongs.