Your AI Can Now Read Your Entire Codebase. That Doesn't Mean It Should.

OpenAI just launched the GPT-5 Pro API with a 400,000-token context window. That's roughly 300,000 words. An entire codebase. A full quarter's worth of Slack messages. Every support ticket from the last year.

The obvious move is to stuff it all in and let the model figure it out. Teams are already doing this. Dumping entire repositories into a single prompt. Pasting full database schemas alongside natural language questions. Feeding months of customer conversations into one completion call.

It works. Until it doesn't.

More context creates a false sense of comprehension

A model that accepts 400K tokens isn't a model that understands 400K tokens equally. Attention degrades over long sequences. Information in the middle of a massive prompt gets less weight than information at the beginning or end. Researchers at Stanford and Berkeley documented this phenomenon back in 2023 and called it "lost in the middle." Larger context windows reduce the problem. They don't eliminate it.

Here's what that looks like in practice. You paste 200 files into a prompt and ask the model to find a bug. It gives you a confident answer. The answer references real code from your repo. It looks right. But it missed the file where the actual bug lives because that file landed in a low-attention zone between token 150,000 and token 200,000.

You trust the answer because it's detailed and specific. That's the dangerous part. The model didn't say "I don't know." It gave you a plausible wrong answer built from real context.

The cost problem nobody's talking about

Large context windows are expensive. Every token you send gets processed. At GPT-5 Pro pricing, a single 400K-token prompt costs roughly 40x what a well-scoped 10K-token prompt costs. Do that a few hundred times a day across a team and you're looking at a serious bill for what amounts to lazy retrieval.

One team we spoke with was spending $15,000 a month on API calls because they were sending their entire monorepo to the model for every code review. They switched to a retrieval layer that pulled only the relevant files and their direct dependencies. Same quality answers. $1,200 a month.

That's not an edge case. That's the default outcome when you treat context size as a substitute for architecture.

When large context actually helps

Large context windows aren't useless. They're a tool. The question is whether they're the right tool for your specific problem.

Long-document analysis. If you need to summarize a 200-page legal contract or extract clauses across an entire filing, a large context window is exactly what you want. The document is the unit of work. Splitting it would lose important cross-references.

One-shot codebase questions. Quick, exploratory queries where you need a rough answer fast and precision isn't critical. "Where does this app handle authentication?" is a fine use case for dumping a repo into context. You're looking for direction, not a production fix.

Multi-turn conversations with long history. If your application needs to reference hours of prior conversation without losing coherence, a larger window helps maintain continuity that summarization would destroy.

When retrieval wins

For everything else, a retrieval layer outperforms raw context size. Here's the pattern that works.

Index your data once. Query it many times. Embed your codebase, documents, or knowledge base into a vector store. When a question comes in, retrieve the 5 to 10 most relevant chunks. Send those to the model with the question. The model gets focused, high-relevance context instead of everything-and-the-kitchen-sink.

Hybrid retrieval catches what embeddings miss. Vector search finds semantically similar content. Keyword search finds exact matches. Use both. A function name like processInvoiceBatch might not be semantically close to a question about "billing errors," but keyword search will find it instantly.

Metadata filtering narrows the search space. Tag your chunks with file paths, dates, authors, or component names. When someone asks about the payments module, filter to payments-related files before running similarity search. You'll get better results and lower costs.

Reranking sorts the final candidates. After retrieval, run a lightweight reranker to sort your chunks by actual relevance to the query. This catches cases where the initial retrieval grabbed something topically related but not actually useful.

The architecture that scales

The teams getting the most out of AI-assisted development aren't the ones with the biggest context windows. They're the ones with the best retrieval pipelines.

A 400K context window is a safety net. It's there for the cases where you genuinely need to process something large in one pass. Building your entire AI workflow around maxing out that window is like building your entire database strategy around SELECT *. It works. It's also the most expensive, least reliable way to get an answer.

Build the retrieval layer first. Use the context window for what it's good at. Your answers will be better and your API bill will thank you.