Claude Fable 5: Past, Present and Future Stories of Informational Retrieval at Scale 

I prod Anthropics latest "Mythos-level" model for hard truths about search in the AI age:

STEVE: I would like to understand the history of information retrieval starting with let's say Yahoo which was a directory and then Alta Vista which was TF/ IDF search and then Google which is pagerank based on inbound link scoring and then alongside that: custom retrieval with elasticsearch where you can have dense nested queries in json like syntax and finally... we have somehow landed on dense vector RAG for AI, which has been a train wreck. Many AI based systems have underperformed or failed based on over-dependence of fuzzy semantic search with embedding models..

So please sum up where we've been... and why IR retrieval is such a moving target … i.e., an unsolved problem. Also: where are we going from here in your opinion.. (And if you say GraphRag I will scream ;-)

12:20 PM

Claude Fable 5 responded: Good arc, and your history is basically right.

Let me fill in some connective tissue and then give you the opinionated part.

The eras, and what each one actually solved

Yahoo was the library model: relevance via human taxonomy. It died for the obvious reason — curation doesn't scale — but also for a subtler one: browsing assumes the user knows which category their intent lives in. They usually don't.

AltaVista (and Lycos, Excite) was the Cranfield/SMART tradition commercialized: inverted indexes, TF-IDF weighting, later BM25. The core theory here is solid — term frequency as evidence of aboutness, inverse document frequency as term informativeness, document length normalization. The fatal flaw wasn't the math, it was that text-only relevance is trivially gameable. Keyword stuffing worked because the document was the sole witness for its own relevance.

Google's insight was to add an exogenous, query-independent prior: PageRank treats the link graph as a citation network, so relevance evidence comes from third parties who are costlier to corrupt. Important nuance: PageRank alone was never the ranker — it was one strong feature combined with anchor text and IR-classic signals, and within a few years Google was really a learning-to-rank system with hundreds of features. The pattern to notice: every retrieval signal decays once it has economic value (Goodhart's law applied to relevance). Links got gamed too; hence the endless arms race.

Elasticsearch is a different branch entirely — it's BM25 plus structured query algebra, commoditized. Its real contribution was admitting that retrieval in applications isn't "find documents about X," it's "find documents about X, filtered by tenant, boosted by recency, faceted by category." Relevance as a composable, engineered function rather than a universal one.

Why RAG disappointed

The embedding-RAG pipeline made three quiet assumptions that are all false: that meaning survives compression into a single ~1000-dim vector (it doesn't — negation, numbers, identifiers, and rare terms get smeared away); that chunking preserves answerability (it destroys document structure and cross-references); and that cosine similarity ≈ relevance ≈ "contains the answer" (three different things). The embarrassing empirical fact the vector DB industry doesn't advertise: hybrid BM25 + dense + cross-encoder reranking beats pure dense retrieval almost everywhere, and on out-of-domain benchmarks like BEIR, plain BM25 embarrassed a lot of dense retrievers. Exact lexical match is high-precision evidence and dense-only systems threw it away.

Why IR is permanently unsolved

Because relevance isn't a property of documents. It's a relation between a query, an unobservable intent, a user's context, and a corpus — and we only see the query. This is the vocabulary mismatch problem identified in the 1980s (Furnas: two people choose the same term for the same concept <20% of the time), and it's never been solved, only attacked from different angles: query expansion, link evidence, click models, embeddings. Add that the corpus is adversarial (SEO), that intents drift, and that evaluation itself (Cranfield-style relevance judgments) is expensive and subjective, and you get a problem that's structurally a moving target, not just an unfinished one.

Where it's going (not GraphRAG, you can unclench)

The interesting shift is that retrieval is stopping being a function and becoming a behavior. The single-shot pipeline — embed query, top-k, stuff context — is being replaced by agentic retrieval: the model issues a query, reads results, notices the vocabulary mismatch itself, reformulates, greps, filters, and decides when it has enough. This is really just Rocchio-style relevance feedback from 1971, except the feedback loop now has a reader with judgment inside it. You see this in your own workflow: Claude Code does shockingly effective retrieval over codebases with zero embeddings — ripgrep plus iterative judgment beats a vector index of the repo.

So my actual prediction: embeddings get demoted from foundation to feature. The stack converges on boring lexical retrieval (BM25/SPLADE-style learned sparse), structured filters, cross-encoder reranking where precision matters, and an LLM in the loop doing query reformulation and sufficiency judgment. Long context erodes the need for surgical top-k. The hard residual problems become when to retrieve, when to stop, and how to evaluate any of it — which are policy and judgment problems, not similarity problems.

The through-line of the whole history: each era added a new, harder-to-fake source of evidence about intent — text, then links, then behavior, then semantics, and now iterative reasoning. The target moves because intent is invisible and the corpus fights back.

STEVE:

Very well said. And here’s a follow-on question which is a whole other can of worms in itself.

How do you know when graph ontology-based retrieval like RDF, OWL or Sparkle are appropriate? It seems like they succeed in certain vertical domains and they even scale, but for Enterprise content retrieval that's large, heterogeneous and messy with many different sub ontologies… graph triple-stores are going to fail. So how do you figure out where that line is between a place where RDF would work and the place where it will suffer a long protracted ugly decline?

12:35 PM

Claude Fable 5 responded: The line is real and you can locate it with a handful of tests, because the failure mode is always the same: the cost of curation grows with heterogeneity and …

... semantic churn, while the value of the graph grows only with query structure. Where those curves cross is your line.

The crisp version first: an ontology works when it describes an existing consensus and dies when it attempts to create one. Gene Ontology, SNOMED, FIBO, CAS registry numbers — these succeeded because a community of practice had already spent decades converging on what the entities are and what the relations mean. The ontologists were scribes, not legislators. Every failed enterprise ontology project inverts this: it tries to use the schema to impose semantic agreement across departments that don't actually agree, and the graph becomes a frozen org-chart of a political negotiation that's obsolete before the triple store finishes loading.

Unpacking that into the specific tests:

1. Identity is cheap. The dominant hidden cost of any KG isn't the triple store or SPARQL — it's entity resolution at the ingestion boundary. Verticals that succeed have natural canonical keys: gene IDs, ISINs, CAS numbers, part numbers, ICD codes. If "the same thing" can be recognized mechanically, curation scales. General enterprise content has no canonical identity — "the Q3 pricing proposal" is seventeen documents, four of them contradictory, and deciding which entities they mention is an interpretive act performed by an expensive human. When identity is interpretation, the graph is hand-built forever.

2. Relations are facts, not readings. inhibits(drug, protein) is a fact someone ran an assay on. supersedes(policyA, policyB) in a typical enterprise is an opinion that legal and ops would litigate. RDF triples carry an implicit claim of assertional truth; if your domain's relations are contestable, you're encoding one party's interpretation with the typographic confidence of a database.

3. Schema half-life exceeds curation cycle time. T-box stability is everything. Biomedical ontologies change, but on a versioned, governed cadence with funded curators (often regulatorily mandated — pharma pays for curation because the FDA makes them). If the domain's conceptual structure churns faster than governance can re-model it, you get the protracted ugly decline you described: the graph doesn't fail loudly, it just goes quietly stale, trust erodes, people route around it, and three years later it's a line item someone is afraid to delete.

4. The query workload actually needs joins. Graphs pay off on multi-hop, typed-relation, constraint-satisfaction queries: "compounds targeting proteins in pathway X with no reported interaction with Y." If 90% of your queries are "find me the document about Z," you've built a reasoning engine to do lookup, and BM25 was right there. Corollary: almost nobody uses OWL's actual reasoning power anyway — full OWL DL inference is computationally brutal, so real deployments run RDFS-plus-a-sprinkle, meaning you're often paying ontology prices for what is effectively a labeled property graph.

5. Curation incentives align. Who creates the metadata vs. who benefits? In Wikidata, volunteers self-select; in pharma, compliance pays; in your general enterprise, tagging is an unfunded tax on document authors who capture none of the retrieval value. Unfunded metadata mandates have a 100% historical failure rate — this killed enterprise taxonomies in the 2000s before it killed enterprise ontologies in the 2010s.

So the heterogeneous-enterprise case fails essentially every test simultaneously: fuzzy identity, interpretive relations, political schema, document-centric queries, misaligned incentives. It's not that RDF "doesn't scale" — Wikidata and Bio2RDF are enormous — it's that scale was never the variable. Heterogeneity and churn are the variables, and triple count is orthogonal to both.

The pattern that actually survives in messy enterprises is the unglamorous hybrid: a small, governed graph for the entities that genuinely have crisp identity and funded ownership (customers, products, contracts, org structure — basically MDM wearing a graph costume), used as a filter and join layer over document retrieval that stays lexical/ranked/messy. The graph holds the fifty thousand things the company truly agrees on; everything else stays text. The classic failure is letting the graph's ambitions leak past the boundary of consensus — every entity type you add beyond that line adds curation cost linearly and value asymptotically approaching zero.

Which suggests the practical heuristic when evaluating any proposal: don't ask "could we model this in OWL?" — you can model anything in OWL, that's the trap. Ask "who is the funded, empowered owner of this schema in five years, and what percentage of our actual query log requires a join?" If either answer is mumbling, you've found your line.