Every February, the Himachal Pradesh Department of Economics & Statistics publishes an Economic Survey alongside the state budget. Fifteen editions are online, 2011-12 through 2025-26 — about 4,400 pages of English text and tables: GSDP, apple production, tourist arrivals, hydropower capacity, per-capita income.
I wanted a RAG project that wasn’t a folder of blog posts. This corpus has a property most don’t: the answers are checkable numbers. If retrieval returns the wrong thing, I can prove it. That’s the difference between an eval that means something and an eval that reports vibes.
Here’s the number that justifies the whole project:
Pure vector search retrieved from the right year 30% of the time.
Worse than keyword search. On a corpus where every question is “what was X in year Y”, the embedding-only baseline is close to useless — and I only know that because I measured it.
Why fifteen documents are harder than one
Fifteen editions of an annual publication contain fifteen near-identical “State Economy” chapters. The 2013-14 paragraph about GSDP growth and the 2019-20 one are around 0.95 cosine-similar, because embeddings encode topic, not year.
Ask “what was GSDP growth in 2019-20?” and top-k comes back as a near-random sample across editions. The model answers with whichever number it saw, and attaches a citation. The citation looks right. The number is from 2013-14.
That failure is worse than getting nothing back, because it survives review. Nobody reads a confident answer with a plausible source and thinks “I should check which edition that came from.”
The fix isn’t subtle — pull the year out of the question and use it as a hard metadata filter before ranking. What’s interesting is how much else has to be right before that filter can work at all.
The Hindi that isn’t Unicode
Here’s a line from the 2025-26 Hindi edition:
vkfFkZd losZ{k.k 2025&26Not corrupted. Not mojibake. It renders on screen as:
आर्थिक सर्वेक्षण 2025-26The Hindi editions are typeset in a legacy Kruti Dev-style transliteration font.
The bytes in the PDF are ASCII; the font maps them to Devanagari glyphs. The
text layer genuinely contains v, k, f, F, i, Z, d.
This breaks language detection in both directions at once:
- Count Devanagari codepoints and you get 0.0 for every Hindi file. All fifteen get classified as English and poured into the English index.
- Check against an English dictionary and it scores as garbage — so my quality gate would have routed roughly 4,000 pages to a vision model. At the rates I was paying, that’s real money to transcribe documents I’d decided not to index.
The detector that works keys on the encoding’s own signature: the digraphs Kruti Dev produces for common Hindi words, plus a low English dictionary hit rate. Ugly, corpus-specific, correct.
The upside: this is recoverable. Kruti Dev → Unicode is a deterministic character mapping. Indexing the Hindi half is a tractable next step, not a dead end — which I would not have guessed from looking at the garbage.
Two editions are one file in two languages
I nearly shipped an index missing two of fifteen years.
The 2021-22 and 2022-23 editions have no separate English PDF. Each is a single combined volume:
| edition | pages | layout |
|---|---|---|
| 2021-22 | 649 | Hindi 6–322, then English 333–649 |
| 2022-23 | 927 | English 3–441, then Hindi 447–927 |
The order flips between them.
If language is a property of the document, both files get one tag, and whichever way it lands you lose a year. Language had to become a property of the page, with contiguous runs rolled up into an English range.
What made this scary is how it would have failed. Not with an error. The manifest would have looked clean, the index would have built, and 2021-22 would simply have had no answers. I’d have blamed retrieval.
The finding I got wrong
Early on I grepped the site’s HTML for PDF links and found something great: the same statistical-tables file linked from seven different year pages, 2019-20 through 2025-26. One file, seven years.
That’s a fantastic trap. Attribute a document’s year from the page linking it and the system confidently cites 2019-20 figures as “2025-26”. I wrote it up as the highest-severity failure mode in the project and designed a four-signal cascade around it.
Then I wrote the crawler with an actual HTML parser, and it found zero.
The links are inside <!-- --> comments. The maintainers had disabled them. My
grep matched raw bytes and didn’t know what a comment was.
The lesson generalises further than the bug does: the tool you audit with has
a model of the data, and if that model is wrong your finding is wrong in a way
that looks like a discovery. grep thinks HTML is lines of text. It was never
going to tell me about comments.
The cascade survived anyway — it just earned its place differently.
Statistical tables are titled by the wrong year
Statistical_Tables2017_18.pdf opens on 2016-17 data.
Not a mistake — a convention. A tables volume is named for the edition it ships with, but titled with the latest fiscal year its tables cover, which is one behind. Take the title-page year as the edition and you misfile all eight tables volumes by a year.
So which signal is authoritative depends on document type. Narrative volumes put
the edition on the title page. Tables volumes don’t, and for those the filename
and linking page win, with the title demoted to a consistency check. The PDF
metadata creationDate does most of the real work: a file created in 2020
cannot be the 2025-26 edition, whatever any page says.
Tables are the whole point, and plain text destroys them
Every interesting question here is numeric, and the numbers live in tables.
Run plain text extraction over one and the column structure is gone:
Apple 1,84,552 2,01,113 1,76,432Which year is which? The header wrapped three lines up and the stub column pushed everything sideways. The reader model will pick one and state it confidently. Wrong numbers under correct-looking citations — the same failure as the year problem, arriving by a different route.
So nothing is accepted on faith. Every candidate table passes the same gate:
- Rectangularity — 90% of rows at the modal cell count.
- Prose rejection — sentence-like cells mean a narrative page got chopped into a grid.
- Header year monotonicity — years out of order means shuffled columns.
- Numeric column purity — a label in a numeric column means a shifted row.
- Row-sum reconciliation — where a “Total” row exists it should equal the column sum. When it does, that’s close to proof the columns were read right.
- Indian number parsing —
1,84,552is 2-2-3 grouping, not 3-3-3.
Three strategies race, first to validate wins: ruled-line detection, unruled text alignment, and a row-wise parser that reads regular tables straight out of the text layer. Across the corpus, 78% of indexed tables come out deterministically.
Four bugs the gate caught in my own code
The gate’s first job turned out to be catching me.
It passed prose. Page 33 of the 2024-25 edition is narrative text. The unruled-text strategy chopped it into a 41×10 grid and my gate accepted it. It was rectangular, after all.
It fabricated a table out of a bar chart. My row-wise parser found Figure 2.1 and paired the axis gridlines with the data labels:
250000 | 232185
150000 | 137320There is no table on that page. I’d taken an honest “needs vision” and turned it into a confident wrong answer. Numeric density cannot distinguish a chart from a table, because a chart’s axis labels are a regular numeric grid. The discriminator that works is the caption, plus a rule that a data row must have a non-empty label.
It split every fiscal year in half. My tokeniser read 1951-56 as 1951
and -56, shifting every value one column left across the entire corpus.
It rejected good tables for having list markers. My reversed-text check —
for pages where (FR) extracts as )RF( — fired on every a) Elementary Education in the corpus. Twenty-five good tables were being sent to a vision
model because I tested for a closing bracket anywhere before an opening one,
rather than within a single cell.
Letting a validation gate choose the model
The 20% of tables that defeat deterministic extraction go to a vision model. My first run used the most capable one available, on the reasoning that numbers matter. It cost $11.97 for 387 pages — and I’d estimated $5.90, because I guessed the output token count and was off by a factor of 2.7.
That’s an unsatisfying way to spend money. So I ran the same pages through cheaper models and compared:
| model | passes the gate | agrees with baseline numbers | cost |
|---|---|---|---|
| most capable | 60% | 100% | $11.97 |
| mid-tier | 56% | 99% | $3.75 |
| cheapest | 60% | 94% | $1.58 |
Mid-tier reproduces the expensive model’s figures essentially exactly at a third of the price. That’s now the default — and when a spend cap left 33 pages unfinished, running them on the cheaper default cost 41 cents instead of a dollar.
I didn’t take the cheapest despite it passing the gate just as often. 94% numeric agreement means one figure in seventeen differs, and the gate catches structural damage — a shifted column, a ragged row — not a misread digit. On a corpus whose entire point is correct numbers, that’s the wrong trade.
The thing that made this experiment safe to run at all is the gate. A cheap model’s mistakes get caught rather than silently accepted, so trying one costs a dollar and an afternoon instead of quietly corrupting the index.
One detail nearly cost me the finding. The cheapest model initially scored
0% — every call failed. Not a capability limit: it doesn’t accept the
effort parameter I was setting, and returned a 400. Had I not chased that,
“the expensive model is the only one that works” would have looked like a
result instead of a bug in my request builder.
Of 414 transcriptions, 168 passed the gate and the rest were rejected — mostly ragged rows where the model dropped or added a cell. I don’t pad those to a modal width. Padding is guessing which column is missing, which is exactly the silent misalignment the gate exists to prevent. The model is a third extraction strategy, not an oracle.
Why I didn’t use a vector database
4,390 chunks. At 512 dimensions that’s 9 MB as a float32 matrix. Scoring the entire corpus is one numpy matmul — single-digit milliseconds, faster than a network round trip to a hosted index.
Lexical search is SQLite’s built-in FTS5, which ships bm25() in the Python
standard library. And BM25 matters here more than it usually does, because these
queries are full of exact tokens embeddings blur: GSDP, 2019-20, Table 4.2, Lahaul-Spiti.
No vector database, no search engine, no orchestration framework. SQLite, a numpy array, and about 400 lines of retrieval code.
I reached the same conclusion building my portfolio’s MCP server — the SDK shim would have been more code than the protocol. It’s becoming a pattern I trust: check the actual size of the problem before adopting infrastructure sized for a different one.
The eval
113 questions across four groups, reported separately and never blended — the auto-generated ones are far easier than the adversarial ones, so a single number would flatter the system by construction.
| config | recall@10 | year-precision | citation | latency |
|---|---|---|---|---|
| dense only, no filter | 0.087 | 0.298 | 0.650 | ~700 ms |
| lexical only, no filter | 0.168 | 0.503 | 0.835 | ~130 ms |
| hybrid, no filter | 0.210 | 0.435 | 0.883 | ~850 ms |
| + year filter | 0.285 | 0.840 | 0.835 | ~840 ms |
| + comparative fan-out | 0.353 | 0.840 | 0.854 | ~980 ms |
Year-precision — of the top-k results, how many actually come from a year the question asked about — is the metric that captures this corpus. Generic RAG evals don’t have it.
Read the first row again. Pure vector search: 0.298. Worse than BM25 alone, at five times the latency. Everything people reach for first — embed the corpus, cosine similarity, top-k — is the weakest thing on this table. The year filter is what fixes it, nearly tripling year-precision to 0.840.
Every retrieval number above reproduces with no API key and no cost. The lexical ladder is complete rather than a single baseline row, because reproducibility that stops at the baseline proves nothing.
Three bugs the eval found in my retrieval
The per-edition cap was firing when it shouldn’t. I cap how many chunks come from one edition so a comparison doesn’t quietly answer for one year. Adding it made recall drop. The cap was binding on single-year questions too — where every candidate is already from that one edition, so it constrains nothing and just perturbs the ranking.
Fan-out was punishing the best possible answer. For a comparison I run one retrieval per year and give each a quota. But every comparison question in my set turns out to be answerable by a single chunk — a table with years as columns. That chunk was competing for one year’s slot instead of ranking on merit. Recall on comparisons went from 0.500 to 0.750 once spanning chunks could win outright.
The dense-only row was cheating. It came back identical to the hybrid row —
same recall, same year-precision, same citation rate. That’s not a coincidence,
it’s a symptom: my config had a lexical flag that was defined but never passed
through to the retriever, so “dense only” was quietly running BM25 alongside the
vectors and being flattered by it. Fixing it dropped that row from 0.435
year-precision to 0.298.
That last one is the one I’d want a reviewer to notice. A mislabelled row in a published eval is worse than a missing one, and the only reason I caught it is that two rows agreeing perfectly is suspicious in a way that a plausible-looking number never is.
What’s still broken
The Hindi half is indexed but not queryable. Two-level table headers get rejected rather than parsed, which is correct (reading them flat would misattribute every value) but isn’t the same as handling them. Chapter titles never get extracted, only numbers — the headings sit on image-only divider pages, and I made the detector strict on purpose, because a wrong chapter title gets embedded into the chunk and actively misleads retrieval.
What I’d tell myself at the start
Spend the first half-day hand-processing three pages before writing any pipeline. I did, and it immediately showed me my validation gate accepting prose and inventing tables out of charts — on day one instead of day four.
Write the eval earlier than feels justified. Mine found three design bugs in code I’d already convinced myself was right.
And measure the thing you’re about to assume. I assumed vector search would be the strong baseline and lexical the fallback; the numbers say the opposite. I assumed the expensive model was worth it; a dollar of experiment said a third of the price buys the same figures. Both assumptions were reasonable. Both were wrong.
Code, the full ablation matrix, and the 113-question golden set are public. A
--no-llm mode reproduces every retrieval number without an API key. Source
documents belong to the Department of Economics & Statistics, Government of
Himachal Pradesh; the project ships the pipeline, not the PDFs.