BM25 vs Sentence-Embedding Retrieval on the data.gov.in Catalogue: A Significance-Tested Query-Level Evaluation

OpenData-IR is an evaluation study that measures, with a defensible protocol, whether transformer-based sentence-embedding retrieval actually finds government datasets better than the classic BM25 lexical ranker when searching the metadata catalogue of data.gov.in — India's national open-government data platform.

data.gov.in publishes a very large catalogue of government datasets — over a lakh API-exposed resources by independent counts — each described by a title, description, sector tags and publishing organisation. Finding the right dataset is the first hurdle every student, journalist and analyst faces, and the portal's search is keyword-based: ask for "rainfall" and a dataset described as "actual precipitation" can rank below weaker matches. Dense retrieval promises to close that vocabulary gap, but adopting heavier infrastructure without measured evidence is engineering folklore.

The study builds a clean corpus of catalogue metadata through the portal's official resource APIs, designs a sampled query set of realistic information needs, collects graded relevance judgments (0/1/2) under a pooling protocol fixed in advance, and scores both systems with nDCG@10, MAP and MRR. Because single-number comparisons mislead, every pairwise claim is tested at query level with the Wilcoxon signed-rank test. Index build time, memory footprint and query latency of both index families are measured on the same laptop, so the accuracy-versus-cost trade-off is stated numerically, not rhetorically.

The deliverable is a paper-style study — corpus snapshot, judgment set, evaluation harness, results tables with confidence statements and an error analysis — executable end to end on a personal machine.

India's open-data catalogue holds government datasets across agriculture, health, education, finance and utilities, but discovery still depends on keyword matching between a user's words and a record's words. Department authors use official terminology — scheme names, acronyms, technical terms — while citizens and analysts phrase needs in everyday language, so relevant records are missed ("crop insurance" against the actual scheme name) and near-duplicates crowd out the right one. Whether sentence-embedding retrieval fixes this on real catalogue metadata is asserted far more often than it is measured, and no graded-relevance benchmark for data.gov.in is public. An organisation upgrading its search must therefore either trust results borrowed from foreign web collections or run an unstructured bake-off. This study produces the missing evidence: a documented corpus, a pre-declared query and judgment protocol, head-to-head nDCG/MAP/MRR results with statistical significance, and measured latency and memory costs — turning the retrieval choice for government catalogue search into a defensible engineering decision rather than an article of faith.

Abstract

This dissertation evaluates lexical (BM25) and sentence-embedding (dense) retrieval over the metadata catalogue of data.gov.in, India's national open-government data platform. A corpus of dataset records (title, description, sector, organisation) is assembled through the portal's official resource APIs; a sampled query set of 30–40 realistic information needs is judged for graded relevance under a pre-declared pooling protocol; and both systems are compared on nDCG@10, MAP and MRR with query-level Wilcoxon signed-rank tests. The study also measures index build time, index size and query latency, and closes with an error analysis locating where dense retrieval earns its cost.

Introduction

Open data creates value only when it can be found. A citizen looking for crop-insurance statistics, a journalist verifying procurement numbers and a researcher building a district panel all start from the catalogue's search box. Metadata records are short, jargon-heavy and written by dozens of departments with inconsistent vocabulary, so keyword overlap is a fragile basis for ranking. The question is empirical: on this catalogue, does embedding-based ranking measurably improve retrieval quality over a well-tuned classical baseline?

Existing system and literature gap

The portal's discovery layer is keyword-led. The information-retrieval literature — from Cranfield through TREC practice to modern neural-ranking work — establishes BM25 as a strong baseline and dense retrieval as strong on large web collections. But published comparisons rarely cover Indian government catalogues: no public graded-relevance benchmark exists for data.gov.in, and web-collection findings do not automatically transfer to short, department-authored records where relevance often hinges on one scheme name or acronym. Adoption arguments for embeddings are asserted far more often than they are measured on this domain.

Proposed work

• Build a provenance-logged corpus via the official resource APIs (paginated harvest, cleaning, deduplication, snapshot date).
• Sample and freeze a query set from four sources: sector names, published scheme names, analyst tasks and everyday paraphrases.
• Collect graded judgments (0/1/2) by pooling the top 10 of each system plus the portal's own first page; double-judge 20% and report inter-assessor agreement.
• Implement BM25 (k1/b tuned only on a development slice of queries) and a transformer-based sentence-embedding retriever (cosine similarity) behind one harness; an optional reciprocal-rank-fusion hybrid is a stretch goal.
• Score nDCG@10, MAP and MRR; test every pairwise difference with the Wilcoxon signed-rank test.
• Profile index build time, index size and p50/p95 latency on identical hardware, and classify per-query failures into an error taxonomy.

Feasibility

• Technical: tens of thousands of short records is a small corpus by IR standards; BM25 via rank-bm25 and embeddings via sentence-transformers run on a 16 GB laptop, with FAISS or NumPy handling vectors.
• Data: the resource APIs are free with a registered key; every request is logged so the corpus is auditable.
• Economic: no licences or paid services; the main cost is judgment effort, bounded by the pooling protocol.
• Operational (research): the harness is fully scripted, so every number regenerates from the pinned corpus snapshot and the frozen judgment sheet.

In scope

• Catalogue metadata records (title + description + sector/organisation fields) as the document unit; English-language metadata.
• Two primary systems — BM25 and one sentence-embedding family — plus an optional reciprocal-rank-fusion hybrid; one pinned evaluation snapshot of the catalogue.
• 30–40 judged queries; 3-point graded judgments via pooling; nDCG@10, MAP and MRR; Wilcoxon signed-rank tests; CPU latency and memory measurements on one machine.

Out of scope

• Full-text search inside the datasets' files (CSV/PDF contents); only metadata is indexed.
• Multilingual retrieval (Hindi, Telugu, Tamil query variants) — listed as future work.
• Learning-to-rank models, query autocompletion, click modelling and user-interface redesign.
• A production deployment; the study stops at measured evidence and a deployment recommendation.
• User studies with real portal visitors; relevance comes from the assessor protocol, not live traffic.

The study uses a quantitative, comparative-experimental research design: the protocol is fixed before scoring, all comparisons are paired at query level, and every artefact (corpus snapshot, judgment sheet, run configuration) is versioned. The evaluation protocol — query sources, pooling depth, grade scale, metrics, statistical test — was written down before the first system ran, following the research-methodology discipline of RM4151, so results cannot be reshaped by re-deciding the protocol after seeing them.

Relevance protocol (declared before scoring). Queries are sampled from four sources — sector names, real scheme names published on the portal, analyst-style tasks and everyday paraphrases — then frozen. For each query, pooling merges the top 10 results from BM25, the top 10 from the embedding retriever and the portal's own first page; each pooled record is graded 0/1/2 against its full metadata page. A second assessor re-judges a 20% stratified sample; Cohen's kappa is reported, and disagreements are resolved by re-reading the record together before any system is scored.

Leakage-free indexing. Indexes are built from the corpus snapshot only — no query text and no judgment information ever enters an index. Embedding-model choice and BM25 tuning touch only the development slice of queries; the judged set is untouched until the final run, and the report states this explicitly.

OpenData-IR is a linear, scripted pipeline with strict artefact boundaries, so that corpus, judgments, runs and statistics cannot contaminate one another.

flowchart TD
  A["data.gov.in resource APIs<br/>official catalogue endpoints"] --> B["Harvest and clean<br/>dedupe, snapshot date, provenance log"]
  B --> C[("Corpus snapshot<br/>JSONL dataset records")]
  C --> D["BM25 indexer<br/>inverted index, k1/b from dev slice"]
  C --> E["Dense indexer<br/>sentence-transformers, cosine, FAISS/NumPy"]
  Q["Frozen query set<br/>30-40 information needs"] --> F["Retrieval harness<br/>one interface, both systems"]
  D --> F
  E --> F
  F --> G["Pooling depth 10<br/>BM25 top-10 + dense top-10 + portal order"]
  G --> H["Graded judgments 0/1/2<br/>20 percent double-judged, kappa reported"]
  H --> I[("Judgment sheet<br/>frozen before test runs")]
  F --> J["Ranked run tables<br/>query id, document id, rank"]
  I --> K["Metrics: nDCG@10, MAP, MRR"]
  J --> K
  K --> L["Wilcoxon signed-rank<br/>paired per-query tests"]
  K --> M["Cost profiler<br/>build time, index size, p50/p95 latency"]
  L --> N["Results tables + error taxonomy"]
  M --> N

Three boundaries carry the study's credibility. First, the corpus snapshot is an immutable artefact with a recorded harvest date; both indexers read only it, which makes the comparison fair and repeatable even as the live catalogue grows. Second, the judgment sheet is produced by the pooling step and frozen before the final scoring run — systems are never modified after judgments exist, which is what separates an evaluation from a leaderboard climb. Third, the cost profiler runs on identical hardware for both index families and reports build time, on-disk size and p50/p95 latency, so every accuracy claim in the results chapter is paired with the price of achieving it.

The harness stores each run as a tidy table (query id, document id, rank, score); metrics are computed only by joining run tables to the frozen judgment sheet, so any number in the dissertation can be recomputed with a single command from two artefacts.

• Multilingual retrieval: Hindi and regional-language query variants against English metadata, evaluated with the same protocol.
• Learning-to-rank on the released judgment set, and a properly powered hybrid-fusion study across several embedding families.
• Scaling the corpus to the full catalogue and re-running annually to track drift.
• Releasing the graded query set as a public benchmark for Indian open-government catalogue search.
• Cross-portal transfer: state-level open-data catalogues evaluated with the identical protocol to test whether conclusions hold beyond the national portal.