KAI Offline RecSys Lab: Retrieval-to-Reranking Evaluation

A personal public-data lab covering candidate retrieval, reranking, calibration, and cohort evaluation. Its aim is to compare models under explicit shared protocols while distinguishing recommendation quality, ANN systems measurements, and business outcomes.

Ranking metrics may not be comparable when candidate sets, populations, negative sampling, or feature sources differ. Frozen retrieval snapshots, dev-only selection, and a defined test-opening sequence reduce the risk of confusing protocol changes with model gains.

Amazon V3 starts from an existing Two-Tower checkpoint and builds Exact / HNSW Top-100 candidates. It compares hard, uniform, and in-batch negatives with DIN / DCN-style reranking. Temperature calibration uses training data only; dev NDCG@100 selects the candidate before paired user-level differences are evaluated on the frozen test set.

Public data and provenance ledger → shared splits and features → retrieval snapshot → reranker training and dev selection → frozen manifest → offline test and bootstrap → interactive report. The million-item HNSW extension is a separate local systems-measurement path.

The main quality experiment uses Amazon Reviews’23 Industrial_and_Scientific with a fixed 5-core protocol. Public reports describe a 25,754-item catalog and 50,653 common test users. Raw third-party records and trained weights are not distributed with the repository.

Compare retrieval, negatives, reranking, calibration, feature dependencies, and cohort support, alongside HNSW recall–latency–index-size diagnostics. Selection uses the development set; configurations, split summaries, and frozen results are recorded together.

In the public V3 summary, dev-selected exact-din-uniform-64-32 achieves higher test NDCG@100 than exact retrieval ordering under the same protocol, with a reported paired bootstrap interval excluding zero. The result applies to that offline population and protocol, not online CTR, CVR, revenue, or service SLAs.

Zero metadata, ID, title, and category inputs to inspect the frozen model's feature dependencies. This is inference-time input zeroing, not a fully retrained ablation. Hard negatives do not win this round of dev selection.

The frozen protocol does not support performance claims for true new users, users without history, or unseen target items; unsupported cohorts are reported explicitly. Aggregate checks cannot replace independent retraining with the unpublished raw data and artifacts. The million-item index experiment is not a million-item relevance benchmark.

The public Playground reads committed reports and presents V3 comparisons, feature dependencies, HNSW trade-offs, and cohort support as static experiment replay. make reproduce-small checks the end-to-end code path on explicitly synthetic data; it does not validate reported public-data performance.