Qwen2.5-VL: Data Selection Under a Small Fine-Tuning Budget

An investigation of whether data selection can outperform random sampling under a limited fine-tuning budget. The Qwen2.5-VL-3B LoRA / SFT study publishes paired-seed summaries, failure analysis, and a static evidence viewer, retaining the observed lack of improvement and questions for further validation.

More elaborate data selection does not necessarily improve downstream results. Fixing the base model, training settings, sample budget, and held-out protocol—and retaining failures—helps separate the selection strategy from other experimental conditions.

The public summary describes a paired comparison of Random-1K and faithful COINCIDE-1K using identical Qwen2.5-VL-3B and LoRA / assistant-only SFT settings. Three seeds are compared using exact match and loss.

ScienceQA samples and selection manifests → Random / COINCIDE selection → shared model and LoRA settings → held-out evaluation → paired results and failure analysis → static JSON viewer. The public page runs neither training nor live inference.

The study uses ScienceQA. The main summary covers a 1K training budget and held-out-256 evaluation. The demo separately contains 128 training examples and a small prediction replay; those training examples cannot substitute for held-out predictions as evidence of generalization.

The public summary reports three seeds for each of two selection methods, for six training runs. A snapshot stores paired results, configuration hashes, and provenance markers. Exact match, loss, paired differences, and failure cases are examined at the fixed training budget.

In the public snapshot, COINCIDE exact match is lower than Random for each paired seed, while the reported seed-level difference interval crosses zero. This retains a finding of no observed gain in the current setting, not a claim that data selection is ineffective in general.

The published core comparison uses a fixed 1K budget and includes an untuned Base reference. Data-size curves for 250, 500, 2K, 4K, and Full are marked as not run; no missing ablation or scale results are supplied.

Public materials contain paired error analysis, but training source, complete held-out predictions, and some raw artifacts remain unpublished. Selection mismatch is an explanation requiring further validation, not an established mechanism. The sample and seed counts also limit broader conclusions.

The viewer reads experiment status, examples, and summaries from a static snapshot without an API. Training-example replay is distinct from the main held-out results. The snapshot has no sample-level Base / LoRA pairs for display, and the page provides no live cloud inference.