MiniClaudeCode: Ordered Trajectory Credit & Agent Evaluation

This study uses MiniClaudeCode tool traces to examine credit assignment when actions have prerequisites. The project also provides a bounded agent loop, tool policies, checkpoints, and evaluation infrastructure; the research focus here is offline ordered attribution.

Vanilla Shapley considers coalitions such as edit without read, even when they violate action dependencies. Preserving order within a subset does not make those counterfactuals executable. The question is how to define and estimate marginal credit under explicit precedence constraints.

Represent dependencies as a DAG and average marginal contributions only over valid linear extensions. Enumerate small sets exactly; for larger sets, use dynamic programming to count completions and sample each next action proportionally, producing uniform samples of valid orders.

A bounded Controller connects the LLM Driver, tool registry, policy, and runtime to recorded ToolObservations. The attribution module separately consumes a trajectory, precedence graph, and replayable utility function, returning credit vectors, standard errors, and efficiency residuals.

The controlled study generates eight variants from each of five structures: chain interactions, failure recovery, redundant evidence, parallel evidence, and policy guards. These are synthetic protocols with declared utility functions, not forty independent real coding tasks or production-user data.

Compare uniform, position, cost, leave-one-out, vanilla Shapley, and precedence-constrained Shapley baselines. Additional checks examine invalid coalitions, sampling budgets, and replay determinism. Scripts and structural-protocol results are published as JSON reports.

The committed report finds lower error for precedence-constrained estimates against the protocol's prefix-sequential reference, and lower average error against the exact reference as sampling increases. These observations concern this estimator protocol, not real coding success, causal identification, or policy-training gains.

Compare credit baselines and convergence across sampling budgets and seeds. A strict chain reduces to prefix-conditioned sequential marginals; a partial order permits independent actions to exchange positions. Correctness of the declared precedence graph remains an assumption.

Attribution loses its basis if replay is nondeterministic or the DAG excludes feasible orders. Public live traces lack per-action workspace snapshots, so they do not support real-task counterfactual coalition replay. Included third-party training code is not evidence that this project trained an RL policy.

Run python -m evaluation.trajectory_credit_study from the repository to reproduce the offline structural protocol. Coding evaluation with a real model follows a separate execution path and is reported separately from offline validation.