Ballot Guide Lab — performance improvement plan
Based on the frozen v2 2B run; October 11, 2026

The model-size control uses Qwen3.5-4B with the current pipeline. The changes below
belong to a separate experiment. They are proposed fixes, not measured gains.

1. Improve retrieval and source-role selection.
   For policy effects, wage timing and fiscal estimates, retrieve the official
   summary/Legislative Analyst analysis first. For supporters/opponents questions,
   retrieve labeled advocacy as additional evidence. For statutory details,
   include relevant proposed law. Preserve every source's role in the prompt.
   V2 jobs retrieval ranked an opponent's argument first despite including the
   official page-39 analysis. Prison-cost retrieval returned five advocacy
   chunks and one official-analysis chunk. Evaluate source recall independently
   before asking the model to explain. In V2-10, the one official chunk starts
   midway through the state dollar estimate and does not include the local-cost
   range continuing on page 61. Preserve complete sentences/paragraphs and table
   structure, and include relevant adjacent passages. Do not assume more top-k passages alone
   fixes selection. Test a section-aware retriever, adjacent passages and reranking.

2. Improve training examples and verify their labels.
   Current 104 examples: 67 with one claim, 7 with two claims, 30 abstentions;
   only 5 examples use a nonempty missing-information field. This is a plausible
   contributor to one-part answers, not a proved causal explanation.
   Add reviewed examples with two-part questions, condition-specific dates,
   numbers with qualifiers, estimates versus guarantees, advocacy attribution,
   and mixed questions with supported and unavailable parts. Review source
   entailment and each answer's completeness before training. More examples or
   more epochs alone do not guarantee improvement. Keep held-out propositions
   and an untouched validation/test split separate from training.

3. Require a complete answer or explicit gap for each requested part.
   Identify requested parts before generation. For a small employer's $17/$18
   timing, cover both rates for that employer. For bond purpose plus amount,
   cover both. For the community-college amount plus local allocation, answer
   the known amount and explicitly state that a local allocation is unavailable.
   The current schema permits at most two claims; review whether that limit and
   the 320-token budget are sufficient before changing them in the new run.

4. Add semantic and attribution checks.
   Existing checks establish valid structure, known source IDs and exact quote
   matches. They do not establish that the claim follows from the quote.
   Verify amounts, dates, qualifying conditions, source roles and uncertainty.
   In particular, campaign arguments must not become Legislative Analyst findings.
   An independent verifier can flag unsupported interpretations for human review;
   its added latency, false alarms and missed errors also need evaluation.
   Consider copying evidence verbatim in the application instead of having the
   model reproduce quotes. Four tuned outputs failed exact-substring matching
   and one failed quote length. This may reduce mechanical failures; it still
   cannot establish that the cited passage entails the generated claim.

5. Handle unsupported questions precisely.
   Distinguish unavailable personal tax amounts, unavailable local allocations
   and later election results from answerable information missed by retrieval.
   Do not turn a partial retrieval failure into 'the guide has no information.'
   Give supported parts and identify gaps. Preserve the no-voting-advice rule.

6. Measure each change, without tailoring to the test.
   Keep the frozen v2 set as a diagnostic benchmark and document any repairs to
   faulty checkpoint metadata. Use fresh held-out voter-style questions to judge
   generalization after fixes; do not train on v2 questions/answers and then call
   a v2 score increase an independent test. Obtain actual user wording before
   claiming common-question coverage. Compare retrieval changes, training-label
   changes and model size separately. Score visible and raw answers independently.
   Maintain real human second ratings and report disagreements only when paired.

Most useful sequence:
- Run the unchanged 4B control to measure whether a larger checkpoint helps.
- Improve official-source selection and reviewed multi-part training labels.
- Evaluate the revised pipeline on a fresh held-out set with blind AI/human review.

Why 4B first:
It is twice the current 2B language-model size, in the same Qwen3.5 family.
Unsloth documents free-Colab 4B LoRA notebooks. Its memory figures are approximately
10 GB for 4B bf16 LoRA and 22 GB for 9B; actual T4 use with this code must be
verified. A 9B run would need more memory or a separately labeled quantization
experiment. The 2B adapter is not compatible with the 4B base; train a new one.

Primary references:
https://huggingface.co/Qwen/Qwen3.5-4B
https://unsloth.ai/docs/models/qwen3.5/fine-tune