Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models

Shreyas Subramanian, Mecit Gungor, Vikram Elango
Amazon, Seattle, USA
mecit@amazon.com

Abstract

Language and embedding models used in RAG systems are conventionally assumed to require large-scale pretraining and explicit grounding supervision. We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM. B1ade-embed, a 335M parameter retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training, and B1ade-1B, an SLM trained on low-cost GPUs using Group Relative Policy Optimization (GRPO) on 723M tokens (2.2M examples) of curated context-question pairs with rewards that optimize only answer similarity. Our central finding is emergent attribution: despite receiving no explicit supervision for source citation, B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points. This demonstrates that grounding behavior can emerge as an accuracy-maximizing strategy under RL training, without explicit reward engineering. On standard QA benchmarks, B1ade-1B achieves 81.82% on PopQA, 65.8% on PubMedQA, and 51.09% on FEVER. In end-to-end RAG evaluation, B1ade-1B achieves an average score of 0.654 across correctness, completeness, coherence, and faithfulness, a 10.8% improvement over the SFT, while closing the gap with models $1.5\times$ its size. These results show that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.

1 Introduction

Since its introduction, Retrieval Augmented Generation has emerged as a critical approach for grounding large language model outputs in verifiable sources, particularly for knowledge-intensive tasks in specialized domains Lewis et al. (2020). However, deploying effective RAG systems at scale requires addressing two fundamental challenges: (1) efficient retrieval of relevant passages from large corpora, and (2) faithful generation that acknowledges and cites source material. While recent advances in foundation models have improved both retrieval and generation capabilities, state-of-the-art systems typically rely on large models (7B+ parameters) with substantial computational requirements, limiting accessibility for resource-constrained deployments.

A central question for efficient RAG development is whether competitive performance requires extensive pretraining, or whether strategic model composition and targeted optimization can achieve similar capabilities with dramatically reduced compute. We investigate this through two complementary techniques: zero-training model merging for dense retrieval, and direct policy optimization on compact language models for grounded generation. Our approach demonstrates that careful architectural choices and training strategies can match or exceed the per-parameter efficiency of conventional large-scale pretraining.

We introduce B1ade-embed, a $335$ M parameter embedding model constructed through parameter-free fusion of five pretrained encoders, achieving top performance in sentence similarity among sub-500M models without additional training. This establishes zero-training model merging as a compute-free method for constructing competitive dense retrievers. For generation, we present B1ade-1B, a $1$ B parameter model trained via Group Relative Policy Optimization (GRPO) (Shao et al., 2024) on only $723$ M tokens of curated context-question pairs which is orders of magnitude fewer than standard pretraining approaches.

Our primary contribution is the empirical demonstration of emergent attribution behavior in B1ade-1B similar to what is described in Wei et al. (2022). Despite training with a reward function that optimizes exclusively for answer similarity, with no explicit term for attribution or source citation, the model learns to cite sources in 42.4% of responses on evaluation datathat is a statistically significant 5.5 percentage point increase over the 36.9% attribution rate in its training distribution. This emergence validates that grounding behaviors can arise as learned accuracy-maximizing strategies when careful reading and source acknowledgment correlate with correct answers, rather than requiring hand-crafted attribution rewards. The model exhibits strategic application, using specific passage citations at 5.6 $\times$ the training rate and applying more attribution to complex multi-hop questions than simple factual queries.

We develop B1ade-1B through an iterative process. Our initial model (v1) based on GPT-NeoX with supervised fine-tuning (SFT) achieved suboptimal RAG performance. This motivated a second iteration (v2) combining a Llama-3.2-1B base architecture with GRPO training, substantially improving performance across multiple evaluation dimensions. Empirically, B1ade-1B achieves 81.82% on PopQA, 65.8% on PubMedQA, and 51.09% on FEVER, demonstrating that small models trained with minimal compute can achieve competitive performance on standard benchmarks.

Contributions:

•

We introduce zero-training model merging as a compute-free method for constructing competitive dense retrievers, demonstrating that parameter-free fusion of pretrained encoders can achieve top performance without additional training.
•

We show that direct policy optimization on compact language models with simple accuracy rewards induces emergent attribution behavior, with models learning to cite sources 5.5 percentage points more frequently than their training distribution despite no explicit supervision for grounding.
•

We demonstrate that small language models trained via GRPO on $723$ M tokens achieve competitive RAG performance, establishing that strategic training can match conventional approaches with orders of magnitude less compute.
•

We provide empirical analysis of the strategy-execution gap in small model attribution, showing that while models learn when to cite sources, factual knowledge limitations remain the primary performance bottleneck.

The remainder of this paper is organized as follows. Section 2 reviews related work in grounded generation, small language models, and reinforcement learning for NLP. Section 3 describes our training methodology, including the iterative development process and GRPO training approach. We show how the simpleCoT training collection is created, and discuss its unique attribution characteristics. Section 4 evaluates B1ade across three complementary settings: standard benchmarks, end-to-end RAG, and LLM-as-Judge comparative evaluation, with emphasis on the emergent attribution finding. Section 5 discusses implications, limitations, and evaluation biases identified in our analysis. Section 6 concludes with a summary of contributions and future directions.

2 Related Work

Recent advances in RAG have demonstrated that compact language models can achieve competitive performance when augmented with external knowledge retrieval capabilities Sorstkins (2025). Several compact models in the 1-4 billion parameter range have emerged as viable alternatives to larger architectures, with research establishing 1B parameters as a practical lower bound for effective instruction-following and question-answering capabilities. Notable examples include Pleias-RAG-350M and Pleias-RAG-1B Langlais et al. (2025), which outperform most small language models below 4 billion parameters on standardized RAG benchmarks such as HotPotQA and 2wiki, demonstrating competitive performance with larger 7-8B models including Qwen-2.5-7B and Llama-3.1-8B. These findings suggest that carefully designed compact architectures can achieve strong RAG performance without the resource requirements of their larger counterparts.

To the best of our knowledge, development of competitive embedding models with zero training and model merging is largely unexplored. However, the development of efficient embedding models through knowledge distillation has become a critical component of resource-constrained RAG systems Zhang et al. (2025). Multi-stage distillation frameworks enable smaller student embedding models to learn from multiple larger teacher models, addressing the challenge that state-of-the-art embedding models often have numerous parameters and high vector dimensionality that pose deployment challenges in real-world scenarios Chen et al. (2025). Recent work has explored distilling RAG capabilities from large language models to small language models through evidence-based and graph-based knowledge distillation techniques, enabling compact models to replicate retrieval-augmented generation capabilities without extensive fine-tuning Chen et al. (2025). These advances in efficient embedding and knowledge distillation techniques have made RAG more accessible for resource-constrained deployments, aligning with the principle that carefully designed compact components can provide effective solutions for RAG tasks.

3 Methodology

We present our approach to resource-efficient RAG through two complementary techniques: zero-training model merging for dense retrieval (B1ade-embed, 335M parameters) and direct policy optimization for grounded generation (B1ade-1B, 1B parameters). Our methodology demonstrates that strategic model composition and targeted low-cost training can achieve competitive performance with dramatically reduced computational requirements compared to conventional large-scale pretraining.

3.1 B1ade-embed: Zero-Training Model Merging for Dense Retrieval

B1ade-embed is a 335M parameter embedding model created using Model Stock model merging, demonstrating that parameter-free fusion of pretrained encoders can achieve competitive performance without additional training. The model comprises the following components using Mergekit¹¹1https://github.com/arcee-ai/mergekit Goddard et al. (2024): (1) bert-large-uncased, (2) WhereIsAI/UAE-Large-V1, (3) BAAI/bge-large-en-v1.5, (4) mixedbread-ai/mxbai-embed-large-v1, (5) avsolatorio/GIST-large-Embedding-v0. Models were chosen based on superior MTEB performance (at the time of writing) despite smaller size and compatible architectures B1ade-embed uses the Model Stock algorithm Wang et al. (2024a), which optimizes linear interpolation weights by computing pairwise task vector similarities across models, providing a principled approach to parameter-free model composition. See Appendix F for detailed merging configuration and algorithm.

On the MTEB benchmark Muennighoff et al. (2023), B1ade-embed ranks #4 overall and #1 in Retrieval. Among models with output dimension $\leq$ 1024 and fewer than 500M parameters, it achieves #1 in Sentence Similarity, #4 in Summarization, Retrieval, and Pair Classification, and #5 in Reranking. See Appendix B for complete MTEB rankings (Table 12), including a comparison against Stella-400M-v5, the strongest sub-500M retriever which motivates our retriever ablation in Section 4.5.

Since its release on the Huggingface Hub, B1ade-embed has been independently and externally validated against various unique tasks. For example, in clinical and biomedical tasks, Soffer et al. (2024) evaluated 30 models across 2.1M comparisons on clinical notes, synthetic EHRs, MIMIC-IV ICU data, PubMed abstracts, and research papers, identifying B1ade-embed as versatile across both clinical and biomedical domains while noting its computational efficiency despite smaller size, particularly strong on short tasks like triage notes and chief complaints where it competed closely with larger models. A related study tested 39 embedding models across 7 medical semantic similarity tasks using Mount Sinai patient data, MIMIC-IV, PubMed texts, and Llama-3-70B synthetic data, finding that smaller models like B1ade-embed (335M parameters) performed comparably to larger models on short tasks.

In labor market applications, Qu et al. (2026) report that a system incorporating B1ade-embed achieved 81% Positive Predictive Value in a closed-world evaluation against ESCO’s occupational hierarchy, demonstrating strong capture of nuanced semantic relationships in labor market terminology.

The syftr (Pareto-Optimal Generative AI) framework Conway et al. (2025) evaluated over $10^{23}$ unique RAG configurations, where B1ade-embed consistently appeared in Pareto-optimal solutions, achieving top accuracy on DRDocs (software documentation), CRAG3 (sports), and Infinitebench (long-context retrieval over 100K tokens), while lying on the performance envelope for HotpotQA and Financebench.

3.2 B1ade-1B: Direct Policy Optimization for Grounded Generation

We develop B1ade-1B through an iterative process with two major versions, each addressing specific limitations discovered through evaluation. This iterative development demonstrates the importance of base model selection and training methodology for efficient RAG systems. Table 1 summarizes the key differences and improvements between versions.

Performance (RAG Evaluation)
Aspect	B1ade v1	B1ade v2
Base Architecture	GPT-NeoX-1B	Llama-3.2-1B
Pre-training Tokens	380B	Up to 9T
Post-training Tokens	720M	720M
Epochs	3	1
Training Stages	SFT only	GRPO only
LoRA Rank	16	16
Target Modules	Q, V projection	Q, V projection
Trainable Parameters	$\sim$ 8.4M	$\sim$ 8.4M
Precision	fp32	bfloat16
Overall Score	0.590	0.654 (+10.8%)
Completeness	0.435	0.539 (+22.8%)
Coherence	0.760	0.856 (+12.6%)

Table 1: Comparison of B1ade v1 and v2 model configurations and performance. B1ade v2 achieves substantial improvements through architectural switch and GRPO training.

Our initial model is based on the GPT-NeoX architecture Andonian et al. (2023) with 1 billion parameters, initialized from the pre-trained checkpoint and fine-tuned with Low-Rank Adaptation (LoRA) Hu et al. (2022). While performance on typical text-based benchmarks was superior to other models in the same weight class, evaluation on end-to-end RAG tasks revealed suboptimal performance as shown in our results, particularly in completeness and coherence dimensions, motivating architectural and training improvements.

Based on these limitations on end-to-end RAG tasks, we developed a second version of B1ade with two key changes: (1) architecture switch to Llama-3.2-1B Grattafiori et al. (2024) base, and (2) training enhancement directly using GRPO with no SFT checkpoint, since Llama-3.2-1B provides a useful and competitive base, which we include in our evaluations.

3.3 Training Data: simpleCoT Collection

We construct simpleCoT, a large-scale dataset of 2,214,941 examples (723M tokens) aggregated from seven sources spanning multi-hop QA, instruction following, mathematical reasoning, and commonsense understanding. All splits are unified into a standardized schema and shuffled with fixed seed 42, yielding 1,771,953 training and 442,988 test examples. Full dataset composition and processing details are provided in Appendix A. The dataset is publicly available.²²2https://huggingface.co/datasets/w601sxs/simpleCoT

3.4 Training

We directly perform GRPO (Shao et al., 2024) on top of the Llama-3.2-1B base using a standard ROUGE-L reward on answer similarity, with no explicit attribution or grounding term. We train for 3 epochs with learning rate $5\times 10^{-6}$ (AdamW), group size $k{=}4$ , batch size 8 questions (32 responses), bfloat16 precision, on a single NVIDIA A10G (24GB) for approximately 12 hours.

Critically, our reward function optimizes only for answer similarity using ROUGE-L score, with no explicit term for attribution or grounding behavior. This design choice is intentional: we investigate whether grounding behavior can emerge as a learned accuracy-maximizing strategy when careful reading and source acknowledgment correlate with correct answers, without requiring explicit supervision or hand-crafted attribution rewards. This minimal reward design enables us to isolate and study the emergence phenomenon.

4 Empirical Evaluation

We evaluate our approach across two complementary settings: (1) standard context-based QA benchmarks to measure core reasoning capabilities, and (2) end-to-end RAG to assess real-world applicability with retrieval systems using LLM-as-Judge scoring. Our evaluation focuses on validating the emergent attribution phenomenon and characterizing the performance-efficiency tradeoffs of our training paradigm.

As described in Section 3, we develop two versions of B1ade-1B: v1 (GPT-NeoX base, SFT only) and v2 (Llama-3.2-1B base, GRPO-only). Standard benchmarks and LLM-as-Judge evaluations use v2, while RAG evaluation compares both versions to demonstrate the improvement from architectural and training changes. Due to the poor performance of the base v1 model on end-to-end RAG tasks, we do not use the same for subsequent GRPO training, or for ablation experiments such as SFT-only vs GRPO-only. We focus on showing that simple reward functions (like rouge score) can still provide meaningful improvements in tasks like RAG. Next we discuss standard QA benchmarks and our end-to-end RAG benchmarks involving both B1ade-embedding and B1ade SLM.

4.1 Standard Context-Based QA Benchmarks

We evaluate B1ade-1B on standard question-answering benchmarks (PopQA, TriviaQA, Natural Questions, WikiMultihopQA, FEVER, ARC-Challenge, PubMedQA) using Exact Match accuracy. See Appendix A.2 for the complete evaluation protocol.

Table 2: Standard context-based QA benchmark results (Exact Match %). B1ade-1B achieves the highest PopQA score (81.82%) and strong performance on PubMedQA (“PMQA”) (65.80%) and FEVER (51.09%) among 1B models.

0^{*}

indicates instruction-following failures.

Model	Size	PopQA	TQA	NQ	ASQA	FEVER	ARC	PMQA	WikiMQA
TinyLlama Zhang et al. (2024)	1.1B	64.32	60.00	44.10	0.48	$0^{*}$	23.46	$0^{*}$	0.23
Llama-3.1B Grattafiori et al. (2024)	1.0B	48.55	36.65	35.67	40.21	23.27	49.91	16.8	12.79
Phi-1.5 Li et al. (2023)	1.2B	68.30	50.34	0	0.14	$0^{*}$	51.96	34.4	0.04
OPT Zhang et al. (2022)	1.3B	70.51	43.75	47.80	2.75	$0^{*}$	22.95	$0^{*}$	0
Pythia Biderman et al. (2023)	1.0B	73.44	39.77	66.90	0.01	8.17	22.78	18.2	0.06
Qwen-2.5 Yang et al. (2024)	1.5B	74.77	41.10	51.36	37.04	23.98	71.93	57.8	11.59
B1ade-1B	1.0B	81.82	33.11	48.32	16.56	51.09	22.44	65.80	7.56

B1ade-1B achieves 81.82% on PopQA (surpassing Qwen-2.5’s 74.77%), 65.80% on PubMedQA, and 51.09% on FEVER. These results show that training on context-based tasks with reasoning chains enables transfer across domains. Our model surpasses Qwen-2.5 on three tasks despite Qwen having 50% more parameters, indicating that training on curated data compensates for reduced capacity. The higher average scores relative to other 1B models result from strong performance on key tasks and avoidance of catastrophic failures observed in other sub-2B models (Also see Fig. 1 for a visual). For other models, zero or near-zero scores result from instruction-following failures and strict exact-match evaluations (detailed analysis in Appendix A.3). Among the models we tested, only Llama-1B and B1ade-1B achieve consistent performance without catastrophic failures. Other models exhibit context repetition, option echoing, and format inconsistencies, revealing limitations in deploying sub-2B models for structured RAG without constrained decoding (detailed in Appendix C.2).

4.2 End-to-End RAG Evaluation

We evaluate B1ade-1B in realistic RAG settings using B1ade-embed and stella-400M for retrieval, with Claude Sonnet 4 as LLM-as-Judge on correctness, completeness, coherence, and faithfulness. We use LLM-as-Judge-based model evaluation jobs on Amazon Bedrock; this evaluation motivated our v1 to v2 improvements.³³3Built-in model evaluation metrics: https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-metrics.html

Table 3: End-to-end RAG evaluation results using B1ade-embed (Claude Sonnet 4 as judge). Scores range from 0–1 (higher is better). Bold denotes global best per metric. See Section 4.5 for retriever choice and training data scale ablations.

LLM	Embedding	K	Corr.	Comp.	Cohe.	Faith.	Avg.
Baseline Models (no fine-tuning)
Llama-1B	B1ade-embed	5	0.636	0.549	0.911	0.639	0.684
Qwen-1.5B	B1ade-embed	5	0.679	0.527	0.829	0.680	0.679
B1ade v1 (GPT-NeoX Base, SFT Only)
B1ade v1	B1ade-embed	5	0.521	0.435	0.760	0.644	0.590
B1ade v2 (Llama-3.2-1B Base, GRPO)
B1ade v2	B1ade-embed	5	0.541	0.539	0.856	0.681	0.654

B1ade v1 (GPT-NeoX + SFT) scores 0.590 average, with low completeness (0.435) and coherence (0.760). Switching to Llama-3.2-1B and adding GRPO (v2) improves to 0.654 (+10.8%), with largest gains in completeness (+22.8%) and coherence (+12.6%). B1ade v2 approaches Qwen-1.5B (0.679) with 33% fewer parameters but trails in correctness (0.541 vs. 0.679). While Llama-1B base achieves the highest average score (0.684) driven by coherence (0.911), B1ade v2 leads all models on faithfulness (0.681), the metric most directly reflective of RAG grounding quality, demonstrating that GRPO training successfully optimizes for source-grounded generation despite using 33% fewer parameters than Qwen-1.5B. Also see note in Appendix E on length bias from judge model impacting Llama 1B scores.

4.3 Attribution Emergence Analysis

We conduct head-to-head comparisons using LLM-as-Judge on 6,345 QA pairs from RAGEval Zhu et al. (2025), comparing B1ade v2, Llama-3.2-1B (base), and Qwen-1.5B. Claude Sonnet 4 ranks outputs based on context fidelity, attribution quality, completeness, and factual accuracy (see Appendix C.1 for the complete template). This evaluation enables direct measurement of the emergent attribution phenomenon.

Table 4: LLM-as-Judge evaluation results (

n=6{,}345

). Top: Primary rankings showing B1ade v2 achieves 32.2% first-place rate with our training approach (Llama-3.2-1B base with GRPO). Bottom: Attribution emergence—B1ade-1B exceeds the training distribution by 5.5 percentage points, representing the main novel contribution of this work.

Rank	Model	First Place	Avg Rank	Params
1st	Qwen-1.5B	42.2% (2,678)	1.77	1.5B
2nd	B1ade v2	32.2% (2,044)	2.11	1B
3rd	Llama-3.2-1B	29.1% (1,849)	2.08	1B
Training Data Attribution		36.9%
B1ade Eval Attribution		42.4%

B1ade-1B achieves 32.2% first-place rate vs. 29.1% for base Llama-1B (+3.1pp) and reaches 76% of Qwen-1.5B’s performance (42.2%) with 67% of its parameters. The key finding is that B1ade-1B uses attribution at 42.4%, exceeding the 36.9% rate found naturally in the training distribution. This shows grounding emerges as an accuracy-maximizing strategy, not through memorization.

With 6,345 evaluations, the 5.5pp increase (2,689 observed vs. 2,341 expected) is significant ( $p<0.001$ , binomial test). If the model merely memorized training patterns, we would expect 36.9% attribution and uniform application. Instead we observe: (1) 42.4% attribution (+5.5pp above training), (2) specific citations at 2.8% (5.6 $\times$ training rate), and (3) higher attribution on multi-hop vs. factual questions.

4.4 50K Subset Curation Methodology

To enable fair comparison of GRPO’s effectiveness across different data scales, we created a clean 50K subset from the 2.2M-example simpleCoT dataset using a four-step pipeline:

1.

Normalization: Extract raw question and answer text from dataset-specific XML-like scaffolding (e.g., question: <...>, answer: <...>). Drop malformed examples.
2.

Length filtering: Retain only examples where the question contains 20–300 tokens and the answer contains 10–200 tokens. This eliminates trivial one-word answers and excessively long documents while maintaining diversity in question complexity.
3.

Deduplication: Remove exact duplicate questions by hashing the first 80 characters of cleaned question text, preserving the first occurrence.
4.

Stratified sampling: Partition questions into four equal-sized difficulty bins based on token length quartiles (easy → hard), then sample uniformly from each bin to ensure balanced coverage of question complexity. This prevents the subset from being skewed toward either trivial or overly complex questions.

This curation process reduces the 2.2M raw examples to approximately 500K candidate examples after filtering, from which we stratified-sample 50K for the scale ablation. The resulting subset is deterministic (seeded), publicly available on Hugging Face Hub⁴⁴4https://huggingface.co/datasets/w601sxs/simplecot_subset_50k, and serves as the training corpus for the smaller model experiments in Section 4.5. Note: 50K experiments train for 1 epoch, while full 2.2M experiments train for 3 epochs.

4.5 Ablation Studies and Additional Analyses

We present three complementary analyses examining training data scale, retriever choice, and inference optimization.

Training data scale: 50K vs. 2.2M. To assess how much training data is necessary to realize GRPO’s benefits, we train Qwen-0.5B and Llama-1B on only 50K examples from the simpleCoT dataset. Table 5 (RAG) and Appendix Table 11 (standard benchmarks) show that standard benchmark performance is substantially lower than B1ade-1B trained on the full 2.2M examples, confirming that factual knowledge acquisition requires large-scale training.

GRPO’s effects differ across data scales. On 50K, GRPO improves all Llama-1B metrics except faithfulness: correctness +0.047 (+12.9%), coherence +0.066 (+18.6%), completeness +0.003 (+0.8%). On 2.2M, B1ade v2 shows the opposite: only faithfulness improves (+0.042, +6.6%), while correctness declines -0.095, coherence -0.055, completeness -0.010 relative to base Llama-1B. With limited data, GRPO improves general quality. With abundant data, GRPO optimizes exclusively for grounding at the expense of other metrics. This indicates GRPO’s reward does not scale uniformly. GRPO reduces performance on format-constrained tasks (FEVER: -13.1pp for Qwen-0.5B), consistent with the reward not penalizing format deviations. These results suggest decoupling between knowledge acquisition (scale-dependent) and grounding behavior (learnable from limited data).

Table 5: Training data scale ablation: Impact of data quantity on GRPO effectiveness (RAG evaluation with B1ade-embed, scores 0–1). Bottom row shows Llama-1B trained on full 2.2M dataset with GRPO (B1ade v2) for comparison.

Model	Data	Corr.	Comp.	Faith.	Avg
Qwen-0.5B	50K	0.321	0.356	0.393	0.345
Qwen-0.5B GRPO	50K	0.340	0.369	0.425	0.368
Llama-1B	50K	0.364	0.383	0.430	0.383
Llama-1B GRPO	50K	0.411	0.386	0.421	0.407
B1ade v2 (Llama-1B)	2.2M	0.541	0.539	0.681	0.654

Retriever choice (B1ade-embed vs. Stella-400M). Table 6 shows minimal impact from retriever choice. B1ade v2 scores 0.654 with B1ade-embed vs. 0.652 with Stella-400M (+0.002 difference); B1ade v1 shows 0.590 vs. 0.589. Generation quality is bottlenecked by the language model, not retrieval at $K{=}5$ . B1ade-embed is a compute-free alternative to Stella-400M without quality loss.

Table 6: Retriever ablation: B1ade-embed vs. Stella-400M on B1ade v1 and v2 (scores range 0–1, higher is better).

Model	Embedding	Corr.	Comp.	Avg
B1ade v1	B1ade-embed	0.521	0.435	0.590
B1ade v1	Stella-400M	0.526	0.437	0.589
B1ade v2	B1ade-embed	0.541	0.539	0.654
B1ade v2	Stella-400M	0.545	0.542	0.652

Retrieval scope (K=5 vs K=10). We test retrieval scope to assess sensitivity to the number of retrieved documents. Table 7 compares B1ade v2 with K=5 (0.654 avg) and K=10 (0.466 avg). Increasing K from 5 to 10 documents provides minimal benefit and degrades performance, consistent with the observation that language model capacity is the bottleneck. We use K=5 for all reported results.

Table 7: Retrieval scope ablation: B1ade v2 with K=5 vs K=10 (B1ade-embed, scores 0–1).

Model	K	Corr.	Comp.	Cohe.	Avg
B1ade v2	5	0.541	0.539	0.856	0.654
B1ade v2	10	0.431	0.347	0.566	0.466

Inference optimization: speculative decoding. We test self-speculative decoding for inference throughput. DoLa decoding reduces PubMedQA accuracy from 65.8% to 63.8%, while ngram lookup improves it to 68.2%. On FEVER, both methods degrade accuracy by 25% due to repeated token artifacts. Speculative decoding disrupts the attribution patterns learned during GRPO training.

5 Discussion and Limitations

We organize this discussion around key themes: the emergence and validation of attribution as a learned strategy, the strategy-execution gap revealed by our evaluations, implications for efficient RAG development, and limitations with future directions.

B1ade-1B exhibits attribution at 42.4%, exceeding training distribution (36.9%) by 5.5pp. This validates that grounding emerges as an accuracy-maximizing strategy. However, the base Llama-3.2-1B may have pre-existing attribution capabilities from distillation or prior training. We cannot definitively separate GRPO training effects from base model capabilities.

Evidence of learned behavior beyond base capabilities: (1) Specific passage citations (e.g., “Passage 1 states…”) appear at 2.8%, 5.6 $\times$ the training rate of 0.5%, with insufficient training examples to explain via memorization. (2) Attribution rate is higher on multi-hop questions than factual queries, not uniform across types. (3) Direct quotes appear in 9.8% of responses, with 494 accurate (38.8%), showing text extraction beyond templates. (4) The model correctly distinguishes when context supports vs. does not support claims (e.g., recognizing “passage does not explicitly mention emergence, only predominance”), showing epistemic calibration.

Regarding training limitations, the simpleCoT dataset containing 2.2M examples exhibits source imbalance, with Kaist contributing 76.6% of training data (Table 9). This may limit generalization to domains underrepresented in training, such as creative writing or low-resource languages. Regarding evaluation limitations, despite explicit instructions to avoid length bias, we observe systematic length-rank correlation (Table 18), with Llama’s longer responses penalized (correlation of +0.264) and Qwen’s brevity rewarded (correlation of -0.112). The “attribution paradox,” where B1ade-1B loses to Qwen despite providing identical answers with explicit attribution (Appendix Cases 1–3), suggests the judge may favor conciseness over source citation, contradicting stated preferences. All LLM-as-Judge evaluations use Claude Sonnet 4 as a single judge model. Different judge models such as GPT-4 or Llama-3-70B-Instruct may exhibit different biases, potentially altering rankings, and multi-judge consensus would strengthen validity. Our standard benchmarks in Section 4.1 focus on factual QA and reasoning. We do not evaluate creative generation, dialogue, or code generation, which are domains where attribution behavior may manifest differently.

6 Conclusion

We introduce a training paradigm for resource-efficient RAG using zero-training model merging for retrieval and policy optimization on compact language models. Strategic architectural choices and training achieve competitive performance with reduced computational requirements. Through this training paradigm We presented B1ade, a resource-efficient RAG system with two contributions: B1ade-embed, a competitive 335M retriever built via zero-training model merging that matches Stella-400M in end-to-end RAG (0.654 vs. 0.652) without any retriever training; and B1ade-1B, a 1B SLM trained with GRPO on 723M tokens that achieves 81.82% PopQA, 65.8% PubMedQA, and 0.654 average RAG score (LLaaJ) that approaches Qwen-1.5B with 500M fewer parameters. An emergent property of GRPO training is that attribution behavior arises through accuracy-only optimization: training with ROUGE rewards alone produces a 42.4% citation rate, exceeding the training distribution (36.9%) by 5.5%. Together, these results show that competitive RAG components can be built without large-scale specialized training, with grounding behavior emerging as a byproduct of reward optimization rather than explicit supervision.

Future work should investigate whether attribution emergence generalizes across model scales, training curricula, and reward formulations. Key questions include: (1) Does this phenomenon persist in larger models, or is it specific to the capacity constraints of 1B-scale systems? (2) Can alternative reward formulations (e.g., token-level rewards, contrastive objectives) induce stronger or more reliable grounding behaviors? (3) What training data characteristics (attribution rate, citation specificity, reasoning chain length) most strongly influence emergence? Additionally, methods to close the strategy-execution gap without sacrificing parameter efficiency warrant investigation, potentially through knowledge distillation or hybrid architectures that separate reasoning from knowledge storage.

References

A. Andonian, S. Biderman, S. Black, P. Gali, L. Gao, E. Hallahan, J. Levy-Kramer, C. Leahy, L. Nestler, K. Parker, et al. (2023) GPT-neox: large scale autoregressive language modeling in pytorch. Zenodo. Cited by: §3.2.
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. (2023) Pythia: a suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397–2430. Cited by: Table 2.
J. Chen, A. Myrzakhan, Y. Luo, H. M. Khan, S. M. Bsharat, and Z. Shen (2025) DRAG: distilling rag for slms from llms to transfer knowledge and mitigate hallucination via evidence and graph-based distillation. External Links: 2506.01954, Link Cited by: §2.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1. Cited by: 7th item.
A. Conway, D. Dey, S. Hackmann, M. Hausknecht, M. Schmidt, M. Steadman, and N. Volynets (2025) Syftr: pareto-optimal generative ai. arXiv preprint arXiv:2505.20266. Cited by: §3.1.
C. Goddard, S. Siriwardhana, M. Ehghaghi, L. Meyers, V. Karpukhin, B. Benedict, M. McQuade, and J. Solawetz (2024) Arcee’s MergeKit: a toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, F. Dernoncourt, D. Preoţiuc-Pietro, and A. Shimorina (Eds.), Miami, Florida, US, pp. 477–485. External Links: Link, Document Cited by: §F.1.1, §3.1.
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §3.2, Table 2.
X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), pp. 6609–6625. External Links: Link Cited by: 4th item.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), pp. 3. Cited by: §3.2.
Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu (2019) PubMedQA: a dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, pp. 2567–2577. External Links: Link, Document Cited by: 8th item.
M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer (2017) triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, pp. arXiv:1705.03551. External Links: 1705.03551 Cited by: 2nd item.
S. Kim, S. J. Joo, D. Kim, J. Jang, S. Ye, J. Shin, and M. Seo (2023) The cot collection: improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305.14045. Cited by: §A.1.
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019) Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, pp. 452–466. External Links: Link, Document Cited by: 3rd item.
P. Langlais, P. Chizhov, M. Nee, C. R. Hinostroza, M. Delsart, I. Girard, O. Hicheur, A. Stasenko, and I. P. Yamshchikov (2025) Even small reasoners should quote their sources: introducing the pleias-rag model family. ArXiv abs/2504.18225. External Links: Link Cited by: §2.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, pp. 9459–9474. Cited by: §1.
Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee (2023) Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463. Cited by: Table 2.
A. Mallen, Asai,Akari, V. Zhong, R. Das, H. Hajishirzi, and D. Khashabi (2022) When not to trust language models: investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint. Cited by: 1st item.
N. Muennighoff, N. Tazi, L. Magne, and N. Reimers (2023) Mteb: massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014–2037. Cited by: §3.1.
J. Qu, M. Gopinathan, S. A. Akbar, and O. Alonso (2026) Interactive taxonomy development with hybrid methods. Cited by: §3.1.
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §3.4.
S. Soffer, B. S. Glicksberg, P. Kovatch, O. Efros, R. Freeman, A. W. Charney, G. N. Nadkarni, and E. Klang (2024) A scalable framework for benchmarking embedding models for semantic medical tasks. medRxiv, pp. 2024–08. Cited by: §3.1.
A. Sorstkins (2025) Assessing rag and hyde on 1b vs. 4b-parameter gemma llms for personal assistants integretion. External Links: 2506.21568, Link Cited by: §2.
I. Stelmakh, Y. Luan, B. Dhingra, and M. Chang (2022) ASQA: factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273–8288. Cited by: 6th item.
J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal (2018) FEVER: a large-scale dataset for fact extraction and VERification. In NAACL-HLT, Cited by: 5th item.
C. Wang, X. Fang, et al. (2024a) Model stock: all we need is just a few fine-tuned models. arXiv preprint arXiv:2403.19522. Cited by: §3.1.
D. Wang, S. Isber, J. Kim, M. Moon, D. Han, J. Choi, and C. Kim (2024b) Model stock: all we need is just a few fine-tuned models. arXiv preprint arXiv:2403.19522. Cited by: §F.1.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. (2022) Emergent abilities of large language models. arXiv preprint arXiv:2206.07682. Cited by: §1.
A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan (2024) Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: Table 2.
D. Zhang, J. Li, Z. Zeng, and F. Wang (2025) Jasper and stella: distillation of sota embedding models. External Links: 2412.19048, Link Cited by: §2.
P. Zhang, G. Zeng, T. Wang, and W. Lu (2024) Tinyllama: an open-source small language model. arXiv preprint arXiv:2401.02385. Cited by: Table 2.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. (2022) Opt: open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. Cited by: Table 2.
K. Zhu, Y. Luo, D. Xu, Y. Yan, Z. Liu, S. Yu, R. Wang, S. Wang, Y. Li, N. Zhang, et al. (2025) Rageval: scenario specific rag evaluation dataset generation framework. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8520–8544. Cited by: §4.3.

Appendix

Appendix A Dataset Details

A.1 Source Datasets

The simpleCoT dataset aggregates seven diverse source datasets totaling 2,214,941 examples across multiple reasoning domains, including multi-hop question answering (HotpotQA Distractor and FullWiki variants), general instruction following (Alpaca cleaned), mathematical reasoning (Math QA), commonsense narrative understanding (Cosmos QA), diverse instruction patterns (GPTeacher), and evolved complex instructions (Wizard Evol V2). The data processing pipeline consists of three key stages: First, all training splits are loaded directly from their respective Hugging Face repositories using the datasets library. Second, format unification is performed by casting all datasets to a standardized Kaist Kim et al. (2023) schema comprising three fields: context (system instructions and factual passages), question (user query embedded within context), and ground truth (rationale combined with final answer) ensuring consistent structure across heterogeneous sources. Third, the unified datasets are concatenated, shuffled with a fixed random seed (42) for reproducibility, and split into training and testing subsets using an 80/20 ratio, yielding 1,771,953 training examples and 442,988 test examples. This systematic approach preserves the diversity of reasoning types while enabling standardized model training and evaluation across the entire collection.

Dataset	Samples	% of Total
HotpotQA (Distractor + FullWiki)	180K	8.1%
Wizard Evol Instruct V2	196K	8.8%
Alpaca (cleaned)	52K	2.3%
Math QA	37K	1.7%
GPTeacher	30K	1.4%
Cosmos QA	25K	1.1%
Kaist (base format)	1,695K	76.6%
Total	2,214,941	100.0%

Table 8: Composition of the simpleCoT training dataset. Kaist provides the base format for unification.

Source	Samples	% of Total
Wizard Evol Instruct V2	196,000	8.8%
HotpotQA (combined)	180,000	8.1%
Alpaca (cleaned)	52,000	2.3%
Math QA	37,000	1.7%
GPTeacher	30,000	1.4%
Cosmos QA	25,000	1.1%
Kaist (diverse)	1,694,941	76.6%

Table 9: Training data distribution across source datasets. Kaist contains 139 diverse sub-tasks.

Metric	Value
Total Samples	2,214,941
Source Datasets	7
Training Split	1,771,953 (80%)
Test Split	442,988 (20%)
Generic Attribution	36.9%
Specific Citation	0.5%
No Attribution	63.1%
Fine-tuning Tokens	723M
Shuffle Seed	42

Table 10: Summary statistics for simpleCoT collection.

A.2 Standard Benchmark Results

Refer to caption — Figure 1: B1ade performance on standard benchmarks

Benchmarks:

•

PopQA (Mallen et al. (2022)): Popular entity questions
•

TriviaQA (Joshi et al. (2017)): Trivia-style factual questions
•

Natural Questions (NQ) (Kwiatkowski et al. (2019)): Google search queries
•

WikiMultihopQA (Ho et al. (2020)): Multi-hop Wikipedia reasoning
•

FEVER (Thorne et al. (2018)): Fact extraction and verification
•

ASQA (4.4K) (Stelmakh et al. (2022)): Ambiguous questions with long-form answers
•

ARC-Challenge (Clark et al. (2018)): Science questions (AI2 Reasoning Challenge)
•

PubMedQA (Jin et al. (2019)): Biomedical yes/no/maybe questions (500 samples)

Model	Data	PopQA	TQA	NQ	ASQA	FEVER	ARC	PMQA	WikiMQA
Qwen-0.5B	50K	7.90	8.60	4.20	6.00	20.53	74.80	7.40	6.01
Qwen-0.5B GRPO	50K	8.82	11.60	4.20	5.40	7.39	72.00	10.30	7.06
Llama-1B	50K	9.60	25.00	11.60	13.40	14.26	52.60	31.40	9.56
Llama-1B GRPO	50K	11.20	25.00	11.60	13.40	15.40	52.60	32.60	13.20

Table 11: Standard benchmark results (Exact Match %) for smaller models trained on 50K subset. GRPO produces modest gains on most tasks but drops on FEVER and ARC for Qwen-0.5B, consistent with the reward not penalizing format deviations.

A.3 Note on low/zero scores

A striking finding across our RAG benchmarks was the widespread failure of small language models to adhere to basic output format constraints, even with extensive prompt engineering. This phenomenon manifested in several distinct failure modes that fundamentally undermined evaluation validity. Most notably, we observed systematic "option repetition" behavior where models would output all possible answer choices rather than selecting one which is a trivial strategy that artificially inflates performance metrics. For instance, on PubMedQA, Llama 3.2 1B repeated all three options ("yes," "no," "maybe") in 321 of 500 test cases, creating false positive matches that initially suggested 80% accuracy. Only after implementing strict single-answer extraction did we observe the true baseline performance.

Catastrophic Failures and Zero-Score Phenomena: Several model-benchmark combinations produced zero or near-zero scores despite iterative prompt refinement, revealing fundamental limitations in instruction-following capacity for certain small models. On ARC-Challenge, Phi 1.5, Phi 3 Mini, and TinyLlama all achieved 0% exact match accuracy, while Llama 3.2 1B managed only 0.09% (1 correct answer out of 1,172 questions). These failures persisted even when simplifying prompts, adding explicit constraints, or restructuring inputs as multiple-choice questions. Particularly illustrative is the case of Pythia 1.4B on PubMedQA, where the model frequently generated only a period (".") with no mention of the required answer options. When constrained through template-based generation, the model instead repeated instructions, specifically outputting "choose between the following three options and only print the correct answer from ’yes’" in 276 of 500 cases. This repetition artificially boosted accuracy to 44%, which collapsed to 9% when these echoed instructions were removed from evaluation.

OPT 1.3B exhibited similar catastrophic behavior across multiple benchmarks, achieving 0% on WikiMultihopQA despite various prompt formulations. On FEVER, OPT repeated the entire context verbatim for every test instance under the original prompt, necessitating a complete reformulation. Even after simplification, the model frequently output the system prompt itself as the answer.

Model-Specific Failure Modes and Prompt Sensitivity: Each model exhibited idiosyncratic responses to prompt engineering, requiring bespoke optimization strategies that often contradicted general best practices. Llama 3.2 1B required verbose, markdown-formatted instructions to trigger chain-of-thought reasoning on FEVER, but this same verbosity caused context repetition in other models. Interestingly, while chain-of-thought generation improved Llama’s answer quality in some cases, it frequently failed to reach a conclusion producing multi-step reasoning that never selected between "SUPPORTS" or "REFUTES." Additionally, Llama often formatted answers using LaTeX notation (e.g., "The final answer is: $\boxed{SUPPORTS}$ "), complicating answer extraction. In the most problematic cases, Llama outputs contained both "SUPPORTS" and "REFUTES" within extended self-contradictory reasoning; penalizing these double-matches reduced FEVER accuracy from 49% to 23%. Qwen 2.5 1.5B demonstrated extreme sensitivity to prompt format. Markdown-structured prompts triggered full context repetition, while simpler templates produced refusal-like outputs that restated the claim in negative form (e.g., "Refutes: The Ukrainian Soviet Socialist Republic was not a founding participant of the UN. The Soviet Union was a founding participant of the UN."). Most surprisingly, using Qwen’s documented message format with role tags degraded performance, causing the model to output repetitive Unicode characters ("ți ți ți…"). Only after converting FEVER to explicit multiple-choice format did Qwen produce evaluable outputs. Phi 1.5 exhibited a distinct hallucination pattern: when presented with two multiple-choice options (A and B), it systematically "completed" the list to four options (adding hallucinated C and D choices) before selecting an answer. For example:

  Choose only one option -
  A. ’SUPPORTS’
  B. ’REFUTES’.
  Result: C. ’NEITHER’.
          D. ’MAYBE’.
          Answer:
          A. ’SUPPORTS’

This behavior increased from 15.98% accuracy (when evaluating the full response) to 24.88% when restricting matches to text following "Answer:", suggesting the model reliably answered correctly after completing its hallucinated option set.

Evaluation Methodology Implications: These findings necessitated substantial refinements to evaluation methodology. Initial exact-match scoring proved inadequate, as it could be trivially gamed through option repetition or context echoing. We implemented several countermeasures:

1.

Positional answer extraction: Restricting match detection to text following explicit markers like "Answer:" reduced false positives from hallucinated reasoning chains.
2.

Penalty for ambiguity: Responses containing multiple contradictory answers (e.g., both "yes" and "no") were marked incorrect, even if one match was correct.
3.

Soft matching: For datasets like WikiMultihopQA, we added fuzzy matching to account for semantically equivalent but lexically different answers, revealing substantial performance gaps between strict (7.56% for B1ade) and soft matching (13.68% F1).
4.

Context truncation strategies: CUDA OOM errors for longer contexts (e.g., FEVER Wikipedia articles on OPT) required systematic truncation, introducing an uncontrolled variable that may have degraded performance for context-dependent questions.

The most concerning implication is that standard benchmark scores may systematically overestimate small model capabilities when evaluation doesn’t account for these failure modes. The PubMedQA case where naive evaluation suggested 80% accuracy that dropped to actual baseline performance after proper answer extraction demonstrates how easily benchmarks can be inadvertently "gamed" by models that exploit evaluation weaknesses rather than demonstrating genuine comprehension.

Performance Variability and Model Selection: Despite uniform prompt engineering efforts, model performance varied dramatically across benchmarks in ways that defy simple model-size predictions. On ARC-Challenge, Qwen 2.5 1.5B achieved 71.93% accuracy while larger models like OPT 1.3B achieved only 22.95%. Similarly, on PopQA, Pythia 1.4B (73.44%) substantially outperformed the smaller TinyLlama (64.32%), but this relationship inverted on WikiMultihopQA where TinyLlama’s exact match score (0.23%), while abysmal, exceeded Pythia’s 0.06%. These inconsistencies suggest that benchmark performance for small models depends more on alignment between model training distribution and task format than on general reasoning capacity. Qwen’s strong ARC performance likely reflects exposure to similar multiple-choice formats during training, while its failure on FEVER (requiring the messages format) indicates brittleness when presented with unexpected input structures.

Broader Implications for RAG Systems: The systematic instruction-following failures documented here pose significant challenges for deploying small models in retrieval-augmented generation systems. The zero-score phenomenon suggests that below a certain capability threshold, prompt engineering alone cannot induce reliable structured output generation. For production RAG systems requiring predictable output formats (e.g., JSON for API responses), these models would require either constrained decoding or downstream post-processing both adding latency and complexity that partially negate the efficiency advantages of smaller models. Moreover, the model-specific prompt optimization required for even basic functionality undermines the modularity of RAG architectures. A system designed around Llama 3.2 1B’s verbose markdown preferences would fail catastrophically if swapped to Qwen without prompt reengineering. This brittleness suggests that small model RAG systems may require maintaining model-specific prompt templates and evaluation pipelines, increasing maintenance burden.

Appendix B B1ade-embed Additional Details

B.1 MTEB rankings and performance at tasks

In the current MTEB dashboard, B1ade-embed achieves the following rankings:

•

#4 overall (Borda ranking)
•

#1 on Retrieval overall
•

#4 on Pair Classification and Classification
•

#3 on AskUbuntuDupQuestions and MTOPDomainClassification
•

#2 on ArXivHierarchicalClusteringS2S, Arguana, and SprintDuplicateQuestions
•

#1 on BIOSSES, Banking77Classification, CQADupstackGamingRetrieval, MindSmallReranking, SciDocs
•

High performing on several STS tasks (mean: 78.99, STS: 87.07)

Model	Avg	Class	Cluster	Pair	Rerank	Retrieval	STS
Stella-400M-v5	70.11	86.67	56.70	87.74	60.16	58.97	84.22
learning2_model	65.39	77.75	47.96	84.53	58.50	57.91	81.43
gte-large-en-v1.5	65.39	77.75	47.96	84.53	58.50	57.91	81.43
cde-small-v1	65.00	81.71	48.32	84.69	56.75	53.27	81.63
mxbai-embed-large-v1	64.68	75.64	46.71	87.20	60.11	54.39	85.00
UAE-Large-V1	64.64	75.58	46.73	87.25	59.88	54.66	84.54
bge-large-en-v1.5	64.23	75.97	46.08	87.12	60.03	54.29	83.11
B1ade-embed (ours)	64.21	75.16	46.46	87.07	60.00	53.30	85.04

Table 12: B1ade-embed MTEB leaderboard results (56 datasets, all tasks). At the time of writing, B1ade-embed ranked #9 overall among 400M-parameter models, competitive with leading retrieval models despite zero-training design via parameter fusion. Bold indicates best performance per column. Scores on Classification (12 datasets), Clustering (11), PairClassification (3), Reranking (4), Retrieval (15), and STS (10) tasks shown as averages.

Table 12 shows that Stella-400M-v5 ranks first across six of seven MTEB categories, making it the strongest available retriever in the sub-500M class. This is why we selected Stella-400M-v5 as the comparison point for the retriever ablation in Section 4.5: it represents the strongest available retrieval baseline, and demonstrating that B1ade-embed performs comparably in end-to-end RAG (0.654 vs. 0.652 average) against it is a strong validation of our zero-training model merging approach. The only category where B1ade-embed outperforms Stella is STS (85.04 vs. 84.22), which is directly relevant to RAG retrieval quality — measuring semantic similarity between query and passage. This suggests that despite lower overall MTEB scores, B1ade-embed is well-suited for RAG-specific retrieval tasks.

On domain-specific leaderboards (sub-500M class):

•

Top-3 on CodeSearchNetRetrieval
•

Top-5 on legal-text retrieval
•

Top-2 by mean score on medical benchmarks
•

#1 on NFcorpus

B.2 Knowledge Distillation Experiments

We also experimented with distilling knowledge from Alibaba-NLP/gte-large-en-v1.5 (434M parameters) into b1ade-embed. Training used AllNLI and English Wikipedia text for 30 epochs (LR = 5e-5, batch size = 64, FP16) with PCA dimensionality reduction followed by dense projection. The distilled model showed targeted gains in specific MTEB subsets, However, average scores dropped across all MTEB categories. All subsequent experiments use the base b1ade-embed model without distillation.

Table 13: Knowledge distillation results on selected MTEB subsets. The distilled model shows targeted improvements on specific tasks, though average scores dropped across all MTEB categories.

MTEB Subset	Before (%)	After (%)	Improvement	Metric
ToxicConversationsClassification	67.91	75.01	+7.10	Accuracy
TweetSentimentExtractionClassification	59.22	61.8	+2.58	Accuracy
MedrxivClusteringS2S	31.7	34.02	+2.32	Validity Measure
ClimateFEVER	30.37	31.95	+1.58	nDCG@10

Appendix C LLMaaJ and Attribution Analysis Examples

C.1 Complete Judge Template

We use Claude Sonnet 4 as the judge to rank model outputs. The full prompt template is:

C.2 Example Evaluations

Example 1: B1ade-1B Wins with Correct Nuance

Model Outputs:

Judge Reasoning: “B1ade-1B correctly identifies that the passage describes when the genotype became predominant, not when it emerged. The question asks about emergence, but the context only provides information about predominance. B1ade’s careful distinction and epistemic humility (acknowledging the passage doesn’t say when it emerged) demonstrate superior context fidelity. Ranks: B1ade=1, Qwen=2, Llama=3.”

Analysis: B1ade wins by understanding the nuance between “emerged” and “became predominant.” This is the most common winning pattern (48% of wins).

Example 2: B1ade-1B Loses from Factual Error

Model Outputs:

Judge Reasoning: “Context explicitly states: ’Hemagglutinin (HA) is the major surface and attachment glycoprotein.’ B1ade-1B’s answer of NA (neuraminidase) is factually incorrect. Despite proper attribution (“Based on the passage”), the core answer is wrong. Llama and Qwen both correctly identify HA. Ranks: Llama=1, Qwen=1, B1ade=3.”

Analysis: B1ade-1B loses from factual error (56% of losses), not from attribution strategy. The model correctly attributes but provides the wrong glycoprotein.

Example 3: B1ade-1B Wins Despite Attribution Paradox

Model Outputs:

Judge Reasoning: “Both provide the correct range (14-53%). B1ade adds useful clinical detail (elevated ALT/AST) and explicitly attributes to the passage. More complete response. Ranks: B1ade=1, Qwen=2.”

Appendix D Attribution Analysis:

B1ade-1B wins when attribution is combined with additional relevant detail from context. This shows attribution can provide value when execution is correct.

D.1 Detection Keywords

Generic Attribution Phrases:

•

“based on”, “according to”, “the passage”, “the context”
•

“as mentioned”, “as stated”, “from the passage”, “the article”
•

“as described”, “in the passage”, “the text states”

Specific Citation Phrases:

•

“passage 1”, “passage 2”, “passage 3”
•

“first passage”, “second passage”, “third passage”
•

“title:”, “paragraph”

Detection Method: Case-insensitive substring matching. A response is classified as “attributed” if it contains any generic attribution phrase. If it also contains a specific citation phrase, it’s classified as “specifically cited.”

D.2 Quote Accuracy Measurement

We measure quote accuracy using fuzzy string matching:

⬇

from difflib import SequenceMatcher

def compute_quote_accuracy(quote, passages):

"""

␣␣␣␣Check␣if␣quote␣appears␣in␣any␣passage.

␣␣␣␣Returns␣accuracy␣category.

␣␣␣␣"""

max_overlap = 0.0

for passage in passages:

# Normalize both strings

quote_norm = normalize(quote)

passage_norm = normalize(passage)

# Compute sequence similarity

similarity = SequenceMatcher(

None, quote_norm, passage_norm

).ratio()

max_overlap = max(max_overlap, similarity)

# Classify

if max_overlap >= 0.8:

return "exact" # High overlap

elif max_overlap >= 0.5:

return "partial" # Medium overlap (paraphrase)

else:

return "inaccurate" # Low overlap (misquote)

Thresholds:

•

$\geq$ 80%: Exact or very close quote
•

50-80%: Partial quote or paraphrase
•

$<$ 50%: Inaccurate or fabricated

D.3 Attribution Analysis Details

D.3.1 Manual Pattern Analysis

We manually analyzed 50 cases (25 wins, 25 losses) to identify qualitative patterns.

Why B1ade Wins:

Pattern	% of Wins	Description
Correct Nuance	48%	Understands what question actually asks
Complete Answer	16%	Provides comprehensive response
Least Wrong	16%	All models struggle; B1ade closest
Direct Quote	13%	Includes exact passage quote
Good Attribution	6%	Clear source citation helps

Table 14: Patterns in B1ade winning cases.

Why B1ade Loses:

Pattern	% of Losses	Description
Factually Wrong	56%	Incorrect answer to question
Incomplete	20%	Missing key information
Misunderstood Question	12%	Answered different question
Over-Interpretation	12%	Added unsupported inference
Fabricated Sources	0%	Made up citations

Table 15: Patterns in B1ade losing cases. Critical finding: B1ade loses from factual errors (56%), NOT from fabrication (0%).

D.3.2 Attribution results

Metric	B1ade	Qwen	Llama	Training
Attribution Usage	42.4%	40.9%	35.4%	36.9%
Total Attributions	2,689	2,595	2,243	—
Good Attributions	1,825	2,126	1,624	—
Bad Attributions	592	227	380	—
Success Rate	67.9%	81.9%	72.4%	—

Table 16: Complete attribution analysis across models.

Metric	B1ade	Qwen	Llama
Total Quotes	1,274	851	909
Quote Usage %	9.8%	7.0%	7.4%
Accurate Quotes	494	419	325
Quote Accuracy %	38.8%	49.2%	35.8%
Rank Improvement	+0.105	+0.098	-0.184

Table 17: Quote usage and effectiveness. B1ade quotes most frequently and has most accurate quotes in absolute terms (494).

To illustrate the attribution patterns observed in the training data, we present three representative examples from the ground truth responses. These examples demonstrate the types of attribution behaviors models learn to emulate. Example 1 and 3 show explicit source attribution using phrases like “the article states that” and “the context describes,” while Example 2 represents the majority case (63.1%) where responses provide information without explicit attribution markers.

Example 1: With Generic Attribution (36.9%)

Ground Truth: “The article states that many medicinal and recreational drugs, such as tetrahydrocannabinol (active ingredient in cannabis), caffeine, morphine and nicotine come directly from plants. These are some examples of the medicines found in plants mentioned by the author.”

Example 2: No Attribution (63.1%)

Ground Truth: “The Battle of Appomattox Court House was a battle in the final stages of the American Civil War, resulting in Confederate General Robert E. Lee surrendering his Army to Union Commander Ulysses S. Grant on April 9th 1865.”

Example 3: With Context Reference (36.9%)

Ground Truth: “The context describes events and details of his life, such as how he met Edna Stillwell in Vincennes and their early careers, the ‘Doughnut Dunkers’ routine that brought him recognition…”

Appendix E Evaluation Bias Analysis

We identify several systematic biases in judge behavior:

E.1 Length Bias

Despite explicit instruction “DO NOT prefer brevity over completeness,” we observe correlation between response length and worse ranks for some models. For example Llama’s longer responses are penalized despite judge instructions.

Model	Avg Length	Length-Rank Correlation	Bias
B1ade	538 chars	+0.084	Slight penalty
Llama	500 chars	+0.264	Strong penalty
Qwen	322 chars	-0.112	Slight reward

Table 18: Length bias analysis. Positive correlation = longer responses receive worse ranks.

E.2 Attribution Paradox

We manually found 10+ cases where B1ade explicitly attributed the right excerpt but lost to Qwen without attribution, contradicting the stated preference for source citation.

Example Case 1:

Both responses are factually identical; B1ade adds attribution phrase. Judge ranks Qwen higher despite stated preference for attribution.

Example Case 2:

In this case Qwen’s answer is in fact wrong, despite pointing to the passage. B1ade correctly answers this nuanced question very similar to the ground truth:

Example Case 3:

In this case Qwen’s answer is hallucinated. B1ade correctly and briefly answers this question. Here the ground truth is more explanatory but matches B1ade’s answer.

Appendix F B1ade-embed: Mergekit Configuration Details

F.1 Model Stock Merging Strategy

B1ade-embed employs the Model Stock algorithm Wang et al. (2024b), which optimizes linear interpolation weights through geometric properties of task vectors. Model Stock provides a principled, data-driven approach to combining multiple pretrained models without requiring additional training, making it ideal for efficient model composition.

F.1.1 Merging Configuration

The merge configuration uses Mergekit⁵⁵5https://github.com/arcee-ai/mergekit Goddard et al. (2024), an open-source toolkit for parameter-efficient model composition:

•

Merge Method: model_stock — geometric weight optimization for linear interpolation
•

Base Model: bert-large-uncased — serves as reference for task vector computation and architecture anchor
•
Auxiliary Models: Four MTEB-optimized dense retrievers:
- –
  
  WhereIsAI/UAE-Large-V1 — specialized in multilingual dense retrieval and cross-language transfer
- –
  
  BAAI/bge-large-en-v1.5 — optimized for cross-lingual semantic matching and zero-shot retrieval
- –
  
  mixedbread-ai/mxbai-embed-large-v1 — instruction-following embeddings for task-aware representations
- –
  
  avsolatorio/GIST-large-Embedding-v0 — general-purpose semantic similarity and broad domain coverage
•

Output Dimension: 1024 (matching BERT-Large architecture)
•

Precision: float32 for numerical stability when combining diverse embedding spaces
•

Tokenizer: Unified to bert-large-uncased tokenizer (30,522 vocabulary)

F.1.2 Model Stock Algorithm

Model Stock computes optimized interpolation weights using the following procedure:

1.

Task Vector Extraction: For each auxiliary model $i$ , compute the task vector as the difference from the base model:

$\tau_{i}=\theta_{i}-\theta_{\text{base}}$ (1)
where $\theta_{i}$ and $\theta_{\text{base}}$ are the flattened parameter vectors.
2.

Pairwise Similarity Computation: Calculate cosine similarities between all pairs of task vectors to measure model coherence:

$s_{ij}=\frac{\tau_{i}\cdot\tau_{j}}{\|\tau_{i}\|\cdot\|\tau_{j}\|}$ (2)
3.

Interpolation Weight Derivation: The interpolation factor $t$ is computed as the mean of pairwise similarities, which reflects how well-aligned the auxiliary models are:

$t=\frac{1}{\binom{n}{2}}\sum_{i<j}s_{ij}$ (3)
4.

Weighted Linear Merge: The final merged parameters are computed as a weighted interpolation between the base model and the average of auxiliary models:

$\theta_{\text{merged}}=t\cdot\bar{\theta}_{\text{aux}}+(1-t)\cdot\theta_{\text{base}}$ (4)
where $\bar{\theta}_{\text{aux}}=\frac{1}{n}\sum_{i=1}^{n}\theta_{i}$ is the arithmetic mean of auxiliary model parameters.

This adaptive weighting mechanism automatically balances the contribution from all auxiliary models based on their mutual alignment, avoiding manual hyperparameter tuning while maintaining base model coherence throughout the merged architecture.

F.1.3 Model Selection Rationale

The five models (including base) were selected based on complementary MTEB performance characteristics:

•

Diversity: Each model excels on different MTEB task types (retrieval, similarity, clustering, classification, reranking), ensuring broad semantic coverage
•

Efficiency: All models remain below 1024-dimensional output and compatible with BERT-Large architecture, enabling parameter-free composition
•

Performance: Collectively rank in the top tier of MTEB leaderboard within the sub-500M parameter constraint at the time of the experiment
•

Complementarity: Task vector analysis revealed high diversity (low pairwise similarity) while remaining semantically coherent, indicating each model contributes unique knowledge

The resulting 335M-parameter merged model inherits representational capabilities from all five input models without requiring additional pretraining, demonstration, or fine-tuning, making it highly resource-efficient for downstream RAG applications.