(Antonio V. Franco)

The selection of language models in production environments has historically been dominated by a single, apparently unequivocal metric: accuracy. Public leaderboards, competitions, and established benchmarks perpetuate this view by ranking models exclusively by their ability to answer a standardized set of questions or tasks correctly. In this framing, the operational cost of inference is treated as a secondary detail, often mentioned in footnotes or in technical specification sections that few consult before making an adoption decision. This approach makes sense in scenarios where the marginal processing cost is irrelevant relative to the project’s total budget, or when the evaluation is aimed exclusively at mapping the absolute frontier of a model’s intellectual capability without considering practical deployment constraints.

For small language models, however, this premise proves deeply inadequate. SLMs are, by definition, models that sacrifice parametric capacity in favor of computational efficiency, lower latency, and reduced cost per processed token. The commercial and technical argument for their adoption rests precisely on this trade-off: offering acceptable performance at a fraction of the operational cost of larger models. When evaluation ignores the cost variable, the SLM loses its very reason for existing in the selection process, being compared at a disadvantage against much more capable and expensive models on terrain it was not designed to win.

The problem, however, is even more subtle. Cost per token alone is also an insufficient metric for guiding engineering decisions. A model that charges cents per million tokens may generate excessively long, structurally invalid, or semantically incorrect answers, completely nullifying the apparent savings of its unit price. Anyone who pays for the volume of tokens produced suffers a double loss when the model is verbose and still wrong: one pays more to obtain less. Similarly, a zero-cost model that never produces correct answers has a cost per token of zero, but equally zero practical value. The efficiency metric must simultaneously consider the correctness of the answer and the cost incurred to obtain it, otherwise it risks guiding decisions that maximize the wrong metric.

The SLM Efficiency Frontier project emerges precisely from this gap. It proposes a central metric, cost per correct answer, which combines correctness and cost into a single auditable ratio, and builds around it a PyTorch-first benchmark for verifiable agentic tasks. This technical report describes the benchmark architecture, the implemented components, the results of the initial validation, and the lessons learned from building an evaluation infrastructure that places economic efficiency at the center of the model selection process.

Metric

The benchmark’s fundamental metric is formally defined as the ratio between the total estimated inference cost and the number of answers considered correct by the automatic validators for each task. The numerator is calculated from the tokens reported by each model’s API, multiplied by the publicly verified prices for that specific model at execution time. The denominator is the number of examples in which the model produced an answer that passed all validation criteria defined for that task. The resulting ratio expresses, in U.S. dollars, the average cost of each correct answer obtained during the evaluation.

This metric has a mathematical and practical property of great relevance: it jointly penalizes error and token waste. A model that is cheap per token but frequently wrong increases the numerator with each call without correspondingly increasing the denominator, resulting in a rising cost per hit. An expensive model that gets little right, even if it produces correct answers in some cases, increases the numerator faster than the denominator, becoming equally inefficient under the metric. Only models that combine low token price with a high accuracy rate are favorably positioned on the benchmark’s efficient frontier.

There is a boundary case that requires careful treatment: when the model produces no correct answers across the entire evaluation set, the denominator of the ratio becomes zero, and the division results in an infinite or undefined value. Instead of serializing the value as Infinity or NaN, the benchmark adopts a deliberate convention: non-finite values are represented as null in the strict output JSON. This decision prevents broken artifacts in leaderboards and dashboards, ensuring that any consumer of the results, including visualization interfaces, can process the artifact without requiring special exception handling. The underlying logic is that a model with zero hits does not have a cost per correct answer; it has an absence of correct answers, and this must be represented explicitly.

Cost per correct answer does not measure subjective quality, stylistic preference, or suitability for open-ended tasks. It measures verifiable efficiency: given a task with an answer objectively determinable by an automatic validator, which model delivers the correct answer at the lowest cost. This restriction is fundamental and defines the scope of the benchmark. The project does not aim to evaluate models on creative, conversational, or open-ended tasks where human judgment would be necessary. Instead, it focuses on agentic tasks with well-defined structure, where correctness can be decided by rules, schemas, exact comparisons, or discrete classifications. This choice is not arbitrary, but a necessary condition for the metric to be objective, reproducible, and auditable across the entire evaluation chain.

The metric also carries an important practical consequence for the development of real applications: it aligns the model selection process with the application’s budget. Instead of optimizing for maximum accuracy without considering marginal cost, or for minimum cost without considering quality, the engineer or data scientist can optimize for the economic efficiency of the correct answer. In scenarios where each API call has a direct cost and call volume is high, this metric offers a rational basis for deployment decisions that balance performance and the system’s financial sustainability.

Benchmark

The developed benchmark was named SLM Efficiency Frontier, and its canonical description is the following: a PyTorch-first benchmark for measuring when small language models outperform larger and inexpensive models in cost per correct answer on verifiable agentic tasks. The current version, called v0.1, is a foundational release that establishes the complete architecture of the system, including the evaluation engine, a sample dataset, local PyTorch baselines, a budget-control mechanism, automatic task validators, strict JSON serialization, and a leaderboard generated by model.

The task coverage in v0.1 spans nine distinct families, all designed to be self-scorable through programmatic validators. The first family, json_validity, validates whether the model output is well-formed JSON and compatible with the expected schema for that task. The second, tool_calling, evaluates the model’s ability to emit correct tool calls with the appropriate parameters, a fundamental skill in agentic systems. The third, short_reasoning, tests short answers to reasoning problems with exact ground truth, offering a measure of rapid inference capability. The fourth, structured_extraction, evaluates the extraction of structured fields from unstructured text. The remaining families include clause_conflict, for detecting conflicts between clauses in contracts; table_reasoning, for reasoning over tabular data; semantic_classification, for discrete semantic classification; judge_reranker, for reranking candidates through an automatic judge; and safe_action_selection, for selecting safe actions in scenarios with constraints.

All tasks share an essential property: the correctness of the answer can be determined automatically, without human intervention. Validators are implemented as functions that receive the model output, apply normalization transformations when necessary, and compare the result with the ground truth or with the validation conditions defined. For tasks such as json_validity, the validator checks whether the output string is parseable as JSON and whether it contains the expected keys and data types. For tool_calling, the validator checks whether the tool name and parameters exactly match the expected values. For short_reasoning, the validator normalizes the answer by extracting the number or short answer and comparing it with the ground truth. This diversity of validation mechanisms ensures that the benchmark is not overly dependent on a single answer format, increasing its robustness as an evaluation tool.

v0.1 remains explicitly restricted to English. This decision was made for practical reasons: the evaluated models are mostly trained on English corpora, OpenRouter prices are quoted in dollars, and the reference literature on SLM efficiency is concentrated in this language. Multilingual extensions are outside the scope of this release and will be considered in future versions if there is demand and technical feasibility. The restriction is rigidly enforced in QA tests, which reject any example in other languages, ensuring that the dataset remains homogeneous.

The project does not intend to prove the general superiority of any model in v0.1. The results obtained come from a validation run with a reduced dataset and serve only to demonstrate that the architecture works end to end: the real OpenRouter API call was exercised, the budget was respected, the output JSON remained strict, and the leaderboard was generated correctly. Any comparative conclusion between models would require a substantially larger evaluation, with more tasks, more examples per task, and a broader set of models.

Architecture

The project architecture distributes responsibilities across five distinct platforms, each fulfilling a specific role in the evaluation and publication chain. GitHub hosts the canonical implementation of the system, including all source code, documentation, configuration files, Jupyter notebooks executed on Kaggle, and automated tests. This repository is the source of truth for the project, maintaining version control over all artifacts that are not data or showcases. The Hugging Face Dataset stores the sample dataset and example schemas, offering a reproducible data package that can be loaded directly by the evaluator. The Hugging Face Space contains the public showcase, with the interactive leaderboard and the results inspection interface. Hugging Face model cards publish the PyTorch baselines that serve as calibration points, including a lightweight classifier and a cross-encoder. Kaggle is the execution environment where all notebooks are actually run, eliminating local runtime dependencies and ensuring reproducibility. Finally, OpenRouter acts as the source of remote models, with prices verified before each evaluation run.

Internally, the evaluation pipeline includes components designed for clear separation of responsibilities. BudgetGuard and CostLedger work together to control the operational budget of the evaluation. Before each API call, the guard evaluates the estimated cost based on verified prices and the average number of tokens per response, approving or blocking the call according to the available budget. The ledger records each actual call, storing the tokens consumed, the cost incurred, and the response status, in order to enable a complete audit of spending. The system supports the definition of a hard cap and a target cost; in v0.1, the cap was set at four dollars, with a target of two dollars, although the validation run used a much lower operational cap of fifty cents.

Task validators are independent components that receive the model’s raw response and return a binary decision of correctness or incorrectness, along with metadata about the validation. Each validator implements the specific logic for its task, but all follow a common interface that allows the evaluator to iterate over the examples and apply the corresponding validation. This architecture facilitates extending the benchmark with new tasks: it is enough to implement a new validator and add examples to the dataset. The separation between the evaluation engine and the validators also makes it possible to test and debug each validator independently, without needing to run the full pipeline.

Metrics per model are aggregated from individual responses and recorded costs. The system calculates overall accuracy as the proportion of correct answers over total attempts; cost per correct answer as the ratio described in the previous section; median latency in milliseconds, computed from response times reported by the API; tokens per correct answer, which measure the model’s lexical efficiency; valid JSON rate, which indicates the model’s adherence to the structured format; refusal rate, when the model refuses to answer; and overgeneration rate, when the model produces excessively long answers or irrelevant content. These complementary metrics offer a more detailed profile of the model than the central metric alone, but the leaderboard ranking is based primarily on cost per correct answer.

Strict JSON serialization is a non-negotiable requirement of the project. The system rejects any attempt to serialize infinite, negative infinite, or NaN values, replacing them with null when necessary. This policy ensures that the output artifact can be processed by all standard JSON manipulation tools without surprises. Raw results are written in JSONL (JSON Lines) format, with a release schema that includes fields such as raw_output, normalized_prediction, validator_name, error, and call metadata. This format allows incremental processing and facilitates integration with data analysis tools.

The leaderboard is generated reproducibly from the aggregated results, with deterministic tie-breakers that ensure successive runs produce the same ordering for the same data. The sorting logic prioritizes lower cost per correct answer, but when two models have null or identical values for the main metric, accuracy is used as the tie-breaker. The leaderboard is published on the Hugging Face Space, where it can be viewed interactively by any visitor.

Validation

The project’s validation methodology consists of four sequential stages: duplication research on Hugging Face, price verification in the OpenRouter catalog, synthetic-mode dry run, and smoke run with real API calls. Each stage was designed to reduce specific risks: duplication research avoids redundant effort, price verification ensures economic eligibility, the dry run validates the pipeline at no cost, and the smoke run confirms real operation under a controlled budget.

The duplication research was conducted with twenty queries over Models, Datasets, and Spaces on Hugging Face, enriching each result with a card summary and, for a subset, README excerpts. Each hit received a score from zero to one hundred against benchmark feature groups, covering theme, metric, task, cost, OpenRouter integration, SLMs, leaderboard, PyTorch ecosystem, Hugging Face, and model pool. The thirteen raw hits resulting after deduplication were all classified in the adjacent_project category, with a maximum risk score of forty-five. No exact or close duplicate was found, and the final decision was to move forward with the project.

Price verification was performed by consulting OpenRouter’s public catalog, available at the API models endpoint, and converting the values to dollars per million tokens. The eligibility criteria were established as numeric values: input price below twelve cents per million and output price below thirty cents per million. Both criteria are exclusive, meaning that missing, ambiguous, or non-numeric prices make the model ineligible. Of the sixty-five models listed in the catalog, twenty-one met the eligibility criteria. High-profile models such as Gemini 2.5 Pro and Magnum V4 72B were ineligible due to their high price, while DeepSeek V4 Flash was confirmed as eligible at 0.09 and 0.18 dollars per million for input and output, respectively.

The dry run, executed before any real call, uses an in-memory synthetic backend that does not perform API calls, assigns a zero price to non-eligible candidates for ranking purposes, and preserves the model identity. This execution processed eight models and ten examples, generating eighty synthetic results, validating strict JSON, and producing the leaderboard without incurring cost or network dependency. The dry run confirmed that the evaluation path, validators, aggregation, and serialization work as expected, serving as a sanity check before the paid execution.

The smoke run was the first real execution with OpenRouter API calls, processing five examples distributed across five task families. Each model received up to five calls, totaling twenty attempts, of which fifteen were successful. The five failures occurred due to HTTP 429, indicating a rate limit on the free layer of the dolphin-mistral-24b model. The total estimated cost of the run was around 88 millionths of a dollar, well below the operational cap of fifty cents and the target of ten cents. The ledger recorded all fifteen successful calls.

The evaluated models were deepseek/deepseek-v4-flash, qwen/qwen-2.5–7b-instruct, meta-llama/llama-3.1–8b-instruct, and cognitivecomputations/dolphin-mistral-24b-venice-edition:free. The resulting leaderboard showed Meta and Qwen models with eighty percent accuracy and costs per hit of 1.78e-06 and 3.72e-06 dollars, respectively. DeepSeek showed sixty percent accuracy with a higher cost, 2.20e-05 dollars per hit. The free model obtained zero percent accuracy and zero cost, resulting in null for the metric. It is crucial to interpret these numbers with caution: this is a smoke run with a tiny dataset, not a definitive ranking.

The final QA, executed without API calls to avoid additional costs, performed one hundred and twenty-one checks, all approved, and forty-four pytest tests, all successful. No issue was classified as BLOCKER, MAJOR, MINOR, or NIT. The final gate was READY_FOR_FINAL_PACKAGING, clearing the project for publication.

Results

The results obtained in the smoke run must be interpreted within the context of pipeline validation. The dataset used five examples, which is insufficient for any statistical inference about the relative capabilities of the models. The observed accuracy for the Meta and Qwen models, both at eighty percent, cannot be generalized to the performance of these models on other tasks or larger datasets. The difference in cost per correct answer between them, a factor of approximately two, is influenced by both token price and answer verbosity. DeepSeek, despite having the lowest latency, presented a higher cost per hit due to the combination of a slightly higher price and greater token consumption per answer. The free model obtained no correct answers not because of cognitive incapacity, but because of rate-limit failures, highlighting one of the practical challenges in evaluating free models: availability and reliability can be as relevant as monetary cost.

The operational results of the execution confirm the robustness of the pipeline under real conditions. BudgetGuard blocked calls when the target budget was reached, CostLedger recorded all costs accurately, and JSON serialization produced a strictly valid artifact. The migration of raw results to the release schema was completed successfully, and the leaderboard was generated without errors. The integration with OpenRouter worked as expected, with appropriate handling of HTTP error codes and exponential retry in the case of temporary failures, although the smoke run was not extensive enough to exhaustively test these mechanisms.

The complementary metrics provide additional insight into model behavior. The Meta model consumed 83.25 tokens per hit, while Qwen consumed 78.75, a modest difference. DeepSeek consumed twice as much, 161 tokens per hit, which contributed significantly to its higher cost. Latencies varied considerably: the Meta model had a median of 264 milliseconds, Qwen 612 milliseconds, and DeepSeek 1229 milliseconds. These differences may reflect both model architecture and provider infrastructure, and should not be attributed solely to the model itself. The valid JSON rate was one hundred percent for all models in successful calls, indicating that, at least in this task set, all produced structurally correct outputs.

The absence of overgeneration and the low refusal rate suggest that the models are reasonably calibrated for the proposed tasks. No model refused to answer any example, and no answer was considered excessively long. These results, although positive, are not surprising given the reduced size of the dataset and the relatively simple nature of the tasks. In future evaluations with more challenging tasks, these metrics are expected to show greater variability across models.

The main contribution of the smoke run is not in the numbers themselves, but in demonstrating that the architecture is functional and ready for evaluation at scale. The total cost of 88 millionths of a dollar to process fifteen successful calls suggests that the budget defined for larger evaluations, on the order of a few dollars, will be sufficient to process hundreds or thousands of examples. This economic efficiency is fundamental to the benchmark’s viability as a continuous research tool.

Limitations

v0.1 is a foundational version, and its limitations must be stated clearly to avoid inadequate expectations. The first and most obvious limitation is the size of the sample dataset. The smoke run used only five examples, and even the complete v0.1 set is considerably smaller than established benchmarks that contain thousands or tens of thousands of examples. This reduction was deliberate in order to keep evaluation costs low during the development and validation phase, but it means that the results obtained do not have the statistical power to compare models robustly. In this version, the project does not offer a definitive model ranking, and any comparative interpretation would be premature.

The second limitation concerns the published PyTorch baselines. Some of the model cards available on Hugging Face are still skeletons, meaning they contain minimal structure and documentation, but not trained model checkpoints. These baselines serve as calibration points and as demonstrations of integration with the PyTorch ecosystem, but they do not yet offer solid empirical references against which the evaluated models can be compared. The training and publication of real baselines is a pending item for future versions, and their absence currently reduces the value of the baselines as a reference.

The project’s execution model also has operational limitations. Dependence on Kaggle as the execution environment, although it ensures reproducibility, imposes runtime, memory, and network-access constraints that may not be suitable for very large evaluations. The system does not yet have a mechanism for private submissions and separation between public and private splits, which makes it susceptible to overfitting if the same examples are repeatedly used to optimize models. This is a common characteristic in benchmarks, but v0.1 does not implement the protections necessary to mitigate this risk.

The task pool and eligible model set are also limited. The nine task families, although diverse, do not cover the full spectrum of agentic applications, and some tasks have few examples. The price eligibility criterion, although rigorous, excludes models that could be relevant in specific contexts, especially those offering promotional or negotiated prices. Dependence on OpenRouter as the sole model source also restricts the generalization of results to other API platforms.

Finally, v0.1 does not offer metrics disaggregated by task family, which limits the ability to diagnose which types of tasks each model is more or less efficient on. The leaderboard aggregates all tasks, producing a single cost-per-correct-answer metric that may hide important differences in performance by domain. These limitations do not invalidate the construction carried out, but they clearly delimit the scope of what v0.1 can claim and point to directions for future evolution.

Conclusion

The SLM Efficiency Frontier project demonstrated that it is possible to build a serious, economical, auditable, and publishable benchmark around a simple and well-defined metric for small language models. For SLMs and inexpensive models, the correct focus is not the glamour of the model or its position on raw-capability leaderboards, but verifiable efficiency: the right answer at the lowest cost, under a task that can be automatically verified. The cost_per_correct_answer metric avoids the two opposite errors that dominate the discussion around small models: treating accuracy as if it were the only relevant criterion and treating token price as if it were the only determinant of effective cost. By combining both factors into a single ratio, the metric offers a rational basis for engineering decisions that balance performance and operational budget.

The implemented architecture, with clearly separated components for budget control, validation, aggregation, serialization, and publication, establishes a solid foundation for the benchmark’s evolution. Integration with platforms such as GitHub, Hugging Face, and Kaggle ensures that the project is publicly accessible, reproducible, and extensible. Price verification in the OpenRouter catalog and the restriction to eligible models ensure that the benchmark remains relevant to the real API-cost landscape, rather than reflecting theoretical or promotional prices that do not represent operational reality.

v0.1 does not settle the question of SLM efficiency. It establishes a starting point, with a functional pipeline, task validators, strict serialization, controlled budget, rigorous QA, and publication across multiple platforms. The smoke-run evaluation confirmed that the system operates as designed, respecting budget, producing valid results, and generating artifacts usable for analysis. The identified limitations are all addressable in future evolutions, and none of them compromise the validity of the fundamental approach.

The next steps include expanding the dataset with more examples per task family, creating public and private splits to reduce overfitting risk, adding more eligible models to the pool, conducting a statistically meaningful evaluation with controlled budget and repetitions, and training and publishing effective PyTorch baselines. Improvements to the Hugging Face Space to support dynamic submissions and the development of metrics by task family are also on the roadmap.

In short, the project offers a practical contribution to the language model research and engineering community: a benchmark centered on economic efficiency, with a transparent metric, open implementation, and reproducible pipeline. v0.1 serves as a proof of concept and a foundation for future evaluations, aligning model selection with the real budget constraints of production applications.

Partnerships and projects: contact@antoniovfranco.com