# Antonio V. Franco — Full Article Archive

Antonio V. Franco is an AI engineer and independent researcher. This blog publishes first-hand technical work on LLMs, SLMs, open-source models, fine-tuning, distillation, agentic AI, and practical model stack decisions.

Canonical: https://antoniovfranco.com/
Sitemap: https://antoniovfranco.com/sitemap.xml
RSS: https://antoniovfranco.com/index.xml

============================================================
ARTICLE
============================================================
URL: https://antoniovfranco.com/posts/2026/quantizing-an-llm-isnt-enough-how-to-prove-that-a-4-bit-model-still-deserves-to-go-into/
Title: Quantizing an LLM Isn’t Enough: How to Prove That a 4-bit Model Still Deserves to Go Into Production
Date: 2026-08-29
Description: When I started designing how quantization should work in Huyawo, the first question was the most obvious one: how much could I reduce the size of a model wi.
Topics: LLM quantization, GPTQ, AWQ, Model compression, Quantization evaluation, Production ML
Author: Antonio V. Franco
============================================================

 When I started designing how quantization should work in Huyawo, the first question was the most obvious one: how much could I reduce the size of a model without compromising its quality?
It’s a good question, that’s true, but I realized it wasn’t enough.
Normally, quantization is presented as a compression problem. You start with a model in FP16 or BF16, convert its weights to 8 or 4 bits, and compare the result with the original. If the model starts consuming much less memory, continues scoring well on the selected benchmarks, and does not show a very large increase in perplexity, the tendency is to consider the conversion successful.
The problem is that none of these signals, in isolation, answers the question that really matters in production: does the quantized model still do well what I had chosen the original model to do?
This difference significantly changed the way I started thinking about quantization.
In theory, reducing weights from 16 to 4 bits decreases the space used to store them by approximately four times. Total memory consumption during inference does not fall by exactly the same proportion, because the system still needs to deal with activations, KV cache, quantization scales, and other structures. Even so, the savings you can achieve can be substantial. In some cases, they are precisely what makes it possible to put a model on a smaller GPU or run locally something that previously required a more expensive machine.
That’s why techniques such as GPTQ and AWQ have become so important. Their goal is to make this reduction while causing as little damage as possible to the model’s behavior. A quantized version can look practically identical to the original model on one benchmark and perform much worse on another. In an evaluation published in 2025 with Llama and Qwen models, for example, 8-bit quantization caused relatively small losses in long-context tasks, while some 4-bit configurations showed much larger regressions. In certain cases, the drop reached 59%.
What interests me in this result is not the extreme number itself, but rather the fact that the regression is so uneven.
There is no simple rule according to which “4 bits costs X% of quality.” The effect depends on the model, the quantization technique, the calibration, and the type of task being evaluated. The same method can preserve one capability very well and noticeably degrade another.
That’s why, for me, the question stopped being “does quantization work?” and became something much more specific: does this quantization preserve the capabilities I need, in this model and in the runtime where it will actually be used?
What Really Changes When a Model Goes to 4 Bits A weight stored in FP16 can assume an enormous number of possible values within a given range. When it is represented in 4 bits, this variety is drastically reduced, meaning that only 16 levels become available within each quantization group. This means that weights that originally had different values need to be approximated to the same level.
In simplified terms, the quantizer determines a scale and maps the original weights to the discrete values that can be represented. During computation, these values are interpreted again using that scale.
The reconstructed weight remains close to the original, but it is not identical to it — this difference is the quantization error.
If all weights were equally important to the model’s behavior, it would be enough to minimize this error uniformly. Transformers, however, do not behave this way. There are weights, channels, and regions of the network that have a much greater impact on the output.
AWQ starts precisely from this observation. Instead of treating all weights in the same way, the technique uses the model’s activations to identify more sensitive channels and tries to protect them during quantization. GPTQ follows a different strategy: it uses an approximation of second-order information to choose the adjustments that introduce less error as the weights are quantized.
These techniques help a lot, but they do not eliminate an important characteristic of the problem: a small error in the weights can produce a behavioral difference much larger than its numerical size would suggest.
This happens because LLMs generate text autoregressively.
At each step, the model calculates a probability distribution for the next token. Imagine that two tokens are almost tied. A very small change in the logits may be enough to change which one is selected. The initial difference may be minimal, but the selected token becomes part of the context used in the next step. From that point on, the two executions no longer have exactly the same input sequence. As generation continues, this difference can accumulate.
That’s why you cannot conclude that a small change in the weights will necessarily produce a small change in the final response, just as you cannot assume that this change will appear uniformly.
A model may continue answering short questions very well and show a much larger regression when it needs to retain information across tens of thousands of tokens. It may preserve conversation and writing, but get worse at certain types of reasoning. Recent studies on aggressive quantization found exactly this kind of behavior: some capabilities remain relatively stable while others suffer much clearer losses.
This is also why I do not consider perplexity sufficient to decide whether a quantized model should replace the original.
Perplexity measures something important: how good the model is, on average, at predicting the next token on that dataset. But that does not mean it represents what the model needs to do within a specific application.
If I’m using an LLM to generate valid JSON, I want to know whether the quantized version continues to respect the schema. If the system works with very long documents, I want to know whether the ability to use context remains stable. If a particular class of reasoning is important to the product, it needs to appear directly in the evaluation.
A general metric can remain practically unchanged while a capability that actually matters gets worse.
A Smaller Model Is Only Better If It Continues Doing the Job The approach I started considering safer is to treat the original model as a baseline and the quantized version as a new candidate. I do not want to evaluate only whether the 4-bit model looks good. I want to know what changed compared with the model that had already been considered suitable.
This requires controlling the comparison.
The prompts need to be the same. The tokenizer needs to be the same. The generation settings as well. The criterion used to judge the response cannot change from one execution to another.
First I run the baseline.
Then I apply quantization.
Then I compare the two.
The difference matters because an isolated score can look good and still hide a regression.
Imagine a model that gets 80% of an evaluation set correct in FP16 and 79% after quantization. The first impulse would be to consider this difference irrelevant, but the average alone does not show where that one percentage point came from. It may be that the drop is distributed almost imperceptibly across the entire set. It may also be that the model preserved practically all the simple tasks and got much worse precisely on a small subset that is essential to the system.
In both cases, the final average would be similar, but the production decision should not be. That’s why I started looking with more interest at quantization evaluations that analyze several datasets and different types of capability. A study on the generalization of quantized LLMs evaluated more than forty datasets and found results showing how the choice of calibration data and method can significantly change the final result.
The number of bits is only one part of the problem, because the source model, the technique used, how calibration was done, and the type of behavior being measured also matter.
And there is still one issue that quality benchmarks, by themselves, do not solve: the reason for quantizing a model is to obtain some operational advantage.
In other words, if the 4-bit version consumes less memory but does not improve execution in a meaningful way in the environment where it will be served, the transformation may not be worth it.
Weight-only quantization, for example, greatly reduces the amount of data that needs to be moved from memory during certain operations. This can improve inference speed, but the gain depends on whether the hardware and runtime have kernels prepared for that format.
AWQ is a good example of this. The quantization technique is part of the story, but practical performance also depends on kernels capable of exploiting the reduced representation. A checkpoint taking up less disk space does not automatically mean it will respond faster.
Do you see what I mean? That’s why I don’t like using “now it fits on the GPU” as the conclusion of a quantization experiment. I want to know how much memory was actually saved during inference. I want to measure latency and throughput. I want to know whether the runtime is taking advantage of that format. And, most importantly, I want to place those gains alongside the quality regression.
If I save a lot of memory, but latency gets worse, I need to understand whether that tradeoff still makes sense.
If I gain a lot of throughput and the quality loss is small precisely in capabilities that are not critical to the application, the quantization may have been excellent.
The point is that there is no universal answer. The decision depends on the tradeoff that specific candidate offers. This distinction ended up significantly influencing the architecture I’m developing in Huyawo.
Quantization is still part of the adaptation and compression layer I’m developing, but from the initial design I did not want the process to simply end with the creation of a new checkpoint.
I want to preserve the identity of the source model, record how calibration was done, know the runtime constraints, and keep evidence of the regression caused by the transformation.
Ah! Most importantly: transformation and promotion are different decisions.
A quantization process can end technically without error and still produce a model that I do not want to put into production.
In Huyawo’s current design, the qualification stage can result in PROMOTE, REJECT, ROLLBACK, or BLOCKED. This separation may seem excessive while everything is working. It starts to make sense when a smaller, cheaper, and apparently equivalent model loses exactly a capability that was important to the application.
Quantization Needs a Regression Contract Another decision I started considering important is defining the acceptance criteria before looking at the result. If I decide what is acceptable only after seeing the numbers, it becomes very easy to adapt the interpretation to the candidate I have just produced.
Suppose I expected a loss of at most 1%, but the model lost 2%. After seeing the result, I may conclude that 2% is still not that bad. If another task gets 4% worse, I can say that perhaps that task was not so important. If the speed gain is lower than expected, I can start emphasizing memory reduction.
None of these justifications necessarily has to be dishonest. The problem is that the criteria start moving along with the result. After some time, it is no longer clear whether I am evaluating the model or trying to find a reason to accept it.
That’s why I like the idea of a regression contract.
Before quantization, I define what needs to remain true. If a certain capability needs to maintain at least a specific score, I record that threshold. If the transformation is only worth it when latency falls below a certain value, that also needs to be defined beforehand.
This contract does not need to be huge. In fact, I prefer that it isn’t.
A real application generally depends on a relatively small set of capabilities. Running fifty benchmarks just to produce a larger table does not automatically make the evaluation more useful. What I want to measure is what would actually justify approving or rejecting the new candidate.
This way of working also helps a lot when something goes wrong. If the 4-bit version fails a criterion that the baseline met, the problem becomes clearer.
Perhaps four bits are too aggressive for that model.
Perhaps another quantization method preserves the lost capability better.
Perhaps some layers need to remain at higher precision.
Perhaps the model is fine and the problem is only in the runtime implementation.
When the baseline, configuration, and criteria are known, these hypotheses can be investigated separately. Without this discipline, everything ends up mixed together in a single vague question: “did the model turn out well?”
Conclusion and Recommendations Today, generating a quantized version of an LLM is relatively simple.
Libraries such as Transformers already offer integrations with different quantization techniques, and tools such as bitsandbytes make it possible to load models in 8 or 4 bits with few changes to the code. NF4, for example, became known mainly for its use in QLoRA and was developed to represent common distributions of neural network weights more appropriately.
That’s why the hardest part of the problem is no longer necessarily producing the quantized checkpoint, but rather deciding whether that checkpoint deserves to replace the original.
To make this decision, I follow five principles:
I establish the baseline before quantizing;
I define the regression limits before knowing the result;
I directly evaluate the capabilities the application depends on;
I measure the gains in the runtime where the model will actually be used;
I treat the quantized version as a new candidate that needs to be qualified.
The last point is what changed the way I think about quantization the most.
Technically, I am compressing an existing representation.
From a production standpoint, however, it does not make sense to assume that I am still dealing with exactly the same model. The weights changed. The numerical error changed. Some generation decisions may change. And the distribution of these differences across the model’s capabilities is not uniform.
That’s why I prefer to treat the quantized version as another candidate. If I can show that it preserves what really matters and delivers a clear operational advantage, I have a good reason to promote it.
If I cannot demonstrate that, knowing that the model now uses much less memory is still a useful result.
It just isn’t, by itself, a justification for putting it into production.
Partnerships and projects: contact@antoniovfranco.com


============================================================
END OF ARTICLE
============================================================

============================================================
ARTICLE
============================================================
URL: https://antoniovfranco.com/posts/2026/deepseek-v4-flash-glm-5-2-combination/
Title: The "DeepSeek V4 Flash + GLM-5.2" Combination Is Currently Enough for Me (and Will Probably Be Enough for You Too)
Date: 2026-07-24
Description: Why DeepSeek V4 Flash handles 70% of my workload and GLM-5.2 covers the rest — a practical AI stack for serious development work
Topics: AI model stack, DeepSeek V4 Flash, GLM-5.2, Practical AI engineering
Author: Antonio V. Franco
============================================================

When I turn on my computer in the morning, one of the first things I do is open whichever Ollama Cloud account I’m currently using and check my consumption. I have three Ollama Cloud accounts, each on the US$20 plan, which gives me considerably more peace of mind regarding usage, even though I still can’t afford to consume resources recklessly.
Ollama Cloud’s billing model isn’t based solely on the number of tokens you use, but also on how demanding a model is in terms of GPU resources. In practice, this means I can get significantly more work done with GLM-5.2 (classified as “High” usage) than with DeepSeek V4 Pro (classified as “Extra High” usage).
As excellent as DeepSeek V4 Pro is, it ultimately doesn’t fit my workflow (even with three Ollama Cloud accounts available). The interesting part is that I don’t actually miss it. My current model stack consists of DeepSeek V4 Flash (an outstanding model) for roughly 70% of my work and GLM-5.2 for the remaining 30%.
Let me explain.
At Huyawo, DeepSeek V4 Flash is my primary working model. It’s the model I use to read large repositories, investigate the current state of an implementation, and turn project specifications into concrete code changes. Most of my day-to-day development flows through it, from the initial implementation to the point where a change is tested, documented, and prepared for remote execution. The word “Flash” in its name may suggest a lightweight model designed only for quick tasks. That couldn’t be further from the truth. DeepSeek V4 Flash can process remarkably large contexts, recognize relationships between distant files, and maintain a sufficiently stable chain of reasoning for work that, not long ago, would have required a much more expensive model. I don’t use it merely to save money. I use it because, most of the time, it gets the job done.
In a project like Huyawo, writing code is only part of the work. A seemingly minor modification can affect how an experiment is reproduced, how results are identified, or how a conclusion will later be supported. That means the model must understand the system as more than just the file it’s currently editing. DeepSeek V4 Flash handles this continuity exceptionally well, especially when the work has been properly divided into stages and every stage has a clearly verifiable objective.
This decomposition matters because it minimizes implicit decisions. First, I determine exactly what needs to change. Then the implementation is carried out within that scope. Afterward, the experiment produces evidence that is committed back to the repository and compared against the expected behavior. The goal isn’t to make software development unnecessarily bureaucratic, but to prevent a subjective impression that “it seems to work” from being mistaken for an implementation that is actually correct.
It’s also the model I rely on to prepare and execute Huyawo’s experiments. Since those experiments typically run on Kaggle, everything must reach the remote environment in a fully reproducible state, with clearly defined inputs and outputs that can later be audited. DeepSeek V4 Flash handles this preparation without turning every iteration into an expensive operation.
This matters because a serious benchmark doesn’t end when the first promising number appears. The result must survive repetition, comparison against a baseline, and investigation into alternative hypotheses that could explain the outcome.
That kind of work represents the majority of my workload, and that’s where roughly 70% of my usage goes.
GLM-5.2 enters the picture when the problem stops being merely large and becomes structurally difficult. I use it when understanding each component individually is no longer enough because the real issue lies in the interaction between them. It might be a refactoring that changes fundamental assumptions within the system, a state that becomes corrupted somewhere along a long execution chain, or a bug that continues to reappear despite multiple seemingly reasonable fixes.
In these situations, I’m not simply paying for a more elaborate answer. I’m paying for a model with greater capacity to sustain a complex investigation without simplifying the problem prematurely.
There are also situations where a change appears perfectly correct during a normal review, yet the cost of being wrong is simply too high. An implementation may execute successfully, generate all the expected artifacts, and even produce plausible metrics while silently violating an assumption that invalidates the entire experiment.
This is precisely where GLM-5.2 becomes particularly valuable. It isn’t just looking for what is obviously broken; it’s looking for what could quietly slip through unnoticed. Once DeepSeek V4 Flash has failed in a concrete and reproducible way, continuing to throw prompts at it stops being an economically rational decision and becomes a waste of your usage limits (I’d simply call it burning money).
The underlying principle, then, isn’t to use the most powerful model for every task.
It’s to use the most powerful model necessary for each stage of the work.
DeepSeek V4 Flash carries the volume and handles the normal pace of Huyawo’s development. GLM-5.2 takes over when the architecture, the risk associated with a modification, or the persistence of a difficult bug demands a deeper level of reasoning.
For what I’m building at Huyawo, this combination isn’t a compromise. It’s a complete AI stack.
Partnerships and projects: contact@antoniovfranco.com


============================================================
END OF ARTICLE
============================================================

============================================================
ARTICLE
============================================================
URL: https://antoniovfranco.com/posts/2026/glm-5-2-sonnet-4-5-and-open-models-were-enough/
Title: GLM-5.2: I thought Sonnet 4.5 and similar open models were enough, but…
Date: 2026-06-25
Description: A first-hand account of testing GLM-5.2 and realizing the open ecosystem has closed the gap on proprietary flagship models — in code quality, cost, and access.
Topics: Open-source LLMs, GLM-5.2, Model comparison, Open AI advocacy
Author: Antonio V. Franco
============================================================

Honestly, it had been a long time since a model surprised me like this — not since the end of 2025.
No, I do not mean it surprised me because I liked a response or was impressed by a benchmark. No, and in this article you will understand where I am going with this.
I remember the launch of Sonnet 4.5 as if it were yesterday. It was one of those moments when the developer community noticed a considerable leap in quality across all existing models — and, make no mistake, the qualitative leap was undeniable. Deeper reasoning, almost human contextual understanding, truly functional code in complex projects. In my head, I said: “Damn! I do not need anything better.” And, in a way, that was not an exaggeration. Sonnet 4.5 elegantly solved problems that previously required workarounds and patches.
That feeling of sufficiency repeated itself with the following open-source models. Then came the Qwen-3.5 family of models — excellent models, by the way! — each version pushing the limits of what was possible with a cost-benefit ratio that truly challenged what we had understood until then as common sense. Qwen3-Coder-Next, specifically, made me realize just how powerful and cheap a model can be — and understand, as I pointed out in this article, that the future of models in the industry will be SLMs. Soon after, the DeepSeek V4 models, although without as much fanfare — the community had expected their release during Chinese New Year, which did not happen — arrived commanding respect, despite the delay. With each release, I had the exact same certainty: we had reached the top of what was satisfactory for a developer. There was nothing left to evolve in what truly mattered for my daily work.
Ah… Pure mistake. And thank God it was a mistake.
When I put GLM-5.2 to work, my first reaction was: “Is it really all that?” The answer came when I analyzed the work it generated. A level of code quality and efficiency that, if there had been any doubt that a model could surprise me more than Sonnet 4.5 and its peers, that doubt vanished completely. At that moment, I understood that there was indeed room — plenty of room — for something to impress me beyond everything I had experienced.
The most fascinating part is that this perception did not come from a closed, extremely expensive, inaccessible model — ah, Fable 5… so good and so expensive… It came from a model running on infrastructure managed by a service that has already become trivial for developers. And this brings us to the elephant-in-the-room question: how close are we, those of us who bet on open models, to having in our hands a Chinese model with capabilities comparable to those of Fable 5?
The answer, surprisingly, is: closer than the proprietary software industry would like to admit. Fable 5 symbolizes, for many, the ceiling of what a closed model can offer: multimodal reasoning, analytical depth, and a veneer of almost antiseptic alignment. But, when testing GLM-5.2 side by side with DeepSeek V4 Pro and Flash variants, and with the latest weights from the Qwen-3.5 family, it becomes evident that the gap is visibly narrowing — and not through imitation, but through independent research paths. Open Chinese models are not merely keeping up; they are defining new frontiers in inference efficiency, knowledge compression, and adaptation to consumer hardware.
And what about costs? Here lies one of the pillars that makes this discussion inseparable from the defense of open artificial intelligence. GLM-5.2 can be run via Ollama Cloud with inference costs that, in many scenarios, fall below a few cents per million tokens — orders of magnitude lower than what one pays to access Fable 5 or other closed models. DeepSeek V4 Pro and Flash follow the same philosophy, with prices that allow an independent researcher like myself to run experiments that previously would have required an excellent corporate budget. The Qwen-3.5 family, in turn, is available in such a wide range of quantizations that it can be adjusted to run on a home GPU, no joke. Fable 5, by contrast, remains under the yoke of premium subscriptions, usage fees, and an access policy that decides, for you, what you are allowed to investigate.
This is exactly where I need to defend something that transcends benchmarks and cost spreadsheets. Artificial intelligence must be open and free to every human being, regardless of the use one intends to make of it. It is about understanding that real progress — the kind that distributes power instead of concentrating it — only sustains itself when the most transformative tools are free and universal.
Some will argue that total openness is dangerous, that certain uses require control and barriers. I disagree, with the conviction of someone who observes history. The conscious and ethical use of intelligence is not born from foolish laws that hinder more than they protect, nor from restrictions that presume bad faith as the default. It is born from the evolution of the principles of human society, from the collective maturation that occurs when people have unrestricted access to knowledge and can experiment, make mistakes, and correct their course in public.
Seeing GLM-5.2 operate with perfect naturalness reminded me that genius is not the monopoly of a laboratory, a country, or a business model. Genius is distributed, and the only way to honor it is to ensure that minds around the world can access it, modify it, and improve it without asking for permission. What Sonnet 4.5, Qwen3-Coder-Next, and DeepSeek V4 proved in the past — and what GLM-5.2 now proves — is that the open ecosystem is not an inferior alternative to the closed one; increasingly, it is the vanguard.
I thought Sonnet 4.5 and similar open models were enough. Fortunately, I was wrong. And, if the current pace continues, I will soon look at Fable 5 with the same distant admiration with which we look at a powerful relic.
Partnerships and projects: contact@antoniovfranco.com


============================================================
END OF ARTICLE
============================================================

============================================================
ARTICLE
============================================================
URL: https://antoniovfranco.com/posts/2026/agentic-ai-slms-why-models-above-50-cents-burning-money/
Title: Agentic AI, SLMs, and Why Models Above US$0.50 Output per 1M Tokens Are Equivalent to Burning Money
Date: 2026-06-13
Description: A first-hand look at why DeepSeek V4 Flash and Qwen3-Coder-Next make anything above US$0.50 per million output tokens a waste of money in agentic pipelines, and why SLMs are the future of industrial agents.
Topics: Small Language Models (SLMs), Agentic AI, LLM API pricing, Model cost optimization, DeepSeek V4 Flash
Author: Antonio V. Franco
============================================================

June 2026. If you are still paying more than fifty cents per million output tokens in any agentic pipeline, forgive my frankness: you are literally burning money. No, this is not an exaggeration on my part, but simply the relationship between API costs and absurd usage — because, indeed, LLMs should be used absurdly, whether you are a solo researcher like me or a startup. And here, the reason has a name, surname, and version: DeepSeek V4 Flash. And what it represents goes far beyond excellent benchmark results — it represents the consolidation of a trend that demonetizes the already psychologically consolidated commercial models and pushes the center of gravity toward SLMs, the highly specialized Small Language Models.
I have been monitoring price-per-token curves since the time when we thought it was normal to use a model like GPT-4o, which, by the way, was the first major model I used. With every Claude model released, I have been monitoring the relationship between token cost and benchmark performance, and comparing it with Chinese models. In an academic and professional context where we seek to maximize the efficiency of our activities — whatever they may be — we tend to be drawn toward what feels like magic: the DeepSeek V4 line.
The official numbers from DeepSeek itself, from its API pricing page, consulted on 06/10/2026, show:
(Antonio V. Franco) The price delta is 60 times. Sixty. The point is not that GPT-5 is bad; it is that what we previously considered “quality” — commercial LLMs — no longer fits that category, which means that any price above US$0.50 per 1M output tokens has become unnecessary.
The DeepSeek phenomenon is not isolated. The Chinese ecosystem has embraced the strategy of saturating the market with SLMs of extremely high task-specific efficiency. While the West is still discussing whether agents need ever-larger models, laboratories in Hangzhou and Beijing are already packaging SLMs for industrial tasks.
The best example I have tested so far is Qwen3-Coder-Next, released in April 2026. The model has only 3.8B active parameters, with a MoE architecture totaling 7B parameters, and was trained exclusively for multilingual code generation and editing. It is not meant to discuss philosophy, summarize emails, or plan trips, but in code, it is a major specialist.
This model crystallizes what I define as an SLM in the agentic era: it is not simply a small model, but a model with strong task alignment, trained on high-quality synthetic data and reinforced by specific RL, namely RLHF applied to tool use, debugging, and bug correction. According to its technical paper, 80% of Qwen3-Coder-Next’s gains came from the curation of coding-agent trajectories, not from the model’s size. In other words, its differentiator lies in specialization — an excellent coder, by the way — and because it is a specialist, its cost and number of parameters drop drastically compared with already traditional commercial LLMs.
The trend I observe in benchmarks points to a clear convergence: by 2027, the standard model for industrial agentic AI will be based on specific SLMs, orchestrated by a slightly larger — and cheap — “conductor” model.
First, the bottleneck of agents in production is not the raw reasoning capacity of the LLM, but reliability in repeatable tasks: calling APIs, extracting schemas, filling spreadsheets, generating queries. An SLM trained for a closed domain gets more things right and consumes fewer tokens — here already lies the benefit of using SLMs.
Second, the cost of fine-tuning also collapses. Tools such as DeepSeek-DistillKit and Qwen-Factory make it possible to generate a specific SLM from a teacher model, such as V4 Flash, in under 48 hours, using datasets of 10,000 examples and a single A100 GPU, which you, like me, can rent with perfect convenience on Runpod. The inference ecosystem for SLMs has also exploded: runtimes such as llama.cpp, vLLM, and the new ExoEngine deliver throughput of 300 tokens/s for 3B models on consumer hardware.
Third, China understood the rules of the game earlier and is flooding the market with SLM families focused on “industrial agents”: Qwen-Agent, DeepSeek-Agent, Baidu Ernie-Lite-Tool, ByteDance Seed-Executor. These models are released with permissive licenses, API prices below US$0.30, and tool benchmarks that already surpass the generalist models of 2025.
Setting a ceiling of US$0.50 output per 1M tokens is not merely a favor I did for my wallet, but a survival habit for every student, every researcher, every entrepreneur, every person who, ultimately, works on building agentic systems. Above fifty cents, you are fully burning your money — perhaps because you want to brag about being a Claude user, like iPhone users do, or perhaps simply because you do not know the real power of SLMs.
In 2026, the best model for agents is not the most famous one, nor the one with the highest benchmark score. It is the one that delivers the best cost-benefit ratio. And when I look toward 2027, I see a scenario in which the overwhelming majority of agents in factories, hospitals, and offices run on SLMs. Whoever realizes this first will have a pleasant surprise in their budget, whether they are a solo professional or a company.
Partnerships and projects: contact@antoniovfranco.com


============================================================
END OF ARTICLE
============================================================

============================================================
ARTICLE
============================================================
URL: https://antoniovfranco.com/posts/2025/qdora-explained-the-new-peft-standard-for-2025/
Title: QDoRA Explained: Why It Became the New PEFT Standard in 2025
Date: 2025-11-11
Description: QDoRA combines DoRA’s weight decomposition with 4-bit quantization for memory-efficient LLM fine-tuning. Learn the mathematics, implementation, and trade-offs.
Topics: QDoRA, PEFT, Parameter-efficient fine-tuning, LoRA, QLoRA, DoRA, LLM quantization
Author: Antonio V. Franco
============================================================

 (Freepik/Antonio Franco) The landscape of parameter-efficient fine-tuning has been going through a quiet revolution since 2021, and most people haven’t noticed yet. While LoRA democratized LLM fine-tuning by drastically reducing memory requirements — letting researchers train models on consumer GPUs instead of needing expensive clusters — it always had this nagging problem: it couldn’t quite match the performance of full fine-tuning. The gap wasn’t huge, maybe 2–5% on most benchmarks, but it was there. And in competitive research or production systems, that gap matters.
Enter QDoRA. I know, I know, another acronym in the PEFT zoo. But this one’s different. QDoRA combines the mathematical elegance of weight decomposition with aggressive quantization in a way that not only matches full fine-tuning performance but sometimes actually exceeds it, all while using less memory than QLoRA. If that sounds too good to be true, I thought so too until I started digging into the math and running experiments.
In this deep dive, I’m going to dissect QDoRA’s architecture from first principles, explore the mathematical foundations that make it work, and show you exactly how I’m planning to leverage it for Uranus-3B — my specialized 3-billion-parameter language model focused on quantum physics and multidimensional theories. This isn’t going to be one of those surface-level “here’s how to pip install” tutorials. We’re going deep, but I promise it’ll be worth it.
The Evolution LoRA Let’s start with where it all began. Low-Rank Adaptation, introduced by Microsoft Research in 2021, revolutionized fine-tuning by exploiting a deceptively simple insight: most fine-tuning updates operate at a low intrinsic rank. In other words, when you fine-tune a massive language model, you’re not really using all those billions of parameters in a truly independent way. The updates you’re making can be approximated by a much lower-dimensional transformation.
The mathematical formulation is elegant. Instead of updating all model parameters directly, LoRA freezes the pre-trained weight matrix W₀ and introduces trainable low-rank decomposition matrices. The forward pass becomes h = W₀x + BAx, where B and A are these low-rank matrices with dimensions d×r and r×k respectively, and crucially, r is much smaller than both d and k (typically between 8 and 64). The term BA approximates the weight update ΔW, but instead of storing and training a d×k matrix, you’re only working with (d+k)×r parameters. For large matrices, this can mean reducing trainable parameters by a factor of 10,000 while maintaining roughly 99% of full fine-tuning quality.
QLoRA Now, LoRA was great, but it still required keeping the full base model in memory, which meant you needed beefy GPUs even for 7B parameter models. QLoRA came along and said “what if we quantize the frozen base model to 4-bit precision?” The innovation wasn’t just in quantization itself — people had been quantizing neural networks for years — but in how QLoRA made it work with LoRA adapters.
The secret sauce was NormalFloat4, or NF4, a data type specifically designed for normally distributed weights. Most neural network weights follow a roughly normal distribution, so NF4 allocates its 16 possible values (4 bits = ²⁴ = 16 values) non-uniformly to match this distribution. You get better precision where most weights cluster around zero and less precision at the tails where weights are rare anyway. Beyond NF4, QLoRA introduced double quantization (quantizing the quantization constants themselves), paged optimizers to handle memory spikes during backpropagation, and a clever trick where computation happens in BFloat16 even though storage is in 4-bit.
The result? You could fine-tune a 65B parameter model on a single 48GB GPU. That was revolutionary. But — and this is the important part — QLoRA still exhibited the same fundamental limitation as LoRA. The learning pattern was different from full fine-tuning. The performance was close, often within a few percentage points, but that gap persisted. And nobody quite understood why until DoRA came along.
DoRA The breakthrough came in February 2024 when a team of NVIDIA researchers led by Shih-Yang Liu published a paper that would fundamentally change how we think about efficient fine-tuning. The paper was later accepted as an oral presentation at ICML 2024, which tells you something about how important the community thought it was — only about 1.5% of ICML submissions get selected for oral presentations.
The core insight came from a careful analysis of how LoRA and full fine-tuning actually update weights during training. The researchers decomposed weights using vector normalization: W = m · (V / ||V||), where m represents the magnitude (just the L2 norm of the weight vector) and V / ||V|| represents the directional unit vector. This decomposition might sound simple — and mathematically it is — but it revealed something profound about the difference between LoRA and full fine-tuning.
When they looked at how full fine-tuning updates weights, they found that magnitude and direction changed independently. The correlation between magnitude changes and directional changes was negative, hovering around -8.0. This means that when the magnitude increased, direction often shifted in a way that partially compensated, and vice versa. It’s like the model was able to adjust the scale of features independently from their semantic content.
LoRA, on the other hand, showed a positive correlation around +2.0. Magnitude and direction changed in lockstep. If LoRA increased the magnitude, it also pushed the direction in the same way proportionally. This coupled updating pattern meant LoRA lacked the nuanced capability for subtle adjustments — it couldn’t do things like make a small directional change while making a large magnitude shift, or vice versa. And the researchers suspected this limitation stemmed from asking LoRA to learn both magnitude and directional adaptation simultaneously, which was apparently too complex a task for the low-rank constraint.
So they proposed DoRA: Weight-Decomposed Low-Rank Adaptation. The architecture decomposes the pre-trained weight and applies LoRA only to the directional component. The new weight becomes W’ = m · (V₀ + BA) / ||V₀ + BA||, where m is now a trainable magnitude vector (one scalar per output dimension), V₀ is the frozen pre-trained directional component, and BA are the familiar LoRA matrices. The normalization denominator ||V₀ + BA|| ensures the directional component remains unit-normalized while LoRA learns directional updates.
The elegance of this approach becomes clear when you think about gradient flow. The gradient with respect to magnitude is straightforward: ∂L/∂m = ∂L/∂W’ · (V + BA) / ||V + BA||. You’re just scaling the direction by the loss gradient. But the gradient with respect to the LoRA matrices involves the chain rule through that normalization term, which creates this interesting dynamic where directional updates are inherently decoupled from magnitude. The math works out to ∂L/∂B = ∂L/∂W’ · [m · (I/||V + BA|| — (V + BA)(V + BA)ᵀ/||V + BA||³)] · A, and that middle term effectively projects the gradient onto the tangent space of the unit sphere. If you’re familiar with optimization on manifolds, you’ll recognize this pattern.
What’s remarkable is that DoRA doesn’t add inference overhead. After training, you can merge the magnitude and the LoRA-adjusted direction back into a single weight matrix, exactly like you can with standard LoRA. The decomposition is purely a training-time construct that shapes the learning dynamics to be more like full fine-tuning.
QDoRA So DoRA was impressive, but it still had the same memory footprint as regular LoRA — you needed to keep the full precision base model in memory. This is where Answer.AI enters the story. Led by Jeremy Howard (co-founder of fast.ai) and featuring exceptional work by Kerem Turgutlu, they asked the obvious question: what if we combine DoRA’s weight decomposition with QLoRA’s quantization?
The result, which they released in April 2024, was QDoRA. The architecture is conceptually straightforward but the implementation details matter enormously. You start with a 4-bit quantized base model — either using bitsandbytes NormalFloat4 or Half-Quadratic Quantization depending on your needs. Then you apply DoRA’s magnitude-direction decomposition to these quantized weights. The magnitude parameters stay in full precision (they’re just scalars per output dimension, so the memory cost is negligible), and the LoRA adapters operate in BFloat16.
The forward pass involves dequantizing the base weights on the fly, applying the LoRA transformation to get the directional update, normalizing to maintain unit direction, then scaling by the magnitude. In pseudocode, it looks something like this: you dequantize W_quantized from NF4 to BF16, compute direction = V + BA where V is the dequantized weight and BA is your LoRA update, normalize it with direction_norm = direction / ||direction||, scale by magnitude m * direction_norm, and finally do your matrix multiplication. The key is that the base weights stay quantized in storage; you only dequantize temporarily for computation.
Let’s talk numbers because they’re striking. For a single linear layer with dimensions d×k, you need roughly d×k×0.5 bytes for the quantized base (4 bits per parameter), d parameters for magnitude in FP32, and (d+k)×r parameters for the LoRA matrices in BF16. The total trainable parameters end up being around 2–3% of the base model size. Taking Llama-2–7B as a concrete example: the base model in 4-bit takes about 3.5 GB, the QDoRA adapters add maybe 300 MB, and your total training memory footprint lands around 12–16 GB. Compare that to 112 GB for full fine-tuning and you start to see why this matters.
But here’s the thing that really caught my attention when I first saw Answer.AI’s results: QDoRA didn’t just match full fine-tuning performance. On the Orca-Math dataset, which tests mathematical reasoning, QDoRA with 100K training samples achieved 31.2% exact match accuracy. Full fine-tuning got 26.0%. QLoRA managed 11.8%. Even full fine-tuning followed by post-quantization — a common deployment strategy — only hit 16.8%. QDoRA outperformed full fine-tuning by nearly 5 percentage points while using roughly nine times less memory. That’s not supposed to happen. Efficient methods aren’t supposed to beat the gold standard.
The explanation, I think, lies in how DoRA’s decomposition interacts with quantization noise. Regular QLoRA asks the low-rank adapters to compensate for both the limitations of the low-rank constraint and the errors introduced by quantization. That’s a lot to ask. DoRA’s magnitude parameters provide an additional degree of freedom that can absorb some of the quantization noise, letting the directional component focus purely on learning the semantic transformations. It’s speculation on my part, but the empirical results are hard to argue with.
Deep Dive: Mathematical Foundations Let’s dig deeper into why this actually works. The mathematical foundations of DoRA draw inspiration from Weight Normalization, a technique introduced by Salimans and Kingma back in 2016. Their original idea was to reparameterize neural network weights as W = g · (v / ||v||), where g is a scalar and v is a vector. This reparameterization decouples the norm of the weight from its direction, and it turns out this improves gradient conditioning — the gradients flow more smoothly during training, leading to faster convergence.
DoRA extends this concept in three crucial ways. First, instead of a single scalar g for the entire weight matrix, DoRA makes the magnitude learnable per output dimension, giving you a vector m of scalars rather than just one global scale. Second, and more importantly, DoRA applies LoRA to the directional component v, letting you learn directional updates efficiently without storing full-rank matrices. Third, the normalization is maintained dynamically through the denominator ||V + BA||, which ensures that no matter how LoRA adjusts the direction, you’re always working with unit vectors.
The gradient flow tells you a lot about why this works better than regular LoRA. When you compute gradients with respect to the magnitude m, you get ∂L/∂m = ∂L/∂W’ · (V + BA) / ||V + BA||. This is clean and direct — the magnitude is learning to scale the effective direction based purely on the loss gradient. There’s no complicated interaction with the directional learning.
The gradient with respect to the LoRA matrices is more interesting. Because of the normalization, you need the chain rule: ∂L/∂B = ∂L/∂W’ · [m · (I/||V + BA|| — (V + BA)(V + BA)ᵀ/||V + BA||³)] · A. That middle term might look intimidating, but geometrically it’s doing something elegant — it’s projecting the gradient onto the tangent space of the unit sphere. If you’ve studied Riemannian optimization or manifold learning, you’ll recognize this as the natural gradient for optimization on the unit sphere. The practical effect is that magnitude adjustments and directional updates become decoupled in gradient space, allowing independent optimization of scale and semantics.
One of the most surprising findings from the DoRA paper was that the method exhibits superior performance at lower ranks compared to LoRA. When you drop LoRA down to rank 8, quality typically degrades significantly. But DoRA maintains quality even at rank 8, sometimes outperforming LoRA at rank 32. My interpretation is that DoRA’s independent magnitude tuning compensates for the reduced expressiveness of the low-rank directional component. You’re effectively getting more representational power without actually adding more parameters to the directional adaptation. The magnitude vector provides an additional degree of freedom that captures something LoRA’s coupled updates can’t.
Practical Implementation with PEFT The Hugging Face PEFT library makes QDoRA remarkably accessible, which is both impressive and slightly dangerous — it’s easy to get something running without understanding what’s actually happening under the hood. Let me walk you through a proper implementation that actually works well in practice, not just in documentation examples.
First, you need to configure your quantization carefully. The BitsAndBytesConfig is where most people make their first mistake. You want load_in_4bit set to True obviously, but the crucial parameter is bnb_4bit_compute_dtype — set this to torch.bfloat16, not float16. BFloat16 has better numerical stability for the kinds of operations QDoRA does during training, and while it’s technically lower precision than float16 in terms of mantissa bits, the wider exponent range matters more here. Double quantization (bnb_4bit_use_double_quant) should definitely be True — it quantizes the quantization constants themselves, saving another chunk of memory with basically no quality impact. And use NF4 (nf4) as your quantization type, not fp4. NF4 is information-theoretically optimal for normally distributed weights, which is exactly what you have in most language models.
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training from transformers import AutoModelForCausalLM, BitsAndBytesConfig import torch bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 ) model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", quantization_config=bnb_config, device_map="auto" ) model = prepare_model_for_kbit_training(model) Now for the LoRA configuration. This is where DoRA differs meaningfully from regular LoRA in terms of hyperparameters. Start with a lower rank than you’d use for LoRA — if you were planning r=32 for LoRA, try r=16 for DoRA. The independent magnitude optimization means you need less directional expressiveness to achieve the same quality. Your alpha should stay at 2× your rank as usual, but pay attention to dropout. DoRA benefits from slightly higher dropout than LoRA, something around 0.05 to 0.1 instead of the 0.0 to 0.05 you might use for regular LoRA. This helps prevent the magnitude parameters from overfitting, which can happen because they’re full precision and not constrained by a low-rank bottleneck.
Target all the linear layers in attention and the MLP blocks. Some tutorials will tell you to just target q_proj, k_proj, and v_proj in attention, but DoRA actually benefits from more comprehensive coverage. Hit the output projection, the gate projection, everything. The memory cost is still reasonable because of the low rank, and you get better quality. The critical parameter, of course, is use_dora=True. Without that, you’re just doing regular QLoRA.
lora_config = LoraConfig( r=16, lora_alpha=32, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj" ], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", use_dora=True ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() When you run print_trainable_parameters(), you’ll typically see something like 0.6–0.8% trainable parameters for a 7B model with these settings. That’s 41–55 million parameters out of ~7 billion, which is astonishingly small but entirely sufficient for most fine-tuning tasks.
The hyperparameter choices matter more for DoRA than for regular LoRA, and this is where experience helps. Learning rate should be roughly 0.5 to 0.7 times what you’d use for LoRA. If you’re used to 2e-4 for LoRA, try 1e-4 for DoRA. The reason is that DoRA converges faster — the decoupled magnitude and direction optimization reaches good solutions more quickly, and if your learning rate is too high, you’ll overshoot and get instability. I’ve seen training runs diverge at epoch 2 or 3 with learning rates that would have been fine for LoRA.
Rank selection follows a different logic too. Use roughly half to two-thirds of the rank you’d choose for LoRA. DoRA at r=16 often matches or beats LoRA at r=32 because of that independent magnitude tuning I keep harping on about. Lower ranks also mean faster training and less memory, which compounds the benefits. And as I mentioned, slightly higher dropout helps — 0.05 to 0.1 versus 0.0 to 0.05 for LoRA. You’re guarding against magnitude overfitting, which is a real failure mode that doesn’t exist in regular LoRA.
Case Study Let me get personal for a moment and talk about why all of this matters for my own work. I’m building Uranus-3B, a specialized 3-billion-parameter language model focused on quantum physics, advanced mathematics, sacred geometry, and multidimensional theories. It’s an unusual mix of topics, I know, but there’s a method to it. The intersection of rigorous physics and more esoteric theoretical frameworks requires a model that can handle both the mathematical precision of quantum field theory and the symbolic reasoning of sacred geometry without treating them as incompatible domains.
The project is currently in data collection and curation, with a target corpus of 50 to 100 GB of high-quality domain-specific content. The base architecture will be modified from Qwen 2.5 3B, with an extended vocabulary to properly handle quantum physics notation and mathematical symbols. We’re targeting 32K token context length and training on RunPod H100 instances when we get to the actual pre-training phase.
Here’s where QDoRA becomes absolutely critical for the project. While Uranus-3B will be pre-trained from scratch on the initial corpus, the real value comes from continued pre-training and domain adaptation on increasingly specialized sub-corpora. This is where most specialized models fail — they either overfit to their narrow domain and forget general capabilities, or they never achieve true specialization because full fine-tuning on small domain-specific corpora isn’t practical. QDoRA offers a way out of this dilemma.
My plan involves three phases of continued pre-training, each narrowing the focus while building on the previous phase. Phase one establishes a general scientific foundation with about 30 GB of physics textbooks, mathematical proofs, and scientific papers. This phase uses full training with gradient checkpointing — we’re still working with a large, diverse corpus, so the benefits of PEFT methods aren’t as pronounced yet. I’m estimating roughly 100 hours on 4×H100 for this phase.
Phase two is where QDoRA enters. This is quantum physics specialization using about 15 GB of quantum mechanics, quantum field theory, and quantum information theory texts. Here I’ll use QDoRA with rank 32, targeting all linear layers. The memory savings compared to full continued pre-training are around 60%, which means instead of needing a full cluster, I can run this on 2×H100 instances. More importantly, the DoRA decomposition helps preserve the general scientific knowledge from phase one while adding quantum-specific capabilities. I’m expecting this to take about 24 hours.
Phase three is the most delicate: integrating the esoteric content without destroying the scientific rigor built in phases one and two. This is roughly 5 GB of sacred geometry, hermetic principles, and multidimensional theory texts. For this phase, I’ll drop to rank 16 — the smaller corpus doesn’t justify higher rank, and the lower rank actually helps prevent overfitting to what is inevitably a more idiosyncratic dataset. The goal here isn’t to make the model believe in any particular metaphysical framework, but to make it fluent in the symbolic languages and reasoning patterns used in these domains. Eight hours on a single H100 should be sufficient.
Beyond the phased training, QDoRA enables something I’m calling task-specific adapters. The idea is to train multiple separate adapters for different use cases, then swap them at inference time. For mathematical proof generation, I’m planning an adapter with rank 24 targeting the attention layers (q_proj, k_proj, v_proj, o_proj). For quantum circuit design, rank 16 targeting attention and the MLP up-projection should work. For symbolic reasoning, which is the most demanding task, rank 32 targeting everything. Each adapter ends up being less than 500 MB, and you can hot-swap them based on what the user is trying to do.
But here’s where it gets interesting. I’m experimenting with what I’m calling hierarchical or multi-resolution QDoRA, where different layers use different ranks. The hypothesis is that deeper layers require higher expressiveness for complex quantum reasoning, while early layers can use more aggressive compression. My initial plan assigns rank 8 to layers 0–7 (input layers capturing surface patterns), rank 24 to layers 8–19 (middle layers handling semantic representations), rank 48 to layers 20–27 (deep layers doing abstract reasoning), and rank 64 to layers 28–31 (output layers where generation quality is most critical). This is speculative — I haven’t seen anyone try this systematically — but the theoretical motivation seems sound.
For the quantization strategy, I’m planning to use Half-Quadratic Quantization instead of bitsandbytes NF4, and there are specific technical reasons for this choice. HQQ preserves outliers better, which matters enormously when you’re dealing with mathematical notation and special symbols. In natural language, weight distributions are relatively well-behaved, but add in quantum physics equations with unusual Unicode characters and the distribution gets heavier tails. HQQ’s approach to handling these outliers means less information loss during quantization. There’s also a practical speed advantage — HQQ dequantization is about 15–20% faster than NF4, which compounds over a long training run. Finally, HQQ lets you configure the group size for quantization, so you can tune the granularity. For Uranus-3B, I’m planning to use group_size=64 rather than the default 128, accepting slightly higher memory cost for better preservation of those mathematical outliers.
from transformers import HqqConfig hqq_config = HqqConfig( nbits=4, group_size=64, quant_zero=True, quant_scale=True, axis=1, offload_meta=True ) model = AutoModelForCausalLM.from_pretrained( "QuanyuanAI/uranus-3b-base", quantization_config=hqq_config, device_map="auto", torch_dtype=torch.bfloat16 ) The training loop needs careful attention to gradient accumulation because we’re memory-constrained even with QDoRA. My plan is to use batch size 1 per device with 32 gradient accumulation steps, giving an effective batch size of 32. This is small enough to fit in VRAM but large enough that optimization is stable. Gradient checkpointing will be enabled, obviously, but there’s a subtlety here that people miss. DoRA’s normalization operation interacts poorly with naive gradient checkpointing because you’re recomputing that denominator ||V + BA|| multiple times, and floating point rounding means the recomputed value isn’t bitwise identical to the original. This can cause optimization instabilities. The solution is selective gradient checkpointing that skips the DoRA layers themselves.
import torch from torch.optim import AdamW from transformers import get_cosine_schedule_with_warmup batch_size = 1 grad_accum_steps = 32 gradient_checkpointing = True optimizer = AdamW( model.parameters(), lr=5e-5, betas=(0.9, 0.999), weight_decay=0.01 ) scheduler = get_cosine_schedule_with_warmup( optimizer, num_warmup_steps=500, num_training_steps=10000 ) model.train() for epoch in range(num_epochs): for step, batch in enumerate(train_loader): outputs = model(**batch) loss = outputs.loss / grad_accum_steps loss.backward() if (step + 1) % grad_accum_steps == 0: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step() scheduler.step() optimizer.zero_grad() if (step + 1) % 100 == 0: magnitude_stats = log_qdora_magnitudes(model) wandb.log(magnitude_stats) That call to log_qdora_magnitudes is something I haven’t seen many people discuss but it’s crucial for QDoRA training. Unlike regular LoRA where you’re basically watching loss curves and maybe looking at gradient norms, QDoRA training benefits enormously from monitoring the magnitude parameters themselves. You want to track mean, standard deviation, max, and min of the magnitude vector for each DoRA layer. What you’re watching for is magnitude collapse (all values approaching zero, which means the layer is effectively learning nothing) or magnitude explosion (values diverging, which means instability is coming). You also want to see that different layers have different magnitude distributions — if they’re all identical, something’s wrong with your initialization or learning rate.
def log_qdora_magnitudes(model): magnitude_stats = {} for name, module in model.named_modules(): if hasattr(module, 'magnitude'): m = module.magnitude.data magnitude_stats[f"{name}_mean"] = m.mean().item() magnitude_stats[f"{name}_std"] = m.std().item() magnitude_stats[f"{name}_max"] = m.max().item() magnitude_stats[f"{name}_min"] = m.min().item() return magnitude_stats Based on the DoRA and QDoRA benchmarks I’ve studied and my preliminary experiments, I’m expecting some specific performance benefits for Uranus-3B. Memory-wise, full continued pre-training would need about 40 GB of VRAM per GPU. With QDoRA, that drops to roughly 14 GB, which is a 65% reduction and means I can train on a single RTX 4090 instead of needing multiple A100s. The cost savings are substantial — we’re talking the difference between affordable and not remotely affordable for an independent research project.
Training speed is interesting. QDoRA has about 15% overhead per step compared to QLoRA because of the additional magnitude operations and the normalization. But because I can use a lower rank (r=16 versus r=32), the total time per step ends up being about 20% faster overall. The per-epoch time is lower even though the per-operation time is slightly higher. For quality metrics, I’m projecting roughly 8–12% improvement over QLoRA on mathematical reasoning tasks, 15–20% better accuracy on quantum physics notation, and 95–98% alignment with the specialized corpus compared to what full fine-tuning would achieve. These are educated guesses based on the Answer.AI results, but I’m confident the direction is right.
For inference, I’ll merge the QDoRA weights after training for production use. Merged weights have no runtime overhead compared to the base model, which is critical for deployment. If I keep the adapters separate for flexibility, there’s a 5–8ms overhead per forward pass, which is acceptable for research use but not for production. For Uranus-3B’s target use cases — detailed quantum physics explanations, mathematical proof assistance, symbolic reasoning over esoteric texts — merged weights are definitely the way to go.
Advanced Topics and Best Practices When you start scaling QDoRA beyond single-GPU training, you need to think carefully about how to combine it with distributed training strategies. Fully Sharded Data Parallel, or FSDP, is the current state-of-the-art for training really large models, and it turns out QDoRA plays nicely with it. The key is that FSDP shards both the model parameters and the optimizer states across GPUs, which means each GPU only needs to hold a fraction of the full model. When you combine this with QDoRA’s already reduced memory footprint, you can train surprisingly large models on relatively modest clusters.
The FSDP configuration needs careful tuning for QDoRA. Use FULL_SHARD as your sharding strategy — this gives maximum memory savings. Mixed precision should be set to BFloat16 for parameters, reductions, and buffers. Don’t use FP16; BFloat16’s wider exponent range matters for the magnitude operations in DoRA. Enable backward prefetching to overlap communication and computation, and keep CPU offloading disabled unless you absolutely have to use it. CPU offloading kills your training speed, and with QDoRA’s already reasonable memory use, you usually don’t need it. This configuration lets you train 70B models with QDoRA on four H100s with 80GB each, which would be completely impossible with full fine-tuning.
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.fsdp import MixedPrecision, BackwardPrefetch from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy fsdp_config = { "sharding_strategy": "FULL_SHARD", "mixed_precision": MixedPrecision( param_dtype=torch.bfloat16, reduce_dtype=torch.bfloat16, buffer_dtype=torch.bfloat16, ), "backward_prefetch": BackwardPrefetch.BACKWARD_PRE, "cpu_offload": False, } model = FSDP(model, auto_wrap_policy=transformer_auto_wrap_policy, **fsdp_config) Catastrophic forgetting is a real problem when you’re doing continued pre-training on specialized corpora, and QDoRA’s magnitude-direction decomposition actually helps mitigate this naturally. The intuition is that magnitude captures something like the importance or scale of features, while direction captures their semantic content. When you train on a new domain, the directional component adapts to the new semantics, but the magnitude can remain relatively stable, preserving the importance structure from the original training. That said, you can explicitly regularize against forgetting with a few strategies.
Magnitude regularization is the simplest approach. You penalize large deviations of the magnitude parameters from their initialization values. The regularization loss is just the L2 norm of the difference between current magnitudes and initial magnitudes, scaled by a small lambda (typically 0.01). Add this to your task loss during training. This keeps the magnitude parameters from drifting too far, which helps preserve the original model’s capability distribution across layers. It’s a soft constraint, not a hard one, so the model can still adjust magnitudes when it really needs to, but there’s a penalty for doing so.
def magnitude_regularization_loss(model, lambda_mag=0.01): reg_loss = 0 for module in model.modules(): if hasattr(module, 'magnitude') and hasattr(module, 'magnitude_init'): reg_loss += lambda_mag * torch.norm( module.magnitude - module.magnitude_init, p=2 ) return reg_loss total_loss = task_loss + magnitude_regularization_loss(model) For more sophisticated forgetting mitigation, you can use Elastic Weight Consolidation specifically on the magnitude parameters. The idea is to compute the Fisher information matrix for the magnitudes after your initial training phase, which tells you which magnitude parameters were most important for the original task. Then, in subsequent training phases, you add an EWC penalty that scales with the Fisher information — parameters that were important for the original task get heavily penalized if they change much, while less important parameters can adapt freely. This is more computationally expensive than simple magnitude regularization, but it’s much more targeted. For Uranus-3B’s phased training, I’m planning to use this between phases two and three to preserve the quantum physics capabilities while adding the esoteric content.
After training your QDoRA adapters, you might want to compress them further for deployment. The standard approach is to merge the adapters back into the base model, then apply post-training quantization to the merged result. GPTQ quantization works well here, especially with activation-aware quantization enabled. You train the quantizer on a small calibration dataset (a few thousand samples from your domain), then save the quantized model. The result is typically around 2 GB for a 3B parameter model, down from 6 GB for the unquantized merged model, with minimal quality loss — usually less than 1% on most benchmarks. The trick is to use a calibration dataset that’s representative of your actual use cases. For Uranus-3B, I’d use a mix of quantum physics problems, mathematical proofs, and symbolic reasoning examples.
For production deployment, inference optimization becomes critical, and you have a few options depending on your requirements. The cleanest approach is to simply merge the adapters back into the base model after training. This gives you a standard model with no overhead at inference time — literally zero. The merged weights are just the original pre-trained weights plus the learned updates, and you serve them exactly like you’d serve any other model. The alternative is to keep adapters separate and load them dynamically, which gives you flexibility (you can swap adapters for different tasks) but costs you roughly 12% in throughput because of the additional LoRA matrix multiplications and the normalization operation.
For really optimized inference, vLLM is the way to go. It’s a production-grade serving framework that implements continuous batching, PagedAttention for efficient KV cache management, and CUDA graphs for minimal Python overhead. The basic configuration sets tensor parallel size based on your GPU count, uses BFloat16 for computation, sets max model length to your context window, and aggressive GPU memory utilization (0.95 is usually safe). With vLLM and merged QDoRA weights, you can typically achieve around 2,000–2,100 tokens per second on a single A100 for a 3B parameter model. Unmerged adapters drop that to around 1,850 tokens per second, which is the 12% overhead I mentioned. The 1% overhead for merged QDoRA versus an unquantized base model is basically negligible and within measurement noise.
from vllm import LLM, SamplingParams llm = LLM( model="QuanyuanAI/uranus-3b-qdora-merged", tensor_parallel_size=2, dtype="bfloat16", max_model_len=32768, gpu_memory_utilization=0.95 ) outputs = llm.generate(prompts, sampling_params) For systems that need to dynamically route between multiple task-specific adapters, you can implement a router that uses a small classifier to predict which adapter to use based on the prompt, then loads that adapter before generation. The classifier can be something simple like a fine-tuned BERT model with 110M parameters, or even just a keyword-based heuristic if your tasks are sufficiently distinct. For Uranus-3B, I’m planning a learned router that classifies prompts into mathematical, quantum physics, or symbolic reasoning categories, then loads the appropriate adapter. The classification overhead is around 2–3ms, which is acceptable given the quality benefits of task-specific adapters.
Common failure modes in QDoRA training are worth discussing because they’re not always obvious from the loss curves. Magnitude explosion is the most dramatic — training loss will converge normally for the first epoch or two, then suddenly diverge. When you look at the magnitude parameters, you’ll find some of them have grown to values like 100 or 1000, which makes the effective weights huge and breaks optimization. The fix is either magnitude clipping (clamp values to a reasonable range like 0.1 to 10.0) or higher weight decay specifically on magnitude parameters. I prefer the weight decay approach because it’s softer — parameters can grow if they really need to, but there’s a penalty.
for name, param in model.named_parameters(): if 'magnitude' in name: param.data.clamp_(min=0.1, max=10.0) Slow convergence is more subtle. Validation loss plateaus early and just sits there, refusing to improve. This usually means your learning rate is too low or your rank is too small for the complexity of your adaptation task. The solution is a better learning rate schedule — use warmup with a cosine decay, but make the warmup longer for DoRA than you would for regular LoRA (maybe 500–1000 steps instead of 100–200). The extended warmup gives the magnitude and direction parameters time to find their relative scales before you start decaying the learning rate.
VRAM spikes during training can cause out-of-memory errors at seemingly random steps, which is frustrating to debug. The culprit is usually gradient checkpointing interacting poorly with DoRA’s normalization operation. The naive checkpointing implementation recomputes the forward pass during the backward pass, but the normalization ||V + BA|| involves non-deterministic floating point operations that give slightly different results on recomputation. This creates gradient noise that can cause memory spikes. The fix is selective gradient checkpointing that skips DoRA layers — you checkpoint the other operations but let DoRA layers keep their activations in memory. It uses a bit more VRAM but prevents the spikes.
Inference latency with unmerged adapters sometimes surprises people because it’s slower than expected. The issue is that DoRA’s forward pass involves a matrix addition (V + BA), a normalization operation (dividing by ||V + BA||), and a magnitude scaling (multiplying by m), and these are three separate CUDA kernel launches. Modern GPUs have amazing throughput but kernel launch overhead is still non-negligible. If you absolutely need to keep adapters separate for flexibility, compile the model with torch.compile using mode “reduce-overhead”, which fuses operations and reduces kernel launches. This gets you back maybe 60–70% of the overhead. Or just merge the adapters and take the zero-overhead path.
The Future of QDoRA and PEFT The research community is already building on QDoRA’s foundations in several promising directions. Answer.AI, along with collaborators in the open-source community, is working on custom CUDA kernels specifically optimized for QDoRA’s forward and backward passes. The current implementation uses separate kernel launches for dequantization, LoRA application, normalization, and magnitude scaling, but a fused kernel could theoretically achieve 5–10× speedup by doing all these operations in a single pass. There’s also active work on integrating QDoRA more tightly with Flash Attention, which would reduce memory further and speed up the attention mechanism during training.
Dynamic rank allocation is another area that’s starting to get attention — pun intended. Instead of choosing a single rank for all layers, you could use neural architecture search to find the optimal rank distribution across the model. Or, even more interestingly, you could make the rank adaptive during training itself, starting with high ranks when the loss surface is complex and gradually reducing ranks as the model converges. This kind of dynamic allocation could give you the best of both worlds: high expressiveness when you need it and aggressive compression when you don’t.
The technique is also starting to spread beyond language models. Early experiments with QDoRA fine-tuning for Stable Diffusion are showing promising results — the weight decomposition seems to help preserve the model’s general image generation capabilities while adding specific styles or concepts. Similarly, there are preliminary results on Whisper model adaptation using QDoRA, which is particularly interesting for low-resource languages where full fine-tuning would overfit catastrophically. The magnitude-direction decomposition might be even more valuable in these domains where catastrophic forgetting is a bigger problem than in language models.
Federated learning with QDoRA is still largely theoretical, but the architecture naturally supports it. In federated training, you typically want to keep raw data on local devices and only aggregate model updates centrally. With QDoRA, you could keep magnitude parameters completely local — they capture device-specific scaling that might leak private information — and only aggregate the directional updates, which capture more general semantic patterns. This could enable much more privacy-preserving fine-tuning for sensitive domains like medical AI or personal assistants.
There’s also interesting work on combining QDoRA with other PEFT techniques. Representation Fine-Tuning, or ReFT, works by modifying intermediate representations rather than weights, and it turns out you can combine this with QDoRA’s weight-space approach in an additive way. The hybrid gets you benefits from both techniques — ReFT’s ability to target specific behaviors and QDoRA’s efficient weight updates. Similarly, Mixture of Experts architectures can use separate QDoRA experts for different domains, with a learned routing mechanism to select which expert handles each input. This is still early-stage research, but the preliminary results suggest substantial quality improvements for multi-domain models.
Conclusion QDoRA represents something more than just an incremental improvement in parameter-efficient fine-tuning — it’s a fundamental rethinking of how we should approach model adaptation. By decomposing weights into magnitude and direction and applying low-rank optimization only to the directional component, QDoRA achieves a learning pattern that mirrors full fine-tuning much more closely than any previous PEFT method. The fact that it does this while using less memory than QLoRA is remarkable. The fact that it sometimes outperforms full fine-tuning is almost absurd.
For my work on Uranus-3B, QDoRA isn’t just convenient — it’s enabling. The memory efficiency means I can do continued pre-training on specialized corpora using consumer hardware instead of needing cluster access. The performance parity with full fine-tuning means I’m not sacrificing quality for feasibility. The flexibility to create multiple task-specific adapters means I can support different use cases without maintaining separate model versions. And the inference compatibility means deployment is straightforward — merge the weights and serve them like any other model, with zero overhead.
As we move deeper into 2025 and beyond, I expect QDoRA or techniques like it to become the default for LLM fine-tuning. Open-source models are getting larger — we’re already seeing 405B parameter models from Meta, and that’s just the beginning. Domain-specific adaptation is becoming more critical as generic models hit their capability ceiling. Hardware costs remain a bottleneck for most research teams and small companies. And deployment efficiency demands are only increasing as these models move into production systems that need to handle thousands of queries per second.
The combination of quantization and weight decomposition isn’t just an optimization trick. It’s a fundamental insight about the structure of neural network learning — that magnitude and direction updates happen at different scales and benefit from different optimization strategies. LoRA was revolutionary when it came out, but it was asking a single low-rank bottleneck to learn too many different kinds of updates simultaneously. QDoRA disentangles these update types and lets each be optimized appropriately.
What excites me most about QDoRA is that it’s still early. The technique was only published in February 2024, and the quantized version came in April. We’re already seeing substantial improvements from better hyperparameter choices and implementation optimizations. The fused kernel work will probably land within the next few months. Dynamic rank allocation and multi-modal extensions are being actively developed. And the integration with other techniques like ReFT and MoE opens up even more possibilities.
If you’re working on LLM research or deployment and you haven’t tried QDoRA yet, now’s the time. The learning curve is gentle — if you understand LoRA and QLoRA, you basically understand QDoRA. The implementation is mature thanks to Hugging Face PEFT. And the results speak for themselves. You’ll get better performance with less memory, which is pretty much the best trade-off you can ask for in machine learning.
For those following Uranus-3B’s development, I’ll be sharing results and code as the project progresses. The phased training approach with QDoRA is something I haven’t seen documented elsewhere, so I’m particularly curious to see how well it works for integrating disparate knowledge domains without catastrophic forgetting. If it works as well as I expect, it could be a template for other specialized models that need to bridge rigorous technical content with more idiosyncratic knowledge.
The future of efficient fine-tuning is here, and it’s quantized, decomposed, and surprisingly elegant. Here’s to building better models with less.
References and Further Reading DoRA Paper: Liu et al. (2024). “DoRA: Weight-Decomposed Low-Rank Adaptation.” ICML 2024 (Oral). arXiv:2402.09353
Answer.AI QDoRA Blog: “Efficient finetuning of Llama 3 with FSDP QDoRA” (April 2024). Link
QLoRA Paper: Dettmers et al. (2023). “QLoRA: Efficient Finetuning of Quantized LLMs.” arXiv:2305.14314
LoRA Paper: Hu et al. (2021). “LoRA: Low-Rank Adaptation of Large Language Models.” arXiv:2106.09685
Weight Normalization: Salimans & Kingma (2016). “Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.”
Hugging Face PEFT: Official Documentation
Sebastian Raschka’s DoRA Tutorial: Magazine Article
Partnerships and projects: contact@antoniovfranco.com


============================================================
END OF ARTICLE
============================================================