The future of Small Language Models in the middle-market

The article discusses a gradual shift in business use of LLMs (Large Language Models) toward smaller equivalents (Small Language Models). The arguments in the article come not from forecasts or expectations but from project experience in SMB’s adoption of Agentic AI Workflows.

Middle-market companies, not enterprises, were first to put language models into core operations, and they now know precisely what they need from them: no black box, no lock-in, and predictable cost. Frontier providers are moving in the opposite direction, competing on breadth of reasoning while running negative unit economics. Small Language Models close that gap. TrafficBench indicates 80.7% of queries can be handled locally, and a Vstorm benchmark for a Middle East client delivered a locally hosted, air-gapped Gemma-4 alternative to cloud LLMs within a week, fully owned by the client.

There is a recurring pattern among CTOs who have successfully deployed Agentic AI based on Large Language Models (LLMs). They eventually ask the same questions: "What does reliance on models from OpenAI, Gemini or Anthropic mean for my business?" And, often, "How could I break the dependency at some point?"

These questions capture several concerns at once: cost, vendor lock-in, and data privacy. But there is a less obvious, and more frightening context for their anxiety. Many of the CTOs I work with are in their forties or fifties, and they do remember the dot-com crash. Because of that, they draw parallels between companies that rushed to embrace "the web" prematurely back in the day and the companies that seem to be making the same mistake today on the crest of the generative AI and Agentic AI wave.

Their concern is real; many of today's LLM offerings might not survive, leaving businesses that have built solutions around their models exposed. Below, we take a look at what is being done to prevent the potential fallout.

Middle-market businesses were first to (truly) use AI: #

Actual adoption of AI in business should not be counted as who had a pilot or a chatbot first, but who successfully used AI to transform key business processes… and is benefitting from it.

The headlines do not highlight this fact, but the hidden truth is that while it was believed that enterprises would spearhead AI adoption, it has actually been middle-market companies that adopted AI-driven agentic workflows first. That is why it is worth considering what they do and what they have seen.

Small and middle-size businesses (SMBs) already use models from OpenAI and Google as part of their core operations, in helping customers place orders, configure their products, or handle the after-sales process, among other uses.

What sets those companies apart from existing "chatbots in Enterprises," is that their systems do not frustrate end-users as chatbots do, but build helpful support workflows… and they learned a ton while doing so.

This reflects the different perspective of SMBs vs Enterprises. While Enterprises actually deployed chatbots first, it was more of a gimmick and showcase than it added real value. But middle-market companies could not afford to just show off. That is why they took their time and developed significantly more reliable systems, as Mixam and Synera have, among many others. As many of their managers shared with us during cooperation, "they had no budget for trial and error and had to nail the project to survive."

A widening gap between Frontier models & needs #

The race to widen LLM capabilities is not what companies actually need.

What, from their perspective, started to show was a growing discrepancy between the AI offered by Anthropic, OpenAI or Google, and the expectations of those companies.

While the model of expanding capabilities operates on the premise of building and providing LLMs with 'superintelligence' (the ability to work in literally every domain), in reality, their models are custom fit to perform tasks within a narrow, but well-defined and well-protected domain use.

Companies like Vstorm, after gaining a deep understanding of the intended use case (and its implications), loom over these "superintelligent" models like an impatient model-dad in order to ensure the model stays well-behaved and performs its duties, imposing rules and guardrails to make it risk and hallucination free.

This creates an inherently unstable situation in which model-providers race to grow in scale and widen abilities, while real world companies are adapting them to handle deep, domain-specific applications. The gap continues to widen with companies struggling to put AI to local, meaningful uses without extra exuberance, while technology providers compete on advancements in multidisciplinary reasoning, detached (to a large degree) from both the costs and the real world applications their customers actually run.

The tipping point of reckoning #

The LLM providers of today are repeating the same mistakes that crashed the market and evaporated tech leaders two decades ago.

And here is where the history of the dot-com bubble comes in handy, with the final lesson culminating in the year 2000. Back then, the Internet and online markets were such a novelty that everyone saw fresh opportunity. This fueled similar dynamics to what we see today: as 'online' companies rushed to increase scale no matter the cost.

As a result, the leaders of innovation ran on negative unit economics, similar to what OpenAI is doing today, with a burn rate forecast to hold at 57% of revenue through 2027, or Anthropic, at roughly a third of revenue in 2026 ( Fortune, reporting on internal documents obtained by the Wall Street Journal).

What makes those forerunners from the 2000s similar to the tech powerhouses of today is that they too invest billions in infrastructure, anticipating increased demand. But it turned out that this rush was disconnected from what users actually wanted and could use.

The offering was just not resonating with the market, and this burst the bubble. With the exception of Amazon (which survived the crash), early companies that rushed to build on ecommerce did not survive to reap the rewards of it in later years.

The same disconnect is visible today. While companies need a model for specific tasks which would perform similarly to a skilled worker, what is shoveled to them is an intimidating triple-Ph.D graduate with a broad wealth of inapplicable knowledge at a higher price tag.

This is simply not what middle-market companies are looking for.

Clearer expectations after using AI #

The businesses using LLMs know what they need (and do not need), setting the stage for the adoption of Small Language Models.

The best way to understand what is expected from language models is to listen to the trailblazers, the businesses that already have language models integrated into crucial parts of their operations. Their perspective is a mix of initial expectations and lessons learned from the productive use of Agentic AI.

No black box: What is most important is that agentic systems, consisting of a language model and harness, cannot be a black box in any business use. An agentic workflow in business is, in a way, the exact opposite of a magic trick: an AI system needs to be deeply understood, with task executions that repeat consistently, to qualify as a business solution.

No lock-in: For language model-based solutions, which became a part of core company processes, dependency on the model provider is a big issue. Worry surfaces with each pricing or availability change. When ChatGPT 4o was retired in August 2025 during the introduction of ChatGPT 5, it caused a major backlash as the model was already widely in use.

This forced OpenAI to reverse course and temporarily reintroduce it ( Business Standard). There is also an observable decrease in model availability. And often, when a successor model comes out, the earlier models are eventually retired, offloading the cost of upkeep by the provider onto the companies actually using the solution, forcing a costly redesigning of the applied system.

Control over costs: The way businesses are billed for model use depends on the providers' strategy. And that strategy continues to change as the market evolves. The fluctuation can be a result of changing market position of the model creators ( OpenAI pricing), and unpredictability in this department is not something that the P&L of businesses welcome, especially if the models are used extensively.

Despite the risks, agentic AI solutions in business are still being built using LLMs. The operating harness needs to be engineered (regardless of the model used), and the lock-in can be avoided by building the harness to be model-agnostic. There are ways to keep rising costs at bay, too. But these friction points set the stage for Small Language Models.

A turn to Small Language Models #

Small Language Models (or just SLMs) are considered part of agentic workflows and their importance is only growing.

Small Language Models (SLMs) are already being described as 'The future of Agentic Workflows' ( NVIDIA Research). Harvard Business Review, relaying that same NVIDIA work, sees in them the potential to become the "true backbone of the next generation of intelligent enterprises" ( Kumar and Davenport, HBR, September 2025). While Forbes had already spotted their potential to redefine the AI race over a year ago ( Forbes, February 2025).

Fast forward to mid-2026, and our practice at Vstorm proves that these are genuine claims. Several of our projects are built around Large Language Models, with a promising path to adopt smaller equivalents in their place.

For example, one of our clients in the Middle East insisted on hosting their model locally to fit into their customers' very specific requirements related to privacy and English-Arabic proficiency.

We designed an end-to-end pipeline leveraging the Berkeley Function Calling Leaderboard for agentic capabilities and FLORES+ datasets to rigorously validate bidirectional English-Arabic proficiency.

The benchmark resulted in a report and suggestions of which SLM may perform best in the customer's scenario. The client then settled on the Gemma-4-26B-A4B model, which was able to outperform even four times larger competitors when it came to overall score. It was exceptionally fast due to its MoE architecture, performed flawlessly on agentic tasks, and is capable of more than one modality.

In a week's time, our Vstorm engineers offered an alternative to cloud-based LLMs. This came along with natural benefits, like full model control, data security and an air-gapped architecture, which are nice things to have given the character and data processed by the whole application. That way, the sovereign solution we built for the client remained under their direct ownership at all times, eliminating both the 'black box' and lock-in issue.

The difference between small and large language models #

As a rule of thumb, 80% of the Agentic AI tasks can be handled by a smaller model, but a larger one is an easier starting point.

In a variety of business uses, companies do not need a frontier model to process their documents and pay ~$25 for 1M output tokens ( Anthropic). The vast number of text operations can be handled with small, local models on local machinery, driving that cost down to below a single US dollar by running the infrastructure on servers the company owns.

In principle, you can think of the difference between Large Language Models and their smaller counterparts as the difference between a Swiss Army knife and a scalpel.

You need an LLM if you do not know what you are going to face, or when complex tasks cut across several domains at once. LLMs are genuinely helpful for our engineers at Vstorm when we are opening a new project and dealing with context and situations we do not fully know how to tackle early on. There they stand, the commandos of AI engineering, facing the enemy of hostile data structures and unfamiliar context of operations. In such a setting, a Swiss Army knife is a useful tool.

But further down the road, when engagement matures, the context becomes well defined, and the AI harness provides the structure to hold on to. The circumstances change from a battlefield to a planned surgical procedure on the data. And now, a well chosen scalpel in the form of a Small Language Model is ideal to get the job done. More highly specialized than the Swiss Army knife, and better for the job at hand.

Such an SLM has been found to be more than enough in the tasks of summarizing, reviewing, answering or checking anything text based (and not this alone). Where a task is narrow and repeatable, such as document classification, fraud detection flagging, or ticket triage, a smaller model fine-tuned on the company's own training data is the more cost effective choice, because artificial intelligence agents built this way are cheaper to run and easier to audit.

With the release of model families like Qwen3.6, Gemma-4 or GPT-OSS, the threshold of what an agent, that executes tasks locally, can do has gone up significantly. As usual in this business, there is a quantification of those thoughts and claims. It is called TrafficBench.

This test reveals that 88.7% of all queries can be successfully handled by local models with quality varying on the domain. This rises up to 90% in creative tasks and drops to nearly 70% in technical ones, and can be solved with local, smaller instances of LLMs ( Intelligence per Watt, arXiv, November 2025).

What is worth noticing is that, between 2023 and 2025, the same study finds that intelligence efficiency (measurement that includes model performance and its power usage) has risen 5.3 times, with the share of locally serviceable queries climbing from 23.2% to 71.3% over the same period. This means that the efficiency case no longer rests on the model alone. Routing those queries to local models, rather than defaulting every one of them to the cloud, cuts energy use by 64.3%, compute by 61.8%, and cost by 59.0% against cloud-only deployment, with no degradation in answer quality.

Summary #

Frontier LLMs are more forgiving when it comes to the overall craftsmanship required to successfully perform a given task. So a prompt does not need to be polished, data can be far from ideal, and logic can have some flaws. The horsepower of the model will kick in and usually solve whatever has to be solved.

However, when you are using smaller models, local ones, the required level of expertise and overall preparation is significantly higher. You will not get away with fuzzy and unclear logic or non-preprocessed data. You need actual knowledge which is not always the case when using the big frontiers.

This does not mean that small models are not capable or are harder to use, they just have requirements, as many things do. Understand your process, map it properly, prepare data for AI usage and you will get comparably good results as while using either Opus or Gemini.

Inherently, SLMs allow you to comply with GDPR, stay secure and process all internal data on your owned machine, since that is where the models live. No need to fear if the data you are sending can be used against you or leak outside the organization. No model will be retired until you give the say so. A well-designed process and solution is properly air-tight, secure and ready to tackle all the tasks you throw at it.

We observe that approximately 8 out of 10 tasks for which ChatGPT-5.5 high reasoning settings are used can be tackled with similar outcomes (and far more cheaply) by Gemma-4-31B or Qwen3.6-27B. And you are free to send internal and classified data there, since it lives in your company's basement. And there are no limits to how much you can use it, no token caps, no cost override.

Sources:

• Korben on GLM 5.2
• Small Language Models are the Future of Agentic AI (NVIDIA Research)
• Towards Data Science: choosing between small and frontier models
• State of Open Source on Hugging Face: Spring 2026
• When to Reason: Semantic Router for vLLM (arXiv)
• Ability.ai on small language models in agentic workflows
• Microsoft Foundry: cost-aware LLM workloads with Model Router
• Ken W. Alger: optimizing AI costs with semantic routing
• NVIDIA Technical Blog: small language models and scalable agentic AI
• Intelligence per Watt (arXiv)
• HBR: The Case for Using Small Language Models
• Forbes on small language models
• Fortune on OpenAI's projected losses
• Anthropic: Introducing Claude Opus 4.5
• OpenAI API pricing
• MindStudio: local AI vs cloud AI in 2026