Small language models: Rethinking enterprise AI architecture

Large language models (LLMs) are the workhorses of AI, supporting ever more sophisticated capabilities and workflows, and approaching near-human level performance.

But sometimes more isn’t always better — it’s just more. Specialized data and limited capabilities are just fine for some workflows.

This realization is driving the evolution of small language models (SLMs), rather than one-size-fits-all LLMs. SLMs — coming in the form of domain-specific models, statistical language models, and neural language models — are faster, cheaper, less resource-intensive, and more private than traditional LLMs, according to experts.

It’s not simply a replacement story, though. “The pattern is closer to a better division of labor,” says Thomas Randall, a research director at Info-Tech Research Group. “A routing architecture sends simple or well-scoped queries to a specialized small model, and complex queries to a large model.”

How are small language models made small?

While LLMs can feature parameter counts in the hundreds of billions — or, increasingly, trillions — SLMs typically fall in the 1 billion to 7 billion parameter range. Generally, anything below 10 billion is considered small.

Whereas LLMs are trained on petabytes of data, SLMs are trained on compact transformer architectures (neural networks) using smaller, specialized, high-quality datasets specific to their intended function. Several techniques help contain model size without compromising performance. These include the following:

• Knowledge distillation: A larger “teacher” model trains a small “student” model so that it can learn to mimic strong reasoning capabilities, but at a much smaller scale.
• Pruning: Redundant or irrelevant parameters are removed from neural network architectures.
• Quantization: Values are reduced from high-precision to lower-precision (that is, floating-point numbers are converted to integers) to reduce data size, speed up processing, and optimize energy consumption.

Larger models can also be modified and distilled into smaller, more specialized models through techniques like retrieval-augmented generation (RAG), when they are trained to pull from trusted sources before generating a response; fine-tuning and prompt tuning to guide responses to specific areas; or LoRa (low-rank adaptation), which adds lightweight pieces to an original model to reduce its size and scope, rather than retraining or modifying the entire model.

Ultimately with SLMs, enterprise data becomes a “key differentiator, necessitating data preparation, quality checks, versioning, and overall management to ensure relevant data is structured to meet fine-tuning requirements,” notes Sumit Agarwal, VP analyst at Gartner.

Benefits of small language models

The core driver of SLMs is economic, analysts note. “For high-volume, repetitive, scoped tasks (such as customer service triage), the costs of using a trillion-parameter generalist cannot be justified,” Info-Tech’s Randall points out.

Modest workflows for GPT-5 at scale, for instance, will generate unsustainable cloud bills. Using a limited, built-for-purpose SLM is “far better” and more efficient for modest workflows, Randall said.

The clearest business advantages emerge when three conditions align for a task, Randall notes: It is narrow in scope, repetitive and high volume, and latency tolerance is low. SLMs perform well when tasks do not require broad general knowledge or novel reasoning. They excel when a task requires a fast, consistent, repetitive application of a well-defined pattern.

The performance is often better in this area than with an LLM, as the SLM has been trained to do “one thing well rather than everything passably,” said Randall. “The SLM also avoids sifting through the noise of the entire internet in its generation of output, decreasing the chances of hallucination.”

Other benefits of SLMs:

• Low compute requirements: SLMs can run on-device (laptops, mobile phones), in edge cases, and even offline.
• Stronger privacy and security: Because they are small enough to run on-device or on-premises, SLMs minimize the risk of data leakage and cybersecurity events. This makes them desirable in highly regulated industries or in organizations handling sensitive data.
• Inference efficiency: Smaller models generate quick responses, which is ideal for real-time applications.
• Cheaper deployment: Hardware and cloud costs are lower.
• Customizability: Models are trained on a specific organization’s data.

Nvidia researchers also point to the adaptability, flexibility, and modular (Lego-like) system design of SLMs. Builders can add new skills and respond to evolving user needs, new formatting requirements, and changing rules and regulations in certain jurisdictions.

Further, SLMs support democratization, the researchers emphasize. When more users and enterprises are involved in building language models, AI can represent a more diverse range of perspectives and societal requirements. And, more people involved in creating and refining models can help the field advance more rapidly.

The Nvidia researchers go so far as to say that SLMs are “sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI.”

IT analyst firm Gartner agrees to an extent, predicting that by 2027, enterprise use of small, task-specific AI models will be threefold more than their use of LLMs.

“The variety of tasks in business workflows and the need for greater accuracy are driving the shift towards specialized models fine-tuned on specific functions or domain data,” said Gartner’s Agarwal.

Use cases for small language models

SLMs shine for a variety of use cases including the following: 

• Boilerplate tasks and simple command parsing and routing based on predefined templates.
• Content summarization and generation: SLMs can build detailed reports, user-tailored copy, web and social media messaging, and marketing materials.
• Chatbots and virtual assistants: Smaller models can provide real-time interaction, handle routine queries from both customers and internal users, and perform live transcription and translation.
• Content analysis: SLMs can perform data analysis and sentiment analysis to surface industry trends and help optimize strategy. 
• Code generation: Small models can work alongside developers to help write and debug code.
• IoT, edge computing scenarios, and low-resource settings: SLMs can run locally on devices without the need for cloud hosting or internet connection.
• Specialized fields (financial, legal, medical) where data privacy is paramount and organizations must comply with changing regulations and laws.

Ultimately, SLMs are optimal for use cases requiring classification or document processing, Info-Tech’s Randall noted. For instance, a help desk might use an SLM to classify a ticket against 200-plus categories, a legal department might use one for contract clause identification, or a finance team might use one to read transaction logs and regulatory texts for fraud detection.

Limitations and trade-offs of small language models

As with anything, of course, SLMs introduce their own challenges.

The largest trade-off is breadth of knowledge and reasoning capabilities, said Randall. SLMs tend to degrade on tasks that require contextual awareness or multi-step reasoning across unfamiliar domains, or when a large context window is required. Smaller models may struggle with edge cases or tangential tasks (such as a help desk ticket requiring a new category) that a generalist LLM can handle.

Analysts call out other disadvantages including the following:

• Narrow scope: SLMs are trained in a specific domain and are constrained by their size and computational abilities. Generalization can be limited; models may struggle with tasks that are more nuanced, require deeper contextual understanding or multifaceted reasoning, or contain high levels of abstraction or intricate data patterns.
• Decreased robustness: SLMs can be prone to errors in areas outside their expertise, or when faced with more advanced adversarial inputs (such as multi-turn social engineering).
• Bias risks: If not carefully curated, smaller datasets could potentially amplify bias.

“General purpose LLMs retain advantages for open-ended reasoning and breadth of knowledge,” said Randall.

Therefore, enterprises should be pragmatic when implementing task-specific models. Gartner recommends piloting small, contextualized models in areas where LLMs have not met expectations around speed or response quality. They should also adopt “composite approaches” involving multiple models and workflow steps in use cases where single model orchestration has fallen short.

Further, enterprises must strengthen skills and data practices. “Prioritize data preparation efforts to collect, curate, and organize the data necessary for fine-tuning,” Gartner advises. 

SLMs will not replace LLMs

Arguably, there will always be a case for both LLMs and SLMs, analysts note.

Randall anticipates continuing growth of SLMs in the enterprise as the volume of AI-mediated tasks expands, particularly for well-defined, highly repetitive tasks.

However, “the SLM versus LLM dichotomy is not a helpful one,” he stressed. “The more accurate picture will be organizations asking how to orchestrate multiple models of different sizes across different deployment contexts.”