The rise of NPU chips: AI processing locally in the workplace

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

The shift from centralized cloud processing to local hardware is one of the most important infrastructural trends in the computing landscape. Where machine learning tasks were handled almost exclusively in large-scale hyperscale data centers for years, modern office laptops and workstations increasingly feature a Neural Processing Unit (NPU). This specialized processor layer is specifically designed to perform tensor calculations and matrix multiplications at a fraction of traditional power consumption.

In the earlier overview article on the global chip and hardware race it became clear how server farms are running up against physical supply and power limits; the integration of NPU silicon in the workplace is a direct response to absorbing computing load at the edge of the network. Instead of sending every audio signal, every search query, or every contextual text suggestion over an external network, inference stays within the housing of the end device. This changes not only the requirements for office hardware, but also forces IT departments to fundamentally rethink their privacy, software, and management strategies.

What is an NPU and how does it differ from a CPU and GPU?

A Neural Processing Unit is a microarchitecture optimized for parallel mathematical operations at low precision, such as INT8, INT4, and FP16 operations. Where a Central Processing Unit (CPU) excels at sequential logic with low latency and a Graphics Processing Unit (GPU) is built for massively parallel floating-point calculations at high precision, the NPU focuses almost exclusively on vector and matrix multiplications (GEMM: General Matrix Multiply) via so-called systolic arrays.

The structural advantage of an NPU lies in energy efficiency per compute operation. A GPU often consumes between 30 and 150 watts during active inference on mobile platforms, causing cooling fans to spin up and battery life to drop sharply. An NPU delivers comparable throughput on quantized models with a power draw of 2.5 to 7 watts. The compute capacity of these chips is expressed in TOPS (Trillion Operations Per Second), with the current standard for modern AI workplaces sitting around 40 to 50 TOPS.

To make effective use of these compact accelerators, it is essential to understand how architecture and model size interact; read the analysis of small language models to see how compressed neural networks align optimally with local compute cores. After all, an NPU is not designed to run a gigantic model with hundreds of billions of parameters, but rather thrives on targeted, streamlined architectures that run continuously in the background.

Why central data centers are not always the answer

The shift to local NPU compute power stems from practical constraints in centralized infrastructures. When hundreds of employees within an organization simultaneously perform microtasks — such as live noise suppression, real-time meeting transcription, OCR on scanned documents, and semantic indexing of emails — the number of API calls to cloud servers explodes. This brings substantial network latency and ongoing operational costs.

Moreover, large data centers are increasingly running into network and power congestion; see the background article on data centers and AI compute capacity in the Netherlands for insight into the physical capacity limits central cloud services are running into. When enterprise software handles routine background tasks locally, this relieves both the external data center infrastructure and the internal network connection of the office building.

In addition, offline availability plays a crucial role. A workplace dependent on cloud inference immediately loses its smart productivity features as soon as an internet connection drops or slows down. A local NPU guarantees that speech recognition, syntax checking, and document classification continue functioning uninterrupted, regardless of the user's network status.

Memory architecture: the real bottleneck of local inference

Although chip manufacturers eagerly advertise impressive TOPS figures, the actual performance limit of local AI inference almost always lies in the memory architecture. Neural networks — and large language models in particular — are, during the token generation phase, so-called *memory-bandwidth bound*. For every generated token, the model's entire weight file must be transported from RAM to the compute cores.

An NPU with a theoretical throughput of 45 TOPS cannot utilize its compute capacity if the memory channel delivers only 60 to 85 gigabytes per second (GB/s) of data over a traditional dual-channel LPDDR5 bus. As a result, token generation speed for larger language models stagnates at around a few words per second, regardless of how fast the systolic arrays can multiply. Systems that use an integrated Unified Memory Architecture (UMA) with wide memory buses (128-bit to 256-bit) and throughput speeds above 120 GB/s therefore perform many times more consistently in practice.

For a detailed overview of system requirements for workplace hardware, the hardware guide for local LLMs offers a technical roadmap for determining which memory bandwidth and RAM capacity are needed for specific model sizes. Without sufficient system memory that can be shared between the OS, GPU, and NPU, local inference inevitably leads to paging to slower storage media.

Privacy, compliance, and the GDPR in the physical workplace

One of the most heavily weighted arguments for local NPU execution is data protection. As soon as employees summarize documents, analyze contracts, or process medical records via cloud-based AI interfaces, a data processor relationship arises in which personal data and trade secrets pass through external infrastructures. This requires data processing agreements, Data Protection Impact Assessments (DPIAs), and strict control over retention periods.

With local processing on an NPU, the raw input data never leaves the physical endpoint. The tokens are processed in the local machine's working memory and released immediately after the operation is completed. This significantly reduces the attack surface for data breaches and simplifies compliance with privacy frameworks.

To assess when a local processing layer is legally necessary under European privacy legislation, the file on GDPR compliance and AI models explains which criteria regulators apply for data minimization and cross-border data transfer. Local NPU inference gives organizations conclusive proof that sensitive personal data was never exposed to the logs of external API providers.

Energy consumption and thermal management on laptops

The thermal design power (TDP) of an average business laptop ranges between 15 and 45 watts for the entire system. When traditional inference tasks are run on a graphics card or central processor, the internal temperature rises quickly, resulting in thermal throttling and noisy fans. An NPU avoids this by maintaining extremely high energy efficiency per watt.

Because an NPU uses fixed compute paths optimized for matrix multiplications without overhead for graphics rendering or general instruction decoding, power draw remains extremely limited. This allows background processes to stay continuously active — think of real-time indexing of all incoming documents for semantic search — without draining the battery within two hours.

For broader context on power consumption and scale, also consult the analysis on energy consumption and compute power in AI, which clearly quantifies the structural difference in consumption between local endpoints and megawatt-intensive data center clusters. At the macro level, moving billions of routine micro-inferences to efficient NPUs delivers a substantial reduction in the total energy load of IT infrastructures.

Software ecosystems: DirectML, ONNX Runtime, and vendor frameworks

The presence of hardware NPU silicon is worthless without a software layer that can translate models into the right instruction sets. Where NVIDIA enjoys a dominant position within data centers thanks to the established CUDA platform, the NPU landscape in the workplace is highly fragmented. Every chip designer supplies its own software development kit (SDK), such as Qualcomm QNN, Intel OpenVINO, and Apple CoreML.

To prevent software developers from having to write separate code for every specific processor, the combination of ONNX Runtime and Microsoft DirectML acts as a universal abstraction layer within Windows environments. DirectML automatically routes tensor operations to the best available accelerator on the system:

// Voorbeeld: Initialisatie van een ONNX Runtime inferentiesessie
// met DirectML-ondersteuning voor automatische NPU/GPU-toewijzing
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "LokaalKantoorModel");
Ort::SessionOptions session_options;

// Activeer DirectML hardware-acceleratie
OrtSessionOptionsAppendExecutionProvider_DML(session_options, 0);
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

// Laad het gekwantiseerde INT8-model in het werkgeheugen
Ort::Session session(env, L"modellen/semantisch_zoeken_int8.onnx", session_options);

Anyone looking for concrete implementations and software packages to drive such models can turn to the overview of tools for local LLMs to directly compare suitable runtime environments and model managers. The challenge for the coming years, however, remains closing the gap between proprietary vendor frameworks and open source machine learning standards.

Practical applications in the office: what already works today?

In current practice, NPU chips excel at targeted, deterministic background tasks with a low to medium parameter size. These tasks require constant processing of data streams without the user experiencing delays in regular office software:

• Real-time audio and video editing: Noise suppression via neural filters, automatic framing of video images, background blur, and real-time gaze correction during video conferences without noticeable load on the central processor.
• Local speech-to-text (ASR): Real-time transcription and translation of meetings via compact speech models (such as optimized variants of Whisper), with audio files staying within the organization's boundaries.
• Semantic search and RAG in the workplace: Generating embeddings over local files, chat history, and emails. Users can search for concepts in natural language instead of exact keywords, with vectorization running continuously on the NPU.
• Small language models for editing: Locally correcting grammar, paraphrasing sentences, or summarizing short documents with compact models between 1 and 3 billion parameters.

What the NPU is explicitly not suited for are heavy reasoning tasks that require models of 70 billion parameters or more, or large-scale generative video tasks. Such workloads exceed both the available memory bandwidth and the memory usage of an average workplace and remain, for the time being, the domain of powerful cloud clusters or dedicated workstation GPUs.

Limitations and pitfalls of the current generation

Despite the commercial marketing around AI PCs, the current generation of NPU hardware has clear technical limitations that IT managers must factor into their procurement policy. First, there is the risk of rapid technological obsolescence. Manufacturers have raised the minimum TOPS requirements for operating system features from 10 to 40+ TOPS within a short period. First-generation laptops therefore quickly fall short for newer operating system features.

Second, model quantization causes quality loss. To fit models within the tight memory and compute budget of an NPU, model weights are often reduced from FP16 to INT4 or even INT3. Although techniques such as AWQ (Activation-aware Weight Quantization) and GPTQ minimize this loss, measurable degradation occurs with complex instructions and nuanced language processing compared to uncompressed cloud models.

Third, mature drivers are often still lacking on Linux and open-source environments. Where Windows and macOS provide integrated drivers for their specific silicon, support for NPU acceleration in enterprise Linux distributions is fragmented, which slows adoption on specialized developer workstations.

Conclusion: The hybrid workplace architecture

The rise of NPU chips does not mark the end of central cloud services, but ushers in the era of hybrid computing architecture. The future business workplace rests on a clear division of tasks across three layers: the NPU handles continuous, energy-efficient background tasks, local privacy-sensitive processing, and real-time media enhancements; discrete local compute power steps in for heavier engineering and creative tasks; and centralized data centers are selectively engaged for complex reasoning tasks and frontier models.

For organizations, this means hardware procurement is no longer purely about CPU clock speeds and RAM capacity. The presence of a capable NPU, supported by a wide memory bus and mature software ecosystems, increasingly determines how productive, secure, and autonomous a workplace can operate in a data-driven business environment.