37
ggml kernels ported from NVIDIA CUDA
Each needed TPC-C on the Gaudi 3 HPU plus host glue on the CPU.
Case Study
Swapping Iron; making AI code designed from Nvidia run on Intel Gaudi
The model was Llama. The metal was Intel Gaudi. llama.cpp had backends for CUDA, Metal, Vulkan — not Gaudi. Vstorm wrote the missing ggml kernels so an on-premise stack could stay on the iron the customer already had.
The outcome
Thirty-seven CUDA kernels, rewritten for Gaudi SIMD
On-premise LLM work is growing. Open-source models and leftover accelerators do not always match. This engagement was a hardware port, not a chatbot: Llama built for NVIDIA, running locally on Intel Gaudi.
37 ggml kernels is this confidential engagement's published count — not llama.cpp's GitHub stars, not STCC triage, not Synera node time. llama.cpp itself sits at 74,500+ stars and 10,800+ forks; those numbers describe the project we ported onto, not a Vstorm KPI.
SIMT → SIMD
Execution model, not a compiler flag
NVIDIA threads vs Gaudi vector units. Divergence and sync do not map 1:1.
Gaudi
The llama.cpp backend that was missing
CPU, CUDA, HIP, Metal, Vulkan, OpenCL were there. Gaudi was not.
About the client
The customer is not named here. They needed Llama on Intel Gaudi, on their own floor — not on a cloud GPU SKU they did not own. Intel is the silicon in the story, not the contracting party.
In the same quarter Vstorm also moved other ML and LLM workloads onto Intel architecture, including Stable Diffusion and BERTopic. This article is the Llama / ggml port.
Vstorm's impact
Vstorm's impact, the TL;DR
- 37 ggml CUDA kernels rewritten for Intel Gaudi 3
- TPC-C device kernels plus CPU host glue for every kernel
- SIMT constructs remapped onto Gaudi SIMD constraints
- Memory transfers reshaped to the new kernel data layout
- Parallelisation kept, not collapsed, across the two architectures
- Llama running locally on the customer's Gaudi iron
The challenge
The model and the metal were not the same machine
llama.cpp is a small C/C++ runtime for local LLMs. Under it sits ggml: tensor ops and hardware backends. By the time of this project those backends already covered CPU and BLAS, NVIDIA CUDA and AMD HIP, Apple Metal, Vulkan, OpenCL, Kompute, some Intel GPUs, Moore Threads, and Huawei CANN. Gaudi was the gap the customer needed closed.
NVIDIA and Gaudi both do matrix and tensor work. They do not think the same way. NVIDIA CUDA is SIMT: many threads, one instruction, each with its own data. Gaudi is SIMD: one instruction over a wide vector on VPUs — 256-bit or 512-bit registers, 8, 16, or 32 elements at once. CUDA hides scheduling and sync. SIMD does not. Branch divergence is expensive on SIMT and largely forbidden on SIMD. Porting is closer to rebuilding a LEGO set from another kit's parts than to flipping a compiler flag.
How we delivered
A ggml backend, not a prompt change
The work was kernels, memory, and workload maps — measured against CUDA behaviour, not against a demo notebook.
Name the missing backend
Inventory of llama.cpp / ggml backends against the customer's Gaudi 3 floor. The gap was specific: Llama inference on HPU, with CUDA kernels as the source of truth.
- Backend gap
- Gaudi 3 HPU target
- 37 CUDA kernels listed
Proof of Value on real ggml kernels
Each kernel was inspected in CUDA, then planned as TPC-C on device plus glue on host. SIMT-only patterns — block sync, thread-local control flow, CUDA memory moves — were marked for rewrite, not translation.
- SIMT vs SIMD delta
- Per-kernel I/O contract
- TPC-C + host glue plan
Ship the Gaudi path
Kernels rewritten, memory exchange matched to the new layouts, workload maps rebuilt so independent ops stayed independent. Input and output checked against the CUDA baseline.
- Gaudi ggml backend
- Reshaped memory path
- Parallelisation preserved
How it works
Kernels, memory, then the map of the work
On paper SIMD is the tighter fit for dense transformer math: one instruction over a wide vector, no warp divergence tax. In practice it is harder to write. Data must pack into vectors. Branches must stay uniform. That is why the port was the job.
| SIMT (NVIDIA) | SIMD (Intel Gaudi) | |
|---|---|---|
| Model | Threads running the same instruction | One instruction runs over wide data |
| Abstraction | Thread-centric | Data-centric |
| Divergence | Handled but costly | Must avoid it |
| Programmer view | Feels like multithreading | Feels like vector math |
| Branching | Flexible, but slows warps | Limited, must be uniform |
37 ggml kernels identified All designed for NVIDIA CUDA
↓
Functional rewrite SIMT constructs rewritten for Gaudi SIMD
↓
Memory path Transfers reshaped to each kernel’s data layout
↓
Workload map Parallelisation kept across the two architectures
Confidential customer × Vstorm — ggml port, NVIDIA CUDA to Intel Gaudi
Results
Llama on Gaudi, on the customer's floor
The missing backend exists. Llama that was written against NVIDIA now runs locally on Intel Gaudi. We do not publish a tokens-per-second number for this customer. The published engineering fact is the port: 37 kernels, SIMD constraints honoured, memory and work maps rebuilt so the iron they bought is the iron they can run.
Work with us
Ready to see how agentic AI runs on the hardware you already own?
Meet directly with our founders and PhD AI engineers. We will walk through real implementations from 30+ agentic projects and the practical steps to integrate them into your workflows.