A GEMM Accelerator on the Cyclone V SoC
Post by Souleymane Dembele on Jun 13, 2026
This is the project at the center of my MS ECE thesis: a custom general matrix-multiply (GEMM) accelerator taken from SystemVerilog RTL, through Qsys integration, to a full Linux bring-up on the ARM Cortex-A9 of a Terasic SoCKit (Cyclone V SX). The software side is full-stack: a custom kernel build (linux-socfpga), a device-tree overlay that enables the FPGA-HPS bridges and binds the accelerator's CSR window to a UIO device, and a user-space driver that uses /dev/mem for register access and /dev/uio0 for done-IRQ synchronization.
The idea
Edge inference spends most of its time in matrix multiply, and the precision a model needs changes layer to layer. Rather than building a separate datapath per precision, I built one multi-precision multiply-accumulate (MAC) processing element that runs INT8, INT16, INT32, FP16, and FP32 in an 8x8 output-stationary systolic array. A 4x4 wide block carries the INT32 and FP32 modes; the surrounding 48 elements carry the narrow precisions.
How it runs
A DMA engine streams tiles from shared DDR3 through a 16-entry load FIFO, packing sub-word operands (four INT8 or two INT16 per 32-bit beat) into on-chip tile buffers. A control plane exposes a CSR register file over Avalon-MM: the host writes the configuration, base addresses, and strides, then launches and either polls or takes an interrupt on completion. Results drain row by row back to DDR.
Results
The design fits and runs on real silicon, with every DSP block in use (so the multipliers are hard DSP, not LUT fallback).
| Metric | Value |
|---|---|
| Device | Cyclone V SX 5CSXFC6D6F31C6 |
| Logic (ALMs) | 19,932 of 41,910 (48%) |
| DSP blocks | 112 of 112 (100%) |
| On-chip RAM (M10K) | 107 of 553 (19%) |
| Fabric clock | 50 MHz |
| INT8 GEMM, 512x512x512 | 3.15 GOP/s at 50 MHz, about 27x an ARM A9 software reference |
| Compute utilization | 60% of cycles, up from 19% on the naive baseline |
| Memory stall | 93% to 76% to 19% across the optimization ladder |
| Correctness | 60 of 60 on-board sweep tests pass, bit-exact |
The design starts memory-bound and works its way off the bandwidth wall in four steps, all on the same 50 MHz fabric and the same DSP array. The naive one-tile-per-launch path reloads both operands from DDR on every output tile and stalls about 93% of cycles, so the 32-bit FPGA-to-HPS bridge, not the PE array, sets the ceiling. A wide-port DMA fast path (a wide-A tile buffer and parallel B-bank fanout) brings the stall down to 76% and throughput to roughly 0.98 GOP/s. Two operand-locality optimizations then close most of the remaining gap. Keeping the A row-block resident across a tile row removes its redundant re-fetch and lifts throughput to 1.55 GOP/s; double-buffering the B tile so the next K-tile prefetches while the array computes the current one lifts it to 2.22 GOP/s and collapses the memory-stall ratio to 19%, moving the design from bandwidth-bound toward compute-bound. Finally, a hardware batch sequencer runs a full row of output tiles from one host launch instead of one launch per tile, cutting the per-launch drain and dispatch overhead: hardware throughput reaches 3.15 GOP/s, compute occupancy climbs to 60% of cycles, and the workload runs about 27x an ARM A9 software reference, bit-exact. Every optimization is CSR-gated and defaults off, so the baseline binding is a strict subset. Functional correctness is verified bit-exact against an ARM A9 software reference across all five precisions and a full K-sweep, with every optimization bit set.
What I took from it
End-to-end ownership of a hardware accelerator: RTL design and verification (leaf, wrapper, integration, and on-board layers), Qsys/Platform Designer integration, timing closure on real silicon, and the Linux bring-up that makes it usable from software, from a custom kernel build and device-tree overlay through the UIO binding and user-space driver.