TinyDeiT FINN Transformer Example

[ollycassidy13]

Add a TinyDeiT VCK190 example flow

Summary

This PR adds an end-to-end FINN example for compiling quantized TinyDeiT models for the AMD VCK190. It showcases W3A3 and W4A4 QONNX checkpoints and provides one 250 MHz folding configuration for each model.

The example is under transformer_examples/tinydeit and is intended to show how a transformer model moves through FINN: checkpoint inspection, graph preparation, loop rolling, software verification, folding, IP generation and VCK190 implementation.

How the flow works

TinyDeiT contains 12 transformer encoder blocks with the same structure. The preparation flow converts the operators to FINN fpgadataflow layers, extracts one encoder block as a FINNLoop body and reuses that hardware for all 12 iterations. Parameters belonging to each encoder block are selected for the corresponding loop iteration.

The flow is split into two main stages:

1. prepare_model.py imports and tidies the QONNX model, streamlines the graph, converts supported operators to hardware layers, specializes them to RTL or HLS implementations, detects the repeated encoder blocks and rolls them into one FINNLoop.
2. build.py validates and applies a folding configuration, performs the remaining hardware transformations, inserts FIFOs, generates and stitches the IP, and produces the requested FINN output.

Supporting tools are included for inspecting a checkpoint and comparing the rolled model with its unrolled specialized reference using C++ simulation.

Main changes

• Add the TinyDeiT example package under transformer_examples/tinydeit.
• Add graph handling for the exported TinyDeiT GELU/PWPolyF forms and repeated transformer-block detection.
• Add explicit FINNLoop preparation for the 12 encoder blocks.
• Add early folding validation so incompatible PE and SIMD values fail before hardware generation.
• Add phase-based estimate, rtl, dcp and full-rtlsim build modes.
• Add VCK190 licence preflight and failed-build cleanup for long DCP builds.
• Add one W3A3 and one W4A4 folding configuration for a 4.0 ns clock:
  • configs/w3a3_vck190_250mhz.json
  • configs/w4a4_vck190_250mhz.json
• Add focused tests for graph preparation, folding validation, build configuration and the two public folding files.

The folding configurations control hardware parallelism and memory choices; they do not change model quantization. A W3A3 configuration must be used with the compatible W3A3 checkpoint, and likewise for W4A4.

Running the example

Run the commands from the FINN repository root inside the FINN container:

./run-docker.sh

The flow expects a QONNX checkpoint with a [1, 3, 224, 224] image input, a [1, 1000] classifier output, and 12 repeated transformer blocks. Checkpoints are not included in the PR. The W3A3 and W4A4 checkpoints are generated using HuangOwen/Quantization-Variation. After exporting the quantized models from that repository, place the QONNX files under onnx-checkpoints/ in the FINN repository. The commands below use:

onnx-checkpoints/deit_tiny_quant.onnx
onnx-checkpoints/deit_tiny_w4a4_quant.onnx

A DCP build requires Vivado 2024.2 and a VCK190 device licence. Generated models, reports, IP and Vivado projects are written under transformer_examples/tinydeit/build/.

W3A3

Prepare the rolled FINN model:

python -m transformer_examples.tinydeit.prepare_model \
  --input onnx-checkpoints/deit_tiny_quant.onnx \
  --output-dir transformer_examples/tinydeit/build/w3a3_prepare \
  --clock-ns 4.0 \
  --target-fps 1000 \
  --save-intermediate

Optionally compare the rolled model with the unrolled specialized graph:

python -m transformer_examples.tinydeit.verify_model \
  --model transformer_examples/tinydeit/build/w3a3_prepare/tinydeit_mlo.onnx \
  --reference \
    transformer_examples/tinydeit/build/w3a3_prepare/07_specialize_layers.onnx \
  --reference-cppsim-prepare

Build the VCK190 DCP:

python -m transformer_examples.tinydeit.build \
  --mode dcp \
  --prepared-model \
    transformer_examples/tinydeit/build/w3a3_prepare/tinydeit_mlo.onnx \
  --output-dir transformer_examples/tinydeit/build/w3a3_250mhz \
  --board VCK190 \
  --clock-ns 4.0 \
  --target-fps 1000 \
  --folding-target-cycles 0 \
  --folding-config-file \
    transformer_examples/tinydeit/configs/w3a3_vck190_250mhz.json \
  --skip-reference-io

W4A4

Prepare the rolled FINN model:

python -m transformer_examples.tinydeit.prepare_model \
  --input onnx-checkpoints/deit_tiny_w4a4_quant.onnx \
  --output-dir transformer_examples/tinydeit/build/w4a4_prepare \
  --clock-ns 4.0 \
  --target-fps 500 \
  --save-intermediate

Build the VCK190 DCP:

python -m transformer_examples.tinydeit.build \
  --mode dcp \
  --prepared-model \
    transformer_examples/tinydeit/build/w4a4_prepare/tinydeit_mlo.onnx \
  --output-dir transformer_examples/tinydeit/build/w4a4_250mhz \
  --board VCK190 \
  --clock-ns 4.0 \
  --target-fps 500 \
  --folding-target-cycles 0 \
  --folding-config-file \
    transformer_examples/tinydeit/configs/w4a4_vck190_250mhz.json \
  --skip-reference-io

Build modes

The same prepared model and folding file can be used with different output modes:

• estimate applies the hardware transformations and writes FINN estimate reports.
• rtl generates and stitches the hardware IP without implementation.
• dcp runs the VCK190 out-of-context implementation flow.
• full-rtlsim builds the stitched design and runs RTL simulation.

Open PR dependencies

This branch currently relies on changes from the following open FINN PRs:

• #1573 — Add PWPolyF piecewise polynomial activation layer or #1629
• #1579 — ONNX Where Operator
• #1611 — DWC wide streams
• #1613 — Allow padded XSI input feed for FIFO simulation
• #1614 — Add stitched RTL-simulation liveness threshold override
• #1615 — Add optional RTL-simulation stage debug logging
• #1620 — Pad1D Operator
• #1629 — PWPolyF thresholding integration
• #1634 — Add ideal-memory MLO RTL-simulation performance measurement
• #1639 — Crop RTL and SelectToken operator