fp4

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

An innovative library for efficient LLM inference via low-bit quantization

Fast and Memory-Efficient Exact Attention (BF16/FP16/FP8/FP4) for Large Headdim, 1.5x~15x speedup over PyTorch SDPA.

Qwen 3.8 27B ROCmFP4 on AMD Strix Halo (Ryzen AI Max+ 395). Up to 36 tok/s via MTP Speculation, TurboQuant & Mesa RADV Wave64.

Hand-written NVFP4 W4A16 CUDA kernels for Volta

An LLM server for a single RTX 5090, built for agent workloads: tool calls, long conversations, reasoning, and many requests at once. Consistently faster than llama.cpp on the same models, with the numbers in the repo. Written end to end by Claude Code.

Run Ornith, Qwen, Nemotron & DeepSeek optimized on AMD Strix Halo — unified OpenAI-compatible server with tuned ROCmFP4 quants, validated 262K context, vision support & hot-swap model zoo (iGPU + NPU)

An stress and benchmark utility for NVIDIA GPUs. Measures performance across various precisions (FP64, FP32, TF32, FP16, INT8) and monitors real-time vitals like power, temperature, and clock speeds.

Narrow precision floating point types

Out-of-tree vLLM plugin and open format spec for NVFP4-CB / FP8-CB product-codebook weights — 2-6 bit-per-weight LLM quantization served on native Blackwell tensor cores.

Base-13 FP4 limbs for Ozaki-scheme DGEMM emulation on FP4 tensor cores

LoRA fine-tune and serve NVFP4 models on one DGX Spark (GB10, 128 GB UMA): text backbones via generic-family onboarding, plus VLMs (vision tower, or LLM+tower jointly via --train-target both) validated end-to-end on Pixtral and Nemotron-Omni. Fused Triton dequant; runtime-LoRA and merge serving.

Python implementations for multi-precision quantization in computer vision and sensor fusion workloads, targeting the XR-NPE Mixed-Precision SIMD Neural Processing Engine. The code includes visual inertial odometry (VIO), object classification, and eye gaze extraction code in FP4, FP8, Posit4, Posit8, and BF16 formats.

The only deployed sparse FP4 GEMM on SM120: beats CUTLASS 80b on every shape, wins end-to-end request latency in 81 of 112 serving regimes vs dense NVFP4.

Three ComfyUI nodes that show the real runtime precision of MODEL, CLIP, and VAE objects.

Unlimited-OCR 的 OpenVINO 本地适配：显式 KV、R-SWA cache、12 层 sparse MoE decode、FP4/INT8 mixed experts

Run Qwen3.6-27B on four Tesla V100s at 366 tok/s using hand-written NVFP4 CUDA kernels and chain-MTP speculation.

FP4 E2M1 Arithmetic Unit — Full RTL-to-GDSII tapeout on SKY130A PDK using OpenLane | 200 MHz timing closure | DRC/LVS clean | AI inference acceleration