The goal of the project is to bring open ABI and FFI for machine learning systems.

- Stable, minimal C ABI designed for kernels, DSLs, and runtime extensibility. - Zero-copy interop across PyTorch, JAX, and CuPy using DLPack protocol. - Compact value and call convention covering common data types for ultra low-overhead ML applications. - Multi-language support out of the box: Python, C++, and Rust (with a path towards more languages).

Scale LLM serving with programmable cross-engine serving patterns, all in a few lines of Python

XGrammar is an open-source library for efficient, flexible, and portable structured generation. Bring 2x-10x speedup in grammar grammar-guided(JSON and CFG) LLM serving.

Comes with ability to do full structured generation with json schema

also a in-browser demo https://chat.webllm.ai/