Luminal AI

About

Luminal builds an AI inference compiler that optimizes models for GPUs and ASICs, delivering the fastest, highest-throughput inference by treating compilation as a search problem.

Featured work

Compiling models to megakernels

Standard GPU inference wastes time on kernel launch overhead, uneven work distribution, and memory access delays between operations. Luminal's compiler fuses entire neural network models into single GPU kernels — megakernels — using a dynamic global instruction queue that schedules work across streaming multiprocessors at runtime, eliminating the gaps that conventional approaches leave behind.

Hardware arbitrage with Luminal and Positron AI

Different inference workloads have fundamentally different hardware needs: prefill is compute-bound while decode is bandwidth-bound. Luminal's compiler automatically routes each phase to the hardware it's best suited for — GPUs for prefill, Positron Atlas systolic arrays for decode — transferring KV cache between systems to squeeze out the maximum performance from heterogeneous infrastructure.

Message from Grant

There were a few phrases that stood out to me after chatting with the Luminal team. One was that "search space design is under-appreciated".

It helped to read through this GitHub repo of theirs to better understand what they were getting at, and more generally to get a concrete sense of their design opinions. For example, you'll find in there a lot of the same philosophy underlying RISC, i.e. efficiency is downstream of reducing to as small a vocabulary of instructions as possible. This goes hand-in-hand with the idea of search space design. The goal is to give a compiler a well-thought-out space to explore, and its job is to find the optimization, rather than relying on our own intuition to bake in hand-tuned heuristics.

There's something pleasingly meta about how this design philosophy is reflected in how the organization is run. The team is lean, and a repeated theme I heard from people there was the pride each one takes in having real responsibility. This strikes me as analogous to the philosophy for how to optimize a kernel, i.e. not by hand-tuning the method, but by designing an effective search space and letting optimizations be found automatically within it. At the level of running an org, you get the most out of a team by setting up the right culture and environment, and from there endowing each person with real agency and trust.

What's clear in talking to the team is how much everyone appreciates that agency. The other phrase that stood out from our conversation was that "you can move mountains with a sufficient combination of ambition and pragmatism."