mesh.applied (2026 to present)
We (Matteo, Janos and me, Petya) make inference faster for specialized tasks (like building GTA7 game engine backends using agents from an imperfect specification, or using agents to design experiments and generate hypotheses in a research lab in experimental physics, etc.). We closed our first deal (we built a cluster and optimized it for local inference). We'll all be in San Francisco for the next 2 months.
We use various techniques (mainly machine learning based, algorithmic, systems engineering) to build an automated framework for the 3 layers of useful inference:
1. the lowest level: quantizations, kernels, computational graph optimizations, KV cache and prefill optimizations, speculative decoding algorithms and drafters, etc.
(a) to make it fit on the wanted hardware if possible while keeping relevant retention high;
(b) to maximize tok/s for the wanted batch size B (concurrency, meaning how many simultaneous users are asking for forward passes) and context/input lengths;
2. scheduling and routing (e.g. task difficulty based LLM or inference time effort choice) to serve users efficiently, minimize dead time, and maximize overlapping processes;
3. the output and harness level, to drastically decrease domain specific time per task.
Additionally we have some experiments and to this date immature prototypes:
1. diffusion based main architectures for building faster hierarchical coding agents;
2. adaptive techniques augmenting the inference engine and the main LLM model for further decreasing time per task post deployment;
3. only very simple prototypes so far: optimizing VLMs and VLAs for fast and scalable feedback when training robotic actions in simulators.