« All talks

How not to build another Megatron, for both humans and agents

October 9, 2026 Lequn Chen — Perplexity AI

Abstract

Megatron is “the” training framework for LLMs. However, its code is intertwined, buggy, and not the fastest. What caused this? How can we build a minimalist training framework for 1T+ params and 1M context, with higher accuracy, better maintainability, and faster speed? Are system design and software engineering still relevant while we pray to the omnipotent AIs?

Speaker Bio

Lequn is a Member of Technical Staff at Perplexity AI. He works on inference systems and more recently on RL training infra. He graduated from syslab 2.5 years ago.