Motivation
Frontier LLMs are too expensive to train more than once, so the model size, token budget and hyperparameters have to be chosen before the full run. Neural scaling laws make this possible: they fit how loss changes with compute, model size and data using small models, then extrapolate to the target scale. This tutorial covers the foundations of scaling laws, from Kaplan et al. and Chinchilla to recent laws for hyperparameters. Drawing on the scaling laws developed for OpenEuroLLM, it then examines the practical choices that decide whether a scaling law can be trusted. It ends with an outlook on how AutoML can improve the construction of scaling laws.
Learning objectives. Attendees will be able to:
- Understand how loss can be modeled as a function of compute, parameters and tokens.
- Compare the three Chinchilla fitting approaches (fixed model sizes, IsoFLOP profiles, fitting a parametric loss) and understand why Kaplan et al. and Hoffmann et al. reach different conclusions.
- Design a scaling-law ladder in practice: model grid, token budgets, warmup and decay schedules, reusing cooldowns, and evaluating on held-out test points.
- Identify where AutoML methods can cut the cost of fitting scaling laws, and why this differs from standard HPO.
Program Outline
Total duration: 90 minutes.
Why hyperparameters can’t be tuned at the target scale, and how to choose them for large runs.
- Loss as a predictable power law in compute (C), parameters (N) and tokens (D).
- History of influential work on scaling laws for LLMs.
- Why Kaplan and Chinchilla disagree (Porian et al.): how parameters are counted, warmup, learning-rate decay, and hyperparameter tuning at each scale.
- Scaling laws for the hyperparameters themselves: learning rate and batch size (Step Law), context length, MoE sparsity.
Key design choices and common pitfalls when running scaling-law experiments in practice, using OpenEuroLLM as an example.
- How fitting a scaling law differs from classical HPO, and why data collection is the bottleneck.
- Recent methods that cut the cost of fitting scaling laws.
- Outlook: the automated hill-climbing machine.
Summary of the key takeaways and open floor for questions.
Target audience
This tutorial is for:
- AutoML and HPO researchers who want to work on LLM pre-training.
- ML engineers planning pre-training runs under a fixed compute budget.
- Graduate students.
You should know the basics of hyperparameter optimization and training deep networks. No experience with large-scale distributed training is needed.
Organizer
Aaron Klein (ELLIS Institute Tübingen) leads the pre-training team at the ELLIS Institute Tübingen as part of OpenEuroLLM, a European initiative developing a family of foundation models for European languages, where his team focuses on automated methods for large-scale pre-training of large language models. He is also a scientific advisor to distil labs, a Member of the ELLIS Network, and co-host of the virtual AutoML Seminar. Previously, he headed the AutoML research group at ScaDS.AI in Leipzig and, until 2024, was a senior scientist at AWS, working on SageMaker and Amazon Q. He earned his PhD at the University of Freiburg in 2019 under the supervision of Frank Hutter, received the Best Paper Award at the AutoML Conference 2022, and won the ChaLearn AutoML Challenge in 2015. He co-organized the Neural Architecture Search workshop at ICLR 2020 and ICLR 2021, and served as local chair for the AutoML Conference 2023. Email: kleiaaro@gmail.com.