Motivation
The premise of AutoML is that model selection, hyperparameter tuning, and pipeline construction can be searched automatically rather than chosen by hand (Hutter et al., 2019), and that search has been driven mostly by Bayesian and evolutionary methods. Using a language model as the search operator changes the core of the loop, yet it remains unsettled where this approach succeeds and where classical methods still dominate. AutoML attendees are exactly the people who will have to make that call, often on real budgets.
The case for a field overview now is that the evidence base is thin relative to the rate of new work. Reported comparisons are frequently not matched for compute, evaluation is noisy, and benchmark contamination is a genuine risk for models trained on public data (MLE-bench). Part I: Introduction to LLM-Driven Search sets out the methods and the conditions under which their published results hold. Part II: A Case Study from Practice reports how the approach behaves once evaluation cost and reproducibility constraints are real, including several attempts that failed. The session deliberately surveys the field as a whole; the case study is one grounding example, not the subject.
Learning outcomes. By the end, attendees should be able to:
- Describe the main ways an LLM enters a search loop: proposing candidates, acting as a mutation operator, synthesizing programs, and orchestrating a pipeline.
- Decide when LLM-driven search is a reasonable choice against Bayesian or evolutionary baselines, given a compute budget and reproducibility requirements.
- Recognize the benchmarking problems specific to these methods, e.g. contamination, evaluation noise and unmatched compute, and set up fair comparisons.
- Outline the components of a production search loop: an evaluation harness, guardrails, cost controls, and human review.
Program Outline / Tentative Schedule
Presentation totals 75 minutes followed by 15 minutes of discussion.
AutoML as a search problem, and the point at which a language model enters the loop as the component that proposes candidates.
The four ways an LLM is used: proposing numeric or prompt configurations, acting as a mutation operator inside an evolutionary loop, synthesizing programs, and orchestrating a pipeline. How candidates are conditioned on the record of past attempts and how feedback is encoded for the model.
Hyperparameter tuning and Bayesian optimization (LLAMBO), neural architecture search (GENIUS), prompt optimization (OPRO, EvoPrompt), program and algorithm discovery (FunSearch,AlphaEvolve), reward design (Eureka), and feature engineering (CAAFE, PromptFE). For each, the relation to Bayesian search, evolutionary search, and learned optimizers (OptFormer).
What the published results do and do not establish: compute-matched comparison against classical baselines, evaluation noise, run-to-run variance, and benchmark contamination (MLE-bench). Includes a short audience exercise (problem cards).
Moving a static AutoML pipeline to a loop in which an LLM proposes configurations, launches experiments, reads metrics, and iterates: the evaluation harness, guardrails, cost control, human review, the cases that did not work, and a short live demonstration. Presented without dependence on a specific model provider.
Where LLM-driven search sits among the classical methods, and what remains open: fair evaluation, reproducibility, cost, and the absence oftheory.
Open floor.
Audience engagement and materials
- Live demonstration. The search loop running end to end, with a recorded version as a fallback if connectivity fails.
- One-page guide. A short decision aid for when LLM-driven search is worth using.
- Slides and the educational code are released publicly after the session.
Target audience
- AutoML researchers interested in how LLM-driven search relates to Bayesian optimization, evolutionary methods, and learned optimizers.
- ML engineers running hyperparameter, architecture, or feature-search pipelines who are weighing this approach.
- Researchers in prompt optimization, program synthesis, and LLM agents.
- Graduate students new to the topic.
Prerequisites: familiarity with basic hyperparameter optimization. No knowledge of LLM internals required.
Organizers
Blaž Škrlj (Teads, Jožef Stefan Institute, Ljubljana) is a machine-learning researcher and engineer at Teads (formerly Outbrain),working on the design and deployment of AutoML systems, with research ties to the Jožef Stefan Institute in Ljubljana, where he completed his PhD on scalable neuro-symbolic machine learning. His research spans automated machine learning —including evolutionary and multimodal AutoML — representation learning, and resource-efficient AI, with more than fifty publications and an ASEF fellowship (2020). He has lectured and given invited talks on AutoML to academic and student audiences, and anchors the survey in Part I. Email: blaz.skrlj@teads.com.
Sebastian Koralewski (Teads, Ljubljana) works on machine learning at Teads (formerly Outbrain), based in the Ljubljana area. Before moving to industry he was a researcher at the Institute for Artificial Intelligence at the University of Bremen, in the group of Prof. MichaelBeetz, working on machine learning and knowledge representation for cognition-enabled robotics — including learning action parameterizations from human demonstration and robot simulation for prospective reasoning, with publications at IROS, ICRA, and IEEE Robotics and Automation Letters. He contributes to the engineering of the search loop and the tutorial’s hands-on materials, and brings the production perspective to Part II from his work on large-scale models for online advertising at Teads. Email: sebastian.koralewski@teads.com.