Reinforcement learning has produced traffic signal controllers that perform very well in simulation. As good as these controllers are, they are rarely tested or evaluated under real-world conditions, and when they are moved from simulation to the field their performance degrades. This is known as the Sim-to-Real gap.
We give a comprehensive overview of sim-to-real foundations, cover the common mitigation methods for gaps in observation, action, transition and reward, and work through a case study on traffic signal control. The tutorial is designed for traffic engineers, transportation researchers and machine learning practitioners, and aims to improve the reliability of simulation-trained methods, reduce deployment risk, and make intelligent transportation systems more trustworthy.
Attendees will leave knowing how a real network is calibrated and exported into simulators for a sim-to-real experiment setup, and how the different mitigation methods address the gap. Bring a laptop: we provide Jupyter notebooks, code and slides to follow along, and the notebooks run in Google Colab with nothing to install.
The four-gap decomposition used throughout the tutorial comes from Sim2Signal, our benchmark for measuring the Sim-to-Real gap in traffic signal control. See the paper on arXiv.
What the Sim-to-Real gap is, why it is a persistent challenge for machine learning methods, and how it can be analyzed systematically by decomposing it into four parts: observation, action, transition and reward, following the MDP formulation.
Hua Wei
Hands-on: take a real dataset from a traffic agency and calibrate it for the simulators we use in our Sim-to-Real experiments.
Run it yourself in Colab. We inspect the data at every step and show where it can go wrong.
Hua Wei, Ferdous Al Rafi
An overview of the mitigation methods that address each gap, applied to a case study in traffic signal control.
Hands-on: a worked sim-to-sim example. Train in CityFlow, transfer to SUMO, induce one gap at a time and measure it.
Ferdous Al Rafi
Recap, open problems, Q&A.
Slides and the hands-on Colab notebooks will be posted in the shared Drive folder below. Nothing to install: open a notebook in Colab and run it. The complete set of materials will be available at the start of the tutorial.
Assistant Professor, School of Computing and Augmented Intelligence, Arizona State University. His research bridges reinforcement learning and transportation, with an emphasis on closing the sim-to-real gap for traffic signal control and multi-agent traffic systems. He has organized community tutorials on traffic-control RL at ITSC 2020 and ITSC 2023, and delivered a keynote at an ITSC 2024 workshop.
Professor of Transportation Systems, School of Sustainable Engineering and the Built Environment, Arizona State University, and Director of the ASU Transportation+AI Lab. He is the principal architect of several open-source packages including DTALite, NEXTA and OSM2GMNS, and serves as Executive Editor-in-Chief of Urban Rail Transit.
Ph.D. student in Computer Science at Arizona State University. His research focuses on mitigating the sim-to-real gap in reinforcement learning through uncertainty-aware learning, grounding and transfer mechanisms, and more realistic simulation environments. He has organized hands-on tutorials at KDD 2025, ITSC 2023 and ICDM 2025.
Ph.D. student in Computer Science at Arizona State University, with a background in Civil Engineering (Transportation) from the Bangladesh University of Engineering and Technology. His research interests include multi-agent reinforcement learning, traffic signal control, and bridging the sim-to-real gap in transportation systems.