Reinforcement learning has produced traffic signal controllers that perform very well in simulation. Almost none of them have been deployed on real signal hardware. The reason is the Sim-to-Real gap: the simulator a policy learns in differs from the street it is meant to run on, and the policy does not survive the move.
This half-day tutorial connects two communities that have been working on the same problem under different names. Traffic engineering has fought this gap for decades under the name calibration. Reinforcement learning attacks it with grounding, domain randomization, domain adaptation, and reward shaping. We show that both are addressing the same four things — mismatches in observation, action, transition, and reward — and that the two halves fit together.
Attendees will leave knowing how a real network is calibrated and exported into a simulator, which RL transfer methods attach to which part of the gap, and how to tell a good transfer from a bad one by reading the evaluation. Two live demos anchor the material: a GMNS network editor for the calibration half, and a phase-time visualizer for the evaluation half.
The four-gap decomposition used throughout the tutorial comes from Sim2Signal, our benchmark for measuring the Sim-to-Real gap in traffic signal control.
What the Sim-to-Real gap is, and why it matters for traffic engineering.
The (S, A, T, R) decomposition: observation, action, transition, and reward gaps mapped onto the MDP tuple — the spine of the rest of the tutorial.
Hua Wei
The gap predates RL: traffic engineering has fought it for decades under the name calibration.
Where the gap comes from in practice, and the state of real deployment — the surveyed RL-for-signal-control literature is simulation-only.
Hua Wei
Signal terminology for an RL audience: movements, phases and phase transitions, min/max green and clearance, pretimed vs. actuated control, NEMA dual-ring.
Data formats and simulators: GMNS as the hub, with UTDF, SUMO and CityFlow as spokes; the UTDF → GMNS → SUMO/CityFlow conversion pipeline and where it breaks.
Live demo: the GMNS Editor — auto-join, merging split intersections, movements, NEMA timing, validate and export.
Hua Wei
The four families of RL solutions — grounding actions, domain randomization, domain adaptation, reward shaping — and where each attaches to the taxonomy.
A worked sim-to-sim example: train in CityFlow, transfer to SUMO, induce gaps and measure them.
The multi-agent extension: why multi-intersection control does not reduce to the single-intersection case, and the cascading invalidation effect.
Live demo: the phase-time visualizer — reading space-time and phase-time diagrams to tell a good transfer from a bad one.
Ferdous Al Rafi
Recap: four gaps, one MDP, calibration and RL as two sides of the same problem.
Open problems: grounding the interface rather than only the dynamics, delay- and constraint-aware MDPs, and the empty deployment cell.
Relevant ITSS Technical Committee: Decision and Control in Transportation Systems. Discussion and Q&A.
Assistant Professor, School of Computing and Augmented Intelligence, Arizona State University. His research bridges reinforcement learning and transportation, with an emphasis on closing the sim-to-real gap for traffic signal control and multi-agent traffic systems. He has organized community tutorials on traffic-control RL at ITSC 2020 and ITSC 2023, and delivered a keynote at an ITSC 2024 workshop.
Professor of Transportation Systems, School of Sustainable Engineering and the Built Environment, Arizona State University, and Director of the ASU Transportation+AI Lab. He is the principal architect of several open-source packages including DTALite, NEXTA and OSM2GMNS, and serves as Executive Editor-in-Chief of Urban Rail Transit.
Ph.D. student in Computer Science at Arizona State University. His research focuses on mitigating the sim-to-real gap in reinforcement learning through uncertainty-aware learning, grounding and transfer mechanisms, and more realistic simulation environments. He has organized hands-on tutorials at KDD 2025, ITSC 2023 and ICDM 2025.
Ph.D. student in Computer Science at Arizona State University, with a background in Civil Engineering (Transportation) from the Bangladesh University of Engineering and Technology. His research interests include multi-agent reinforcement learning, traffic signal control, and bridging the sim-to-real gap in transportation systems.