The problem
Online reinforcement learning assumes you can let a policy act in the environment and learn from what happens. In a live commercial game that assumption is unusable: exploration means a half-trained agent behaving visibly badly in front of players, and the sample budget an on-policy method needs is far beyond what a production title will give up.
What Wargaming does have is an enormous archive of logged battles. The question this thesis takes on is whether that archive is enough. Can you learn a competent policy purely from behaviour that was already recorded, and, harder, can you trust your evaluation of it without ever deploying it?
Why offline RL is the hard version
Offline RL removes the feedback loop that normally keeps value estimates honest. A Q-function trained on a fixed dataset will happily assign high value to state-action pairs the data never covered, and because there is no environment to contradict it, that error compounds through bootstrapping instead of being corrected. Most of the engineering is about constraining the learned policy toward the support of the logged data without collapsing it into pure imitation.
Approach
- Built the training stack on TorchRL and PyTorch, with CUDA acceleration on NVIDIA hardware, so the offline algorithms and replay structures came from a maintained library rather than bespoke implementations.
- Framed the logged battle archive as a static dataset of trajectories, with the state, action and reward representation derived from telemetry rather than designed for an agent.
- Used off-policy evaluation as the primary measurement channel, estimating how a candidate policy would perform from data generated by a different behaviour policy, since running the candidate to find out was not an option.
Results
TODO. Add the headline numbers here: which offline algorithms you compared, the OPE estimator(s) used, and how the learned policies scored against the behaviour policy baseline. This is the section a reader will look for first.
What I took away
TODO. Two or three sentences on what surprised you, what you would do differently, and what this says about applying offline RL to shipped products generally.