Technical report
From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks
RL post-training for language models · January 2025 to June 2026
1,598 RL training methods on arXiv. 37 reached framework code.
Each dot is one paper that proposed a general RL training method, stacked by the month of its first arXiv version. A dot is green when the method reached the main branch of at least one of seven open-source frameworks. Every date comes from the frameworks' own commit histories.
Look up a method or a paper
Search by method name, paper title, arXiv id or the setting a framework uses, such as clip_ratio_high. Every method and paper has its own link to share.
/
Methods by training failure
Pick the failure you see in training. Each cell links to the line of code that offers the method in that framework, at the snapshot commit. The tracker lists implementations and does not test whether a method helps.
Latest landings
A landing is the first commit that put a method on a framework's main branch. Days run from the method's source to that commit. The list grows with the weekly update.
Seven frameworks
How many methods each framework took in, how fast, and what happened to public proposals for the methods no framework included. The guide to each framework lists its contribution rules and the concerns its reviewers raised.
Methods counts landings of catalog methods with a source from 2025 or later, and first anywhere counts the methods with their first landing there. Median days run from the source to the landing and leave out upper bounds. A landing pull request counts as opened by a prior merger when its author had merged a pull request in the framework before. Declined proposals include those redirected elsewhere, and lapsed proposals were closed as inactive. ROLL publishes code in bulk releases without pull requests.
Report and data
Every number in the report is generated from the files in the repository. Code is under the MIT License and data under CC BY 4.0.
Reproduce
git clone https://github.com/pth2002/rl-method-tracker
cd rl-method-tracker
git checkout v1.0
pip install -r requirements.txt
python tools/reproduce_report.py --pdfThe last command checks the frozen inputs and rebuilds every number, table and figure without network access. The PDF step needs pdflatex.
Cite
@misc{pan2026rlframeworks,
title = {From {RL} Papers to Framework Code:
Inclusion and Review in Seven
Open-Source Training Frameworks},
author = {Pan, Taiheng},
year = {2026},
eprint = {2610.28027},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arXiv.2610.28027},
url = {https://arxiv.org/abs/2610.28027}
}