Skip to content

Benchmarks

Reproduced scores from examples/benchmark_scores.py, single seed (0), CPU. Return is the mean +/- std of the final deterministic policy over evaluation episodes; random is a uniform-random policy on the same task for reference.

Algorithm Environment Steps Return (mean ± std) Random Time (s)
QLearning GridWorld 30,000 0.95 ± 0.00 0.49 0
SARSA GridWorld 30,000 0.95 ± 0.00 0.44 1
ExpectedSARSA GridWorld 30,000 0.95 ± 0.00 0.35 1
DQN CartPole 40,000 103.20 ± 2.77 21.35 93
C51 CartPole 40,000 500.00 ± 0.00 21.25 131
QRDQN CartPole 40,000 222.40 ± 183.09 20.80 159
SACDiscrete CartPole 15,000 389.40 ± 86.25 22.50 95
PPO CartPole 40,000 500.00 ± 0.00 21.80 39
A2C CartPole 40,000 500.00 ± 0.00 23.75 36
REINFORCE CartPole 30,000 473.00 ± 44.85 22.10 11
DDPG Pendulum 15,000 -146.88 ± 89.75 -1318.11 113
TD3 Pendulum 15,000 -156.05 ± 100.58 -1314.10 98
SAC Pendulum 15,000 -148.98 ± 97.51 -1311.03 160
TD3BC PointMass 10,000 -2.47 ± 1.24 -36.92 70
IQL PointMass 10,000 -2.74 ± 1.25 -36.84 114

Total wall-clock: 1119s. GridWorld optimal ≈ 0.95; CartPole max = 500; Pendulum optimal ≈ -150 (higher is better); PointMass random ≈ -42.

Comparison vs Stable-Baselines3 / CleanRL

The scores above are on the built-in environments. To compare decisionrl head-to-head against established libraries on the same Gymnasium tasks, use examples/benchmark_vs_baselines.py. It trains matched algorithms on the same env, over several seeds and an identical step budget, and reports mean ± std return and wall-clock side by side (results saved to JSON).

pip install stable_baselines3          # the SB3 side is skipped if not installed
python examples/benchmark_vs_baselines.py --algos ppo --env CartPole-v1 \
    --seeds 5 --steps 100000

Methodology

  • Same env, same budget, same seeds. Both libraries train on the identical Gymnasium id for the identical total_timesteps, then evaluate the greedy policy over 20 episodes; each library's own evaluate_policy is used.
  • Library defaults. Each side uses its own default hyperparameters (this measures the out-of-the-box experience, not a tuned bake-off). For a fair tuned comparison, pass matched hyperparameters to both.
  • Multiple seeds. Report the mean and std of the per-seed evaluation returns.
  • CleanRL. CleanRL ships single-file reference scripts rather than an installable package, so compare by running the corresponding script (e.g. ppo.py) with the same --env-id, --total-timesteps and --seed, and drop its reported return into the table below.

Results

Run the script on a machine with SB3 installed and paste the emitted table here (placeholder — SB3 was not installed in the environment these docs were generated in):

Algorithm Environment Steps Seeds decisionrl (return) SB3 (return) CleanRL (return)
PPO CartPole-v1 100,000 5 run to fill run to fill run to fill
DQN CartPole-v1 100,000 5 run to fill run to fill run to fill
SAC Pendulum-v1 30,000 5 run to fill run to fill run to fill

Atari and MuJoCo tasks work the same way once their extras are installed (pip install "gymnasium[atari,accept-rom-license,mujoco]"); decisionrl reaches them via make_env("gym:ALE/Breakout-v5") and make_atari.