Environments¶
All environments follow the Gymnasium API
(reset(seed=...) -> (obs, info), step(action) -> (obs, reward, terminated,
truncated, info)) and need no external dependencies.
Classic control¶
| Env | Class | Obs | Action |
|---|---|---|---|
| Grid navigation | GridWorld |
Discrete or one-hot Box | Discrete(4) |
| Multi-armed bandit | MultiArmedBandit |
Box(1) | Discrete(k) |
| CartPole | CartPole |
Box(4) | Discrete(2) |
| Pendulum swing-up | Pendulum |
Box(3) | Box(1) |
| Point-mass reach | PointMass |
Box(n) | Box(n) |
| Mountain Car | MountainCar |
Box(2) | Discrete(3) |
| Mountain Car (continuous) | MountainCarContinuous |
Box(2) | Box(1) |
| Acrobot | Acrobot |
Box(6) | Discrete(3) |
| Bit-flipping (goal, sparse) | BitFlipping |
Box(2n) | Discrete(n) |
Complex scenarios¶
Higher-dimensional, harder tasks spanning distinct domains — richer observations, non-linear dynamics and real credit-assignment / exploration challenges.
| Env | Class | Obs | Action | Domain |
|---|---|---|---|---|
| Two-link reaching arm | ReacherArm |
Box(10) | Box(2) | robotic manipulation |
| 2-D maze navigation (lidar) | Navigation2D |
Box(14) | Box(2) | navigation / hard exploration |
| Lunar lander | LunarLander |
Box(8) | Discrete(4) | rocket soft-landing control |
| Portfolio allocation | PortfolioAllocation |
Box(4·n) | Box(n) | finance / sequential allocation |
- ReacherArm — 2-DoF torque-controlled arm; dense negative-distance reward with a control-effort penalty. Non-linear kinematics; solve with SAC / TD3.
- Navigation2D — point robot with momentum and lidar range sensors must reach a
goal past walls with gaps; progress reward, collision penalty, terminal bonus.
Pairs well with the
decisionrl.explorationcuriosity bonuses. - LunarLander — self-contained 2-D rigid-body lander; potential-based shaping (distance, speed, tilt, legs, fuel) plus a large land/crash terminal reward.
- PortfolioAllocation — allocate across correlated assets with AR(1) momentum returns and transaction costs; recent returns are predictive, so the optimal policy is not static and should beat equal-weight.
Applied (operational decisions)¶
The flagship set: environments modelling the decisions businesses actually make.
Each pairs with a classic operations-research baseline so a learned policy can be
proved better, not just asserted (see examples/applied_rl_demo.py).
| Env | Class | Obs | Action | Problem | Baseline to beat |
|---|---|---|---|---|---|
| Inventory management | InventoryManagement |
Box(1) | Discrete | order under stochastic demand | base-stock ("order up to S") |
| Non-stationary inventory | NonstationaryInventory |
Box(2) | Discrete | order as the demand rate drifts between regimes | best fixed base-stock (RL beats it) |
| Thermostat / HVAC | Thermostat |
Box(2) | Box(1) | hold a setpoint at minimal energy | bang-bang |
| Dynamic pricing | DynamicPricing |
Box(2) | Discrete | price a finite stock over a deadline | best fixed price |
| Queue admission control | QueueAdmissionControl |
Box(2) | Discrete(2) | admit/reject jobs at a busy server | admit-all |
| Energy microgrid | EnergyMicrogrid |
Box(6) | Box(1) | charge/discharge a battery vs price & solar | no battery |
| Supply chain (2-echelon) | SupplyChain |
Box(5) | Box(2) | coordinate orders across retailer + warehouse | per-echelon base-stock |
- NonstationaryInventory — the case where the classic formula breaks: the demand rate switches between a low and a high regime, so no single base-stock level is right. The agent sees inventory plus an EWMA of recent demand (a read on the current regime) and learns an adaptive order-up-to level that beats the best fixed base-stock — the clearest "why RL, not a solver" example.
- DynamicPricing — revenue management: sell limited inventory before a deadline under price-elastic, stochastic demand; the optimal price rises as stock gets scarce relative to time (airline/hotel pricing).
- QueueAdmissionControl — admit a high-value job or shed it to protect a finite buffer from congestion; the optimal policy is a value threshold that tightens as the queue fills (load shedding / call admission).
- EnergyMicrogrid — store cheap/surplus solar energy and discharge it into the evening price peak (battery arbitrage + self-consumption).
- SupplyChain — the "beer game": order across a serial two-echelon chain with shipment lead times, balancing holding vs stockout cost while avoiding bullwhip.
See the applied solutions in the README for trained results.
Gymnasium interop (optional)¶
from decisionrl.envs import make_gym
env = make_gym("CartPole-v1") # requires: pip install "decisionrl[gym]"
Gymnasium environments already match this library's API, so agents consume them
directly; make_gym simply wraps one with decisionrl's Box/Discrete spaces.
For vectorized Gymnasium training use make_gym_vec:
from decisionrl.envs import make_gym_vec
from decisionrl.algorithms import PPO
venv = make_gym_vec("CartPole-v1", num_envs=8, asynchronous=True)
PPO(venv, n_steps=256, seed=0).learn(200_000)
It vectorizes Gymnasium single envs with decisionrl's own vector envs, which use
correct immediate-autoreset and final_observation bootstrapping — stable across
Gymnasium autoreset-API changes.
For Atari, make_atari applies the standard DQN preprocessing (grayscale, resize
to 84×84, frame-skip, 4-frame stack) — ready for the built-in CNN:
from decisionrl.envs import make_atari
from decisionrl.algorithms import DQN
agent = DQN(make_atari("ALE/Pong-v5"), seed=0) # needs: pip install "gymnasium[atari]" ale-py
make_minigrid("MiniGrid-Empty-5x5-v0") (needs pip install minigrid) wraps
MiniGrid navigation envs, and decisionrl.multiagent.make_pettingzoo(...) (needs
pip install pettingzoo) adapts PettingZoo parallel envs to MultiAgentEnv.
Wrappers¶
TimeLimit, NormalizeObservation, NormalizeReward, FrameStack,
FlattenObservation, OneHotObservation, SyncVectorEnv, AsyncVectorEnv.