Overview

Full Changelog: v5.4.0...v5.4.1

Fixed

• Fixes the CNNModel to support 2D observations only by @ClemensSchwarke in #215

Overview

This release introduces a new LogWriter base class that makes it easy to add custom logging services (e.g., ClearML, SwanLab, MLFlow, ...). A new subclass can be passed via a class_name key in the configuration dictionary, the same way custom models and algorithms are passed.

Full Changelog: v5.3.0...v5.4.0

Added

• Adds passing custom logging services by @ClemensSchwarke and @MarkusPorti in #211

Fixed

• Preserve observation tensors dtype in the rollout storage by @juliabeliaeva in #212
• Include day count in long training log durations by @masahiro-kubota in #210

Breaking Changes

• WandbSummaryWriter/NeptuneSummaryWriter were renamed to WandbLogWriter/NeptuneLogWriter.

New Contributors

• @MarkusPorti made their first contribution in #211
• @juliabeliaeva made their first contribution in #212
• @masahiro-kubota made their first contribution in #210

Overview

This release adds a Beta distribution that is beneficial for bounded action spaces, as samples are guaranteed to be inside the specified bounds.

Full Changelog: v5.2.0...v5.3.0

Added

• Adds a Beta distribution by @ClemensSchwarke in #202

Fixed

• Fixes the logger by taking the union of keys across the window instead of sampling from index 0 by @kevinzakka in #205
• Fixes bug #130 by saving and restoring the critic's hidden state in compute_returns() by @ClemensSchwarke in #206

Overview

This release adds the option to keep a fixed standard deviation for the Gaussian distribution, and adds standard deviation clamping to all distributions.

Full Changelog: 5.1.0...v5.2.0

Added

• Adds clip and constant std functionalities to Gaussian Distribution by @epalmaEth in #201

New Contributors

• @epalmaEth made their first contribution in #201

Overview

This release introduces model compilation via torch.compile, which can speed up training especially for large networks like CNNs. For example, the Isaac-Dexsuite-Kuka-Allegro-Lift-v0 task in Isaac Lab trains 1.3x faster with the compilation mode default. For simple networks like MLPs no speed up is expected. The release also includes a clean up of PPO, moving a large part of the extension logic to their respective files rnd.py and symmetry.py.

Full Changelog: v5.0.1...v5.1.0

Added

• Adds Torch compile for PPO and Distillation by @ClemensSchwarke and @adenzler-nvidia in #199
• Cleans up PPO by moving extension logic to the extension files by @ClemensSchwarke in #200

New Contributors

• @adenzler-nvidia made their first contribution in #199

Overview

Full Changelog: v5.0.0...v5.0.1

Fixed

• Fixes shared CNN modules in _OnnxCNNModel and _TorchCNNModel by @kevinzakka in #189
• Fixes _OnnxCNNModel forward signature for ONNX export by @kevinzakka in #187

Overview

This release introduces a new Batch class and a new Distribution class. The Batch class eliminates the possibility of switching tensors due to incorrect positional argument ordering. The Distribution class makes it easy to add new distributions without modifying the models directly. Furthermore, the library now has a small documentation that can be found here, as well as a test suite. Lastly, a new NAN check makes it easier to detect and debug NAN values from the environment. With this release, the main structural changes to the library are complete, and the library will be more stable going forward.

Isaac Lab users can refer to this PR until it is merged, which automatically converts old configurations to the new structure.

Full Changelog: v4.0.1...v5.0.0

Added

• Adds a batch class to avoid positional arguments in generators by @ClemensSchwarke in #172
• Adds a distribution class for easier adaptability by @ClemensSchwarke in #173
• Adds a small documentation by @ClemensSchwarke in #183
• Adds tests to the library by @ClemensSchwarke in #184
• Adds NAN check to avoid ambiguous std >= 0.0 error by @ClemensSchwarke in #185

Fixed

• Fixes code state logging by @DreaverZhao in #178
• Fixes multi gpu unbounded growth due to logging by @VineetTambe in #179
• Fixes missing raise and wrong RND broadcast index by @jashshah999 in #180
• Fixes mutable default arguments by @jashshah999 in #181

Breaking Changes

The configuration setup has changed. Instead of requiring noise parameters (stochastic, init_noise_std, noise_std_type, state_dependent_std), models now require a DistributionCfg. The following example demonstrates the necessary changes using the configuration style of Isaac Lab:

Instead of defining:

 actor = RslRlMLPModelCfg(
        hidden_dims=[512, 256, 128],
        activation="elu",
        obs_normalization=False,
        stochastic=True,
        init_noise_std=1.0,

    )

define:

 actor = RslRlMLPModelCfg(
        hidden_dims=[512, 256, 128],
        activation="elu",
        obs_normalization=False,
        distribution_cfg=RslRlMLPModelCfg.GaussianDistributionCfg(init_std=1.0),
    )

More details can be found in the new documentation.

New Contributors

• @DreaverZhao made their first contribution in #178
• @VineetTambe made their first contribution in #179
• @jashshah999 made their first contribution in #180

Overview

Full Changelog: v4.0.0...v4.0.1

Fixed

• Fixes CNNModel init ordering bugs by @kevinzakka in #175

Overview

This release introduces a new library structure and many new features described in the following. These changes require a new configuration setup, detailed in the breaking changes section below. Isaac Lab users can refer to this PR until it is merged, which automatically converts old configurations to the new structure.

1. The ActorCritic and StudentTeacher classes got split, allowing all models to be an instance of the same class, e.g., MLPModel. This way, a new architecture has to be implemented only once and can be reused for any purpose. This makes expecially distillation more streamlined, as new architectures do not have to be implemented twice, once for the actor and once for the teacher. Additionally, all code duplication that existed in the ActorCritic and StudentTeacher classes could be removed.

2. Some directories got renamed for a more consistent structure:

   • RND and Symmetry got moved to a new "extensions" folder
   • The "modules" folder got renamed to "models" (following the new naming of models)
   • The "networks" folder got renamed to "modules" (as, e.g., normalization is not a network)

3. Some functions got moved from the Runner to the Algorithm, as they are algorithm specific. Those include: train_mode(), eval_mode(), save(), load(), construct_algorithm().

4. Video logging for W&B was added, such that videos recorded in, e.g., IsaacLab, are automatically uploaded.

5. Policy export functions were added, such that one function call to the runner returns either a JIT or ONNX model of the policy via export_policy_to_jit() and export_policy_to_onnx()

6. An option for sharing the CNN encoders between actor and critic was added.

7. A load_cfg argument was added to the load() function, allowing the user to specify what models/states to load.

Full Changelog: v3.3.0...v4.0.0

Added

• Separates actor and critic by @ClemensSchwarke in #159
• Moves functions from Runners to Algorithms by @ClemensSchwarke in #164
• Adds video logging for W&B by @ClemensSchwarke in #167
• Adds model exports to ONNX and JIT files by @ClemensSchwarke in #170
• Adds option to share CNN encoders between actor and critic models by @ClemensSchwarke in #171

Fixed

• Fixes multi-gpu initialization with W&B logging by @ClemensSchwarke in #166

Breaking Changes

The configuration setup has changed. Instead of requiring a policy configuration, separate actor and critic / student and teacher configurations are needed. The following example demonstrates the necessary changes using the configuration style of Isaac Lab:

Instead of defining:

policy = RslRlPpoActorCriticCfg(
        init_noise_std=1.0,
        actor_obs_normalization=False,
        critic_obs_normalization=False,
        actor_hidden_dims=[512, 256, 128],
        critic_hidden_dims=[512, 256, 128],
        activation="elu",
    )

define:

 actor = RslRlMLPModelCfg(
        hidden_dims=[512, 256, 128],
        activation="elu",
        obs_normalization=False,
        stochastic=True,
        init_noise_std=1.0,
    )
 critic = RslRlMLPModelCfg(
        hidden_dims=[512, 256, 128],
        activation="elu",
        obs_normalization=False,
        stochastic=False,
    )

Overview

This release enables passing custom classes without having to modify the library. For example, you can now define a custom actor-critic class in a different repository and directly pass it to RSL-RL.

Full Changelog: v3.2.0...v3.3.0

Added

• Adds direct class passing by @ClemensSchwarke in #145

Fixed

• Fixes a small typo in the cnn.py by @CCanxue in #153

New Contributors

• @CCanxue made their first contribution in #153