Context

#3265 added a try/except around trainer.fit in src/gluonts/torch/model/estimator.py so that training can survive the StudentTOutput NaN ValueError described in the PR description. The current implementation (dev @ a0af774, lines 210-240) is:

try:
    trainer.fit(
        model=training_network,
        train_dataloaders=training_data_loader,
        val_dataloaders=validation_data_loader,
        ckpt_path=ckpt_path,
    )
except Exception as e:
    if checkpoint.best_model_path == "":
        logger.error("Training failed with no checkpoint available")
        raise
    else:
        logger.warning(
            "Recovering from a checkpoint after training failed "
            f"with the following exception:\n {e}"
        )

if checkpoint.best_model_path != "":
    logger.info(f"Loading best model from {checkpoint.best_model_path}")
    best_model = training_network.__class__.load_from_checkpoint(
        checkpoint.best_model_path
    )
else:
    best_model = training_network

Observations

1. The catch is too wide. except Exception matches the targeted ValueError, but also CUDA OOM (torch.cuda.OutOfMemoryError), disk-full errors from dataloader checkpoints, shape mismatches from mis-configured user features, pickling failures, and Lightning's own MisconfigurationException. After any of these the training log shows a single WARNING and the function returns a stale checkpoint from a much earlier epoch, which users then evaluate and possibly promote to production. The failure signal that should have been a hard stop becomes a silent accuracy regression.

2. The original traceback is discarded. e is formatted with f\"{e}\", so only the exception's args are logged. Users lose the file/line/frames they need to decide whether the exception was NaN-loss (fine to recover from) or a real bug. logger.exception(...) or logger.warning(..., exc_info=True) both preserve the stack.

3. Recovery restores only model weights. training_network.__class__.load_from_checkpoint(...) rebuilds the module state dict; it does not restore the optimizer, LR scheduler, RNG, or dataloader iterator. For a short run that is fine — but the warning message does not say so, and users reading "recovering from a checkpoint" reasonably expect the full training state. A longer-running pipeline that wraps train(...) and then continues with additional gradient steps will silently diverge from a run that never hit the exception.

4. No retry / attempt counter. Nothing prevents a user from wrapping train(...) in their own retry loop and silently burning through N failed runs, each of which recovers from the very first checkpoint and reports success.

Suggested direction

• Narrow the catch to the exceptions this code was designed to recover from. (ValueError, RuntimeError) covers both the NaN-loss ValueError described in Recover from a checkpoint if training failed with an exception #3265 and the "Expected parameter ... to satisfy the constraint" RuntimeError variant that PyTorch sometimes raises instead. Let torch.cuda.OutOfMemoryError, KeyboardInterrupt, SystemExit, and Lightning's MisconfigurationException propagate.
• Log with logger.exception(...) / exc_info=True so the traceback lives in the log.
• If full trainer state is worth restoring, Lightning already does it: instead of load_from_checkpoint after the catch, pass ckpt_path=checkpoint.best_model_path on a re-trainer.fit(...) call. If that is out of scope, at minimum document (in the warning message and the estimator docstring) that only weights are restored.
• Consider an optional max_recoveries knob — default 1 — so a flapping training is surfaced to the user by the second failure instead of silently truncating every attempt.

I am happy to send a PR if the direction above is agreeable; I wanted to open the issue first since the trade-off around "narrow the catch vs. keep the safety net wide" is a design call for the maintainers.