A PyTorch implementation of a Contrastive Denoising Autoencoder that compresses frozen BERT sentence embeddings into a low-dimensional latent space, trained on SNLI. The model combines a denoising reconstruction objective with an InfoNCE contrastive objective (SimCSE-style dual-pass augmentation) to learn a compact, semantically structured, and noise-robust sentence representation.

Each training step processes a sentence twice:

1. Clean pass — the original sentence is embedded with a frozen BERT backbone and encoded into a 128-dim latent vector z_original.
2. Perturbed pass — a noisy twin of the sentence (via synonym replacement, word dropout, or masking) is embedded and encoded into z_perturbed.

Two losses are computed on top of these:

• InfoNCE (contrastive) loss pulls z_original and z_perturbed together relative to other sentences in the batch. Gradients from this loss reach only the encoder.
• Reconstruction (denoising) loss trains the decoder to reconstruct the original clean 768-dim embedding from both z_original and z_perturbed. Gradients from this loss flow through both the decoder and the encoder.

The final sentence representation used downstream is the 128-dimensional bottleneck vector z, not the 768-dim decoder reconstruction.

sentence ──► perturbation──► BERT (frozen)──► 768-dim embedding ──Encoder──► z (128-dim latent)
                                                              │
                                                    ┌─────────┴─────────┐
                                              InfoNCE loss         Decoder
                                           (encoder only)     ──► 768-dim reconstruction
                                                                   (reconstruction loss →
                                                                    encoder + decoder)

git clone <your-repo-url>
cd <your-repo-name>
pip install torch transformers datasets pandas numpy scikit-learn nltk tqdm matplotlib umap-learn

NLTK resources (WordNet, stopwords, POS tagger, punkt) are downloaded automatically on first run of perturbation.py.

All commands are run through main.py:

# Train from scratch
python main.py train --epochs 20 --batch-size 512

# Resume training from a checkpoint
python main.py train --resume outputs/checkpoints/epoch_10.pt

# Run the robustness-to-perturbation sweep
python main.py robustness --checkpoint outputs/checkpoints/best.pt

# Run the full pipeline: train -> robustness
python main.py all --epochs 20

Common overrides available on every subcommand: --model-key, --batch-size, --max-samples, --seed.

• Backbone: BERT (bert-base-uncased), frozen by default (FREEZE_ENCODER = True); RoBERTa, SimCSE, and E5 are also configured as swappable options.
• Pooling: mean pooling over token embeddings (configurable to CLS pooling).
• Encoder: 768 → 512 → 256 → 128, with BatchNorm + ReLU + Dropout on hidden layers; the final latent projection is a plain linear layer (no activation), since InfoNCE similarity is computed directly on z.
• Decoder: 128 → 256 → 512 → 768, mirroring the encoder.
• Contrastive loss: symmetric InfoNCE with in-batch negatives, temperature 0.07.
• Reconstruction loss: MSE between recon(perturbed) and the clean target embedding (denoising term), plus an identity term on recon(original) to keep the clean pathway well-behaved.

robustness.py sweeps perturbation strength (0.1–0.9) across synonym/dropout/mask methods, comparing cosine stability of the learned 128-dim latent space against the raw 768-dim BERT embedding baseline. Because these two spaces have different dimensionality, raw cosine similarities are not directly comparable across them (concentration-of-measure effects) — a random-pair cosine baseline should be used to calibrate each space before comparing the two.

If you use this code in your research, please cite the accompanying ICCKE submission (details to be added upon acceptance).

Add a license of your choice (e.g. MIT) before publishing.