Part 1
Overfit test — full V + T
Same recipe, three backbones. Does the model memorize the ~30 training episodes?
v3 · baseline
Bidirectional temporal attention (full-window). The original Cosmos-tokenizer world model.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0955 | 0.0827 | -13% |
| SSIM ↑ | 0.8190 | 0.8529 | -4% |
| PSNR ↑ | 25.3119 | 26.5329 | -5% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1508 | 0.0839 | -44% |
| LPIPS · right ↓ | 0.1933 | 0.0661 | -66% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.3559 | 0.1712 | +52% |
| contact MSE ↓ | 131.7636 | 34.5099 | -74% |
| onset err ↓ | 0.0000 | 104.3420 | — |
| intensity spearman ↑ | 0.3031 | 0.5875 | -94% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
v3 · causal
Causal temporal attention (per-frame tril mask) — matches the causal tokenizer and AR rollout regime.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0530 | 0.0454 | -14% |
| SSIM ↑ | 0.8789 | 0.9016 | -3% |
| PSNR ↑ | 28.0306 | 29.3914 | -5% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1612 | 0.0856 | -47% |
| LPIPS · right ↓ | 0.2013 | 0.0667 | -67% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.2755 | 0.1542 | +44% |
| contact MSE ↓ | 144.7365 | 33.4357 | -77% |
| onset err ↓ | 27.4073 | 0.0000 | -100% |
| intensity spearman ↑ | 0.3281 | 0.5312 | -62% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
v3 · contact-aware
Adds the contact-aware tactile aux loss (x0-space, alpha_bar-weighted) on top of v3.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0659 | 0.0508 | -23% |
| SSIM ↑ | 0.8553 | 0.8887 | -4% |
| PSNR ↑ | 26.7893 | 28.5571 | -7% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1350 | 0.0735 | -46% |
| LPIPS · right ↓ | 0.1952 | 0.0639 | -67% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.3481 | 0.2051 | +41% |
| contact MSE ↓ | 130.1456 | 32.8804 | -75% |
| onset err ↓ | 0.0000 | 0.3817 | — |
| intensity spearman ↑ | 0.4219 | 0.5844 | -39% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
Part 2
Cross-architecture — causal vs noncausal
The same causal-backbone swap under three modality configurations. Val metrics; the winning backbone per metric is highlighted.
Full V + T — causal vs noncausal
Three views + two tactile sensors. Swapping the backbone from bidirectional (v3) to causal.
noncausal vs causal (val)
| metric (val) | noncausal | causal |
|---|---|---|
| Visual — middle view | ||
| LPIPS ↓ | 0.0827 | 0.0454 |
| SSIM ↑ | 0.8529 | 0.9016 |
| PSNR ↑ | 26.5329 | 29.3914 |
| Tactile | ||
| LPIPS · left ↓ | 0.0839 | 0.0856 |
| LPIPS · right ↓ | 0.0661 | 0.0667 |
| mask IoU (τ8) ↑ | 0.1712 | 0.1542 |
| contact MSE ↓ | 34.5099 | 33.4357 |
| onset err ms ↓ | 104.3420 | 0.0000 |
Rollouts — noncausal | causal
noncausal · val_loss 0.1234
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
causal · val_loss 0.1647
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
Vision-only — causal vs noncausal
Tactile stream removed — isolates how the causal backbone affects pure visual prediction.
noncausal vs causal (val)
| metric (val) | noncausal | causal |
|---|---|---|
| Visual — middle view | ||
| LPIPS ↓ | 0.0687 | 0.0584 |
| SSIM ↑ | 0.8555 | 0.8767 |
| PSNR ↑ | 26.5801 | 27.8866 |
Rollouts — noncausal | causal
noncausal · val_loss 0.0297
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
causal · val_loss 0.0347
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
Tactile-only — causal vs noncausal
Visual stream removed — isolates the causal backbone's effect on tactile / contact prediction.
noncausal vs causal (val)
| metric (val) | noncausal | causal |
|---|---|---|
| Tactile | ||
| LPIPS · left ↓ | 0.0790 | 0.0790 |
| LPIPS · right ↓ | 0.0667 | 0.0676 |
| mask IoU (τ8) ↑ | 0.1748 | 0.1674 |
| contact MSE ↓ | 34.2550 | 34.4423 |
| onset err ms ↓ | 0.0000 | 87.2179 |
Rollouts — noncausal | causal
noncausal · val_loss 0.1084
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
causal · val_loss 0.1208
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
Part 3
Tactile reference frame — fixed vs random
Does sampling a random no-contact reference (vs one fixed p01 anchor) fix the tactile train/val divergence? The gap collapses and visual val LPIPS roughly halves — but held-out tactile contact quality (val IoU / spearman) stays flat: the pathology is removed, the data ceiling remains. The V→T aux head adds nothing.
fixed p01 ref · baseline
Tactile delta measured against ONE fixed no-contact anchor per episode. The suspected-overfit baseline (train↓ / val plateau).
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0955 | 0.0827 | -13% |
| SSIM ↑ | 0.8190 | 0.8529 | -4% |
| PSNR ↑ | 25.3119 | 26.5329 | -5% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1508 | 0.0839 | -44% |
| LPIPS · right ↓ | 0.1933 | 0.0661 | -66% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.3559 | 0.1712 | +52% |
| contact MSE ↓ | 131.7636 | 34.5099 | -74% |
| onset err ↓ | 0.0000 | 104.3420 | — |
| intensity spearman ↑ | 0.3031 | 0.5875 | -94% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
random no-contact ref
A random pixel-labeled no-contact frame is sampled as the reference per window (train aug). Collapses the tactile train/val loss gap 1.39→1.09 and roughly halves visual val LPIPS.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0395 | 0.0419 | +6% |
| SSIM ↑ | 0.9061 | 0.9001 | +1% |
| PSNR ↑ | 29.2115 | 29.1317 | +0% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.0839 | 0.0813 | -3% |
| LPIPS · right ↓ | 0.1151 | 0.0698 | -39% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.4778 | 0.1734 | +64% |
| contact MSE ↓ | 92.2412 | 35.5656 | -61% |
| onset err ↓ | 0.0000 | 0.0000 | — |
| intensity spearman ↑ | 0.3500 | 0.5219 | -49% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
random ref + V→T aux
Same random reference plus the V→T auxiliary head (weight 0.5). Adds no gain — slightly worse tactile/visual than random-ref alone; V→T can be dropped.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0473 | 0.0460 | -3% |
| SSIM ↑ | 0.8962 | 0.8977 | -0% |
| PSNR ↑ | 28.5951 | 28.8901 | -1% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1010 | 0.0765 | -24% |
| LPIPS · right ↓ | 0.1582 | 0.0687 | -57% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.4096 | 0.1762 | +57% |
| contact MSE ↓ | 115.7422 | 34.2311 | -70% |
| onset err ↓ | 27.3989 | 0.0000 | -100% |
| intensity spearman ↑ | 0.3625 | 0.5844 | -61% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
Part 4
Reference refinements — clean bank & difference-image
Two principled attempts to push past the random-reference result: (1) guarantee the reference latent is clean via a replicate-16 bank; (2) target the encoded pixel-difference instead of the latent-difference. Neither beats plain random-reference — clean-bank is a wash (pool contamination was negligible), and difference-image is worst (its decode+ref reconstruction adds error, even though the model predicts the diff latent near-perfectly). The tactile ceiling is data + tokenizer fidelity, not reference cleanliness or diff-space.
random ref (in-video)
The winner so far: random in-video no-contact reference, difference-LATENT target. Baseline for the two refinements.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0395 | 0.0419 | +6% |
| SSIM ↑ | 0.9061 | 0.9001 | +1% |
| PSNR ↑ | 29.2115 | 29.1317 | +0% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.0839 | 0.0813 | -3% |
| LPIPS · right ↓ | 0.1151 | 0.0698 | -39% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.4778 | 0.1734 | +64% |
| contact MSE ↓ | 92.2412 | 35.5656 | -61% |
| onset err ↓ | 0.0000 | 0.0000 | — |
| intensity spearman ↑ | 0.3500 | 0.5219 | -49% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
clean reference bank
Guarantee the reference latent is clean via a replicate-16 bank (no in-video ti≥1 contamination). No gain — decoded LPIPS slightly worse, contact IoU tied: pool contamination was negligible in practice.
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0476 | 0.0433 | -9% |
| SSIM ↑ | 0.8970 | 0.9040 | -1% |
| PSNR ↑ | 28.8151 | 29.3747 | -2% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1007 | 0.0876 | -13% |
| LPIPS · right ↓ | 0.1427 | 0.0743 | -48% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.4012 | 0.1763 | +56% |
| contact MSE ↓ | 122.4160 | 35.8125 | -71% |
| onset err ↓ | 0.0000 | 51.4189 | — |
| intensity spearman ↑ | 0.3344 | 0.4750 | -42% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
difference image
Target = encode((tac−ref)/255) on the fly; reconstruct tac ≈ decode(pred)+ref. The model predicts the diff latent near-perfectly, but the decode+ref roundtrip adds error → WORST decoded quality. (Its latent contact metrics are in diff-space, not comparable.)
Overfit test — train vs val
| metric | train (n=16) | val (n=16) | gap |
|---|---|---|---|
| Visual — middle view | |||
| LPIPS ↓ | 0.0505 | 0.0453 | -10% |
| SSIM ↑ | 0.8939 | 0.8965 | -0% |
| PSNR ↑ | 28.7100 | 28.8675 | -1% |
| Tactile — perceptual | |||
| LPIPS · left ↓ | 0.1271 | 0.0928 | -27% |
| LPIPS · right ↓ | 0.1670 | 0.0779 | -53% |
| Tactile — contact / physics | |||
| mask IoU (τ8) ↑ | 0.2804 | 0.0672 | +76% |
| contact MSE ↓ | 149.4884 | 46.5032 | -69% |
| onset err ↓ | 0.0000 | — | — |
| intensity spearman ↑ | 0.2656 | 0.5062 | -91% |
Short-horizon rollouts
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16
sample 000
0/16
sample 001
0/16
sample 002
0/16
sample 003
0/16