CLAP: Cross-Embodiment Video World Models as Zero-Shot Physical Simulators

Introduction

The Challenge of Cross-Embodiment Physical Simulation

Imagine you're a game developer tasked with creating a physics system for a procedurally generated creature—say, a six-legged insect with articulated antennae. Your engine has a perfectly tuned ragdoll system for humanoids, a servomotor model for quadrupedal robots, and a fluid dynamics solver for aquatic life. None of it transfers. You either hand-author the insect's movement or spend weeks tweaking constraints until it stops clipping through the floor.

Robotics researchers face the same problem, but with higher stakes. A manipulation policy trained on a 7-DoF robot arm doesn't transfer to a humanoid hand. A locomotion controller for a quadruped is useless on a wheeled rover. Each new embodiment requires collecting new data, training new models, and debugging new failure modes. The field has largely accepted this as the cost of doing business.

But what if a single world model could simulate any embodiment—seen or unseen—without a single parameter update?

What is CLAP? A Brief Overview

CLAP (Cross-embodiment Latent Action Pretraining) is a method for training video world models that generalize across different robot embodiments without requiring paired cross-embodiment data. The core insight is deceptively simple: instead of conditioning a video prediction model on embodiment-specific action commands (joint torques, end-effector velocities, etc.), CLAP learns a universal latent action space—a shared, low-dimensional representation of motion primitives that abstracts away the physical specifics of any particular body.

The training pipeline works as follows: you collect unpaired videos from multiple embodiments (human hands, robot arms, quadrupeds, quadcopters). Each video is processed by an encoder that extracts latent actions—a compressed representation of what is happening between frames, independent of who is doing it. A contrastive loss pulls together latent actions from different embodiments that produce similar visual transitions. The world model learns to predict future frames conditioned on these latent actions.

At inference time, you feed the model a video of a new embodiment it has never seen. The model encodes the current state, searches over latent actions to find one that produces the desired visual outcome, and generates plausible future trajectories. Zero-shot. No fine-tuning. No paired data.

Why This Matters for Gaming and Robotics

For robotics, CLAP suggests a path toward general-purpose physical reasoning. A robot could observe a human demonstration, infer the latent action, and plan its own execution without needing a cross-embodiment dataset. For gaming, the implications are even broader: a single world model could simulate any character morphology, from bipedal knights to snake-like bosses to amorphous blob enemies, using the same latent action space.

This isn't just about saving development time—it's about enabling emergent physics. Game characters could be designed procedurally, and the world model would handle their dynamics automatically. Physics-based animation could be generated on the fly for any creature, without hand-authored state machines.

Article Scope and Structure

This deep-dive covers the CLAP framework from first principles to implementation details. We'll examine the architecture, walk through the training pipeline, analyze the experimental results, and explore how this technology could transform game development workflows. We'll also be honest about the limitations—because there are several.

---

Background and Related Work

World Models in Reinforcement Learning and Simulation

World models have been a staple of model-based RL since the 2010s. The idea: learn a predictive model of the environment dynamics, then use that model for planning or policy optimization instead of interacting with the real environment. Classic approaches like Dreamer (Hafner et al., 2020) and PlaNet (Hafner et al., 2019) learn latent state representations from image sequences and use them for control.

The key limitation of these models is embodiment-specificity. Dreamer trained on a Cheetah environment can't simulate a Walker. The latent state space encodes the specific physics of the agent's body, and the action space is tied to the specific actuator configuration. Transfer requires retraining from scratch.

Latent Action Spaces: From Single-Embodiment to Cross-Embodiment

The concept of latent action spaces emerged from inverse RL and imitation learning. Instead of using ground-truth actions (which require instrumentation), researchers learned to infer actions from visual observations alone. The Latent Action Model (LAM) (Schmidt et al., 2024) trains an encoder to map a pair of frames to a latent action, and a decoder to predict the next frame given the current frame and the latent action.

These models work well for single embodiments, but they don't naturally extend across different bodies. The latent action learned for a robot arm—"move end-effector 5cm left"—has no meaning for a quadcopter. CLAP's contribution is to align these latent spaces across embodiments using contrastive learning.

Contrastive Learning for Representation Alignment

Contrastive learning has been a dominant paradigm for representation learning in computer vision (SimCLR, MoCo, CLIP). The idea is to pull positive pairs together and push negative pairs apart in embedding space. CLAP applies this principle to latent actions: if a visual transition in a human hand video (fingers closing) produces a similar visual transition in a robot gripper video (jaws closing), the corresponding latent actions should be close in embedding space.

The critical design choice is what constitutes a positive pair. CLAP uses a clever trick: the same visual transition rendered in different embodiments. But since we don't have paired data, the model learns to align latent actions that produce similar visual outcomes—measured by the similarity of the predicted next-frame features.

Video Prediction as a Physical Simulator

Video prediction models (VideoGPT, VideoPoet, etc.) have shown impressive results in generating plausible future frames. The question is whether these models actually learn physics or just statistical regularities. CLAP's claim is stronger: by training on diverse embodiments with a shared latent action space, the model learns abstract physics (gravity, momentum, contact dynamics) that transfer across bodies.

Limitations of Prior Approaches

• Single-embodiment world models (Dreamer, PlaNet): Cannot generalize to new morphologies without retraining.
• Paired cross-embodiment datasets: Extremely expensive to collect; require the same task to be performed by multiple robots with synchronized observations.
• Hand-crafted action spaces: Require human expertise to design for each embodiment; don't scale to novel morphologies.
• Video prediction without latent actions: Can generate plausible frames but lacks the controllability needed for planning and simulation.

---

The CLAP Framework: Core Concepts

Problem Formulation: Learning a Universal Latent Action Space

Formally, CLAP learns a world model $p_\theta(s_{t+1} | s_t, a^{latent})$ where $a^{latent} \in \mathbb{R}^d$ is a latent action vector. The model is trained on data from $K$ embodiments, each with its own observation distribution $\mathcal{O}_k$. The training data is unpaired: we have videos from each embodiment, but no cross-embodiment correspondences.

The objective has two components:

1. Video prediction loss: The model must predict future frames accurately for each embodiment.
2. Contrastive alignment loss: Latent actions that produce visually similar transitions in different embodiments must be pulled together in embedding space.

The key insight is that the video prediction loss provides the signal (what latent actions do), and the contrastive loss provides the alignment (which latent actions are equivalent across embodiments).

Architecture Overview: Transformer-Based Video Prediction

CLAP uses a transformer-based architecture for the world model. The input is a sequence of frames (patchified and embedded) plus a latent action token. The transformer processes this sequence and outputs a predicted next frame.

Specifically: - Encoder: A ViT-based encoder processes the current frame $s_t$ and produces a set of patch embeddings. - Latent action token: A learned embedding $\mathbf{a} \in \mathbb{R}^d$ is prepended to the sequence. - Transformer backbone: A standard causal transformer processes the sequence, producing contextualized embeddings. - Decoder: A lightweight decoder (e.g., a transposed convolution or a second ViT) maps the output embeddings back to pixel space.

The model is trained to minimize the mean squared error (or a perceptual loss) between the predicted and actual next frame.

Training Objective: Contrastive Alignment of Latent Actions

The contrastive loss operates on the latent action space. During training, we sample a batch of transitions $(s_t, a^{latent}, s_{t+1})$ from different embodiments. For a given anchor transition, we define:

• Positive pairs: Transitions from different embodiments that produce similar visual changes (e.g., an object being pushed left).
• Negative pairs: Transitions that produce different visual changes.

Since we don't have ground-truth labels for "similar visual change," CLAP uses a feature-space similarity measure. The encoder produces a feature vector for each frame; the difference between $s_t$ and $s_{t+1}$ features captures the visual change. Transitions with similar feature differences are treated as positive pairs.

The contrastive loss is a standard InfoNCE loss:

$$\mathcal{L}{contrast} = -\log \frac{\exp(\text{sim}(a_i, a_j^+) / \tau)}{\sum \exp(\text{sim}(a_i, a_k) / \tau)}$$

where $\text{sim}$ is cosine similarity and $\tau$ is a temperature parameter (set to 0.07 in the paper).

Key Takeaway: The contrastive loss is what makes CLAP cross-embodiment. Without it, the latent actions would be embodiment-specific. With it, the model learns a shared "motion vocabulary" that abstracts away body morphology.

Zero-Shot Transfer: How CLAP Generalizes to New Embodiments

At inference time, CLAP receives a video from a novel embodiment. The process:

1. Encode: The current frame is passed through the encoder to get a state representation.
2. Latent action search: The model searches over the latent action space (e.g., via random shooting or CEM) to find a latent action that produces a desired visual goal.
3. Predict: The transformer generates the next frame conditioned on the current state and the chosen latent action.

The zero-shot capability comes from the fact that the latent action space is shared across embodiments. A latent action that means "push object right" in the training data means the same thing for the novel embodiment, even though the specific motor commands would be completely different.

Key Design Choices and Hyperparameters

• Latent action dimension: 32 (from the paper)
• Transformer depth: 12 layers (with 8 attention heads)
• Patch size: 16x16 for video frames
• Batch size: 256
• Contrastive temperature: 0.07
• Optimizer: AdamW with learning rate 1e-4 and cosine decay

The choice of 32 dimensions for the latent action space is a sweet spot: large enough to capture complex motions, small enough for efficient search during planning.

---

Implementation Details

Data Pipeline: Collecting and Preprocessing Multi-Embodiment Videos

The paper uses a dataset of over 1 million video frames from 5 embodiments:

1. 7-DoF robot arm (simulation)
2. Quadcopter (simulation)
3. Human hand (real-world recordings)
4. Quadruped (simulation)
5. Object manipulation with a stick (real-world recordings)

Each video is preprocessed to 64x64 RGB frames at 10 FPS. The frames are normalized to [-1, 1]. No action labels are used—the model learns entirely from pixels.

Model Architecture: Encoder, Decoder, and Latent Action Predictor

class CLAPWorldModel(nn.Module):
    def __init__(self, latent_dim=32, patch_size=16, hidden_dim=512, num_layers=12):
        super().__init__()
        self.patch_embed = nn.Conv2d(3, hidden_dim, kernel_size=patch_size, stride=patch_size)
        self.latent_embed = nn.Linear(latent_dim, hidden_dim)
        self.transformer = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=8),
            num_layers=num_layers
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(hidden_dim, 256, kernel_size=patch_size, stride=patch_size),
            nn.ReLU(),
            nn.ConvTranspose2d(256, 3, kernel_size=patch_size, stride=patch_size),
            nn.Tanh()
        )

    def forward(self, x, latent_action):
        # x: (B, T, C, H, W) - input frames
        # latent_action: (B, latent_dim)
        B, T, C, H, W = x.shape
        # Encode each frame
        patches = self.patch_embed(x.reshape(B*T, C, H, W))  # (B*T, hidden_dim, H/p, W/p)
        patches = patches.flatten(2).transpose(1, 2)  # (B*T, num_patches, hidden_dim)

        # Add latent action token
        action_token = self.latent_embed(latent_action).unsqueeze(1)  # (B, 1, hidden_dim)
        action_token = action_token.unsqueeze(1).expand(B, T, 1, hidden_dim).reshape(B*T, 1, hidden_dim)
        sequence = torch.cat([action_token, patches], dim=1)  # (B*T, num_patches+1, hidden_dim)

        # Transformer
        output = self.transformer(sequence)  # (B*T, num_patches+1, hidden_dim)

        # Decode next frame
        output = output[:, 0]  # Take the action token position
        output = output.reshape(B*T, hidden_dim, 1, 1)
        next_frame = self.decoder(output)  # (B*T, C, H, W)
        return next_frame.reshape(B, T, C, H, W)

Training Procedure: Loss Functions and Optimization

The total loss is a weighted sum of the prediction loss and the contrastive loss:

$$\mathcal{L} = \mathcal{L}{pred} + \lambda \mathcal{L}$$

where $\lambda$ is a hyperparameter (set to 1.0 in the paper).

The prediction loss is a perceptual loss (LPIPS) rather than raw MSE, which produces sharper images. The contrastive loss is computed on a batch of latent actions with their associated visual transitions.

Inference: Generating Future Frames and Planning

For planning, CLAP uses cross-entropy method (CEM) over the latent action space:

def plan_with_cem(model, current_frame, goal_frame, num_iterations=10, num_samples=100):
    # Initialize distribution over latent actions
    mean = torch.zeros(latent_dim)
    std = torch.ones(latent_dim)

    for _ in range(num_iterations):
        # Sample candidate latent actions
        candidates = mean + std * torch.randn(num_samples, latent_dim)

        # Predict next frames
        predicted_frames = model(current_frame.unsqueeze(0), candidates)

        # Compute distance to goal
        distances = torch.norm(predicted_frames - goal_frame, dim=(2, 3, 4))

        # Select top-k
        top_k = distances.topk(10, largest=False).indices
        mean = candidates[top_k].mean(dim=0)
        std = candidates[top_k].std(dim=0)

    return mean

Code Walkthrough: Key Components and Pseudocode

The training loop pseudocode:

for batch in dataloader:
    # batch contains videos from multiple embodiments
    frames = batch['frames']  # (B, T, C, H, W)

    # Encode latent actions using the latent action encoder
    latent_actions = latent_encoder(frames[:, :-1], frames[:, 1:])  # (B, T-1, latent_dim)

    # Predict next frames
    predicted = model(frames[:, :-1], latent_actions)  # (B, T-1, C, H, W)

    # Prediction loss
    pred_loss = perceptual_loss(predicted, frames[:, 1:])

    # Contrastive loss
    # Extract visual transition features
    trans_features = frame_encoder(frames[:, 1:]) - frame_encoder(frames[:, :-1])
    contrast_loss = info_nce_loss(latent_actions, trans_features)

    # Total loss
    loss = pred_loss + contrast_loss
    loss.backward()
    optimizer.step()

Key Takeaway: The latent action encoder is trained jointly with the world model. It learns to compress visual transitions into a compact action representation that the world model can decode back into pixels.

---

Experimental Evaluation

Experimental Setup: Embodiments, Baselines, and Metrics

Embodiments for evaluation: - Trained on: 7-DoF robot arm, quadcopter, human hand, quadruped, stick manipulation - Zero-shot tested on: Snake-like robot (simulation), soft gripper (simulation), and a bipedal robot (simulation)

Baselines: 1. DreamerV2: Single-embodiment world model trained on each embodiment separately 2. LAM (Latent Action Model): Single-embodiment latent action model 3. VideoGPT: Video prediction without latent actions (unconditioned)

Metrics: - Prediction error: LPIPS distance between predicted and actual future frames - Physical plausibility: Human evaluation (whether the generated motion looks physically realistic) - Goal-reaching success: For planning tasks, whether the model can find latent actions that achieve a visual goal

Main Results: Zero-Shot Prediction Performance

The headline result: CLAP achieves a 30% reduction in prediction error on unseen embodiments compared to single-embodiment baselines.

Model	Human Hand (trained)	Snake Robot (unseen)	Soft Gripper (unseen)	Biped (unseen)
DreamerV2 (per-embodiment)	0.052	0.183	0.194	0.178
LAM (per-embodiment)	0.048	0.171	0.186	0.169
VideoGPT (unconditioned)	0.089	0.142	0.151	0.147
CLAP (zero-shot)	0.044	0.112	0.118	0.109

The improvement is most dramatic on unseen embodiments—exactly where single-embodiment models fail entirely.

Ablation Studies: Importance of Contrastive Learning and Latent Space Size

Without contrastive loss (i.e., training the world model with only the prediction loss on multi-embodiment data): The model fails to generalize to unseen embodiments. Prediction error on the snake robot jumps from 0.112 to 0.167. The latent actions become embodiment-specific, and the model essentially memorizes each embodiment's dynamics without learning a shared representation.

Latent space dimension:

Latent dim	Prediction error (unseen)	Planning success rate
8	0.134	62%
16	0.121	71%
32	0.112	78%
64	0.115	74%

The sweet spot is 32 dimensions. Too small, and the latent actions can't capture complex motions. Too large, and the contrastive alignment becomes harder and planning search space grows.

Qualitative Results: Visualizing Cross-Embodiment Simulations

The paper includes striking qualitative examples:

1. Object pushing: A human hand pushes a block. CLAP generates the same pushing motion for a snake robot's head, even though the snake has no "hand" or "fingers." The model abstracts the motion as "apply force to the left side of the block" rather than "curl fingers and extend wrist."

2. Locomotion: A quadruped's galloping motion is transferred to a snake-like body. The snake undulates in a way that produces forward motion, despite never having seen snake locomotion in training.

3. Grasping: A robot arm's pinch grasp is applied to a soft gripper, which conforms to the object rather than clamping rigidly.

Comparison with Single-Embodiment World Models

Single-embodiment models fail catastrophically on unseen embodiments. DreamerV2 trained on a quadruped produces nonsense predictions when fed a snake robot video—it tries to apply quadruped joint dynamics to a body that doesn't have legs. CLAP, by contrast, learns abstract physics (object permanence, gravity, contact dynamics) that transfer across morphologies.

Key Takeaway: The 30% error reduction on unseen embodiments is the core evidence for CLAP's cross-embodiment capability. This isn't a small incremental improvement—it's the difference between a model that works and one that doesn't.

---

Applications in Gaming

Unified Physics Simulation for Diverse Game Characters

Game engines like Unity and Unreal use hand-authored physics constraints for each character type. A humanoid uses a ragdoll with specified joint limits; a snake uses a spline-based system; a spider uses inverse kinematics. CLAP offers a unified alternative: train a world model on a diverse set of motions, then use it to simulate any character morphology.

For example, a game with procedurally generated monsters could use CLAP as the physics backend. The monster's morphology is defined by a mesh, and the world model predicts how it moves based on latent actions. No per-character tuning required.

Procedural Animation and Interaction

Current procedural animation systems (e.g., Motion Matching, learned locomotion controllers) require large motion capture datasets for each character type. CLAP could generate plausible animations for any body plan from a single pretrained model. The latent action space provides a natural interface: game designers could specify a motion as a latent action (e.g., "roll forward," "push object"), and the world model generates the corresponding animation for any character.

Zero-Shot Character Control and Planning

In an open-world game, you might have a player character with a unique morphology (e.g., a slime, a dragon, a shapeshifter). CLAP enables zero-shot control: the game engine encodes the current visual state, searches over latent actions to find one that achieves the player's goal (e.g., "move toward the door"), and generates the resulting animation. No need to train a custom controller for each new character.

Case Study: Simulating a Snake-Like Character Using Humanoid and Quadruped Data

This is the paper's most striking demonstration. The authors train CLAP on humanoid and quadruped videos, then use it to simulate a snake-like robot. The model generates convincing snake locomotion—undulating waves that produce forward motion—without ever seeing a snake in training.

For a game, this means you could create a snake enemy using the same world model that powers your humanoid NPCs. The latent action for "move forward" is the same across both, even though the pixel-level implementation is completely different.

Potential Impact on Game Development Workflows

• Reduced animation costs: No need to hand-author animations for every creature type.
• Emergent physics: Characters can interact with objects in ways that aren't pre-programmed.
• Procedural content: New character morphologies can be added without physics tuning.
• Networked simulation: A single world model could run on the client and server, ensuring consistent physics.

Key Takeaway: The gaming industry's physics pipelines are a collection of bespoke solutions. CLAP points toward a future where one model handles all of it.

---

Limitations and Challenges

Data Requirements and Quality

CLAP requires large amounts of video data from multiple embodiments. The paper uses 1 million frames, which is substantial. For a game studio, collecting this data would require either extensive simulation runs or motion capture sessions. The quality of the data matters enormously—if the training videos have artifacts or unrealistic physics, the world model will learn those artifacts.

Fine-Grained Physical Interaction Accuracy

CLAP excels at coarse-grained prediction (object being pushed, body moving through space) but struggles with fine-grained interactions. A hand grasping a fragile object and applying variable pressure is not captured well. The latent action space is too coarse to encode the subtle differences between a firm grip and a delicate pinch.

Scalability to Complex Environments

The paper evaluates CLAP in relatively simple environments—single objects, flat ground, minimal clutter. Real game environments have complex geometry, multiple interacting objects, and dynamic lighting. Whether CLAP scales to these conditions is an open question.

Ethical and Safety Considerations

For robotics, a world model that generalizes to unseen embodiments raises safety concerns. A model trained on benign manipulation tasks could potentially be used to plan actions for dangerous robots. The paper doesn't address this directly, but it's a real consideration.

Open Problems for Future Research

• Long-horizon prediction: CLAP generates short-term predictions (a few frames). Long-horizon simulation (minutes or hours) accumulates error.
• Multi-object interactions: The model struggles when multiple objects are interacting simultaneously.
• Partial observability: CLAP assumes full observation of the state. Partial occlusion breaks it.

---

Future Directions

Extending CLAP to More Diverse Embodiments and Tasks

The current evaluation covers a modest set of embodiments. Scaling to more diverse morphologies (flying creatures, underwater swimmers, modular robots) would test the limits of the shared latent action space. The hypothesis is that the latent space becomes more abstract and general as more embodiments are added.

Integrating CLAP with Reinforcement Learning for Control

CLAP is currently a simulator, not a controller. Integrating it with RL could enable zero-shot policy transfer: train a policy in the CLAP latent action space, then deploy it on any embodiment. The policy would output latent actions, and the world model would translate them into embodiment-specific motions.

Improving Physical Fidelity with Hybrid Approaches

CLAP is purely learned from pixels, which means it can violate physics in subtle ways (e.g., slight object penetration, momentum conservation violations). Combining CLAP with a lightweight physics engine (e.g., for contact resolution) could give the best of both worlds: the generalization of learned models with the precision of analytic physics.

Real-Time Simulation for Interactive Applications

Current CLAP inference is not real-time—generating a single frame takes ~50ms on an A100 GPU. For gaming, you'd need at least 30 FPS, which requires either model compression, distillation, or specialized hardware. This is a significant engineering challenge.

Community and Open-Source Opportunities

The paper's code and models are open-source. There's an opportunity for the gaming community to build tools on top of CLAP: Unity plugins, Unreal Engine integrations, and asset pipelines that use CLAP for procedural animation.

---

Conclusion

Recap of CLAP's Contributions

CLAP introduces a method for training video world models that generalize across embodiments. The key innovations:

1. Latent action space: A universal, low-dimensional representation of motion primitives that abstracts away body morphology.
2. Contrastive alignment: A training objective that aligns latent actions across embodiments without paired data.
3. Zero-shot transfer: The ability to simulate novel embodiments without fine-tuning.

Implications for Robotics and Gaming

For robotics, CLAP offers a path toward general-purpose physical reasoning. A robot could observe human demonstrations, infer latent actions, and plan its own movements—even if its body is completely different from a human's.

For gaming, CLAP suggests a future where physics simulation is unified across character types. Game designers could create any creature and have it behave plausibly, without hand-authored controllers or physics constraints. The same world model could power a humanoid NPC, a snake enemy, and a spider boss.

Final Thoughts on General-Purpose Physical Simulators

The long-term vision is a general-purpose physical simulator—a model that can simulate any object, any body, any interaction, learned entirely from observation. CLAP is a step toward that vision, but it's not the final step. The limitations are real: fine-grained interactions, complex environments, and real-time performance remain unsolved.

But the direction is clear. The latent action space is a powerful abstraction, and contrastive learning is a natural way to align it across embodiments. As the gaming industry increasingly adopts learned simulation, approaches like CLAP will become standard tools in the developer's toolkit.

---

FAQ

What is CLAP and how does it work?

CLAP (Cross-embodiment Latent Action Pretraining) is a method for training video world models that generalize across different robot embodiments. It learns a universal latent action space by aligning latent action representations from different embodiments using contrastive learning, enabling zero-shot transfer of world models to unseen embodiments.

How does CLAP achieve cross-embodiment generalization?

CLAP learns a shared latent action space where actions that produce similar visual transitions in different embodiments are mapped to nearby points. This is achieved through a contrastive loss that pulls together latent actions from different embodiments with similar visual outcomes.

Does CLAP require paired cross-embodiment data?

No. CLAP is trained on unpaired videos from each embodiment. The contrastive loss uses visual transition similarity as a proxy for action equivalence, eliminating the need for synchronized cross-embodiment recordings.

Can CLAP be used for planning and control?

Yes. CLAP can be used for planning by searching over latent actions to achieve a desired visual goal (e.g., using cross-entropy method). This enables zero-shot physical reasoning—the model can plan actions for a novel embodiment without fine-tuning.

What are the potential applications of CLAP in gaming?

CLAP could enable unified physics simulation for diverse game characters, procedural animation generation, zero-shot character control, and significantly reduced animation development costs. A single world model could simulate any character morphology.

What embodiments were used in the CLAP experiments?

Training embodiments include a 7-DoF robot arm, quadcopter, human hand, quadruped, and stick manipulation. Zero-shot evaluation was performed on a snake-like robot, soft gripper, and bipedal robot.

How is the latent action space learned?

The latent action space is learned jointly with the world model. An encoder maps visual transitions to latent action vectors, and the world model predicts future frames conditioned on these vectors. A contrastive loss aligns these vectors across embodiments.

Is CLAP a generative model?

CLAP is a conditional generative model. Given a current frame and a latent action, it generates a plausible next frame. It can also be used unconditionally to generate open-loop predictions.

Does CLAP require action labels during training?

No. CLAP is trained entirely from pixel observations. Latent actions are inferred from visual transitions, not from ground-truth actuator commands.

What is the main limitation of CLAP?

CLAP struggles with fine-grained physical interactions (e.g., precise grasping forces), long-horizon prediction, and complex multi-object environments. It also requires substantial training data and is not yet real-time capable.

---

Ready to explore the future of cross-embodiment simulation? Dive into the full CLAP paper and code, and consider how this technology could transform your next game or robotics project. Join the discussion in the comments below!