World Model

World Model

The world model is a space-time factorized transformer with a hidden dimension of $C$.

Where a factorized transformer is a transformer that performs attention over different dimensions of the input, in this case being over $T$ and $H \times W$.

Let

$$X \in \mathbb{R}^{T \times N \times H \times W \times L}$$

be the set of input latent vectors in the form of a tensor, where

• $T$ is the temporal window or the number of frames in the input,
• $N$ is the number of cameras,
• $H$ is the height of the frame,
• $W$ is the width of the frame,
• $L$ is the dimension of the individual latent vector.

Assume $N = 1$, then we simplify:

$$X \in \mathbb{R}^{T \times 1 \times H \times W \times L}.$$