Gaussian Splatting

Summary

Gaussian Splatting is a way to represent 3D scenes by using soft, colored blobs (Gaussians) to quickly and realistically create images from different camera views.

Gaussian

• A gaussian is basically a “soft point”, which has a position, shape/orientation, color, opacity, and view-dependent effect in 3D space.
• Mathematically, it is defined as $$\large G(x) = \exp(\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu))

- $\mu$ is the center of the Gaussian (its position) - $\Sigma$ is the covariance matrix, which controls the size and direction (3d ellipse)

Projecting to the Camera

• Each 3D Gaussian appears as a 2D ellipse on the camera sensor $\to$ its shape changes based on how far it is and what angle you’re looking at it.
• Math: Covariance projection
  • Project a 3D gaussian to the image plane using the camera projection and a Jacobian
  • $\Pi (\cdot )$: the perspective projection function from 3D to 2D
  • $\mu$: the mean of the 3D Gaussian
  • $\Sigma$: the 3D covariance matrix The projected 2D covariance ${\Sigma }_{2D}$ is given by:

${\Sigma }_{2D}=J\Sigma J^T$

Where: >$>J={\frac{\partial \Pi }{\partial x}|}_{\mu }>$

• $J$: The Jacobian matrix of the projection function, evaluated at the mean $\mu$
  • $J^T$: The transpose of $J$