3D Matrices in Nuke: Local, World and Projection Matrix

Matrices in Nuke, the series: Convolution Matrix · Transform Matrix · ColorMatrix · Convolve & Deconvolution · 3D Matrices · Morphology · CornerPin & Homography · BlinkScript Matrices

The Transform Matrix page moved pixels on a flat image. Nuke's 3D system does the same thing with one more dimension: every Axis, Camera, Card and TransformGeo carries a 4x4 matrix that says where it is, how it's rotated and how big it is. Learn to read it and you can link anything to anything: the world position of an object hidden inside a parent chain, the distance between the camera and a card to drive a defocus, a 3D point projected on the 2D image.

Useful Links

Index

01. From 3x3 to 4x4

In 2D we needed a 3x3 matrix: 2x2 for rotation and scale, plus one column for the translation. In 3D it's the same with one more dimension: 3x3 for rotation and scale, one column for the translation, and a last row that stays 0 0 0 1.

	X axis	Y axis	Z axis	position
x	r00	r01	r02	tx
y	r10	r11	r12	ty
z	r20	r21	r22	tz
	0	0	0	1

The nicest way to read it: the first three columns are the object's own X, Y and Z axes, written in world coordinates. Their direction is the rotation, their length is the scale. The fourth column is simply where the object is. A point is transformed by multiplying the matrix by (x, y, z, 1): the 1 is what lets the last column add the translation.

Numbering. Nuke shows its matrix knobs row by row, so in expressions the 16 values are world_matrix.0 to world_matrix.15: 0 1 2 3 the first row, 4 5 6 7 the second, and so on. The translation is in 3, 7 and 11, the rotation and scale in 0 1 2, 4 5 6, 8 9 10. You will use these numbers a lot.

02. Local and world matrix

The 3D nodes that can move (Axis, Camera, Card, TransformGeo, lights...) have two matrices, which you can see in their properties:

- local matrix: built from the node's own translate, rotate, scale, skew and pivot knobs. It describes the move relative to its parent.
- world matrix: the final position in the scene. If the node has a parent (an Axis plugged into its axis input), it's the parent's world matrix multiplied by the local one. If there's no parent, world and local are the same.

This is why the world matrix is so useful: a camera inside a chain of three Axis nodes can show translate (0, 0, 0) in its knobs while it's flying around the set. Its world_matrix always tells the truth.

03. The order matters

A 3D transform is several matrices multiplied together, and as we saw in 2D, the order changes the result. Nuke gives you two knobs for it:

- transform order (default SRT): scale first, then rotate, then translate. With TRS the object would be moved first and then rotated around the origin of the world, like a planet around the sun.
- rotation order (default ZXY): which axis rotates first. The same three angles in XYZ or ZXY give a different orientation. When you bring a camera from Maya or Houdini, check that the rotation order matches.

In the Lab below you can change both orders and watch the cube and the matrix change, with exactly the same numbers in the knobs.

3D Matrix Lab

A cube transformed by translate, rotate and scale, seen from a fixed camera. The orange axes are the cube's own X, Y and Z: compare them with the first three columns of the matrix.

Translate X 0 Translate Y 0 Translate Z 0 Rotate X 0 Rotate Y 30 Rotate Z 0 Uniform scale 1

transform order rotation order

world position (values 3, 7, 11):

04. Expressions with matrices

Every element of a matrix knob can be used in an expression. Some of the ones I use the most:

# the real position in the scene of an Axis, even inside a parent chain
Axis1.world_matrix.3     # x
Axis1.world_matrix.7     # y
Axis1.world_matrix.11    # z

# distance between the camera and a card (for example to drive the focus of a ZDefocus)
sqrt(pow(Card1.world_matrix.3 - Camera1.world_matrix.3, 2) + pow(Card1.world_matrix.7 - Camera1.world_matrix.7, 2) + pow(Card1.world_matrix.11 - Camera1.world_matrix.11, 2))

# the same position one frame before (motion, speed, trails)
Axis1.world_matrix.3(frame - 1)

And the rotation part (0, 1, 2, 4, 5, 6, 8, 9, 10) can go straight into a ColorMatrix to rotate a normals pass from world space to camera space: it's explained step by step in the ColorMatrix page.

05. Matrices in Python

Reading a matrix knob gives you the 16 numbers in the same order as the expressions, row by row. With a few lines of plain Python you can move points around:

import nuke

def world_matrix(name, frame=None):
    """The 16 values of a node's world matrix, row by row."""
    knob = nuke.toNode(name)["world_matrix"]
    if frame is None:
        return [knob.getValue(i) for i in range(16)]
    return [knob.getValueAt(frame, i) for i in range(16)]

def transform_point(m, p):
    """Apply a 4x4 matrix (16 values, row by row) to a point (x, y, z)."""
    x, y, z = p
    return (m[0] * x + m[1] * y + m[2] * z + m[3],
            m[4] * x + m[5] * y + m[6] * z + m[7],
            m[8] * x + m[9] * y + m[10] * z + m[11])

def rigid_inverse(m):
    """Inverse of a matrix with only rotation and translation (scale 1), like a camera."""
    r = [m[0], m[4], m[8], m[1], m[5], m[9], m[2], m[6], m[10]]   # transposed rotation
    t = (m[3], m[7], m[11])
    return [r[0], r[1], r[2], -(r[0] * t[0] + r[1] * t[1] + r[2] * t[2]),
            r[3], r[4], r[5], -(r[3] * t[0] + r[4] * t[1] + r[5] * t[2]),
            r[6], r[7], r[8], -(r[6] * t[0] + r[7] * t[1] + r[8] * t[2]),
            0, 0, 0, 1]

# where is a point 1 unit in front of Axis1, in the world?
print(transform_point(world_matrix("Axis1"), (0, 0, -1)))

Nuke also has its own maths module, nuke.math, with Matrix4, Vector3 and Vector4: inverse(), transpose(), transform() for points, vtransform() for directions, rotateX/Y/Z(), translate(), scale(), and even rotationsZXY() to get the angles back from a matrix. It's faster for heavy work; just remember to check the order of the values when you copy them from a knob into a Matrix4.

06. From 3D to 2D: the projection

A camera adds one last matrix: the projection. First the point goes into camera space (the inverse of the camera's world matrix), then it's divided by its distance: far things become small. That division is what makes parallel lines meet at the horizon, and it's why a projection matrix doesn't end with 0 0 0 1. The size of the image depends only on the ratio between focal and horizontal aperture.

pixel x = width/2 + width · (focal / haperture) · x / −z
pixel y = height/2 + width · (focal / haperture) · y / −z

(x, y, z) is the point in camera space; Nuke's cameras look down the −Z axis, so points in front of the camera have a negative z. In Python, with the functions of chapter 05:

def project(camera, point, frame=None):
    """2D pixel position of a 3D world point, seen by a camera (no lens distortion)."""
    cam = nuke.toNode(camera)
    x, y, z = transform_point(rigid_inverse(world_matrix(camera, frame)), point)
    if z >= 0:
        return None                                  # behind the camera
    w, h = nuke.root().width(), nuke.root().height()
    f = cam["focal"].getValue() / cam["haperture"].getValue()
    return (w / 2.0 + w * f * x / -z, h / 2.0 + w * f * y / -z)

print(project("Camera1", (0, 1.5, -10)))

This is what Reconcile3D does for you, frame by frame, to turn a 3D point into a 2D track (and it also handles the window and aperture knobs). Use the function when you need many points at once, or to build a CornerPin from the four corners of a card. And the CornerPin itself is a projection in 2D: a 3x3 matrix with a last row that isn't 0 0 1. nuke.math.Matrix4 can build it from four corners with mapUnitSquareToQuad(), the same matrix you can see in the Transform Matrix page.

Nodes to paste in Nuke

Click a box, copy it (Ctrl+C / Cmd+C) and paste it in Nuke's Node Graph (Ctrl+V / Cmd+V).

They point to nodes called Axis1, Camera1 and Card1: rename them in the expressions (or rename your nodes).

World position. A NoOp that shows where Axis1 really is in the scene, even inside a parent chain. You can link other knobs to its world position.

NoOp {
 name WorldPosition_Axis1
 label "x \[value world_position.x]\ny \[value world_position.y]\nz \[value world_position.z]"
 addUserKnob {20 User}
 addUserKnob {13 world_position l "world position"}
 world_position {{Axis1.world_matrix.3} {Axis1.world_matrix.7} {Axis1.world_matrix.11}}
}

Camera to card distance. A NoOp with the distance between Camera1 and Card1, updated every frame. Link it to a focus knob, a fog density, a blur size...

NoOp {
 name Distance_Camera1_Card1
 label "distance \[value distance]"
 addUserKnob {20 User}
 addUserKnob {7 distance R 0 100}
 distance {{sqrt(pow(Card1.world_matrix.3-Camera1.world_matrix.3,2)+pow(Card1.world_matrix.7-Camera1.world_matrix.7,2)+pow(Card1.world_matrix.11-Camera1.world_matrix.11,2))}}
}

Normals from world to camera space. A ColorMatrix linked to the rotation of Camera1, with invert on. Set channels to your normals layer. Untick invert for camera to world.

ColorMatrix {
 matrix {
     {{Camera1.world_matrix.0} {Camera1.world_matrix.1} {Camera1.world_matrix.2}}
     {{Camera1.world_matrix.4} {Camera1.world_matrix.5} {Camera1.world_matrix.6}}
     {{Camera1.world_matrix.8} {Camera1.world_matrix.9} {Camera1.world_matrix.10}}
   }
 invert true
 name Normals_WorldToCamera
 label "Camera1 rotation, inverted"
}

07. The whole family

Five kinds of matrices, five pages:

- Convolution matrix: mixes a pixel with its neighbours (blur, sharpen, edges). → Convolution Matrix
- Convolve: the kernel is a whole image (bokeh), and the Fourier transform to undo a blur. → Convolve & Deconvolution
- ColorMatrix: mixes the channels of the same pixel. → ColorMatrix
- Transform matrix: moves the pixels in 2D, 3x3 inside Nuke's 4x4. → Transform Matrix
- 3D matrices: local, world and projection, this page.

3D Matrices

Local, world and projection matrix in Nuke