Nothing to write here. Here you watch what PyTorch does — with small numbers, so you can check it with a pencil. Every printed value below is real, captured by running the code.

A tensor, its shape and its type

a = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])

shape: torch.Size([2, 3]) dtype: torch.float32 device: cpu

Three properties you will see in every error message:

• shape — [2, 3]: two rows, three columns;
• dtype — the number type. float32 is the currency of learning: gradients live only on floating-point numbers;
• device — where the tensor lives. cpu is the processor. On your machine it will stay that way.

One operation, every element

a * 2 + 1 = tensor([[ 3.,  5.,  7.],
                    [ 9., 11., 13.]])

Every element multiplied by 2, then 1 added. Not a single for.

When shapes differ: broadcasting

a   [2, 3]     [[ 1.,  2.,  3.],
                [ 4.,  5.,  6.]]
row    [3]     [10., 20., 30.]

a + row =      [[11., 22., 33.],
                [14., 25., 36.]]

The [3] row was "poured" over both rows of a. The rule: shapes are compared from the right, and an axis that has only 1 (or is missing altogether) stretches to the other tensor's size. Here [2, 3] and [3] agree in the last axis, and row simply has no first one.

Matrix product

a @ w   [2, 3] @ [3, 1] -> [2, 1]

w = [[ 0.5],
     [ 0.0],
     [-0.5]]

a @ w = [[-1.],       1*0.5 + 2*0.0 + 3*(-0.5) = -1
         [-1.]]       4*0.5 + 5*0.0 + 6*(-0.5) = -1

Every row of the result is a weighted sum of a row of a: each input multiplied by its weight, everything added up. That is exactly the definition of lesson one's neuron. Keep the shape rule in mind: the inner numbers (3 and 3) must match and "disappear", the outer ones (2 and 1) remain.

Gradient: what autograd says

w = torch.tensor(3.0, requires_grad=True)
x = torch.tensor(2.0)
b = torch.tensor(1.0, requires_grad=True)

loss = (w * x + b - 10.0) ** 2
loss.backward()

loss: 9.0
w.grad: -12.0 b.grad: -6.0

The prediction w*x + b = 3*2 + 1 = 7, but we wanted 10. The error is 7 − 10 = −3, the loss is its square, 9.

A gradient answers "how would the loss change if I increased this number?". Here w.grad = −12: increasing w lowers the loss, and quite steeply. That makes sense: the prediction is too small and w makes it bigger. By hand: 2 · (−3) · x = 2 · (−3) · 2 = −12, and for b — without the factor x: 2 · (−3) = −6.

Autograd does not know "formulas". It follows every operation that produced loss, and when you ask for .backward() it walks back and builds the gradients. That is why requires_grad=True means "keep an eye on this number".

A picture is a tensor too

The FashionMNIST file holds a [60000, 28, 28] tensor of type uint8 — whole numbers from 0 to 255. Part of one row of the first picture (an ankle boot, label 9), columns 10–17:

in the file (uint8):   [0, 0, 237, 226, 217, 223, 222, 219]
after dividing by 255: [0.0000, 0.0000, 0.9294, 0.8863, 0.8510, 0.8745, 0.8706, 0.8588]

A neuron takes not a table but a row: [28, 28] becomes [784], row after row. The answers to the first questions: 784 numbers, and a stack of 64 pictures has shape [64, 28, 28] (or [64, 784] when each is flattened).

Three observations

1. Before doing anything, look at the shape. Matrix product, broadcasting, flattening — all of them are just shape arithmetic. If the shape equation does not balance, PyTorch tells you exactly so (you will see the messages in step 6).

2. A gradient has a sign and a size. −12 says "increase w, and increase it hard"; −6 — "increase b, but half as much". In lesson 4 a number appears that turns all these "directions and sizes" into one correction. For now it is enough to understand that a gradient is advice, not an order.

3. Everything here is numbers and shapes. Picture, weights, loss, gradient — one and the same kind of thing. That is why the rest of the course fits into a few dozen lines.

What next

In the next step you write two of these pieces as Python: the gradient and the flattening of a picture. You have already seen both at work.