Tensors & PyTorch — Programuoki

Code

Four functions — write them yourself

The gradient and the flattening work. The remaining four functions you write without holes this time: the skeleton stands, the body does not.

You met three of the four in step 2, in the tables; the fourth is the same flattening, only for a whole stack of pictures.

Four functions

Each is a single line, but each teaches one thing about shapes.

affine(a, scale, shift) — a * scale + shift for every element at once. And one condition the test checks: a must not change. Return a new tensor. It is the same rule you will keep throughout the course: never touch the input data.

add_row(a, row) — add a [3] row to every row of a [2, 3]. Broadcasting does all the work here; no copying by hand.

weighted_sum(x, w, b) — x is [n, k], w is [k, m], b is [m]. The result is [n, m]. This is lesson one's neuron definition, only written for many neurons and many inputs at once.

flatten_batch(images) — the whole stack at once: [n, 28, 28] → [n, 784]. Row i must exactly match flatten_image(images[i]). Here is the trap: do not write n by hand. The batch size varies — there will be thousands of training pictures and one test picture.

Your job

tensors_demo.py (right panel) has grown: four functions with raise NotImplementedError were added at the bottom. Above, your loss_and_grads and flatten_image are unchanged.

The panel shows the skeleton, not your file: the HOLE places are still empty in it. In your copy they are filled in — and stay that way.

test_tensors_demo.py grew as well: it now checks all six functions. One of the new tests runs flatten_batch with n = 1 and n = 5: only one of two "correct-looking" versions survives both.

Run python -m unittest. If you run it before writing anything (with your holes filled in), you get:

$ python -m unittest
....EEEEE
ERROR: test_add_row_broadcasts (...)
NotImplementedError
ERROR: test_affine (...)
NotImplementedError
ERROR: test_affine_does_not_change_its_input (...)
NotImplementedError
ERROR: test_flatten_batch (...)
NotImplementedError
ERROR: test_weighted_sum (...)
NotImplementedError
----------------------------------------------------------------------
Ran 9 tests in 0.002s

FAILED (errors=5)

Four dots are your earlier tests, still passing. Five Es are five functions still empty (test_affine and test_affine_does_not_change_its_input both touch affine). Write them one at a time and the Es go away.

When it does not pass

affine passes, but does_not_change_its_input fails — you wrote a *= scale. *= on a tensor works in place: it changes the very a the caller owns. Write a * scale, no =.
The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 0 — add_row with row.reshape(3, 1). Now the row is a column, trying to add 3 to 2 rows. Shapes compare from the right: [2, 3] and [3] — fine, [2, 3] and [3, 1] — not.
The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 0 with weighted_sum — you wrote x * w + b instead of x @ w + b. * is element by element, not the matrix product.
mat1 and mat2 shapes cannot be multiplied (3x1 and 2x3) — w @ x instead of x @ w. The product is not commutative.
shape '[5, 784]' is invalid for input of size 784 — flatten_batch with reshape(5, 784). Passes one test, falls over on the other. Use len(images) or -1.

tensors_demo.py MODIFY 32 lines changed

Edit this file in ./

"""tensors_demo.py — tensors, vectorized ops, autograd, and an image as a tensor."""

import torch


def loss_and_grads(w0: float, x: float, b0: float, target: float):
    """Squared error of the line  y = w * x + b  against a target, plus gradients.

    Returns (loss, w.grad, b.grad): the loss as a plain float and the two
    gradients as the 0-dimensional tensors PyTorch fills in.
    """
    w = torch.tensor(w0, requires_grad=True)
    b = torch.tensor(b0, requires_grad=True)

    loss = (w * x + b - target) ** 2

    # HOLE 1: ask PyTorch for the gradients of `loss` with respect to w and b.
    # Without it, w.grad and b.grad stay None.

    return loss.item(), w.grad, b.grad


def flatten_image(image: torch.Tensor) -> torch.Tensor:
    """Turn one [28, 28] image into the [784] vector a neuron takes as input.

    The pixels keep their row-by-row order.
    """
    # HOLE 2: reshape the image into ONE dimension of length 784.
    return image


def affine(a: torch.Tensor, scale: float, shift: float) -> torch.Tensor:
    """Return a * scale + shift, for every element at once.

    No Python loop, and `a` itself must not be changed: build a new tensor.
    """
    raise NotImplementedError


def add_row(a: torch.Tensor, row: torch.Tensor) -> torch.Tensor:
    """Add a [3] row to every row of a [2, 3] tensor and return the [2, 3] sum.

    No loop, no copying the row two times by hand: let broadcasting stretch it.
    """
    raise NotImplementedError


def weighted_sum(x: torch.Tensor, w: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """Matrix product of x and w, plus b added to every row.

    x is [n, k], w is [k, m], b is [m]; the result is [n, m].
    """
    raise NotImplementedError


def flatten_batch(images: torch.Tensor) -> torch.Tensor:
    """Turn a batch of images [n, 28, 28] into [n, 784]: one row per image.

    Row i must equal flatten_image(images[i]). Do not hard-code n.
    """
    raise NotImplementedError

test_tensors_demo.py MODIFY 35 lines changed

Edit this file in ./

"""test_tensors_demo.py — the verdict for tensors_demo.py.

Run it from this folder:   python -m unittest
"""

import unittest

import torch

import tensors_demo as td


class TestAutograd(unittest.TestCase):
    def test_gradients_match_the_hand_calculation(self):
        # loss = (w*x + b - target)^2 at w=3, x=2, b=1, target=10
        #      = (6 + 1 - 10)^2 = 9
        # d loss / d w = 2 * (w*x + b - target) * x = 2 * (-3) * 2 = -12
        # d loss / d b = 2 * (w*x + b - target)     = 2 * (-3)     = -6
        loss, dw, db = td.loss_and_grads(3.0, 2.0, 1.0, 10.0)
        self.assertEqual(loss, 9.0)
        self.assertIsNotNone(dw, "w.grad is None - did you call backward() on the loss?")
        self.assertIsNotNone(db, "b.grad is None - did you call backward() on the loss?")
        self.assertEqual(dw.item(), -12.0)
        self.assertEqual(db.item(), -6.0)

    def test_gradient_is_zero_when_the_prediction_is_perfect(self):
        # w*x + b = 4*2 + 2 = 10 = target: nothing to correct
        loss, dw, db = td.loss_and_grads(4.0, 2.0, 2.0, 10.0)
        self.assertEqual(loss, 0.0)
        self.assertIsNotNone(dw, "w.grad is None - did you call backward() on the loss?")
        self.assertEqual(dw.item(), 0.0)
        self.assertEqual(db.item(), 0.0)


class TestImage(unittest.TestCase):
    def setUp(self):
        # a fake image whose pixel value is its own position: 0, 1, 2 ... 783
        self.image = torch.arange(28 * 28, dtype=torch.float32).reshape(28, 28)

    def test_flatten_gives_one_dimension_of_784(self):
        flat = td.flatten_image(self.image)
        self.assertEqual(tuple(flat.shape), (784,), f"got shape {tuple(flat.shape)}, want (784,)")

    def test_flatten_keeps_row_order(self):
        flat = td.flatten_image(self.image)
        self.assertEqual(flat.dim(), 1, f"got {flat.dim()} dimensions, want 1")
        self.assertEqual(flat[1].item(), 1.0, "second value should be pixel (0, 1)")
        self.assertEqual(flat[28].item(), 28.0, "value 28 should be the first pixel of row 1")
        self.assertEqual(flat[783].item(), 783.0)


class TestVectorized(unittest.TestCase):
    def setUp(self):
        self.a = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])

    def test_affine(self):
        out = td.affine(self.a, 2.0, 1.0)
        self.assertEqual(out.tolist(), [[3.0, 5.0, 7.0], [9.0, 11.0, 13.0]])

    def test_affine_does_not_change_its_input(self):
        td.affine(self.a, 2.0, 1.0)
        self.assertEqual(self.a.tolist(), [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])

    def test_add_row_broadcasts(self):
        row = torch.tensor([10.0, 20.0, 30.0])
        out = td.add_row(self.a, row)
        self.assertEqual(out.tolist(), [[11.0, 22.0, 33.0], [14.0, 25.0, 36.0]])

    def test_weighted_sum(self):
        w = torch.tensor([[0.5], [0.0], [-0.5]])
        b = torch.tensor([0.25])
        out = td.weighted_sum(self.a, w, b)
        self.assertEqual(tuple(out.shape), (2, 1))
        self.assertEqual(out.tolist(), [[-0.75], [-0.75]])

    def test_flatten_batch(self):
        for n in (1, 5):  # the batch size must not be baked in
            images = torch.arange(n * 28 * 28, dtype=torch.float32).reshape(n, 28, 28)
            flat = td.flatten_batch(images)
            self.assertEqual(tuple(flat.shape), (n, 784), f"got shape {tuple(flat.shape)}")
            for i in range(n):
                self.assertTrue(
                    torch.equal(flat[i], td.flatten_image(images[i])), f"row {i} differs"
                )


if __name__ == "__main__":
    unittest.main()