PyTorch Fundamentals

Before delving into the depths of deep learning—pun intended—it's crucial to grasp the foundational aspects that pave the way. This will be a journal of sorts to document my learning experience as I figure out how to build an eye blink tracker using PyTorch from scratch.

What is PyTorch?

PyTorch is a powerful open-source machine learning library that provides a flexible and efficient platform for building deep learning models. It is particularly well-suited for tasks involving computer vision, natural language processing, and reinforcement learning. PyTorch operates on tensors, akin to numpy arrays but optimized for GPU-accelerated computing. This efficiency boost—up to 100 times faster than CPUs—is due to GPUs' inherent computational prowess. hugo

Attributes and Operations:

Similar to numpy arrays, tensors support indexing, slicing, reshaping, and mathematical operations. The concept of broadcasting enables simultaneous operations across compatible elements, enhancing computational efficiency.

• Building Tensors: python import torch import numpy as np m = torch.tensor([[1,2,3], [4,5,6]]) print(m) output: output tensor([[1, 2, 3], [4, 5, 6]])

• Types of Tensors: ```python np_array = np.array([[1,2,3], [7,8,9]]) torch_tensor = torch.from_numpy(np_array) print(f"Numpy array to tensor: \n {torch_tensor}") zeros_tensor = torch.zeros(2,5) print(f"zeros_tensor: \n {zeros_tensor}") one_tesnors = torch.ones(2,5) print(f"one_tesnors: \n {one_tesnors}") rand_tensor = torch.rand(2,3) print(f" rand_tensor: \n {rand_tensor}") randn_tensor = torch.randn(2,3) print(f"randn_tensor: \n {randn_tensor}") float_tensor =torch.tensor([1.0,2.0,3.0]) print(f"Float tesnor: \n {float_tensor.dtype}") int_tensor = torch.tensor([1,2,3,4], dtype=torch.int64) print(f"Int tensor: \n {int_tensor.dtype}")

  output Numpy array to tensor: tensor([[1, 2, 3], [7, 8, 9]]) zeros_tensor: tensor([[0., 0., 0., 0., 0.], [0., 0., 0., 0., 0.]]) one_tesnors: tensor([[1., 1., 1., 1., 1.], [1., 1., 1., 1., 1.]]) rand_tensor: tensor([[0.7409, 0.5551, 0.9490], [0.6028, 0.1038, 0.5413]]) randn_tensor: tensor([[-1.3301, -0.3333, -0.4628], [-1.0383, -0.8357, -0.2252]]) Float tesnor: torch.float32 Int tensor: torch.int64 ```

• Checking the device and shape of tensors: python if torch.cuda.is_available(): device = torch.device("cuda") gpu_tensor = torch.tensor([1,2,3], device=device) print(gpu_tensor.device) else: print("CUDA is not available, tensor on CPU") device = torch.device("cpu") # which is the default print(f"Shape of tensor: {m.shape}") print(f"Datatype of tensor:{m.dtype}") print(f"Device tensor is on {m.device}") ````output CUDA is not available, tensor on CPU Shape of tensor: torch.Size([2, 3]) Datatype of tensor:torch.int64 Device tensor is on cpu ```

• PyTorch Tensor operations: extracting rows and columns, changing shapes with .view() and .reshape(), and performing element wise and matrix operations (addition, subtraction, multiplication, division, and matrix multiplication) ```python # 1. Tensor creation and basic indexing/slicing tensor = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) print("First row: ", tensor[0]) print("First column: ", tensor[:, 0]) print("Element at (1,2):", tensor[1, 2])

  2. Slicing + in-place update

  sub = tensor[:2, :2] sub[0, :] = 100 print("After in-place update:\n", tensor)

  3. Random tensor + reshape/flatten

  t = torch.rand(4, 4) print("Original shape:", t.shape) print("Reshaped (2×8):", t.view(2, 8).shape) print("Flattened: ", t.view(-1).shape)

  4. Element-wise arithmetic

  x = torch.tensor([1., 2., 3.]) y = torch.tensor([4., 5., 6.]) print("\nAdd: ", x + y) print("Sub: ", x - y) print("Mul: ", x * y) print("Div: ", x / y)

  5. Matrix multiplication

  mat1 = torch.randn(2, 3) mat2 = torch.randn(3, 4) print("\nMatrix multiplication:\n", torch.matmul(mat1, mat2))

  6. Broadcasting examples

  a = torch.tensor([[1, 2, 3], [4, 5, 6]]) print("\nAdd scalar: \n", a + 10) print("Add row vector: \n", a + torch.tensor([1, 2, 3])) print("Add column vector:\n", a + torch.tensor([[1], [2]]))

  7. In-place operation

  x.add_(y) print("\nIn-place add result:", x) output:output First row: tensor([1, 2, 3]) First column: tensor([1, 4, 7]) Element at (1,2): tensor(6) After in-place update: tensor([[100, 100, 3], [ 4, 5, 6], [ 7, 8, 9]])

  Original shape: torch.Size([4, 4]) Reshaped (2×8): torch.Size([2, 8]) Flattened: torch.Size([16])

  Add: tensor([5., 7., 9.]) Sub: tensor([-3., -3., -3.]) Mul: tensor([ 4., 10., 18.]) Div: tensor([0.2500, 0.4000, 0.5000])

  Matrix multiplication: tensor([[ 0.6673, 0.5182, 1.2387, -0.9455], [ 0.0869, 0.3411, 0.6695, -0.4641]])

  Add scalar:
  tensor([[11, 12, 13], [14, 15, 16]]) Add row vector:
  tensor([[2, 4, 6], [5, 7, 9]]) Add column vector: tensor([[2, 3, 4], [6, 7, 8]])

  In-place add result: tensor([5., 7., 9.]) ```

  Automatic Differentiation:

  Central to PyTorch's appeal is its automatic differentiation capability. By tracking operations on tensors marked with requires_grad=True, it computes gradients (.grad)—crucial for understanding how changes in weights affect model output. Clearing gradients prevents unintended accumulation, ensuring accurate subsequent calculations.

• Torch.nn.Module: Enter nn.Module, the cornerstone of PyTorch neural networks. This template empowers developers to construct models equipped with essential functionalities—a scaffold for innovation in deep learning architectures.

From common layers and activation functions to loss functions that gauge predictive accuracy, PyTorch offers a robust toolkit for constructing and refining neural networks.

• Optimizers and Training Loop Optimizers, such as SGD and Adam, fine-tune model parameters by minimizing computed gradients. The training loop—comprising forward passes, loss calculation, backpropagation for gradient computation, and model updates—forms the backbone of neural network training.

In essence, mastering these fundamentals equips us to embark on building an eye blink tracker. Stay tuned as we translate theory into practice, harnessing PyTorch's capabilities to innovate in real-world applications.

Gradients & backward propagation

    x = torch.tensor(2.0, requires_grad=True)
    w = torch.tensor(3.0, requires_grad=True)
    b = torch.tensor(1.0, requires_grad=True)
    y = w * x + b
    target = torch.tensor(10.0)
    loss = (y - target)**2

    loss.backward(retain_graph=True)
    print(f"Gradient of loss w.r.t x: {x.grad}")
    print(f"Gradient of loss w.r.t w: {w.grad}")
    print(f"Gradient of loss w.r.t b: {b.grad}")

    # Reset gradients
    x.grad = None
    w.grad = None
    b.grad = None
    print(f"Gradients reset for x, w, b: {x.grad}, {w.grad}, {b.grad}")

    # Gradients with no tracking
    with torch.no_grad():
        y_no_grad = w * x + b
        print(f"y_no_grad requires_grad: {y_no_grad.requires_grad}")

Gradient of loss w.r.t x: -18.0
Gradient of loss w.r.t w: -12.0
Gradient of loss w.r.t b: -6.0
Gradients reset for x, w, b: None, None, None
y_no_grad requires_grad: False

Exercise: Gradient computation with new tensors

    a = torch.tensor(4.0, requires_grad=True)
    b = torch.tensor(5.0, requires_grad=True)
    c = torch.tensor(6.0, requires_grad=True)
    d = a * b + c
    l = d**2
    l.backward()
    print(f"Gradient of loss w.r.t a: {a.grad}")
    print(f"Gradient of loss w.r.t b: {b.grad}")
    print(f"Gradient of loss w.r.t c: {c.grad}")

    # Verify gradients are correct
    print(f"Gradient of loss w.r.t a (dl/da): {2 * d * b}")
    print(f"Gradient of loss w.r.t b (dl/db): {2 * d * a}")
    print(f"Gradient of loss w.r.t c (dl/dc): {2 * d}")

    # Reset gradients and calculate for new operations
    a.grad = None
    b.grad = None
    c.grad = None
    d1 = a * b
    d1.backward()
    print(f"Gradient of d1: {a.grad}, {b.grad}")

Output:

Gradient of loss w.r.t a: 260.0
Gradient of loss w.r.t b: 208.0
Gradient of loss w.r.t c: 52.0
Gradient of loss w.r.t a (dl/da): 260.0
Gradient of loss w.r.t b (dl/db): 208.0
Gradient of loss w.r.t c (dl/dc): 52.0
Gradient of d1: 5.0, 4.0

Creating a simple neural network (Linear Regression Model)
Essentially, a linear regression model is a simple neural network with one layer and no activation function. It learns to map input features to output values by adjusting weights and biases during training. To create a simple neural network using Pytorch, we can follow the a simple structure or template as follows:
class YourModelName(nn.Module):
          def __init__(self):
             super().__init__()
              self.layer1 = ...
              self.layer2 = ...
          def forward(self, x):
              x = self.layer1(x)
              x = self.layer2(x)
          return x

    class LinearRegressionModel(nn.Module):
        def __init__(self, input_dim, output_dim):
            super(LinearRegressionModel, self).__init__()
            self.linear = nn.Linear(input_dim, output_dim)

        def forward(self, x):
            return self.linear(x)

    # Instantiate model
    input_features = 5
    output_features = 1
    model = LinearRegressionModel(input_features, output_features)

    # Print model and parameters
    print(f"Model: {model}")
    for name, param in model.named_parameters():
        if param.requires_grad:
            print(f"Parameter name: {name}, shape: {param.shape}")

    # Dummy input and model output
    dummy_input = torch.randn(10, input_features)
    output = model(dummy_input)
    print(f"Output shape: {output.shape}")     

Output:

Model: LinearRegressionModel(
  (linear): Linear(in_features=5, out_features=1, bias=True)
)
Parameter name: linear.weight, shape: torch.Size([1, 5])
Parameter name: linear.bias, shape: torch.Size([1])
Output shape: torch.Size([10, 1])

Exercises:

1. Simple Custom Module:
   • Create a custom nn.Module called SimpleNet.
   • In its __init__, define two nn.Linear layers:
     • fc1: Takes input_size to hidden_size
     • fc2: Takes hidden_size to output_size
   • In its forward method, pass the input x through fc1, then apply a non-linearity (you'll learn nn.ReLU next, for now just pass it to fc2), and then pass the result through fc2.
   • Instantiate SimpleNet with input_size=10, hidden_size=20, output_size=2.
   • Print the model architecture.
   • Pass a dummy input of shape (1, 10) through the model and print the output shape.

Answer:

class SimpleNet(nn.Module):
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)
    def forward (self, x):
        intermidiate_layer = self.fc1(x)
        y = self.fc2(intermidiate_layer)
        return y
input_size = 10
hidden_size = 20
output_size = 2
model = SimpleNet()
print(model)

dummy_input = torch.randn([1,10])
output = model(dummy_input)
print(f"Output shape: {output.shape}")

output:

SimpleNet(
  (fc1): Linear(in_features=10, out_features=20, bias=True)
  (fc2): Linear(in_features=20, out_features=2, bias=True)
)
Output shape: torch.Size([1, 2])

Regression, Classification, Neural Networks, and Optimization

import torch
import torch.nn as nn
import torch.optim as optim

# --- Regression Example ---
predictions_reg = torch.randn(5, 1, requires_grad=True)  # 5 samples, 1 output feature
targets_reg = torch.randn(5, 1)

mse_loss_fn = nn.MSELoss()
loss_reg = mse_loss_fn(predictions_reg, targets_reg)
print(f"MSE Loss: {loss_reg.item()}")

# --- Classification Example (Multi-class) ---
predictions_clf = torch.randn(5, 3, requires_grad=True)  # 5 samples, 3 classes
targets_clf = torch.tensor([0, 1, 2, 1, 0])  # Actual class labels

ce_loss_fn = nn.CrossEntropyLoss()
loss_clf = ce_loss_fn(predictions_clf, targets_clf)
print(f"Cross Entropy Loss: {loss_clf.item()}")

# --- Neural Network Model ---
class Mynetwork(nn.Module):
    def __init__(self):
        super(Mynetwork, self).__init__()
        self.fc1 = nn.Linear(50, 100)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(100, 60)
        self.tanh = nn.Tanh()
        self.output_layer = nn.Linear(60, 5)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.tanh(x)
        x = self.output_layer(x)
        return x

model = Mynetwork()
print(model)

dummy_input = torch.randn(10, 50)
output = model(dummy_input)

# Mean Squared Error Example
preds = torch.tensor([[0.5], [1.5], [2.8], [3.2]])
actuals = torch.tensor([[0.8], [1.2], [3.1], [3.0]])
mse = nn.MSELoss()
loss = mse(preds, actuals)
print(f"Mean Squared Error Loss: {loss.item()}")

# Cross Entropy Loss Example
logits = torch.tensor([[2.0, 1.0, 0.1], [-1.0, 3.0, 0.5]])
true_labels = torch.tensor([0, 1])
cross_entropy = nn.CrossEntropyLoss()
loss_clf = cross_entropy(logits, true_labels)
print(f"Cross Entropy Loss: {loss_clf.item()}")

# --- Optimizer Examples ---
optimizer_sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
optimizer_adam = optim.Adam(model.parameters(), lr=0.001)

# --- Linear Regression with Adam Optimizer ---
x = torch.linspace(0, 10, 100).unsqueeze(1)  # Input data
y = 2 * x + 1 + torch.randn(100, 1) * 0.5  # Target data

class LinearRegressionModel(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressionModel, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        return self.linear(x)

input_size = x.shape[1]
output_size = y.shape[1]
model = LinearRegressionModel(input_size, output_size)

criterion = nn.MSELoss()
optim = optim.Adam(model.parameters(), lr=0.01)

num_epochs = 200
for epoch in range(num_epochs):
    outputs = model(x)
    loss = criterion(outputs, y)
    optim.zero_grad()
    loss.backward()
    optim.step()

    if (epoch) % 20 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

Output:

MSE Loss: 0.5887360572814941
Cross Entropy Loss: 1.4941388368606567
Mynetwork(
  (fc1): Linear(in_features=50, out_features=100, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=100, out_features=60, bias=True)
  (tanh): Tanh()
  (output_layer): Linear(in_features=60, out_features=5, bias=True)
)
Mean Squared Error Loss: 0.07749999314546585
Cross Entropy Loss: 0.25635215640068054
Epoch [1/200], Loss: 211.3235
Epoch [21/200], Loss: 174.6047
Epoch [41/200], Loss: 142.3446
Epoch [61/200], Loss: 114.7111
Epoch [81/200], Loss: 91.4159
Epoch [101/200], Loss: 72.0290
Epoch [121/200], Loss: 56.0981
Epoch [141/200], Loss: 43.1791
Epoch [161/200], Loss: 32.8476
Epoch [181/200], Loss: 24.7049

Resources:

• PyTorch Documentation
• Torch.nn Module Documentation