Final Eye Blink Tracker Implementation / Etsub Feleke

Eye Tracking for Everyone: Final Implementation Phase

In this final blog post of our iTracker series, I'll walk you through the complete implementation of our eye-tracking model using PyTorch. This post covers the full architecture, training process, and results of our gaze estimation system.

Project Overview

The implementation provides a complete pipeline for eye gaze tracking, built on PyTorch and utilizing the GazeCapture dataset. The system can predict where a person is looking on a screen using just their face and eye images. You can find the complete implementation in my GitHub repository.

Key Components

1. Data Pipeline

The data pipeline consists of two main components:

Data Preparation (prepareDataset.py)

Processes the raw GazeCapture dataset
Extracts and crops face and eye images
Generates face grid representations
Creates a metadata.mat file containing frame indices, recording numbers, and gaze targets

Custom Dataset Implementation (DataLoader.py)

Implements a PyTorch Dataset class for efficient data loading
Handles image preprocessing and normalization
Manages train/validation/test splits
Implements face grid generation on the fly

2. Model Architecture

The iTracker model (iTrackerModel.py) implements a multi-stream CNN architecture:

class iTracker(nn.Module):
    def __init__(self, dropout_rate=0.5):
        # Three parallel CNN branches:
        self.left_eye_branch = nn.Sequential(...)
        self.right_eye_branch = nn.Sequential(...)
        self.face_branch = nn.Sequential(...)
        self.face_grid_branch = nn.Sequential(...)
        self.fully_con_layers = nn.Sequential(...)

Key features:

Separate CNN branches for left eye, right eye, and face processing
Face grid processing branch for spatial awareness
Feature fusion layer combining all streams
Final regression layers for gaze prediction

3. Training Pipeline

The training system (main.py) includes:

Apple Silicon GPU support via MPS backend
Automatic checkpointing of best models
Comprehensive evaluation metrics
Hyperparameters:
Batch size: 50
Learning rate: 0.001
Weight decay: 1e-4
Number of epochs: 100

Implementation Highlights

GPU Acceleration

One of the key features of our implementation is the support for Apple Silicon GPUs through PyTorch's MPS (Metal Performance Shaders) backend. This optimization allows for significantly faster training times on modern Mac hardware:

if torch.backends.mps.is_available():
    device = torch.device('mps')
    print('Using MPS backend for Apple Silicon GPU.')
else:
    device = torch.device('cpu')

This implementation choice provides several benefits:

Native support for Apple M1/M2 chips
Reduced training time compared to CPU-only processing
Efficient memory utilization through Metal's optimization

Data Augmentation and Preprocessing

Our data pipeline includes careful preprocessing steps to ensure optimal model performance. Each image stream (face, left eye, right eye) undergoes the following transformations:

self.transformFace = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    scalarmean(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
])

Key preprocessing features include:

Consistent image resizing to 224x224 pixels
Normalization using ImageNet statistics for transfer learning benefits
Custom face grid generation for spatial awareness
Efficient data loading through PyTorch's DataLoader

Model Training

The training pipeline is designed for robustness and reproducibility, incorporating several best practices:

Loss Function: We use Mean Square Error (MSE) loss, which is particularly suitable for regression tasks like gaze estimation:
Provides smooth gradients for optimization
Natural choice for coordinate prediction
Easy to interpret in terms of pixel-level accuracy
Optimization Strategy:
Adam optimizer with carefully tuned parameters
Learning rate of 0.001 for stable convergence
Weight decay (1e-4) for regularization
Batch size of 50 for good gradient estimates
Training Loop Features:
Regular validation to monitor overfitting
Automatic model checkpointing
Early stopping based on validation loss
Comprehensive logging of metrics

Results and Performance

📈 Training Evolution

Milestone	Training Loss	Validation Loss	Gaze Error
Initial (Epoch 1)	33.37	31.15	55.89°
Mid-Training (Epoch 50)	~4.5	~2.5	~10°
Best (Epoch 95)	3.04	1.66	8.18°
Final (Epoch 100)	3.09	2.12	8.79°

🏃‍♂️ Training Progress Timeline


Loss     │                                  Best Model (Epoch 95)
33.37    │█                                      ↓
         │█
         │█
~15.0    │ █
         │  █
         │   █
~7.86    │    █
         │     █        Steady Optimization
         │      ██████████████████████████████████
3.04     │                                      █
         │
         └───────────────────────────────────────────────-───────────────
           1    10    20    30    40    50    60    70    80    90   100
                                  Epochs

Hardware & Dataset Statistics

Training Environment

Platform: Apple Silicon GPU (MPS backend)
Dataset Split: ├── Training: 9,089 records (≈2% of full dataset) ├── Validation: 9,089 records └── Test: 2,682 records

Key Performance Metrics

Best Model Performance (Epoch 95)

📊 Training Loss: 3.04
📊 Validation Loss: 1.66
📊 Average Gaze Error: 8.18°

Test Set Performance

⚠️ Test Loss: 51.82
⚠️ Average Gaze Error: 53.63°

📊 Training Analysis

Major Milestones

Rapid Initial Improvement 📈
First 10 epochs showed dramatic improvement
Training loss dropped from 33.37 → 7.86
Optimization Phase ⚡
Steady improvement between epochs 10-50
Gradual refinement of model parameters
Fine-tuning Phase 🎯
Best performance at epoch 95
Validation loss reached minimum of 1.66

Performance Insights

✅ Successes:

Reduced gaze error by 85% (55.89° → 8.18°)
Achieved stable training progression
Maintained consistent validation performance

⚠️ Challenges:

Significant gap between validation and test performance
Signs of potential overfitting
Limited dataset utilization (~2% of full dataset)

Areas for Improvement

Data Enhancement
Expand training to full dataset
Implement additional augmentation techniques
Enhance data preprocessing pipeline
Model Optimization
Add regularization layers
Tune learning rate schedule
Explore architectural modifications
Performance Tuning
Address validation-test gap
Improve generalization
Optimize inference speed

Future Work

The current implementation, while functional, represents only the beginning of what's possible with this eye-tracking system. Here are the key areas for future development:

Expanded Dataset Utilization
Current implementation uses only ~2% of the full GazeCapture dataset
Plan to extend training to the complete dataset (>2.4M frames)
Expect significant improvement in model robustness and accuracy
Real-time Implementation
Develop webcam integration for live gaze tracking
Optimize inference pipeline for real-time performance
Implement efficient frame processing and prediction
Technical Improvements
Integration of attention mechanisms
Performance optimization strategies
Mobile deployment solutions
Cross-platform GPU support
Model Architecture Enhancements
Explore architectural modifications to reduce overfitting
Implement additional data augmentation techniques
Fine-tune hyperparameters with larger dataset

The immediate focus will be on expanding the training data utilization, as this forms the foundation for more robust real-time applications. Once we achieve better generalization through comprehensive training, we'll move on to implementing real-time webcam tracking capabilities.

Resources

Acknowledgments

This implementation builds upon the work presented in:

K.Krafka, A. Khosla, P. Kellnhofer, H. Kannan, S. Bhandarkar, W. Matusik and A. Torralba, "Eye Tracking for Everyone", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

Special thanks to the CSAIL Vision group for making their dataset and original implementation publicly available.

This completes our journey of implementing the iTracker model. The full source code, along with detailed documentation, is available in the GitHub repository. Feel free to explore, experiment, and contribute to the project!