Deja Vu: Near-Duplicate Video Detection with Deep Learning / Etsub Feleke

The Question That Started This

Something that always bugged me about content moderation: what happens when someone takes a banned video, crops it, shifts the colors, maybe sticks it in a reaction layout, and re-uploads it? The platform has technically seen this content before, but now it looks just different enough to slip through.

I wanted to explore that for my CSCE 5218 (Deep Learning) course at UNT. I had never worked with video data before, so this felt like a good excuse to figure out transfer learning on something that isn't just image classification. The name "Deja Vu" came from the core idea: the system has seen this before, even if it looks a little different now.

How Do You Even Compare Videos?

This was my first real question. I didn't know how to approach video similarity programmatically. My initial thought was to break it down frame by frame. A video is just a sequence of images, right? So if I could get a meaningful representation of each frame, maybe I could aggregate those into a single "fingerprint" for the whole video.

That led me to transfer learning. Training something from scratch wasn't realistic with the data I had, but pretrained models like CLIP (ViT-B/32) and ResNet-50 already understand visual content at a deep level. The idea: extract embeddings from sampled frames, average them into one video-level vector, and then use cosine similarity to compare against a database of known banned content.

Here's the pipeline I ended up with:

Sample frames at 1 fps
Extract embeddings with CLIP or ResNet
Mean-pool and L2-normalize into a single video embedding
Compare against the banned database using cosine similarity
Map the similarity score to a decision: Auto-flag (>= 0.90), Human Review (0.70-0.89), or Allow (< 0.70)

Making My Own Dataset

There's no "banned videos and their re-uploads" dataset on Kaggle, so I had to make my own. This ended up taking more time than I expected.

I started with 52 source videos from Pexels and Pixabay (Creative Commons), then wrote scripts to generate two tiers of transformations:

Easy (8 types, 416 videos): Flips, brightness changes, cropping, speed adjustments. The kind of quick edits someone might make in 30 seconds.

Hard (6 types, 312 videos): Reaction layouts (video shrunk into a corner of a bigger frame), heavy text overlays, aggressive combined transforms. These are what actually break simple detection.

I also generated 104 negative videos (totally unrelated content) to test for false positives.

Splitting into easy vs. hard turned out to be one of the most important decisions, because it completely changed the results story. More on that in a second.

Transformation Examples (Click to Expand)

Results

Method	Easy mAP	Hard mAP	Best F1
CLIP	0.972	0.828	0.877
ResNet-50	0.979	0.905	0.901
pHash baseline	-	-	easy: 0.889, hard: 0.857

A few things surprised me here.

ResNet-50 outperformed CLIP overall. I honestly expected CLIP to win because of its more sophisticated vision-language training, but ResNet's pure visual features were better at discriminating in most cases.

CLIP did better on reaction layouts and heavy combined edits though. I think its training gives it a more semantic sense of "what's in this frame" that holds up when the visual structure gets rearranged. That was interesting to see.

pHash worked surprisingly well on easy edits. If I had only tested easy transformations, I might have concluded deep learning was overkill for this problem. The hard set is what showed the real difference. This is why splitting into two tiers mattered so much.

The Live Demo

I wanted this to feel tangible, not just metrics in a notebook. So I built a demo with FastAPI on the backend and React + Vite on the frontend. You upload an MP4, the system samples frames, extracts CLIP embeddings, compares against the banned database, and returns a similarity score with a moderation decision.

Getting the frontend and backend talking smoothly involved the usual headaches (CORS, file upload handling, async processing), but having an actual interface where you drag and drop a video and see "Auto-flag: 0.94 similarity" made the whole project click in a way that charts and tables don't.

Demo Video

What I Took Away

Transfer learning goes further than I expected. I didn't fine-tune anything. Pretrained CLIP and ResNet, straight out of the box, hit over 90% F1 on hard near-duplicate detection. The representations these models learn are incredibly general.

How you build your test set shapes the story you tell. If I'd skipped the hard transformations, I would have drawn totally different (and misleading) conclusions about what methods work. I'll probably think about this every time I set up an evaluation from now on.

I also learned that a single threshold doesn't really cut it for this kind of task. I tried binary duplicate/not-duplicate at first and it didn't work well. The tiered approach (auto-flag / review / allow) was much more practical. And more frames doesn't always mean better, since accuracy plateaus pretty early when you increase the sampling rate.

What I'd Explore Next

Fine-tuning the encoders on actual near-duplicate pairs
Adding audio fingerprinting (the system is purely visual right now)
Temporal modeling, since mean-pooling throws away all frame ordering

Tech Stack

Deep Learning: PyTorch, CLIP (ViT-B/32), ResNet-50, torchvision
Similarity Search: FAISS, cosine similarity
Evaluation: scikit-learn (mAP, precision, recall, F1)
Demo: FastAPI (backend), React + Vite (frontend)
Data Processing: OpenCV, NumPy, imagehash
Visualization: Matplotlib, Seaborn

See It in Action

Course project for CSCE 5218 (Deep Learning) at UNT