Case Study 4: Duplicate File Detector & Storage Reclaimer

Scenario

A shared drive has accumulated years of duplicate files due to repeated copying and folder migrations. Your task is to identify exact duplicate files and estimate the amount of storage that can be reclaimed.

Suggested Libraries

• hashlib
• pathlib
• collections
• os

Sample Files You May Create

• shared_drive/ — a folder tree containing duplicate files
• duplicate_finder.py — your solution script
• duplicate_report — the output report

Requirements

• Scan folders recursively.
• Group files efficiently (don't hash every file up front — that's expensive).
• Detect duplicates based on file contents, not just filename.
• Avoid excessive memory usage, even for very large files.
• Produce a duplicate file report.
• Estimate reclaimable storage space.
• Support a dry-run mode before deletion.

Have a genuine attempt at this before revealing the solution below. Think about why comparing file sizes first, before hashing, matters for performance on a drive with millions of files.

💡 Show Answer

import os
import hashlib
from collections import defaultdict
from pathlib import Path

def file_hash(path, block_size=65536):
    """Stream the file so we never load huge files fully into memory."""
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(block_size), b""):
            h.update(chunk)
    return h.hexdigest()

def find_duplicates(root_dir):
    size_map = defaultdict(list)
    # Step 1: group by size first (cheap) before hashing (expensive)
    for dirpath, _, filenames in os.walk(root_dir):
        for name in filenames:
            path = Path(dirpath) / name
            try:
                size_map[path.stat().st_size].append(path)
            except OSError:
                continue

    duplicates = defaultdict(list)
    for size, paths in size_map.items():
        if len(paths) < 2:
            continue  # unique size => can't be a duplicate
        hash_map = defaultdict(list)
        for p in paths:
            hash_map[file_hash(p)].append(p)
        for h, group in hash_map.items():
            if len(group) > 1:
                duplicates[h] = group
    return duplicates

def reclaim_space(duplicates, dry_run=True):
    freed = 0
    for h, paths in duplicates.items():
        keep, *remove = sorted(paths)  # keep the alphabetically first path
        for p in remove:
            freed += p.stat().st_size
            print(f"{'[DRY RUN] Would delete' if dry_run else 'Deleting'}: {p}")
            if not dry_run:
                p.unlink()
    print(f"Total space {'reclaimable' if dry_run else 'reclaimed'}: {freed / 1e9:.2f} GB")

if __name__ == "__main__":
    dupes = find_duplicates("./shared_drive")
    reclaim_space(dupes, dry_run=True)

How it scales: The size-first grouping avoids hashing every file — it's an O(n) pass of cheap stat() calls before the expensive I/O of hashing. For millions of files, swap the in-memory dict for a SQLite table keyed on (size, partial_hash) and only compute full hashes on collisions.