🔧 Worker Pipeline Architecture

Production-ready Pokemon card processing pipeline with CLIP similarity search and autonomous recovery.

🚀 System Overview

The Project Arceus worker pipeline combines YOLO detection with CLIP similarity search for fast, accurate card identification with autonomous operation.

🎯 Performance Metrics

  • • Processing: CLIP-only identification
  • • Speed: Fast embedding-based matching
  • • Cost: Free processing (no API calls)
  • • Uptime: 99.9%+ with auto-recovery

🤖 Autonomous Features

  • • Auto-Recovery: 30s detection, 10s resolution
  • • Smart Retries: Max 3 attempts with backoff
  • • Process Monitoring: Auto-restart on crash
  • • Health Checks: Continuous system monitoring

🔄 Processing Pipeline

📊 Step-by-Step Process

  1. 1
    Image Upload: User uploads Pokemon card images (JPEG, PNG, HEIC)
    Frontend stores images in Supabase Storage and creates job queue entry
  2. 2
    Job Dequeue: Python worker polls for new jobs every 5 seconds
    Worker uses dequeue_job() with visibility timeout for atomic processing
  3. 3
    YOLO Detection: Custom YOLOv8 model detects individual cards
    Trained specifically for Pokemon cards with high detection accuracy
  4. 4
    CLIP Similarity: Fast embedding-based card matching
    OpenAI CLIP ViT-B-32-quickgelu model with Pokemon card embeddings
  5. 5
    Card Database Lookup: Match identified cards to database
    Automatic card creation and inventory updates with confidence tracking
  6. 6
    Result Storage: Save detection results and update job status
    Store bounding boxes, confidence scores, and processing metadata

🧠 CLIP Card Identification System

Our production CLIP similarity search system provides fast, cost-effective card identification with high accuracy.

🔍 CLIP Similarity Search

  • • Model: ViT-B-32-quickgelu (optimized)
  • • Speed: Fast embedding-based matching
  • • Cost: Free processing (no API calls)
  • • Database: 19k+ Pokemon card embeddings
  • • Accuracy: High confidence matches for known cards

💡 System Benefits

• Cost-Effective: No API costs, completely free processing

• Fast Performance: Embedding-based similarity search

• Scalable: No per-card costs, unlimited processing

• Reliable: Self-hosted model with local embeddings

🔧 Auto-Recovery System

Autonomous monitoring and recovery system eliminates manual intervention for stuck jobs.

🚨 Stuck Job Detection

  • • Monitor Interval: Every 30 seconds
  • • Timeout Threshold: 10 minutes in processing
  • • Detection Query: get_stuck_jobs() function
  • • Recovery Trigger: Automatic retry or permanent failure

🔄 Smart Retry Logic

  • • Max Retries: 3 attempts per job
  • • Retry Tracking: Database column with attempt counter
  • • Exponential Backoff: Increasing delays between retries
  • • Permanent Failure: After 3 failed attempts

🏥 Process Health Monitoring

  • • Worker Heartbeat: Regular job processing confirmation
  • • Process Restart: Auto-restart crashed workers
  • • Health Metrics: Job queue statistics and trends
  • • Alert System: Real-time problem notifications

💾 Database Schema

📊 Key Tables

job_queue
  • • id, scan_id, status
  • • created_at, updated_at
  • • visibility_timeout
  • • retry_count, error_message
scan_uploads
  • • id, user_id, storage_path
  • • status, created_at
  • • processing metadata
detected_cards
  • • id, scan_id, card_id
  • • confidence, bbox coordinates
  • • identification_method, cost
worker_logs
  • • id, level, message
  • • scan_id, created_at
  • • processing context

🔍 Health Monitoring Functions

SELECT * FROM job_queue_health;

Returns job statistics: total, processing, failed, average wait time

SELECT * FROM get_stuck_jobs();

Identifies jobs stuck longer than timeout threshold

SELECT * FROM auto_recover_stuck_jobs();

Automatically retries or marks failed stuck jobs

🚀 Production Deployment

🎛️ Complete System Startup

python start_production_system.py

Starts worker + auto-recovery + process monitoring + health checks

⚡ Individual Components

cd worker && python worker.py

Main processing worker (YOLO + CLIP identification)

cd worker && python auto_recovery_system.py

Autonomous stuck job recovery monitor

npm run dev

Next.js frontend development server

📋 Environment Configuration

Required: SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY

Optional: HUGGING_FACE_TOKEN (for model downloads)

Settings: Confidence thresholds, retry counts

📊 Performance & Monitoring

💰 Cost Analysis

  • • Processing Cost: $0.00 (completely free)
  • • No API Dependencies: Self-hosted model
  • • Scalability: Unlimited processing capacity
  • • Infrastructure: Only hosting costs

📈 System Metrics

  • • Training Data: 4-category feedback system
  • • Confidence Tracking: CLIP similarity scores
  • • Error Analysis: Misidentification patterns
  • • Model Performance: Embedding quality monitoring