api/archive/auto_processing/README.md

Auto-Processing Code Archive

This directory contains the complex auto-processing system that was previously used for automatic document processing after file upload.

Archived Components

Core Processing Files

• files_with_auto_processing.py - Original files.py router with automatic processing
• pipeline_controller.py - Complex multi-phase pipeline orchestration
• task_processors.py - Document processing task handlers

Advanced Queue Management (Created but not deployed)

• memory_aware_queue.py - Memory-based intelligent queue management
• enhanced_upload_handler.py - Advanced upload handler with queuing
• enhanced_upload.py - API endpoints for advanced upload system

What This System Did

Automatic Processing Pipeline

1. File Upload → Immediate processing trigger
2. PDF Conversion (synchronous, blocking)
3. Phase 1: Structure discovery (Tika, Page Images, Document Analysis, Split Map)
4. Phase 2: Docling processing (NO_OCR → OCR → VLM pipelines)
5. Complex Dependencies: Phase coordination, task sequencing
6. Redis Queue Management: Service limits, rate limits, dependency tracking

Features

• Multi-phase processing pipelines
• Complex task dependency management
• Memory-aware queue limits
• Multi-user capacity management
• Real-time processing status
• WebSocket status updates
• Service-specific resource limits
• Task recovery on restart

Why Archived

The system was overly complex for the current needs:

• Complexity: Multi-phase pipelines with complex dependencies
• Blocking Operations: Synchronous PDF conversion causing timeouts
• Resource Management: Over-engineered for single-user scenarios
• User Experience: Users had to wait for processing to complete

New Simplified Approach

The new system focuses on:

• Simple Upload: Just store files and create database records
• No Auto-Processing: Users manually trigger processing when needed
• Directory Support: Upload entire folders with manifest tracking
• Immediate Response: Users get instant confirmation without waiting

If You Need to Restore

To restore the auto-processing functionality:

1. Copy files_with_auto_processing.py back to routers/database/files/files.py
2. Ensure pipeline_controller.py and task_processors.py are in modules/
3. Update imports and dependencies
4. Re-enable background processing in upload handlers

Migration Notes

The database schema and Redis structure remain compatible. The new simplified system can coexist with the archived processing logic if needed.

Date Archived: $(date) Reason: Simplification for directory upload implementation