# ACH File Processing Pipeline - Deployment Checklist ## Implementation Complete ✅ This document summarizes what has been implemented and the deployment checklist. --- ## Files Created ### Core Application Files #### Configuration & Entry Point - **config.py** - Configuration management (loads .env variables) - **main.py** - Updated application entry point - **scheduler.py** - Main 30-minute polling scheduler #### Database Module (db/) - **db/__init__.py** - Module initialization - **db/oracle_connector.py** - Oracle connection pooling - **db/models.py** - TransactionRecord and ProcessedFile data models - **db/repository.py** - Data access layer (CRUD operations) #### SFTP Module (sftp/) - **sftp/__init__.py** - Module initialization - **sftp/sftp_client.py** - SFTP client for file operations - **sftp/file_monitor.py** - File discovery and monitoring #### Processing Module (processors/) - **processors/__init__.py** - Module initialization - **processors/data_mapper.py** - Field mapping and transformations - **processors/file_processor.py** - End-to-end file processing #### Testing - **tests/__init__.py** - Tests module initialization - **tests/test_data_mapper.py** - Unit tests for data mapper - **tests/test_file_monitor.py** - Unit tests for file monitor ### Configuration Files - **.env** - Environment configuration (for testing) - **.env.example** - Configuration template - **docker-compose.yml** - Mock SFTP server setup for testing - **requirements.txt** - Updated with all dependencies ### Documentation - **SETUP.md** - Complete setup and installation guide - **IMPLEMENTATION.md** - Detailed implementation documentation - **DEPLOYMENT.md** - This file --- ## Key Features Implemented ### 1. ACH File Processing Pipeline - ✅ SFTP file monitoring (multi-bank support) - ✅ File parsing using existing ACHParser - ✅ Field mapping to database format - ✅ Batch database insertion (configurable size) - ✅ Duplicate detection by filename - ✅ Error handling with detailed logging - ✅ Graceful shutdown on SIGTERM/SIGINT ### 2. Database Integration - ✅ Oracle connection pooling (min=2, max=10) - ✅ Transaction safety (commit/rollback) - ✅ Processed file tracking table - ✅ Batch insert to `ach_api_log` table - ✅ Duplicate detection in `ach_processed_files` table - ✅ Error message storage for failed files ### 3. Field Mapping - ✅ `remarks` → `narration` - ✅ `sys` → `status` - ✅ `jrnl_no` → `jrnl_id` - ✅ `date` (DD/MM/YY) → `tran_date` (DATE) - ✅ `cust_acct` → `cbs_acct` - ✅ `amount` → `tran_amt` (absolute value, Decimal) - ✅ `amount` → `TXNIND` ('CR' if ≥0, 'DR' if <0) ### 4. Scheduling - ✅ Configurable poll interval (default: 30 minutes) - ✅ Multi-bank file processing - ✅ Graceful shutdown handling - ✅ Processing statistics logging ### 5. Configuration Management - ✅ Environment variable loading (.env) - ✅ Configuration validation - ✅ Bank codes as comma-separated list - ✅ Flexible polling interval ### 6. Error Handling - ✅ SFTP connection failures (logged) - ✅ File parsing errors (marked as failed) - ✅ Database transaction errors (rolled back) - ✅ Duplicate files (skipped, logged as info) - ✅ Partial failures (continue processing) ### 7. Testing - ✅ Unit tests for data mapper - ✅ Unit tests for file monitor - ✅ Integration test structure - ✅ Mock SFTP server setup --- ## Deployment Checklist ### Pre-Deployment - [ ] Read SETUP.md for complete installation steps - [ ] Install Python dependencies: `pip install -r requirements.txt` - [ ] Install Oracle Instant Client (21.12 or later) - [ ] Set LD_LIBRARY_PATH for Oracle Instant Client - [ ] Create Oracle tables (ach_api_log, ach_processed_files) - [ ] Verify database connectivity with sqlplus - [ ] Verify SFTP connectivity with sftp command - [ ] Copy .env.example to .env - [ ] Update .env with production credentials - [ ] Run tests: `pytest tests/ -v` - [ ] Test manual run: `python main.py` (should complete one cycle) ### Testing (Development Environment) - [ ] Use mock SFTP with Docker (see SETUP.md, Step 5) - [ ] Place test ACH file in SFTP data directory - [ ] Run scheduler for one cycle - [ ] Verify file was downloaded and processed - [ ] Verify records in ach_api_log table - [ ] Verify file marked in ach_processed_files table - [ ] Run same file again (should be skipped) - [ ] Check logs for expected messages - [ ] Test CTRL+C for graceful shutdown ### Production Deployment - [ ] Deploy to production server - [ ] Create systemd service file (see SETUP.md, Step 7) - [ ] Test service: `sudo systemctl start ach_processor` - [ ] Verify service is running: `sudo systemctl status ach_processor` - [ ] Check logs: `journalctl -u ach_processor -f` - [ ] Enable on boot: `sudo systemctl enable ach_processor` - [ ] Monitor for first 24 hours - [ ] Set up log rotation if needed - [ ] Document any custom configurations --- ## Quick Start ### For Testing (with Mock SFTP) ```bash # 1. Install dependencies pip install -r requirements.txt # 2. Start mock SFTP docker-compose up -d mkdir -p sftp_data/HDFC/NACH cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/ # 3. Update .env for testing SFTP_HOST=127.0.0.1 SFTP_PORT=2222 POLL_INTERVAL_MINUTES=1 # 4. Run application python main.py # 5. Stop mock SFTP when done docker-compose down ``` ### For Production ```bash # 1. Install Oracle Instant Client # See SETUP.md for detailed instructions # 2. Create database tables # See SETUP.md, Step 3 # 3. Create and edit .env cp .env.example .env # Edit with production credentials # 4. Create systemd service # See SETUP.md, Step 7 # 5. Start service sudo systemctl start ach_processor sudo systemctl status ach_processor ``` --- ## Configuration Summary ### Key Environment Variables ``` # Database (required) DB_USER=pacs_db DB_PASSWORD=pacs_db DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com DB_PORT=1521 DB_SERVICE_NAME=IPKSDB # SFTP (required) SFTP_HOST=192.168.1.100 SFTP_PORT=22 SFTP_USERNAME=ipks SFTP_PASSWORD=secure_password SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS # Processing (optional) POLL_INTERVAL_MINUTES=30 BATCH_SIZE=100 BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB LOG_LEVEL=INFO ``` ### Database Schema **ach_api_log** (existing table) - narration: VARCHAR2(500) - status: VARCHAR2(100) - bankcode: VARCHAR2(20) - jrnl_id: VARCHAR2(50) - tran_date: DATE - cbs_acct: VARCHAR2(50) - tran_amt: NUMBER(15,2) - TXNIND: VARCHAR2(2) **ach_processed_files** (created by app) - filename: VARCHAR2(500) UNIQUE NOT NULL - bankcode: VARCHAR2(20) - file_path: VARCHAR2(1000) - processed_at: TIMESTAMP - transaction_count: NUMBER - status: VARCHAR2(20) - error_message: VARCHAR2(2000) --- ## System Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Scheduler (30 min interval) │ └──────────────────────────┬──────────────────────────────────┘ │ ├─────────────┬────────────────┐ ▼ ▼ ▼ SFTP Bank 1 SFTP Bank 2 SFTP Bank N (HDFC/NACH) (ICICI/NACH) (SBI/NACH) │ │ │ └─────────────┼────────────────┘ ▼ ┌─────────────────────────┐ │ File Monitor │ │ - Scan directories │ │ - Check duplicates │ └────────┬────────────────┘ ▼ ┌─────────────────────────┐ │ File Processor │ │ - Download file │ │ - Parse ACH │ │ - Map fields │ └────────┬────────────────┘ ▼ ┌─────────────────────────┐ │ Data Mapper │ │ - Convert dates │ │ - Calculate TXNIND │ │ - Format amounts │ └────────┬────────────────┘ ▼ ┌─────────────────────────┐ │ Repository │ │ - Batch insert │ │ - Mark as processed │ │ - Check duplicates │ └────────┬────────────────┘ ▼ ┌─────────────────────────┐ │ Oracle Database │ │ - ach_api_log │ │ - ach_processed_files │ └─────────────────────────┘ ``` --- ## Processing Flow ``` Start Scheduler (30-min interval) │ ├─> Database Connection Test │ └─> Create ach_processed_files table if needed │ ├─> For Each Bank Code (HDFC, ICICI, SBI, etc.) │ │ │ ├─> SFTP Connect │ │ │ ├─> Scan Directory: /bank_code/NACH/ │ │ └─> List files: ACH_*.txt │ │ │ ├─> For Each File Found │ │ │ │ │ ├─> Check if Already Processed │ │ │ └─> If yes: Skip and log as info │ │ │ │ │ ├─> Download File to Temp Directory │ │ │ │ │ ├─> Parse ACH File │ │ │ └─> Extract transactions (178 in sample) │ │ │ │ │ ├─> Map Each Transaction │ │ │ ├─> Convert date DD/MM/YY → DATE │ │ │ ├─> Calculate TXNIND from amount │ │ │ └─> Create TransactionRecord │ │ │ │ │ ├─> Batch Insert to Database (every 100 records) │ │ │ ├─> BEGIN TRANSACTION │ │ │ ├─> INSERT batch into ach_api_log │ │ │ ├─> INSERT into ach_processed_files │ │ │ └─> COMMIT (or ROLLBACK on error) │ │ │ │ │ ├─> Mark File as Processed │ │ │ │ │ └─> Clean Up Local File │ │ │ └─> SFTP Disconnect │ ├─> Log Processing Summary │ └─> Total/Successful/Failed counts │ └─> Sleep 30 Minutes (or configured interval) │ └─> Repeat... ``` --- ## Monitoring ### Log Location ``` logs/app.log ``` ### Key Log Messages | Event | Log Level | Example | |-------|-----------|---------| | Scheduler started | INFO | "ACH File Processing Scheduler Started" | | Database connected | INFO | "Database connection test successful" | | File found | INFO | "Found new file: ACH_99944_..." | | File skipped | INFO | "File already processed: ACH_99944_..." | | Processing started | INFO | "Starting processing: ACH_99944_..." | | Processing complete | INFO | "Successfully processed ACH_99944_..." | | Processing failed | ERROR | "Error processing ACH_99944_..." | | Database error | ERROR | "Error inserting transactions: ..." | | SFTP error | ERROR | "Failed to connect to SFTP server" | ### Metrics to Monitor 1. **File Processing Rate** - How many files processed per cycle - Success vs. failure rate 2. **Transaction Processing** - Number of transactions per file - Records inserted vs. parsed 3. **Processing Time** - Time per file - Time per cycle (should be << 30 min) 4. **Error Rate** - Failed files - Database errors - SFTP errors ### Health Checks ```bash # Check service status sudo systemctl status ach_processor # Check recent logs journalctl -u ach_processor -n 50 # Check database connectivity sqlplus pacs_db/pacs_db@... # Check SFTP connectivity sftp -P 22 user@host # Check processed file count sqlplus -s pacs_db/pacs_db@... <