ach_ui_dbtl_file_based/DEPLOYMENT.md

ACH File Processing Pipeline - Deployment Checklist

Implementation Complete ✅

This document summarizes what has been implemented and the deployment checklist.

---

Files Created

Core Application Files

Configuration & Entry Point

• config.py - Configuration management (loads .env variables)
• main.py - Updated application entry point
• scheduler.py - Main 30-minute polling scheduler

Database Module (db/)

• db/init.py - Module initialization
• db/oracle_connector.py - Oracle connection pooling
• db/models.py - TransactionRecord and ProcessedFile data models
• db/repository.py - Data access layer (CRUD operations)

SFTP Module (sftp/)

• sftp/init.py - Module initialization
• sftp/sftp_client.py - SFTP client for file operations
• sftp/file_monitor.py - File discovery and monitoring

Processing Module (processors/)

• processors/init.py - Module initialization
• processors/data_mapper.py - Field mapping and transformations
• processors/file_processor.py - End-to-end file processing

Testing

• tests/init.py - Tests module initialization
• tests/test_data_mapper.py - Unit tests for data mapper
• tests/test_file_monitor.py - Unit tests for file monitor

Configuration Files

• .env - Environment configuration (for testing)
• .env.example - Configuration template
• docker-compose.yml - Mock SFTP server setup for testing
• requirements.txt - Updated with all dependencies

Documentation

• SETUP.md - Complete setup and installation guide
• IMPLEMENTATION.md - Detailed implementation documentation
• DEPLOYMENT.md - This file

---

Key Features Implemented

1. ACH File Processing Pipeline

• ✅ SFTP file monitoring (multi-bank support)
• ✅ File parsing using existing ACHParser
• ✅ Field mapping to database format
• ✅ Batch database insertion (configurable size)
• ✅ Duplicate detection by filename
• ✅ Error handling with detailed logging
• ✅ Graceful shutdown on SIGTERM/SIGINT

2. Database Integration

• ✅ Oracle connection pooling (min=2, max=10)
• ✅ Transaction safety (commit/rollback)
• ✅ Processed file tracking table
• ✅ Batch insert to ach_api_log table
• ✅ Duplicate detection in ach_processed_files table
• ✅ Error message storage for failed files

3. Field Mapping

• ✅ remarks → narration
• ✅ sys → status
• ✅ jrnl_no → jrnl_id
• ✅ date (DD/MM/YY) → tran_date (DATE)
• ✅ cust_acct → cbs_acct
• ✅ amount → tran_amt (absolute value, Decimal)
• ✅ amount → TXNIND ('CR' if ≥0, 'DR' if <0)

4. Scheduling

• ✅ Configurable poll interval (default: 30 minutes)
• ✅ Multi-bank file processing
• ✅ Graceful shutdown handling
• ✅ Processing statistics logging

5. Configuration Management

• ✅ Environment variable loading (.env)
• ✅ Configuration validation
• ✅ Bank codes as comma-separated list
• ✅ Flexible polling interval

6. Error Handling

• ✅ SFTP connection failures (logged)
• ✅ File parsing errors (marked as failed)
• ✅ Database transaction errors (rolled back)
• ✅ Duplicate files (skipped, logged as info)
• ✅ Partial failures (continue processing)

7. Testing

• ✅ Unit tests for data mapper
• ✅ Unit tests for file monitor
• ✅ Integration test structure
• ✅ Mock SFTP server setup

---

Deployment Checklist

Pre-Deployment

• Read SETUP.md for complete installation steps
• Install Python dependencies: pip install -r requirements.txt
• Install Oracle Instant Client (21.12 or later)
• Set LD_LIBRARY_PATH for Oracle Instant Client
• Create Oracle tables (ach_api_log, ach_processed_files)
• Verify database connectivity with sqlplus
• Verify SFTP connectivity with sftp command
• Copy .env.example to .env
• Update .env with production credentials
• Run tests: pytest tests/ -v
• Test manual run: python main.py (should complete one cycle)

Testing (Development Environment)

• Use mock SFTP with Docker (see SETUP.md, Step 5)
• Place test ACH file in SFTP data directory
• Run scheduler for one cycle
• Verify file was downloaded and processed
• Verify records in ach_api_log table
• Verify file marked in ach_processed_files table
• Run same file again (should be skipped)
• Check logs for expected messages
• Test CTRL+C for graceful shutdown

Production Deployment

• Deploy to production server
• Create systemd service file (see SETUP.md, Step 7)
• Test service: sudo systemctl start ach_processor
• Verify service is running: sudo systemctl status ach_processor
• Check logs: journalctl -u ach_processor -f
• Enable on boot: sudo systemctl enable ach_processor
• Monitor for first 24 hours
• Set up log rotation if needed
• Document any custom configurations

---

Quick Start

For Testing (with Mock SFTP)

# 1. Install dependencies
pip install -r requirements.txt

# 2. Start mock SFTP
docker-compose up -d
mkdir -p sftp_data/HDFC/NACH
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/

# 3. Update .env for testing
SFTP_HOST=127.0.0.1
SFTP_PORT=2222
POLL_INTERVAL_MINUTES=1

# 4. Run application
python main.py

# 5. Stop mock SFTP when done
docker-compose down

For Production

# 1. Install Oracle Instant Client
# See SETUP.md for detailed instructions

# 2. Create database tables
# See SETUP.md, Step 3

# 3. Create and edit .env
cp .env.example .env
# Edit with production credentials

# 4. Create systemd service
# See SETUP.md, Step 7

# 5. Start service
sudo systemctl start ach_processor
sudo systemctl status ach_processor

---

Configuration Summary

Key Environment Variables

# Database (required)
DB_USER=pacs_db
DB_PASSWORD=pacs_db
DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com
DB_PORT=1521
DB_SERVICE_NAME=IPKSDB

# SFTP (required)
SFTP_HOST=192.168.1.100
SFTP_PORT=22
SFTP_USERNAME=ipks
SFTP_PASSWORD=secure_password
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS

# Processing (optional)
POLL_INTERVAL_MINUTES=30
BATCH_SIZE=100
BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB
LOG_LEVEL=INFO

Database Schema

ach_api_log (existing table)

• narration: VARCHAR2(500)
• status: VARCHAR2(100)
• bankcode: VARCHAR2(20)
• jrnl_id: VARCHAR2(50)
• tran_date: DATE
• cbs_acct: VARCHAR2(50)
• tran_amt: NUMBER(15,2)
• TXNIND: VARCHAR2(2)

ach_processed_files (created by app)

• filename: VARCHAR2(500) UNIQUE NOT NULL
• bankcode: VARCHAR2(20)
• file_path: VARCHAR2(1000)
• processed_at: TIMESTAMP
• transaction_count: NUMBER
• status: VARCHAR2(20)
• error_message: VARCHAR2(2000)

---

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Scheduler (30 min interval)              │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ├─────────────┬────────────────┐
                           ▼             ▼                ▼
                      SFTP Bank 1   SFTP Bank 2    SFTP Bank N
                    (HDFC/NACH)   (ICICI/NACH)   (SBI/NACH)
                           │             │                │
                           └─────────────┼────────────────┘
                                         ▼
                           ┌─────────────────────────┐
                           │  File Monitor           │
                           │  - Scan directories     │
                           │  - Check duplicates     │
                           └────────┬────────────────┘
                                    ▼
                           ┌─────────────────────────┐
                           │  File Processor         │
                           │  - Download file        │
                           │  - Parse ACH            │
                           │  - Map fields           │
                           └────────┬────────────────┘
                                    ▼
                           ┌─────────────────────────┐
                           │  Data Mapper            │
                           │  - Convert dates        │
                           │  - Calculate TXNIND     │
                           │  - Format amounts       │
                           └────────┬────────────────┘
                                    ▼
                           ┌─────────────────────────┐
                           │  Repository             │
                           │  - Batch insert         │
                           │  - Mark as processed    │
                           │  - Check duplicates     │
                           └────────┬────────────────┘
                                    ▼
                           ┌─────────────────────────┐
                           │  Oracle Database        │
                           │  - ach_api_log          │
                           │  - ach_processed_files  │
                           └─────────────────────────┘

---

Processing Flow

Start Scheduler (30-min interval)
    │
    ├─> Database Connection Test
    │    └─> Create ach_processed_files table if needed
    │
    ├─> For Each Bank Code (HDFC, ICICI, SBI, etc.)
    │    │
    │    ├─> SFTP Connect
    │    │
    │    ├─> Scan Directory: /bank_code/NACH/
    │    │    └─> List files: ACH_*.txt
    │    │
    │    ├─> For Each File Found
    │    │    │
    │    │    ├─> Check if Already Processed
    │    │    │    └─> If yes: Skip and log as info
    │    │    │
    │    │    ├─> Download File to Temp Directory
    │    │    │
    │    │    ├─> Parse ACH File
    │    │    │    └─> Extract transactions (178 in sample)
    │    │    │
    │    │    ├─> Map Each Transaction
    │    │    │    ├─> Convert date DD/MM/YY → DATE
    │    │    │    ├─> Calculate TXNIND from amount
    │    │    │    └─> Create TransactionRecord
    │    │    │
    │    │    ├─> Batch Insert to Database (every 100 records)
    │    │    │    ├─> BEGIN TRANSACTION
    │    │    │    ├─> INSERT batch into ach_api_log
    │    │    │    ├─> INSERT into ach_processed_files
    │    │    │    └─> COMMIT (or ROLLBACK on error)
    │    │    │
    │    │    ├─> Mark File as Processed
    │    │    │
    │    │    └─> Clean Up Local File
    │    │
    │    └─> SFTP Disconnect
    │
    ├─> Log Processing Summary
    │    └─> Total/Successful/Failed counts
    │
    └─> Sleep 30 Minutes (or configured interval)
        │
        └─> Repeat...

---

Monitoring

Log Location

logs/app.log

Key Log Messages

Event	Log Level	Example
Scheduler started	INFO	"ACH File Processing Scheduler Started"
Database connected	INFO	"Database connection test successful"
File found	INFO	"Found new file: ACH_99944_..."
File skipped	INFO	"File already processed: ACH_99944_..."
Processing started	INFO	"Starting processing: ACH_99944_..."
Processing complete	INFO	"Successfully processed ACH_99944_..."
Processing failed	ERROR	"Error processing ACH_99944_..."
Database error	ERROR	"Error inserting transactions: ..."
SFTP error	ERROR	"Failed to connect to SFTP server"

Metrics to Monitor

1. File Processing Rate

   • How many files processed per cycle
   • Success vs. failure rate

2. Transaction Processing

   • Number of transactions per file
   • Records inserted vs. parsed

3. Processing Time

   • Time per file
   • Time per cycle (should be << 30 min)

4. Error Rate

   • Failed files
   • Database errors
   • SFTP errors

Health Checks

# Check service status
sudo systemctl status ach_processor

# Check recent logs
journalctl -u ach_processor -n 50

# Check database connectivity
sqlplus pacs_db/pacs_db@...

# Check SFTP connectivity
sftp -P 22 user@host

# Check processed file count
sqlplus -s pacs_db/pacs_db@... <<EOF
SELECT COUNT(*) FROM ach_processed_files;
SELECT COUNT(*) FROM ach_api_log;
EXIT;
EOF

---

Rollback Plan

If issues arise in production:

1. Stop the scheduler

   sudo systemctl stop ach_processor
   

2. Investigate the issue

   journalctl -u ach_processor --since "30 min ago"
   tail -n 1000 logs/app.log | grep ERROR
   

3. Fix the issue

   • Update .env configuration
   • Restart services if needed
   • Check database/SFTP connectivity

4. Restart the scheduler

   sudo systemctl start ach_processor
   

5. Verify processing resumes

   journalctl -u ach_processor -f
   

---

Support & Troubleshooting

See SETUP.md for detailed troubleshooting guide covering:

• ImportError: No module named 'cx_Oracle'
• Database Connection Refused
• SFTP Connection Refused
• Application Hangs or Doesn't Process Files
• Permission Denied errors
• Performance tuning

---

Next Steps

1. Follow the Quick Start section above
2. Complete the Deployment Checklist
3. Deploy to production
4. Monitor logs and metrics
5. Set up additional monitoring/alerting as needed
6. Consider enhancements (see IMPLEMENTATION.md for ideas)

---

Contact & Support

For issues:

1. Check this document
2. Review SETUP.md troubleshooting
3. Check application logs with LOG_LEVEL=DEBUG
4. Review IMPLEMENTATION.md for architectural details

---

Deployment Date: [Insert date when deployed] Deployed By: [Insert name/team] Version: 1.0 Last Updated: 2026-01-30