482 lines
15 KiB
Markdown
482 lines
15 KiB
Markdown
# ACH File Processing Pipeline - Deployment Checklist
|
|
|
|
## Implementation Complete ✅
|
|
|
|
This document summarizes what has been implemented and the deployment checklist.
|
|
|
|
---
|
|
|
|
## Files Created
|
|
|
|
### Core Application Files
|
|
|
|
#### Configuration & Entry Point
|
|
- **config.py** - Configuration management (loads .env variables)
|
|
- **main.py** - Updated application entry point
|
|
- **scheduler.py** - Main 30-minute polling scheduler
|
|
|
|
#### Database Module (db/)
|
|
- **db/__init__.py** - Module initialization
|
|
- **db/oracle_connector.py** - Oracle connection pooling
|
|
- **db/models.py** - TransactionRecord and ProcessedFile data models
|
|
- **db/repository.py** - Data access layer (CRUD operations)
|
|
|
|
#### SFTP Module (sftp/)
|
|
- **sftp/__init__.py** - Module initialization
|
|
- **sftp/sftp_client.py** - SFTP client for file operations
|
|
- **sftp/file_monitor.py** - File discovery and monitoring
|
|
|
|
#### Processing Module (processors/)
|
|
- **processors/__init__.py** - Module initialization
|
|
- **processors/data_mapper.py** - Field mapping and transformations
|
|
- **processors/file_processor.py** - End-to-end file processing
|
|
|
|
#### Testing
|
|
- **tests/__init__.py** - Tests module initialization
|
|
- **tests/test_data_mapper.py** - Unit tests for data mapper
|
|
- **tests/test_file_monitor.py** - Unit tests for file monitor
|
|
|
|
### Configuration Files
|
|
- **.env** - Environment configuration (for testing)
|
|
- **.env.example** - Configuration template
|
|
- **docker-compose.yml** - Mock SFTP server setup for testing
|
|
- **requirements.txt** - Updated with all dependencies
|
|
|
|
### Documentation
|
|
- **SETUP.md** - Complete setup and installation guide
|
|
- **IMPLEMENTATION.md** - Detailed implementation documentation
|
|
- **DEPLOYMENT.md** - This file
|
|
|
|
---
|
|
|
|
## Key Features Implemented
|
|
|
|
### 1. ACH File Processing Pipeline
|
|
- ✅ SFTP file monitoring (multi-bank support)
|
|
- ✅ File parsing using existing ACHParser
|
|
- ✅ Field mapping to database format
|
|
- ✅ Batch database insertion (configurable size)
|
|
- ✅ Duplicate detection by filename
|
|
- ✅ Error handling with detailed logging
|
|
- ✅ Graceful shutdown on SIGTERM/SIGINT
|
|
|
|
### 2. Database Integration
|
|
- ✅ Oracle connection pooling (min=2, max=10)
|
|
- ✅ Transaction safety (commit/rollback)
|
|
- ✅ Processed file tracking table
|
|
- ✅ Batch insert to `ach_api_log` table
|
|
- ✅ Duplicate detection in `ach_processed_files` table
|
|
- ✅ Error message storage for failed files
|
|
|
|
### 3. Field Mapping
|
|
- ✅ `remarks` → `narration`
|
|
- ✅ `sys` → `status`
|
|
- ✅ `jrnl_no` → `jrnl_id`
|
|
- ✅ `date` (DD/MM/YY) → `tran_date` (DATE)
|
|
- ✅ `cust_acct` → `cbs_acct`
|
|
- ✅ `amount` → `tran_amt` (absolute value, Decimal)
|
|
- ✅ `amount` → `TXNIND` ('CR' if ≥0, 'DR' if <0)
|
|
|
|
### 4. Scheduling
|
|
- ✅ Configurable poll interval (default: 30 minutes)
|
|
- ✅ Multi-bank file processing
|
|
- ✅ Graceful shutdown handling
|
|
- ✅ Processing statistics logging
|
|
|
|
### 5. Configuration Management
|
|
- ✅ Environment variable loading (.env)
|
|
- ✅ Configuration validation
|
|
- ✅ Bank codes as comma-separated list
|
|
- ✅ Flexible polling interval
|
|
|
|
### 6. Error Handling
|
|
- ✅ SFTP connection failures (logged)
|
|
- ✅ File parsing errors (marked as failed)
|
|
- ✅ Database transaction errors (rolled back)
|
|
- ✅ Duplicate files (skipped, logged as info)
|
|
- ✅ Partial failures (continue processing)
|
|
|
|
### 7. Testing
|
|
- ✅ Unit tests for data mapper
|
|
- ✅ Unit tests for file monitor
|
|
- ✅ Integration test structure
|
|
- ✅ Mock SFTP server setup
|
|
|
|
---
|
|
|
|
## Deployment Checklist
|
|
|
|
### Pre-Deployment
|
|
|
|
- [ ] Read SETUP.md for complete installation steps
|
|
- [ ] Install Python dependencies: `pip install -r requirements.txt`
|
|
- [ ] Install Oracle Instant Client (21.12 or later)
|
|
- [ ] Set LD_LIBRARY_PATH for Oracle Instant Client
|
|
- [ ] Create Oracle tables (ach_api_log, ach_processed_files)
|
|
- [ ] Verify database connectivity with sqlplus
|
|
- [ ] Verify SFTP connectivity with sftp command
|
|
- [ ] Copy .env.example to .env
|
|
- [ ] Update .env with production credentials
|
|
- [ ] Run tests: `pytest tests/ -v`
|
|
- [ ] Test manual run: `python main.py` (should complete one cycle)
|
|
|
|
### Testing (Development Environment)
|
|
|
|
- [ ] Use mock SFTP with Docker (see SETUP.md, Step 5)
|
|
- [ ] Place test ACH file in SFTP data directory
|
|
- [ ] Run scheduler for one cycle
|
|
- [ ] Verify file was downloaded and processed
|
|
- [ ] Verify records in ach_api_log table
|
|
- [ ] Verify file marked in ach_processed_files table
|
|
- [ ] Run same file again (should be skipped)
|
|
- [ ] Check logs for expected messages
|
|
- [ ] Test CTRL+C for graceful shutdown
|
|
|
|
### Production Deployment
|
|
|
|
- [ ] Deploy to production server
|
|
- [ ] Create systemd service file (see SETUP.md, Step 7)
|
|
- [ ] Test service: `sudo systemctl start ach_processor`
|
|
- [ ] Verify service is running: `sudo systemctl status ach_processor`
|
|
- [ ] Check logs: `journalctl -u ach_processor -f`
|
|
- [ ] Enable on boot: `sudo systemctl enable ach_processor`
|
|
- [ ] Monitor for first 24 hours
|
|
- [ ] Set up log rotation if needed
|
|
- [ ] Document any custom configurations
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### For Testing (with Mock SFTP)
|
|
|
|
```bash
|
|
# 1. Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# 2. Start mock SFTP
|
|
docker-compose up -d
|
|
mkdir -p sftp_data/HDFC/NACH
|
|
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/
|
|
|
|
# 3. Update .env for testing
|
|
SFTP_HOST=127.0.0.1
|
|
SFTP_PORT=2222
|
|
POLL_INTERVAL_MINUTES=1
|
|
|
|
# 4. Run application
|
|
python main.py
|
|
|
|
# 5. Stop mock SFTP when done
|
|
docker-compose down
|
|
```
|
|
|
|
### For Production
|
|
|
|
```bash
|
|
# 1. Install Oracle Instant Client
|
|
# See SETUP.md for detailed instructions
|
|
|
|
# 2. Create database tables
|
|
# See SETUP.md, Step 3
|
|
|
|
# 3. Create and edit .env
|
|
cp .env.example .env
|
|
# Edit with production credentials
|
|
|
|
# 4. Create systemd service
|
|
# See SETUP.md, Step 7
|
|
|
|
# 5. Start service
|
|
sudo systemctl start ach_processor
|
|
sudo systemctl status ach_processor
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration Summary
|
|
|
|
### Key Environment Variables
|
|
|
|
```
|
|
# Database (required)
|
|
DB_USER=pacs_db
|
|
DB_PASSWORD=pacs_db
|
|
DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com
|
|
DB_PORT=1521
|
|
DB_SERVICE_NAME=IPKSDB
|
|
|
|
# SFTP (required)
|
|
SFTP_HOST=192.168.1.100
|
|
SFTP_PORT=22
|
|
SFTP_USERNAME=ipks
|
|
SFTP_PASSWORD=secure_password
|
|
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS
|
|
|
|
# Processing (optional)
|
|
POLL_INTERVAL_MINUTES=30
|
|
BATCH_SIZE=100
|
|
BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB
|
|
LOG_LEVEL=INFO
|
|
```
|
|
|
|
### Database Schema
|
|
|
|
**ach_api_log** (existing table)
|
|
- narration: VARCHAR2(500)
|
|
- status: VARCHAR2(100)
|
|
- bankcode: VARCHAR2(20)
|
|
- jrnl_id: VARCHAR2(50)
|
|
- tran_date: DATE
|
|
- cbs_acct: VARCHAR2(50)
|
|
- tran_amt: NUMBER(15,2)
|
|
- TXNIND: VARCHAR2(2)
|
|
|
|
**ach_processed_files** (created by app)
|
|
- filename: VARCHAR2(500) UNIQUE NOT NULL
|
|
- bankcode: VARCHAR2(20)
|
|
- file_path: VARCHAR2(1000)
|
|
- processed_at: TIMESTAMP
|
|
- transaction_count: NUMBER
|
|
- status: VARCHAR2(20)
|
|
- error_message: VARCHAR2(2000)
|
|
|
|
---
|
|
|
|
## System Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Scheduler (30 min interval) │
|
|
└──────────────────────────┬──────────────────────────────────┘
|
|
│
|
|
├─────────────┬────────────────┐
|
|
▼ ▼ ▼
|
|
SFTP Bank 1 SFTP Bank 2 SFTP Bank N
|
|
(HDFC/NACH) (ICICI/NACH) (SBI/NACH)
|
|
│ │ │
|
|
└─────────────┼────────────────┘
|
|
▼
|
|
┌─────────────────────────┐
|
|
│ File Monitor │
|
|
│ - Scan directories │
|
|
│ - Check duplicates │
|
|
└────────┬────────────────┘
|
|
▼
|
|
┌─────────────────────────┐
|
|
│ File Processor │
|
|
│ - Download file │
|
|
│ - Parse ACH │
|
|
│ - Map fields │
|
|
└────────┬────────────────┘
|
|
▼
|
|
┌─────────────────────────┐
|
|
│ Data Mapper │
|
|
│ - Convert dates │
|
|
│ - Calculate TXNIND │
|
|
│ - Format amounts │
|
|
└────────┬────────────────┘
|
|
▼
|
|
┌─────────────────────────┐
|
|
│ Repository │
|
|
│ - Batch insert │
|
|
│ - Mark as processed │
|
|
│ - Check duplicates │
|
|
└────────┬────────────────┘
|
|
▼
|
|
┌─────────────────────────┐
|
|
│ Oracle Database │
|
|
│ - ach_api_log │
|
|
│ - ach_processed_files │
|
|
└─────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Processing Flow
|
|
|
|
```
|
|
Start Scheduler (30-min interval)
|
|
│
|
|
├─> Database Connection Test
|
|
│ └─> Create ach_processed_files table if needed
|
|
│
|
|
├─> For Each Bank Code (HDFC, ICICI, SBI, etc.)
|
|
│ │
|
|
│ ├─> SFTP Connect
|
|
│ │
|
|
│ ├─> Scan Directory: /bank_code/NACH/
|
|
│ │ └─> List files: ACH_*.txt
|
|
│ │
|
|
│ ├─> For Each File Found
|
|
│ │ │
|
|
│ │ ├─> Check if Already Processed
|
|
│ │ │ └─> If yes: Skip and log as info
|
|
│ │ │
|
|
│ │ ├─> Download File to Temp Directory
|
|
│ │ │
|
|
│ │ ├─> Parse ACH File
|
|
│ │ │ └─> Extract transactions (178 in sample)
|
|
│ │ │
|
|
│ │ ├─> Map Each Transaction
|
|
│ │ │ ├─> Convert date DD/MM/YY → DATE
|
|
│ │ │ ├─> Calculate TXNIND from amount
|
|
│ │ │ └─> Create TransactionRecord
|
|
│ │ │
|
|
│ │ ├─> Batch Insert to Database (every 100 records)
|
|
│ │ │ ├─> BEGIN TRANSACTION
|
|
│ │ │ ├─> INSERT batch into ach_api_log
|
|
│ │ │ ├─> INSERT into ach_processed_files
|
|
│ │ │ └─> COMMIT (or ROLLBACK on error)
|
|
│ │ │
|
|
│ │ ├─> Mark File as Processed
|
|
│ │ │
|
|
│ │ └─> Clean Up Local File
|
|
│ │
|
|
│ └─> SFTP Disconnect
|
|
│
|
|
├─> Log Processing Summary
|
|
│ └─> Total/Successful/Failed counts
|
|
│
|
|
└─> Sleep 30 Minutes (or configured interval)
|
|
│
|
|
└─> Repeat...
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring
|
|
|
|
### Log Location
|
|
```
|
|
logs/app.log
|
|
```
|
|
|
|
### Key Log Messages
|
|
|
|
| Event | Log Level | Example |
|
|
|-------|-----------|---------|
|
|
| Scheduler started | INFO | "ACH File Processing Scheduler Started" |
|
|
| Database connected | INFO | "Database connection test successful" |
|
|
| File found | INFO | "Found new file: ACH_99944_..." |
|
|
| File skipped | INFO | "File already processed: ACH_99944_..." |
|
|
| Processing started | INFO | "Starting processing: ACH_99944_..." |
|
|
| Processing complete | INFO | "Successfully processed ACH_99944_..." |
|
|
| Processing failed | ERROR | "Error processing ACH_99944_..." |
|
|
| Database error | ERROR | "Error inserting transactions: ..." |
|
|
| SFTP error | ERROR | "Failed to connect to SFTP server" |
|
|
|
|
### Metrics to Monitor
|
|
|
|
1. **File Processing Rate**
|
|
- How many files processed per cycle
|
|
- Success vs. failure rate
|
|
|
|
2. **Transaction Processing**
|
|
- Number of transactions per file
|
|
- Records inserted vs. parsed
|
|
|
|
3. **Processing Time**
|
|
- Time per file
|
|
- Time per cycle (should be << 30 min)
|
|
|
|
4. **Error Rate**
|
|
- Failed files
|
|
- Database errors
|
|
- SFTP errors
|
|
|
|
### Health Checks
|
|
|
|
```bash
|
|
# Check service status
|
|
sudo systemctl status ach_processor
|
|
|
|
# Check recent logs
|
|
journalctl -u ach_processor -n 50
|
|
|
|
# Check database connectivity
|
|
sqlplus pacs_db/pacs_db@...
|
|
|
|
# Check SFTP connectivity
|
|
sftp -P 22 user@host
|
|
|
|
# Check processed file count
|
|
sqlplus -s pacs_db/pacs_db@... <<EOF
|
|
SELECT COUNT(*) FROM ach_processed_files;
|
|
SELECT COUNT(*) FROM ach_api_log;
|
|
EXIT;
|
|
EOF
|
|
```
|
|
|
|
---
|
|
|
|
## Rollback Plan
|
|
|
|
If issues arise in production:
|
|
|
|
1. **Stop the scheduler**
|
|
```bash
|
|
sudo systemctl stop ach_processor
|
|
```
|
|
|
|
2. **Investigate the issue**
|
|
```bash
|
|
journalctl -u ach_processor --since "30 min ago"
|
|
tail -n 1000 logs/app.log | grep ERROR
|
|
```
|
|
|
|
3. **Fix the issue**
|
|
- Update .env configuration
|
|
- Restart services if needed
|
|
- Check database/SFTP connectivity
|
|
|
|
4. **Restart the scheduler**
|
|
```bash
|
|
sudo systemctl start ach_processor
|
|
```
|
|
|
|
5. **Verify processing resumes**
|
|
```bash
|
|
journalctl -u ach_processor -f
|
|
```
|
|
|
|
---
|
|
|
|
## Support & Troubleshooting
|
|
|
|
See **SETUP.md** for detailed troubleshooting guide covering:
|
|
- ImportError: No module named 'cx_Oracle'
|
|
- Database Connection Refused
|
|
- SFTP Connection Refused
|
|
- Application Hangs or Doesn't Process Files
|
|
- Permission Denied errors
|
|
- Performance tuning
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. Follow the Quick Start section above
|
|
2. Complete the Deployment Checklist
|
|
3. Deploy to production
|
|
4. Monitor logs and metrics
|
|
5. Set up additional monitoring/alerting as needed
|
|
6. Consider enhancements (see IMPLEMENTATION.md for ideas)
|
|
|
|
---
|
|
|
|
## Contact & Support
|
|
|
|
For issues:
|
|
1. Check this document
|
|
2. Review SETUP.md troubleshooting
|
|
3. Check application logs with `LOG_LEVEL=DEBUG`
|
|
4. Review IMPLEMENTATION.md for architectural details
|
|
|
|
---
|
|
|
|
**Deployment Date**: [Insert date when deployed]
|
|
**Deployed By**: [Insert name/team]
|
|
**Version**: 1.0
|
|
**Last Updated**: 2026-01-30
|