product
This commit is contained in:
481
DEPLOYMENT.md
Normal file
481
DEPLOYMENT.md
Normal file
@@ -0,0 +1,481 @@
|
||||
# ACH File Processing Pipeline - Deployment Checklist
|
||||
|
||||
## Implementation Complete ✅
|
||||
|
||||
This document summarizes what has been implemented and the deployment checklist.
|
||||
|
||||
---
|
||||
|
||||
## Files Created
|
||||
|
||||
### Core Application Files
|
||||
|
||||
#### Configuration & Entry Point
|
||||
- **config.py** - Configuration management (loads .env variables)
|
||||
- **main.py** - Updated application entry point
|
||||
- **scheduler.py** - Main 30-minute polling scheduler
|
||||
|
||||
#### Database Module (db/)
|
||||
- **db/__init__.py** - Module initialization
|
||||
- **db/oracle_connector.py** - Oracle connection pooling
|
||||
- **db/models.py** - TransactionRecord and ProcessedFile data models
|
||||
- **db/repository.py** - Data access layer (CRUD operations)
|
||||
|
||||
#### SFTP Module (sftp/)
|
||||
- **sftp/__init__.py** - Module initialization
|
||||
- **sftp/sftp_client.py** - SFTP client for file operations
|
||||
- **sftp/file_monitor.py** - File discovery and monitoring
|
||||
|
||||
#### Processing Module (processors/)
|
||||
- **processors/__init__.py** - Module initialization
|
||||
- **processors/data_mapper.py** - Field mapping and transformations
|
||||
- **processors/file_processor.py** - End-to-end file processing
|
||||
|
||||
#### Testing
|
||||
- **tests/__init__.py** - Tests module initialization
|
||||
- **tests/test_data_mapper.py** - Unit tests for data mapper
|
||||
- **tests/test_file_monitor.py** - Unit tests for file monitor
|
||||
|
||||
### Configuration Files
|
||||
- **.env** - Environment configuration (for testing)
|
||||
- **.env.example** - Configuration template
|
||||
- **docker-compose.yml** - Mock SFTP server setup for testing
|
||||
- **requirements.txt** - Updated with all dependencies
|
||||
|
||||
### Documentation
|
||||
- **SETUP.md** - Complete setup and installation guide
|
||||
- **IMPLEMENTATION.md** - Detailed implementation documentation
|
||||
- **DEPLOYMENT.md** - This file
|
||||
|
||||
---
|
||||
|
||||
## Key Features Implemented
|
||||
|
||||
### 1. ACH File Processing Pipeline
|
||||
- ✅ SFTP file monitoring (multi-bank support)
|
||||
- ✅ File parsing using existing ACHParser
|
||||
- ✅ Field mapping to database format
|
||||
- ✅ Batch database insertion (configurable size)
|
||||
- ✅ Duplicate detection by filename
|
||||
- ✅ Error handling with detailed logging
|
||||
- ✅ Graceful shutdown on SIGTERM/SIGINT
|
||||
|
||||
### 2. Database Integration
|
||||
- ✅ Oracle connection pooling (min=2, max=10)
|
||||
- ✅ Transaction safety (commit/rollback)
|
||||
- ✅ Processed file tracking table
|
||||
- ✅ Batch insert to `ach_api_log` table
|
||||
- ✅ Duplicate detection in `ach_processed_files` table
|
||||
- ✅ Error message storage for failed files
|
||||
|
||||
### 3. Field Mapping
|
||||
- ✅ `remarks` → `narration`
|
||||
- ✅ `sys` → `status`
|
||||
- ✅ `jrnl_no` → `jrnl_id`
|
||||
- ✅ `date` (DD/MM/YY) → `tran_date` (DATE)
|
||||
- ✅ `cust_acct` → `cbs_acct`
|
||||
- ✅ `amount` → `tran_amt` (absolute value, Decimal)
|
||||
- ✅ `amount` → `TXNIND` ('CR' if ≥0, 'DR' if <0)
|
||||
|
||||
### 4. Scheduling
|
||||
- ✅ Configurable poll interval (default: 30 minutes)
|
||||
- ✅ Multi-bank file processing
|
||||
- ✅ Graceful shutdown handling
|
||||
- ✅ Processing statistics logging
|
||||
|
||||
### 5. Configuration Management
|
||||
- ✅ Environment variable loading (.env)
|
||||
- ✅ Configuration validation
|
||||
- ✅ Bank codes as comma-separated list
|
||||
- ✅ Flexible polling interval
|
||||
|
||||
### 6. Error Handling
|
||||
- ✅ SFTP connection failures (logged)
|
||||
- ✅ File parsing errors (marked as failed)
|
||||
- ✅ Database transaction errors (rolled back)
|
||||
- ✅ Duplicate files (skipped, logged as info)
|
||||
- ✅ Partial failures (continue processing)
|
||||
|
||||
### 7. Testing
|
||||
- ✅ Unit tests for data mapper
|
||||
- ✅ Unit tests for file monitor
|
||||
- ✅ Integration test structure
|
||||
- ✅ Mock SFTP server setup
|
||||
|
||||
---
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
### Pre-Deployment
|
||||
|
||||
- [ ] Read SETUP.md for complete installation steps
|
||||
- [ ] Install Python dependencies: `pip install -r requirements.txt`
|
||||
- [ ] Install Oracle Instant Client (21.12 or later)
|
||||
- [ ] Set LD_LIBRARY_PATH for Oracle Instant Client
|
||||
- [ ] Create Oracle tables (ach_api_log, ach_processed_files)
|
||||
- [ ] Verify database connectivity with sqlplus
|
||||
- [ ] Verify SFTP connectivity with sftp command
|
||||
- [ ] Copy .env.example to .env
|
||||
- [ ] Update .env with production credentials
|
||||
- [ ] Run tests: `pytest tests/ -v`
|
||||
- [ ] Test manual run: `python main.py` (should complete one cycle)
|
||||
|
||||
### Testing (Development Environment)
|
||||
|
||||
- [ ] Use mock SFTP with Docker (see SETUP.md, Step 5)
|
||||
- [ ] Place test ACH file in SFTP data directory
|
||||
- [ ] Run scheduler for one cycle
|
||||
- [ ] Verify file was downloaded and processed
|
||||
- [ ] Verify records in ach_api_log table
|
||||
- [ ] Verify file marked in ach_processed_files table
|
||||
- [ ] Run same file again (should be skipped)
|
||||
- [ ] Check logs for expected messages
|
||||
- [ ] Test CTRL+C for graceful shutdown
|
||||
|
||||
### Production Deployment
|
||||
|
||||
- [ ] Deploy to production server
|
||||
- [ ] Create systemd service file (see SETUP.md, Step 7)
|
||||
- [ ] Test service: `sudo systemctl start ach_processor`
|
||||
- [ ] Verify service is running: `sudo systemctl status ach_processor`
|
||||
- [ ] Check logs: `journalctl -u ach_processor -f`
|
||||
- [ ] Enable on boot: `sudo systemctl enable ach_processor`
|
||||
- [ ] Monitor for first 24 hours
|
||||
- [ ] Set up log rotation if needed
|
||||
- [ ] Document any custom configurations
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### For Testing (with Mock SFTP)
|
||||
|
||||
```bash
|
||||
# 1. Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 2. Start mock SFTP
|
||||
docker-compose up -d
|
||||
mkdir -p sftp_data/HDFC/NACH
|
||||
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/
|
||||
|
||||
# 3. Update .env for testing
|
||||
SFTP_HOST=127.0.0.1
|
||||
SFTP_PORT=2222
|
||||
POLL_INTERVAL_MINUTES=1
|
||||
|
||||
# 4. Run application
|
||||
python main.py
|
||||
|
||||
# 5. Stop mock SFTP when done
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
### For Production
|
||||
|
||||
```bash
|
||||
# 1. Install Oracle Instant Client
|
||||
# See SETUP.md for detailed instructions
|
||||
|
||||
# 2. Create database tables
|
||||
# See SETUP.md, Step 3
|
||||
|
||||
# 3. Create and edit .env
|
||||
cp .env.example .env
|
||||
# Edit with production credentials
|
||||
|
||||
# 4. Create systemd service
|
||||
# See SETUP.md, Step 7
|
||||
|
||||
# 5. Start service
|
||||
sudo systemctl start ach_processor
|
||||
sudo systemctl status ach_processor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Summary
|
||||
|
||||
### Key Environment Variables
|
||||
|
||||
```
|
||||
# Database (required)
|
||||
DB_USER=pacs_db
|
||||
DB_PASSWORD=pacs_db
|
||||
DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com
|
||||
DB_PORT=1521
|
||||
DB_SERVICE_NAME=IPKSDB
|
||||
|
||||
# SFTP (required)
|
||||
SFTP_HOST=192.168.1.100
|
||||
SFTP_PORT=22
|
||||
SFTP_USERNAME=ipks
|
||||
SFTP_PASSWORD=secure_password
|
||||
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS
|
||||
|
||||
# Processing (optional)
|
||||
POLL_INTERVAL_MINUTES=30
|
||||
BATCH_SIZE=100
|
||||
BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB
|
||||
LOG_LEVEL=INFO
|
||||
```
|
||||
|
||||
### Database Schema
|
||||
|
||||
**ach_api_log** (existing table)
|
||||
- narration: VARCHAR2(500)
|
||||
- status: VARCHAR2(100)
|
||||
- bankcode: VARCHAR2(20)
|
||||
- jrnl_id: VARCHAR2(50)
|
||||
- tran_date: DATE
|
||||
- cbs_acct: VARCHAR2(50)
|
||||
- tran_amt: NUMBER(15,2)
|
||||
- TXNIND: VARCHAR2(2)
|
||||
|
||||
**ach_processed_files** (created by app)
|
||||
- filename: VARCHAR2(500) UNIQUE NOT NULL
|
||||
- bankcode: VARCHAR2(20)
|
||||
- file_path: VARCHAR2(1000)
|
||||
- processed_at: TIMESTAMP
|
||||
- transaction_count: NUMBER
|
||||
- status: VARCHAR2(20)
|
||||
- error_message: VARCHAR2(2000)
|
||||
|
||||
---
|
||||
|
||||
## System Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Scheduler (30 min interval) │
|
||||
└──────────────────────────┬──────────────────────────────────┘
|
||||
│
|
||||
├─────────────┬────────────────┐
|
||||
▼ ▼ ▼
|
||||
SFTP Bank 1 SFTP Bank 2 SFTP Bank N
|
||||
(HDFC/NACH) (ICICI/NACH) (SBI/NACH)
|
||||
│ │ │
|
||||
└─────────────┼────────────────┘
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ File Monitor │
|
||||
│ - Scan directories │
|
||||
│ - Check duplicates │
|
||||
└────────┬────────────────┘
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ File Processor │
|
||||
│ - Download file │
|
||||
│ - Parse ACH │
|
||||
│ - Map fields │
|
||||
└────────┬────────────────┘
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ Data Mapper │
|
||||
│ - Convert dates │
|
||||
│ - Calculate TXNIND │
|
||||
│ - Format amounts │
|
||||
└────────┬────────────────┘
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ Repository │
|
||||
│ - Batch insert │
|
||||
│ - Mark as processed │
|
||||
│ - Check duplicates │
|
||||
└────────┬────────────────┘
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ Oracle Database │
|
||||
│ - ach_api_log │
|
||||
│ - ach_processed_files │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Processing Flow
|
||||
|
||||
```
|
||||
Start Scheduler (30-min interval)
|
||||
│
|
||||
├─> Database Connection Test
|
||||
│ └─> Create ach_processed_files table if needed
|
||||
│
|
||||
├─> For Each Bank Code (HDFC, ICICI, SBI, etc.)
|
||||
│ │
|
||||
│ ├─> SFTP Connect
|
||||
│ │
|
||||
│ ├─> Scan Directory: /bank_code/NACH/
|
||||
│ │ └─> List files: ACH_*.txt
|
||||
│ │
|
||||
│ ├─> For Each File Found
|
||||
│ │ │
|
||||
│ │ ├─> Check if Already Processed
|
||||
│ │ │ └─> If yes: Skip and log as info
|
||||
│ │ │
|
||||
│ │ ├─> Download File to Temp Directory
|
||||
│ │ │
|
||||
│ │ ├─> Parse ACH File
|
||||
│ │ │ └─> Extract transactions (178 in sample)
|
||||
│ │ │
|
||||
│ │ ├─> Map Each Transaction
|
||||
│ │ │ ├─> Convert date DD/MM/YY → DATE
|
||||
│ │ │ ├─> Calculate TXNIND from amount
|
||||
│ │ │ └─> Create TransactionRecord
|
||||
│ │ │
|
||||
│ │ ├─> Batch Insert to Database (every 100 records)
|
||||
│ │ │ ├─> BEGIN TRANSACTION
|
||||
│ │ │ ├─> INSERT batch into ach_api_log
|
||||
│ │ │ ├─> INSERT into ach_processed_files
|
||||
│ │ │ └─> COMMIT (or ROLLBACK on error)
|
||||
│ │ │
|
||||
│ │ ├─> Mark File as Processed
|
||||
│ │ │
|
||||
│ │ └─> Clean Up Local File
|
||||
│ │
|
||||
│ └─> SFTP Disconnect
|
||||
│
|
||||
├─> Log Processing Summary
|
||||
│ └─> Total/Successful/Failed counts
|
||||
│
|
||||
└─> Sleep 30 Minutes (or configured interval)
|
||||
│
|
||||
└─> Repeat...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Log Location
|
||||
```
|
||||
logs/app.log
|
||||
```
|
||||
|
||||
### Key Log Messages
|
||||
|
||||
| Event | Log Level | Example |
|
||||
|-------|-----------|---------|
|
||||
| Scheduler started | INFO | "ACH File Processing Scheduler Started" |
|
||||
| Database connected | INFO | "Database connection test successful" |
|
||||
| File found | INFO | "Found new file: ACH_99944_..." |
|
||||
| File skipped | INFO | "File already processed: ACH_99944_..." |
|
||||
| Processing started | INFO | "Starting processing: ACH_99944_..." |
|
||||
| Processing complete | INFO | "Successfully processed ACH_99944_..." |
|
||||
| Processing failed | ERROR | "Error processing ACH_99944_..." |
|
||||
| Database error | ERROR | "Error inserting transactions: ..." |
|
||||
| SFTP error | ERROR | "Failed to connect to SFTP server" |
|
||||
|
||||
### Metrics to Monitor
|
||||
|
||||
1. **File Processing Rate**
|
||||
- How many files processed per cycle
|
||||
- Success vs. failure rate
|
||||
|
||||
2. **Transaction Processing**
|
||||
- Number of transactions per file
|
||||
- Records inserted vs. parsed
|
||||
|
||||
3. **Processing Time**
|
||||
- Time per file
|
||||
- Time per cycle (should be << 30 min)
|
||||
|
||||
4. **Error Rate**
|
||||
- Failed files
|
||||
- Database errors
|
||||
- SFTP errors
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl status ach_processor
|
||||
|
||||
# Check recent logs
|
||||
journalctl -u ach_processor -n 50
|
||||
|
||||
# Check database connectivity
|
||||
sqlplus pacs_db/pacs_db@...
|
||||
|
||||
# Check SFTP connectivity
|
||||
sftp -P 22 user@host
|
||||
|
||||
# Check processed file count
|
||||
sqlplus -s pacs_db/pacs_db@... <<EOF
|
||||
SELECT COUNT(*) FROM ach_processed_files;
|
||||
SELECT COUNT(*) FROM ach_api_log;
|
||||
EXIT;
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise in production:
|
||||
|
||||
1. **Stop the scheduler**
|
||||
```bash
|
||||
sudo systemctl stop ach_processor
|
||||
```
|
||||
|
||||
2. **Investigate the issue**
|
||||
```bash
|
||||
journalctl -u ach_processor --since "30 min ago"
|
||||
tail -n 1000 logs/app.log | grep ERROR
|
||||
```
|
||||
|
||||
3. **Fix the issue**
|
||||
- Update .env configuration
|
||||
- Restart services if needed
|
||||
- Check database/SFTP connectivity
|
||||
|
||||
4. **Restart the scheduler**
|
||||
```bash
|
||||
sudo systemctl start ach_processor
|
||||
```
|
||||
|
||||
5. **Verify processing resumes**
|
||||
```bash
|
||||
journalctl -u ach_processor -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support & Troubleshooting
|
||||
|
||||
See **SETUP.md** for detailed troubleshooting guide covering:
|
||||
- ImportError: No module named 'cx_Oracle'
|
||||
- Database Connection Refused
|
||||
- SFTP Connection Refused
|
||||
- Application Hangs or Doesn't Process Files
|
||||
- Permission Denied errors
|
||||
- Performance tuning
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Follow the Quick Start section above
|
||||
2. Complete the Deployment Checklist
|
||||
3. Deploy to production
|
||||
4. Monitor logs and metrics
|
||||
5. Set up additional monitoring/alerting as needed
|
||||
6. Consider enhancements (see IMPLEMENTATION.md for ideas)
|
||||
|
||||
---
|
||||
|
||||
## Contact & Support
|
||||
|
||||
For issues:
|
||||
1. Check this document
|
||||
2. Review SETUP.md troubleshooting
|
||||
3. Check application logs with `LOG_LEVEL=DEBUG`
|
||||
4. Review IMPLEMENTATION.md for architectural details
|
||||
|
||||
---
|
||||
|
||||
**Deployment Date**: [Insert date when deployed]
|
||||
**Deployed By**: [Insert name/team]
|
||||
**Version**: 1.0
|
||||
**Last Updated**: 2026-01-30
|
||||
Reference in New Issue
Block a user