product
This commit is contained in:
492
SETUP.md
Normal file
492
SETUP.md
Normal file
@@ -0,0 +1,492 @@
|
||||
# ACH File Processing Pipeline - Setup Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.8+
|
||||
- Oracle Database (or access to Oracle instance)
|
||||
- SFTP Server (or Docker for local testing)
|
||||
- Linux/Unix environment (for systemd integration)
|
||||
|
||||
## Step 1: Install Python Dependencies
|
||||
|
||||
The project requires several new packages. Install them using:
|
||||
|
||||
```bash
|
||||
cd /home/asif/projects/ach_ui_dbtl_file_based
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
This will install:
|
||||
- `cx_Oracle==8.3.0` - Oracle database driver
|
||||
- `paramiko==3.4.0` - SFTP client
|
||||
- `schedule==1.2.0` - Job scheduling
|
||||
- `python-decouple==3.8` - Configuration management
|
||||
- `cryptography==41.0.7` - For paramiko SSH
|
||||
- `pytz==2023.3` - Timezone support
|
||||
- Existing packages: `python-dotenv`, `pytest`, `black`, `flake8`
|
||||
|
||||
## Step 2: Oracle Client Setup (Optional)
|
||||
|
||||
The application uses **oracledb**, which includes two modes:
|
||||
|
||||
### Option A: Thin Mode (Recommended - No Installation Needed)
|
||||
|
||||
oracledb Thin mode connects directly to Oracle Database without any Oracle Instant Client:
|
||||
|
||||
```bash
|
||||
# No installation needed - Thin mode works out of the box!
|
||||
python -c "import oracledb; print('oracledb ready')"
|
||||
```
|
||||
|
||||
This is the default mode and requires no additional setup.
|
||||
|
||||
### Option B: Thick Mode (Requires Oracle Instant Client)
|
||||
|
||||
If you prefer Thick mode or have an existing Oracle Instant Client installation:
|
||||
|
||||
**On Linux (Ubuntu/Debian):**
|
||||
|
||||
```bash
|
||||
# Download Oracle Instant Client (version 21.12 or later)
|
||||
cd /tmp
|
||||
wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basic-linux.x64-21.12.0.0.0dbru.zip
|
||||
|
||||
# Unzip and move to system location
|
||||
unzip instantclient-basic-linux.x64-21.12.0.0.0dbru.zip
|
||||
sudo mkdir -p /opt/oracle
|
||||
sudo mv instantclient_21_12 /opt/oracle/
|
||||
|
||||
# Setup library path
|
||||
echo '/opt/oracle/instantclient_21_12' | sudo tee /etc/ld.so.conf.d/oracle.conf
|
||||
sudo ldconfig
|
||||
```
|
||||
|
||||
**On macOS:**
|
||||
|
||||
```bash
|
||||
# Using Homebrew
|
||||
brew install instantclient-basic
|
||||
```
|
||||
|
||||
### Set Environment Variable (Thick Mode Only):
|
||||
|
||||
Add to your shell profile (`~/.bashrc` or `~/.zshrc`):
|
||||
|
||||
```bash
|
||||
export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
Then reload:
|
||||
```bash
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
### Summary:
|
||||
|
||||
| Mode | Installation | Best For |
|
||||
|------|-------------|----------|
|
||||
| **Thin** | None needed ✓ | Default, simplest |
|
||||
| **Thick** | Oracle Instant Client | Legacy apps, specific features |
|
||||
|
||||
## Step 3: Database Schema Setup
|
||||
|
||||
Login to your Oracle database and create the required tables:
|
||||
|
||||
```sql
|
||||
-- Login to database
|
||||
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB
|
||||
|
||||
-- Create ACH transaction log table (if not already exists)
|
||||
CREATE TABLE ach_api_log (
|
||||
id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
narration VARCHAR2(500),
|
||||
status VARCHAR2(100),
|
||||
bankcode VARCHAR2(20),
|
||||
jrnl_id VARCHAR2(50),
|
||||
tran_date DATE,
|
||||
cbs_acct VARCHAR2(50),
|
||||
tran_amt NUMBER(15, 2),
|
||||
TXNIND VARCHAR2(2),
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Create indexes for performance
|
||||
CREATE INDEX idx_ach_jrnl_id ON ach_api_log(jrnl_id);
|
||||
CREATE INDEX idx_ach_bankcode ON ach_api_log(bankcode);
|
||||
|
||||
-- Verify table was created
|
||||
DESC ach_api_log;
|
||||
|
||||
-- Exit
|
||||
EXIT;
|
||||
```
|
||||
|
||||
**Note**: The `ach_processed_files` table will be created automatically by the application on first run.
|
||||
|
||||
## Step 4: Environment Configuration
|
||||
|
||||
### Create .env File:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
### Edit .env for Your Environment:
|
||||
|
||||
```bash
|
||||
# Database Configuration
|
||||
DB_USER=pacs_db
|
||||
DB_PASSWORD=pacs_db
|
||||
DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com
|
||||
DB_PORT=1521
|
||||
DB_SERVICE_NAME=IPKSDB
|
||||
DB_POOL_MIN=2
|
||||
DB_POOL_MAX=10
|
||||
|
||||
# SFTP Configuration (update with your SFTP credentials)
|
||||
SFTP_HOST=192.168.1.100
|
||||
SFTP_PORT=22
|
||||
SFTP_USERNAME=ipks_user
|
||||
SFTP_PASSWORD=your_secure_password
|
||||
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS
|
||||
|
||||
# Processing Configuration
|
||||
POLL_INTERVAL_MINUTES=30
|
||||
BATCH_SIZE=100
|
||||
BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
```
|
||||
|
||||
**For Testing with Mock SFTP**, see Step 5 below.
|
||||
|
||||
## Step 5: Testing with Mock SFTP (Optional)
|
||||
|
||||
If you don't have a real SFTP server, you can use Docker to run a mock SFTP server locally.
|
||||
|
||||
### Requirements:
|
||||
- Docker and Docker Compose installed
|
||||
|
||||
### Setup:
|
||||
|
||||
```bash
|
||||
# Create SFTP directory structure
|
||||
mkdir -p sftp_data/HDFC/NACH
|
||||
mkdir -p sftp_data/ICICI/NACH
|
||||
mkdir -p sftp_data/SBI/NACH
|
||||
mkdir -p sftp_data/AXIS/NACH
|
||||
mkdir -p sftp_data/PNB/NACH
|
||||
|
||||
# Copy sample ACH file to test directory
|
||||
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/
|
||||
# Also copy to other bank directories if needed
|
||||
cp ACH_99944_19012026103217_001.txt sftp_data/ICICI/NACH/
|
||||
|
||||
# Start SFTP server
|
||||
docker-compose up -d
|
||||
|
||||
# Verify it's running
|
||||
docker ps | grep sftp
|
||||
|
||||
# Test SFTP connection
|
||||
sftp -P 2222 ipks@127.0.0.1
|
||||
# When prompted for password, enter: ipks_password
|
||||
# Commands to try:
|
||||
# ls
|
||||
# cd /home/ipks/IPKS_FILES/REPORTS/HDFC/NACH
|
||||
# ls
|
||||
# exit
|
||||
```
|
||||
|
||||
### Update .env for Mock SFTP:
|
||||
|
||||
```bash
|
||||
# For Docker SFTP testing
|
||||
SFTP_HOST=127.0.0.1
|
||||
SFTP_PORT=2222
|
||||
SFTP_USERNAME=ipks
|
||||
SFTP_PASSWORD=ipks_password
|
||||
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS
|
||||
|
||||
# Shorter poll interval for testing
|
||||
POLL_INTERVAL_MINUTES=1
|
||||
```
|
||||
|
||||
## Step 6: Verify Installation
|
||||
|
||||
Before running the application, verify all components are working:
|
||||
|
||||
### Test Database Connection:
|
||||
|
||||
```bash
|
||||
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB
|
||||
|
||||
-- In SQL*Plus:
|
||||
SELECT COUNT(*) FROM ach_api_log;
|
||||
EXIT;
|
||||
```
|
||||
|
||||
### Test SFTP Connection:
|
||||
|
||||
```bash
|
||||
sftp -P 22 your_sftp_user@your_sftp_host
|
||||
# Or for mock Docker SFTP:
|
||||
sftp -P 2222 ipks@127.0.0.1
|
||||
```
|
||||
|
||||
### Test Python Import:
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
python -c "from config import get_config; cfg = get_config(); print('Config OK'); cfg.validate()"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Config OK
|
||||
Configuration validated. Bank codes: HDFC, ICICI, SBI, AXIS, PNB
|
||||
```
|
||||
|
||||
## Step 7: Run the Application
|
||||
|
||||
### Development Mode (Foreground):
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
python main.py
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
2026-01-30 12:00:00 - scheduler - INFO - ================================================================================
|
||||
2026-01-30 12:00:00 - scheduler - INFO - ACH File Processing Scheduler Started
|
||||
2026-01-30 12:00:00 - scheduler - INFO - Poll Interval: 30 minutes
|
||||
2026-01-30 12:00:00 - scheduler - INFO - Bank Codes: HDFC, ICICI, SBI, AXIS, PNB
|
||||
2026-01-30 12:00:00 - scheduler - INFO - ================================================================================
|
||||
2026-01-30 12:00:01 - db.oracle_connector - INFO - Oracle connection pool initialized
|
||||
2026-01-30 12:00:01 - db.oracle_connector - INFO - Database connection test successful
|
||||
2026-01-30 12:00:01 - db.repository - INFO - Created ach_processed_files table
|
||||
2026-01-30 12:00:01 - scheduler - INFO - === Starting processing cycle 1 ===
|
||||
...
|
||||
```
|
||||
|
||||
To stop, press `CTRL+C` for graceful shutdown.
|
||||
|
||||
### Production Mode (Background Service):
|
||||
|
||||
```bash
|
||||
# Create systemd service file
|
||||
sudo nano /etc/systemd/system/ach_processor.service
|
||||
```
|
||||
|
||||
Paste the following content:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=ACH File Processor
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=appuser
|
||||
WorkingDirectory=/opt/ach_processor
|
||||
Environment="PATH=/opt/ach_processor/venv/bin"
|
||||
Environment="LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH"
|
||||
ExecStart=/opt/ach_processor/venv/bin/python main.py
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Then start the service:
|
||||
|
||||
```bash
|
||||
# Reload systemd configuration
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Enable service to start on boot
|
||||
sudo systemctl enable ach_processor
|
||||
|
||||
# Start the service
|
||||
sudo systemctl start ach_processor
|
||||
|
||||
# Check status
|
||||
sudo systemctl status ach_processor
|
||||
|
||||
# View logs
|
||||
journalctl -u ach_processor -f
|
||||
```
|
||||
|
||||
## Step 8: Running Tests
|
||||
|
||||
### Unit Tests:
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
|
||||
# Run all tests
|
||||
pytest tests/ -v
|
||||
|
||||
# Run specific test file
|
||||
pytest tests/test_data_mapper.py -v
|
||||
|
||||
# Run with coverage report
|
||||
pytest tests/ --cov=processors --cov=db --cov=sftp -v
|
||||
```
|
||||
|
||||
### Integration Tests:
|
||||
|
||||
```bash
|
||||
# With mock SFTP running (see Step 5)
|
||||
source venv/bin/activate
|
||||
|
||||
# Create test file
|
||||
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/ACH_99944_01010101010101_001.txt
|
||||
|
||||
# Run application for one cycle
|
||||
python main.py
|
||||
|
||||
# Verify file was processed by checking logs
|
||||
tail -f logs/app.log
|
||||
|
||||
# Verify data in database
|
||||
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB
|
||||
SELECT COUNT(*) FROM ach_api_log;
|
||||
SELECT * FROM ach_processed_files;
|
||||
EXIT;
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
After setup, your project structure should look like:
|
||||
|
||||
```
|
||||
ach_ui_dbtl_file_based/
|
||||
├── venv/ # Virtual environment
|
||||
├── logs/ # Log files (created on first run)
|
||||
├── sftp_data/ # Mock SFTP data (for testing)
|
||||
│ ├── HDFC/NACH/
|
||||
│ ├── ICICI/NACH/
|
||||
│ └── SBI/NACH/
|
||||
├── config.py # Configuration management
|
||||
├── main.py # Application entry point
|
||||
├── scheduler.py # Main scheduler
|
||||
├── ach_parser.py # Existing parser
|
||||
├── logging_config.py # Existing logging
|
||||
├── db/ # Database module
|
||||
├── sftp/ # SFTP module
|
||||
├── processors/ # Processing module
|
||||
├── tests/ # Test files
|
||||
├── requirements.txt # Dependencies
|
||||
├── .env # Configuration (created)
|
||||
├── .env.example # Configuration template
|
||||
├── docker-compose.yml # Mock SFTP config
|
||||
├── SETUP.md # This file
|
||||
├── IMPLEMENTATION.md # Implementation details
|
||||
└── README.md # Original README
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### ImportError: No module named 'cx_Oracle'
|
||||
|
||||
**Solution**: Install Oracle Instant Client (Step 2) and ensure `LD_LIBRARY_PATH` is set.
|
||||
|
||||
```bash
|
||||
# Check if installed
|
||||
python -c "import cx_Oracle; print(cx_Oracle.version)"
|
||||
|
||||
# If error, check LD_LIBRARY_PATH
|
||||
echo $LD_LIBRARY_PATH
|
||||
|
||||
# If not set, add to ~/.bashrc
|
||||
export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
### Database Connection Refused
|
||||
|
||||
**Solution**: Verify database credentials and network connectivity.
|
||||
|
||||
```bash
|
||||
# Test with sqlplus
|
||||
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB
|
||||
|
||||
# If network timeout, check firewall
|
||||
# Database may require security group rules for your IP
|
||||
```
|
||||
|
||||
### SFTP Connection Refused
|
||||
|
||||
**Solution**: Verify SFTP credentials and check if server is running.
|
||||
|
||||
```bash
|
||||
# Test SFTP connection
|
||||
sftp -P 22 your_user@your_host
|
||||
|
||||
# For Docker, ensure container is running
|
||||
docker-compose up -d
|
||||
docker ps | grep sftp
|
||||
```
|
||||
|
||||
### Application Hangs or Doesn't Process Files
|
||||
|
||||
**Solution**: Check logs and verify database/SFTP availability.
|
||||
|
||||
```bash
|
||||
# Watch logs
|
||||
tail -f logs/app.log
|
||||
|
||||
# Enable debug logging
|
||||
LOG_LEVEL=DEBUG in .env
|
||||
```
|
||||
|
||||
### Permission Denied on /opt/oracle
|
||||
|
||||
**Solution**: Check directory permissions.
|
||||
|
||||
```bash
|
||||
# Verify Oracle client is readable
|
||||
ls -la /opt/oracle/instantclient_21_12
|
||||
|
||||
# If needed, adjust permissions
|
||||
sudo chmod -R +r /opt/oracle/instantclient_21_12
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Database
|
||||
- Adjust `DB_POOL_MIN/MAX` for concurrent load
|
||||
- Increase `BATCH_SIZE` if database can handle it
|
||||
- Monitor indexes: `idx_ach_jrnl_id`, `idx_ach_bankcode`
|
||||
|
||||
### Polling
|
||||
- Adjust `POLL_INTERVAL_MINUTES` based on file arrival rate
|
||||
- Default 30 minutes should handle most cases
|
||||
- Lower for high-volume processing
|
||||
|
||||
### Network
|
||||
- Ensure low-latency connection to SFTP and database
|
||||
- Use VPN or direct network path if possible
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Verify all setup steps are complete
|
||||
2. Run tests to ensure everything works
|
||||
3. Deploy to production following Step 7
|
||||
4. Monitor logs regularly
|
||||
5. Set up log rotation (handled by `RotatingFileHandler`)
|
||||
6. Consider adding alerting for failures
|
||||
|
||||
## Support
|
||||
|
||||
For issues:
|
||||
1. Check logs: `tail -f logs/app.log`
|
||||
2. Enable debug: `LOG_LEVEL=DEBUG` in `.env`
|
||||
3. Review error messages and stack traces
|
||||
4. Verify database and SFTP connectivity
|
||||
5. Check this guide for troubleshooting section
|
||||
Reference in New Issue
Block a user