ach_ui_dbtl_file_based/SETUP.md

ACH File Processing Pipeline - Setup Guide

Prerequisites

• Python 3.8+
• Oracle Database (or access to Oracle instance)
• SFTP Server (or Docker for local testing)
• Linux/Unix environment (for systemd integration)

Step 1: Install Python Dependencies

The project requires several new packages. Install them using:

cd /home/asif/projects/ach_ui_dbtl_file_based
source venv/bin/activate
pip install -r requirements.txt

This will install:

• cx_Oracle==8.3.0 - Oracle database driver
• paramiko==3.4.0 - SFTP client
• schedule==1.2.0 - Job scheduling
• python-decouple==3.8 - Configuration management
• cryptography==41.0.7 - For paramiko SSH
• pytz==2023.3 - Timezone support
• Existing packages: python-dotenv, pytest, black, flake8

Step 2: Oracle Client Setup (Optional)

The application uses oracledb, which includes two modes:

Option A: Thin Mode (Recommended - No Installation Needed)

oracledb Thin mode connects directly to Oracle Database without any Oracle Instant Client:

# No installation needed - Thin mode works out of the box!
python -c "import oracledb; print('oracledb ready')"

This is the default mode and requires no additional setup.

Option B: Thick Mode (Requires Oracle Instant Client)

If you prefer Thick mode or have an existing Oracle Instant Client installation:

On Linux (Ubuntu/Debian):

# Download Oracle Instant Client (version 21.12 or later)
cd /tmp
wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basic-linux.x64-21.12.0.0.0dbru.zip

# Unzip and move to system location
unzip instantclient-basic-linux.x64-21.12.0.0.0dbru.zip
sudo mkdir -p /opt/oracle
sudo mv instantclient_21_12 /opt/oracle/

# Setup library path
echo '/opt/oracle/instantclient_21_12' | sudo tee /etc/ld.so.conf.d/oracle.conf
sudo ldconfig

On macOS:

# Using Homebrew
brew install instantclient-basic

Set Environment Variable (Thick Mode Only):

Add to your shell profile (~/.bashrc or ~/.zshrc):

export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH

Then reload:

source ~/.bashrc

Summary:

Mode	Installation	Best For
Thin	None needed ✓	Default, simplest
Thick	Oracle Instant Client	Legacy apps, specific features

Step 3: Database Schema Setup

Login to your Oracle database and create the required tables:

-- Login to database
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB

-- Create ACH transaction log table (if not already exists)
CREATE TABLE ach_api_log (
    id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    narration VARCHAR2(500),
    status VARCHAR2(100),
    bankcode VARCHAR2(20),
    jrnl_id VARCHAR2(50),
    tran_date DATE,
    cbs_acct VARCHAR2(50),
    tran_amt NUMBER(15, 2),
    TXNIND VARCHAR2(2),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create indexes for performance
CREATE INDEX idx_ach_jrnl_id ON ach_api_log(jrnl_id);
CREATE INDEX idx_ach_bankcode ON ach_api_log(bankcode);

-- Verify table was created
DESC ach_api_log;

-- Exit
EXIT;

Note: The ach_processed_files table will be created automatically by the application on first run.

Step 4: Environment Configuration

Create .env File:

cp .env.example .env

Edit .env for Your Environment:

# Database Configuration
DB_USER=pacs_db
DB_PASSWORD=pacs_db
DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com
DB_PORT=1521
DB_SERVICE_NAME=IPKSDB
DB_POOL_MIN=2
DB_POOL_MAX=10

# SFTP Configuration (update with your SFTP credentials)
SFTP_HOST=192.168.1.100
SFTP_PORT=22
SFTP_USERNAME=ipks_user
SFTP_PASSWORD=your_secure_password
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS

# Processing Configuration
POLL_INTERVAL_MINUTES=30
BATCH_SIZE=100
BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB

# Logging
LOG_LEVEL=INFO

For Testing with Mock SFTP, see Step 5 below.

Step 5: Testing with Mock SFTP (Optional)

If you don't have a real SFTP server, you can use Docker to run a mock SFTP server locally.

Requirements:

• Docker and Docker Compose installed

Setup:

# Create SFTP directory structure
mkdir -p sftp_data/HDFC/NACH
mkdir -p sftp_data/ICICI/NACH
mkdir -p sftp_data/SBI/NACH
mkdir -p sftp_data/AXIS/NACH
mkdir -p sftp_data/PNB/NACH

# Copy sample ACH file to test directory
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/
# Also copy to other bank directories if needed
cp ACH_99944_19012026103217_001.txt sftp_data/ICICI/NACH/

# Start SFTP server
docker-compose up -d

# Verify it's running
docker ps | grep sftp

# Test SFTP connection
sftp -P 2222 ipks@127.0.0.1
# When prompted for password, enter: ipks_password
# Commands to try:
# ls
# cd /home/ipks/IPKS_FILES/REPORTS/HDFC/NACH
# ls
# exit

Update .env for Mock SFTP:

# For Docker SFTP testing
SFTP_HOST=127.0.0.1
SFTP_PORT=2222
SFTP_USERNAME=ipks
SFTP_PASSWORD=ipks_password
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS

# Shorter poll interval for testing
POLL_INTERVAL_MINUTES=1

Step 6: Verify Installation

Before running the application, verify all components are working:

Test Database Connection:

sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB

-- In SQL*Plus:
SELECT COUNT(*) FROM ach_api_log;
EXIT;

Test SFTP Connection:

sftp -P 22 your_sftp_user@your_sftp_host
# Or for mock Docker SFTP:
sftp -P 2222 ipks@127.0.0.1

Test Python Import:

source venv/bin/activate
python -c "from config import get_config; cfg = get_config(); print('Config OK'); cfg.validate()"

Expected output:

Config OK
Configuration validated. Bank codes: HDFC, ICICI, SBI, AXIS, PNB

Step 7: Run the Application

Development Mode (Foreground):

source venv/bin/activate
python main.py

Expected output:

2026-01-30 12:00:00 - scheduler - INFO - ================================================================================
2026-01-30 12:00:00 - scheduler - INFO - ACH File Processing Scheduler Started
2026-01-30 12:00:00 - scheduler - INFO - Poll Interval: 30 minutes
2026-01-30 12:00:00 - scheduler - INFO - Bank Codes: HDFC, ICICI, SBI, AXIS, PNB
2026-01-30 12:00:00 - scheduler - INFO - ================================================================================
2026-01-30 12:00:01 - db.oracle_connector - INFO - Oracle connection pool initialized
2026-01-30 12:00:01 - db.oracle_connector - INFO - Database connection test successful
2026-01-30 12:00:01 - db.repository - INFO - Created ach_processed_files table
2026-01-30 12:00:01 - scheduler - INFO - === Starting processing cycle 1 ===
...

To stop, press CTRL+C for graceful shutdown.

Production Mode (Background Service):

# Create systemd service file
sudo nano /etc/systemd/system/ach_processor.service

Paste the following content:

[Unit]
Description=ACH File Processor
After=network.target

[Service]
Type=simple
User=appuser
WorkingDirectory=/opt/ach_processor
Environment="PATH=/opt/ach_processor/venv/bin"
Environment="LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH"
ExecStart=/opt/ach_processor/venv/bin/python main.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Then start the service:

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable service to start on boot
sudo systemctl enable ach_processor

# Start the service
sudo systemctl start ach_processor

# Check status
sudo systemctl status ach_processor

# View logs
journalctl -u ach_processor -f

Step 8: Running Tests

Unit Tests:

source venv/bin/activate

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_data_mapper.py -v

# Run with coverage report
pytest tests/ --cov=processors --cov=db --cov=sftp -v

Integration Tests:

# With mock SFTP running (see Step 5)
source venv/bin/activate

# Create test file
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/ACH_99944_01010101010101_001.txt

# Run application for one cycle
python main.py

# Verify file was processed by checking logs
tail -f logs/app.log

# Verify data in database
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB
SELECT COUNT(*) FROM ach_api_log;
SELECT * FROM ach_processed_files;
EXIT;

Directory Structure

After setup, your project structure should look like:

ach_ui_dbtl_file_based/
├── venv/                      # Virtual environment
├── logs/                       # Log files (created on first run)
├── sftp_data/                 # Mock SFTP data (for testing)
│   ├── HDFC/NACH/
│   ├── ICICI/NACH/
│   └── SBI/NACH/
├── config.py                  # Configuration management
├── main.py                    # Application entry point
├── scheduler.py               # Main scheduler
├── ach_parser.py             # Existing parser
├── logging_config.py         # Existing logging
├── db/                       # Database module
├── sftp/                     # SFTP module
├── processors/               # Processing module
├── tests/                    # Test files
├── requirements.txt          # Dependencies
├── .env                      # Configuration (created)
├── .env.example              # Configuration template
├── docker-compose.yml        # Mock SFTP config
├── SETUP.md                  # This file
├── IMPLEMENTATION.md         # Implementation details
└── README.md                 # Original README

Troubleshooting

ImportError: No module named 'cx_Oracle'

Solution: Install Oracle Instant Client (Step 2) and ensure LD_LIBRARY_PATH is set.

# Check if installed
python -c "import cx_Oracle; print(cx_Oracle.version)"

# If error, check LD_LIBRARY_PATH
echo $LD_LIBRARY_PATH

# If not set, add to ~/.bashrc
export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH
source ~/.bashrc

Database Connection Refused

Solution: Verify database credentials and network connectivity.

# Test with sqlplus
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB

# If network timeout, check firewall
# Database may require security group rules for your IP

SFTP Connection Refused

Solution: Verify SFTP credentials and check if server is running.

# Test SFTP connection
sftp -P 22 your_user@your_host

# For Docker, ensure container is running
docker-compose up -d
docker ps | grep sftp

Application Hangs or Doesn't Process Files

Solution: Check logs and verify database/SFTP availability.

# Watch logs
tail -f logs/app.log

# Enable debug logging
LOG_LEVEL=DEBUG in .env

Permission Denied on /opt/oracle

Solution: Check directory permissions.

# Verify Oracle client is readable
ls -la /opt/oracle/instantclient_21_12

# If needed, adjust permissions
sudo chmod -R +r /opt/oracle/instantclient_21_12

Performance Tuning

Database

• Adjust DB_POOL_MIN/MAX for concurrent load
• Increase BATCH_SIZE if database can handle it
• Monitor indexes: idx_ach_jrnl_id, idx_ach_bankcode

Polling

• Adjust POLL_INTERVAL_MINUTES based on file arrival rate
• Default 30 minutes should handle most cases
• Lower for high-volume processing

Network

• Ensure low-latency connection to SFTP and database
• Use VPN or direct network path if possible

Next Steps

1. Verify all setup steps are complete
2. Run tests to ensure everything works
3. Deploy to production following Step 7
4. Monitor logs regularly
5. Set up log rotation (handled by RotatingFileHandler)
6. Consider adding alerting for failures

Support

For issues:

1. Check logs: tail -f logs/app.log
2. Enable debug: LOG_LEVEL=DEBUG in .env
3. Review error messages and stack traces
4. Verify database and SFTP connectivity
5. Check this guide for troubleshooting section