ach_ui_dbtl_file_based/SETUP.md

# ACH File Processing Pipeline - Setup Guide

## Prerequisites

- Python 3.8+
- Oracle Database (or access to Oracle instance)
- SFTP Server (or Docker for local testing)
- Linux/Unix environment (for systemd integration)

## Step 1: Install Python Dependencies

The project requires several new packages. Install them using:

```bash
cd /home/asif/projects/ach_ui_dbtl_file_based
source venv/bin/activate
pip install -r requirements.txt
```

This will install:
- `cx_Oracle==8.3.0` - Oracle database driver
- `paramiko==3.4.0` - SFTP client
- `schedule==1.2.0` - Job scheduling
- `python-decouple==3.8` - Configuration management
- `cryptography==41.0.7` - For paramiko SSH
- `pytz==2023.3` - Timezone support
- Existing packages: `python-dotenv`, `pytest`, `black`, `flake8`

## Step 2: Oracle Client Setup (Optional)

The application uses **oracledb**, which includes two modes:

### Option A: Thin Mode (Recommended - No Installation Needed)

oracledb Thin mode connects directly to Oracle Database without any Oracle Instant Client:

```bash
# No installation needed - Thin mode works out of the box!
python -c "import oracledb; print('oracledb ready')"
```

This is the default mode and requires no additional setup.

### Option B: Thick Mode (Requires Oracle Instant Client)

If you prefer Thick mode or have an existing Oracle Instant Client installation:

**On Linux (Ubuntu/Debian):**

```bash
# Download Oracle Instant Client (version 21.12 or later)
cd /tmp
wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basic-linux.x64-21.12.0.0.0dbru.zip

# Unzip and move to system location
unzip instantclient-basic-linux.x64-21.12.0.0.0dbru.zip
sudo mkdir -p /opt/oracle
sudo mv instantclient_21_12 /opt/oracle/

# Setup library path
echo '/opt/oracle/instantclient_21_12' | sudo tee /etc/ld.so.conf.d/oracle.conf
sudo ldconfig
```

**On macOS:**

```bash
# Using Homebrew
brew install instantclient-basic
```

### Set Environment Variable (Thick Mode Only):

Add to your shell profile (`~/.bashrc` or `~/.zshrc`):

```bash
export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH
```

Then reload:
```bash
source ~/.bashrc
```

### Summary:

| Mode | Installation | Best For |
|------|-------------|----------|
| **Thin** | None needed ✓ | Default, simplest |
| **Thick** | Oracle Instant Client | Legacy apps, specific features |

## Step 3: Database Schema Setup

Login to your Oracle database and create the required tables:

```sql
-- Login to database
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB

-- Create ACH transaction log table (if not already exists)
CREATE TABLE ach_api_log (
    id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    narration VARCHAR2(500),
    status VARCHAR2(100),
    bankcode VARCHAR2(20),
    jrnl_id VARCHAR2(50),
    tran_date DATE,
    cbs_acct VARCHAR2(50),
    tran_amt NUMBER(15, 2),
    TXNIND VARCHAR2(2),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create indexes for performance
CREATE INDEX idx_ach_jrnl_id ON ach_api_log(jrnl_id);
CREATE INDEX idx_ach_bankcode ON ach_api_log(bankcode);

-- Verify table was created
DESC ach_api_log;

-- Exit
EXIT;
```

**Note**: The `ach_processed_files` table will be created automatically by the application on first run.

## Step 4: Environment Configuration

### Create .env File:

```bash
cp .env.example .env
```

### Edit .env for Your Environment:

```bash
# Database Configuration
DB_USER=pacs_db
DB_PASSWORD=pacs_db
DB_HOST=testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com
DB_PORT=1521
DB_SERVICE_NAME=IPKSDB
DB_POOL_MIN=2
DB_POOL_MAX=10

# SFTP Configuration (update with your SFTP credentials)
SFTP_HOST=192.168.1.100
SFTP_PORT=22
SFTP_USERNAME=ipks_user
SFTP_PASSWORD=your_secure_password
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS

# Processing Configuration
POLL_INTERVAL_MINUTES=30
BATCH_SIZE=100
BANK_CODES=HDFC,ICICI,SBI,AXIS,PNB

# Logging
LOG_LEVEL=INFO
```

**For Testing with Mock SFTP**, see Step 5 below.

## Step 5: Testing with Mock SFTP (Optional)

If you don't have a real SFTP server, you can use Docker to run a mock SFTP server locally.

### Requirements:
- Docker and Docker Compose installed

### Setup:

```bash
# Create SFTP directory structure
mkdir -p sftp_data/HDFC/NACH
mkdir -p sftp_data/ICICI/NACH
mkdir -p sftp_data/SBI/NACH
mkdir -p sftp_data/AXIS/NACH
mkdir -p sftp_data/PNB/NACH

# Copy sample ACH file to test directory
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/
# Also copy to other bank directories if needed
cp ACH_99944_19012026103217_001.txt sftp_data/ICICI/NACH/

# Start SFTP server
docker-compose up -d

# Verify it's running
docker ps | grep sftp

# Test SFTP connection
sftp -P 2222 ipks@127.0.0.1
# When prompted for password, enter: ipks_password
# Commands to try:
# ls
# cd /home/ipks/IPKS_FILES/REPORTS/HDFC/NACH
# ls
# exit
```

### Update .env for Mock SFTP:

```bash
# For Docker SFTP testing
SFTP_HOST=127.0.0.1
SFTP_PORT=2222
SFTP_USERNAME=ipks
SFTP_PASSWORD=ipks_password
SFTP_BASE_PATH=/home/ipks/IPKS_FILES/REPORTS

# Shorter poll interval for testing
POLL_INTERVAL_MINUTES=1
```

## Step 6: Verify Installation

Before running the application, verify all components are working:

### Test Database Connection:

```bash
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB

-- In SQL*Plus:
SELECT COUNT(*) FROM ach_api_log;
EXIT;
```

### Test SFTP Connection:

```bash
sftp -P 22 your_sftp_user@your_sftp_host
# Or for mock Docker SFTP:
sftp -P 2222 ipks@127.0.0.1
```

### Test Python Import:

```bash
source venv/bin/activate
python -c "from config import get_config; cfg = get_config(); print('Config OK'); cfg.validate()"
```

Expected output:
```
Config OK
Configuration validated. Bank codes: HDFC, ICICI, SBI, AXIS, PNB
```

## Step 7: Run the Application

### Development Mode (Foreground):

```bash
source venv/bin/activate
python main.py
```

Expected output:
```
2026-01-30 12:00:00 - scheduler - INFO - ================================================================================
2026-01-30 12:00:00 - scheduler - INFO - ACH File Processing Scheduler Started
2026-01-30 12:00:00 - scheduler - INFO - Poll Interval: 30 minutes
2026-01-30 12:00:00 - scheduler - INFO - Bank Codes: HDFC, ICICI, SBI, AXIS, PNB
2026-01-30 12:00:00 - scheduler - INFO - ================================================================================
2026-01-30 12:00:01 - db.oracle_connector - INFO - Oracle connection pool initialized
2026-01-30 12:00:01 - db.oracle_connector - INFO - Database connection test successful
2026-01-30 12:00:01 - db.repository - INFO - Created ach_processed_files table
2026-01-30 12:00:01 - scheduler - INFO - === Starting processing cycle 1 ===
...
```

To stop, press `CTRL+C` for graceful shutdown.

### Production Mode (Background Service):

```bash
# Create systemd service file
sudo nano /etc/systemd/system/ach_processor.service
```

Paste the following content:

```ini
[Unit]
Description=ACH File Processor
After=network.target

[Service]
Type=simple
User=appuser
WorkingDirectory=/opt/ach_processor
Environment="PATH=/opt/ach_processor/venv/bin"
Environment="LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH"
ExecStart=/opt/ach_processor/venv/bin/python main.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

Then start the service:

```bash
# Reload systemd configuration
sudo systemctl daemon-reload

# Enable service to start on boot
sudo systemctl enable ach_processor

# Start the service
sudo systemctl start ach_processor

# Check status
sudo systemctl status ach_processor

# View logs
journalctl -u ach_processor -f
```

## Step 8: Running Tests

### Unit Tests:

```bash
source venv/bin/activate

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_data_mapper.py -v

# Run with coverage report
pytest tests/ --cov=processors --cov=db --cov=sftp -v
```

### Integration Tests:

```bash
# With mock SFTP running (see Step 5)
source venv/bin/activate

# Create test file
cp ACH_99944_19012026103217_001.txt sftp_data/HDFC/NACH/ACH_99944_01010101010101_001.txt

# Run application for one cycle
python main.py

# Verify file was processed by checking logs
tail -f logs/app.log

# Verify data in database
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB
SELECT COUNT(*) FROM ach_api_log;
SELECT * FROM ach_processed_files;
EXIT;
```

## Directory Structure

After setup, your project structure should look like:

```
ach_ui_dbtl_file_based/
├── venv/                      # Virtual environment
├── logs/                       # Log files (created on first run)
├── sftp_data/                 # Mock SFTP data (for testing)
│   ├── HDFC/NACH/
│   ├── ICICI/NACH/
│   └── SBI/NACH/
├── config.py                  # Configuration management
├── main.py                    # Application entry point
├── scheduler.py               # Main scheduler
├── ach_parser.py             # Existing parser
├── logging_config.py         # Existing logging
├── db/                       # Database module
├── sftp/                     # SFTP module
├── processors/               # Processing module
├── tests/                    # Test files
├── requirements.txt          # Dependencies
├── .env                      # Configuration (created)
├── .env.example              # Configuration template
├── docker-compose.yml        # Mock SFTP config
├── SETUP.md                  # This file
├── IMPLEMENTATION.md         # Implementation details
└── README.md                 # Original README
```

## Troubleshooting

### ImportError: No module named 'cx_Oracle'

**Solution**: Install Oracle Instant Client (Step 2) and ensure `LD_LIBRARY_PATH` is set.

```bash
# Check if installed
python -c "import cx_Oracle; print(cx_Oracle.version)"

# If error, check LD_LIBRARY_PATH
echo $LD_LIBRARY_PATH

# If not set, add to ~/.bashrc
export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12:$LD_LIBRARY_PATH
source ~/.bashrc
```

### Database Connection Refused

**Solution**: Verify database credentials and network connectivity.

```bash
# Test with sqlplus
sqlplus pacs_db/pacs_db@testipksdb.c7q7defafeea.ap-south-1.rds.amazonaws.com:1521/IPKSDB

# If network timeout, check firewall
# Database may require security group rules for your IP
```

### SFTP Connection Refused

**Solution**: Verify SFTP credentials and check if server is running.

```bash
# Test SFTP connection
sftp -P 22 your_user@your_host

# For Docker, ensure container is running
docker-compose up -d
docker ps | grep sftp
```

### Application Hangs or Doesn't Process Files

**Solution**: Check logs and verify database/SFTP availability.

```bash
# Watch logs
tail -f logs/app.log

# Enable debug logging
LOG_LEVEL=DEBUG in .env
```

### Permission Denied on /opt/oracle

**Solution**: Check directory permissions.

```bash
# Verify Oracle client is readable
ls -la /opt/oracle/instantclient_21_12

# If needed, adjust permissions
sudo chmod -R +r /opt/oracle/instantclient_21_12
```

## Performance Tuning

### Database
- Adjust `DB_POOL_MIN/MAX` for concurrent load
- Increase `BATCH_SIZE` if database can handle it
- Monitor indexes: `idx_ach_jrnl_id`, `idx_ach_bankcode`

### Polling
- Adjust `POLL_INTERVAL_MINUTES` based on file arrival rate
- Default 30 minutes should handle most cases
- Lower for high-volume processing

### Network
- Ensure low-latency connection to SFTP and database
- Use VPN or direct network path if possible

## Next Steps

1. Verify all setup steps are complete
2. Run tests to ensure everything works
3. Deploy to production following Step 7
4. Monitor logs regularly
5. Set up log rotation (handled by `RotatingFileHandler`)
6. Consider adding alerting for failures

## Support

For issues:
1. Check logs: `tail -f logs/app.log`
2. Enable debug: `LOG_LEVEL=DEBUG` in `.env`
3. Review error messages and stack traces
4. Verify database and SFTP connectivity
5. Check this guide for troubleshooting section