270 lines
7.1 KiB
Markdown
270 lines
7.1 KiB
Markdown
# FargoImport Project - Complete Module Overview
|
|
|
|
## Project Structure
|
|
|
|
This project provides utilities for extracting Wells Fargo transaction data and integrating it with YNAB (You Need A Budget) for financial reconciliation.
|
|
|
|
## Core Modules
|
|
|
|
### 1. **wf_debit_extractor.py** - Wells Fargo Data Extraction
|
|
Extracts proprietary Wells Fargo JSON payloads from HAR-style JSON files.
|
|
|
|
**Classes:**
|
|
- `WellsFargoPayloadExtractor` - Main extraction engine
|
|
- `extract_from_file(path)` - Load and parse HAR JSON file
|
|
- `extract_from_data(har_data)` - Parse raw HAR data
|
|
- `get_objects()` - Access parsed objects
|
|
- Properties: `objects`, `errors`
|
|
|
|
**Functions:**
|
|
- `extract_wf_objects(har_data)` - Backward-compatible functional API
|
|
|
|
**Usage:**
|
|
```python
|
|
from wf_debit_extractor import WellsFargoPayloadExtractor
|
|
|
|
extractor = WellsFargoPayloadExtractor()
|
|
payloads = extractor.extract_from_file("posted.dat")
|
|
print(len(payloads)) # Number of extracted payloads
|
|
```
|
|
|
|
---
|
|
|
|
### 2. **transaction_processor.py** - Wells Fargo Transaction Processing
|
|
Transforms raw Wells Fargo payloads into structured transaction data.
|
|
|
|
**Classes:**
|
|
- `TransactionProcessor` - Transaction extraction and transformation
|
|
- `extract_transactions(extracted_objects)` - Extract from payloads
|
|
|
|
**Data Structure:**
|
|
- `ProcessedTransaction` TypedDict with fields:
|
|
- `id` - Transaction ID
|
|
- `transaction_amount_cents` - Amount as integer cents
|
|
- `transaction_date_timestamp` - Milliseconds since epoch
|
|
- `transaction_date_datetime` - Python datetime object
|
|
- `post_date_timestamp` - Milliseconds since epoch
|
|
- `post_date_datetime` - Python datetime object
|
|
- `transaction_description` - Transaction description
|
|
|
|
**Usage:**
|
|
```python
|
|
from transaction_processor import TransactionProcessor
|
|
from wf_debit_extractor import WellsFargoPayloadExtractor
|
|
|
|
extractor = WellsFargoPayloadExtractor()
|
|
payloads = extractor.extract_from_file("posted.dat")
|
|
|
|
processor = TransactionProcessor()
|
|
transactions = processor.extract_transactions(payloads)
|
|
```
|
|
|
|
---
|
|
|
|
### 3. **ynab_client.py** - YNAB Integration
|
|
Authenticates with YNAB API and fetches transactions using official SDK.
|
|
|
|
**Classes:**
|
|
- `YNABClient` - YNAB API wrapper
|
|
- `__init__(config_path)` - Initialize from YAML config
|
|
- `get_transactions_for_days(days)` - Fetch transactions for past N days
|
|
- `get_account_balance(account_id)` - Get account balance
|
|
|
|
**Data Structure:**
|
|
- `YNABTransaction` TypedDict with fields:
|
|
- `id` - Transaction ID
|
|
- `date` - Date (YYYY-MM-DD format)
|
|
- `payee_name` - Payee name
|
|
- `category_name` - Category name
|
|
- `amount_milliCurrency` - Amount in milliCurrency units
|
|
- `memo` - Transaction memo
|
|
- `account_id` - YNAB account ID
|
|
- `account_name` - Account name
|
|
- `cleared` - Cleared status
|
|
|
|
**Configuration (ynab.yml):**
|
|
```yaml
|
|
ynab:
|
|
- token: "your_api_token"
|
|
plan_id: "your_budget_id"
|
|
accounts:
|
|
- name: "Credit Card"
|
|
id: "account_id"
|
|
```
|
|
|
|
**Usage:**
|
|
```python
|
|
from ynab_client import YNABClient
|
|
|
|
client = YNABClient("ynab.yml")
|
|
transactions = client.get_transactions_for_days(days=30)
|
|
balance = client.get_account_balance("account_id")
|
|
```
|
|
|
|
---
|
|
|
|
### 4. **reconcile.py** - Reconciliation Integration
|
|
Combines Wells Fargo and YNAB transactions for comparison and reporting.
|
|
|
|
**Functions:**
|
|
- `reconcile_transactions()` - Fetch both data sources and create report
|
|
- `main()` - CLI interface with summary and JSON output
|
|
|
|
**Data Structure:**
|
|
- `ReconciliationReport` TypedDict with:
|
|
- `timestamp` - Report timestamp
|
|
- `wellsfargo_transactions` - List of WF transactions
|
|
- `ynab_transactions` - List of YNAB transactions
|
|
- `wellsfargo_total` - Total WF amount
|
|
- `ynab_total` - Total YNAB amount
|
|
- `transaction_count_wf` - Count of WF transactions
|
|
- `transaction_count_ynab` - Count of YNAB transactions
|
|
|
|
**Usage:**
|
|
```bash
|
|
python reconcile.py
|
|
```
|
|
|
|
Output: `reconciliation_report.json` with full comparison
|
|
|
|
---
|
|
|
|
## Example Scripts
|
|
|
|
### **example_usage.py** - Wells Fargo Transaction Example
|
|
Demonstrates extraction and processing of Wells Fargo data.
|
|
|
|
**Output:**
|
|
- Console display of first 5 transactions
|
|
- Saves all transactions to `processed_transactions.json`
|
|
|
|
**Run:**
|
|
```bash
|
|
python example_usage.py
|
|
```
|
|
|
|
### **ynab_example.py** - YNAB Transaction Example
|
|
Demonstrates fetching and displaying YNAB transactions.
|
|
|
|
**Output:**
|
|
- Console display of first 10 transactions
|
|
- Saves all transactions to `ynab_transactions.json`
|
|
|
|
**Run:**
|
|
```bash
|
|
python ynab_example.py
|
|
```
|
|
|
|
---
|
|
|
|
## CLI Interface
|
|
|
|
### **main.py** - Command-line Entry Point
|
|
Thin wrapper around extraction functionality.
|
|
|
|
**Arguments:**
|
|
- `--input` - Input HAR file (default: posted.dat)
|
|
- `--output` - Output JSON file (optional)
|
|
- `--show-errors` - Display parse errors
|
|
|
|
**Examples:**
|
|
```bash
|
|
# Extract to console
|
|
python main.py --input posted.dat
|
|
|
|
# Extract to file
|
|
python main.py --input posted.dat --output extracted.json
|
|
|
|
# Show parsing errors
|
|
python main.py --input posted.dat --show-errors
|
|
```
|
|
|
|
---
|
|
|
|
## Data Flow
|
|
|
|
```
|
|
posted.dat (HAR JSON)
|
|
↓
|
|
wf_debit_extractor.py (Extract proprietary payloads)
|
|
↓
|
|
Extracted payload objects
|
|
↓
|
|
transaction_processor.py (Transform to structured format)
|
|
↓
|
|
Processed transaction list
|
|
↓
|
|
├─→ example_usage.py (Save to processed_transactions.json)
|
|
└─→ reconcile.py (Compare with YNAB)
|
|
↓
|
|
ynab_client.py (Fetch from YNAB API)
|
|
↓
|
|
reconciliation_report.json
|
|
```
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
- **ynab** (v2.1.0) - Official YNAB Python SDK
|
|
- **pyyaml** (v6.0.3) - YAML configuration parsing
|
|
|
|
**Install:**
|
|
```bash
|
|
pip install ynab pyyaml
|
|
```
|
|
|
|
---
|
|
|
|
## File Manifest
|
|
|
|
| File | Purpose | Lines |
|
|
|------|---------|-------|
|
|
| `wf_debit_extractor.py` | Core extraction logic | 91 |
|
|
| `transaction_processor.py` | Transaction transformation | 70 |
|
|
| `ynab_client.py` | YNAB API integration | 150 |
|
|
| `reconcile.py` | Reconciliation orchestration | 117 |
|
|
| `main.py` | CLI wrapper | ~40 |
|
|
| `example_usage.py` | WF usage example | ~60 |
|
|
| `ynab_example.py` | YNAB usage example | ~60 |
|
|
| `ynab.yml` | YNAB configuration | 6 |
|
|
| `README.md` | Project documentation | ~100 |
|
|
|
|
---
|
|
|
|
## Key Features
|
|
|
|
✓ Extracts Wells Fargo proprietary JSON from HAR files
|
|
✓ Transforms transactions with amount normalization
|
|
✓ Converts millisecond timestamps to Python datetime objects
|
|
✓ Integrates with YNAB API for budget tracking
|
|
✓ Supports date range filtering (past X days)
|
|
✓ Generates reconciliation reports
|
|
✓ Error handling and reporting
|
|
✓ Type-safe with TypedDict definitions
|
|
✓ Configuration via YAML
|
|
✓ Modular, reusable architecture
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
- ✅ All modules created and tested
|
|
- ✅ No static errors
|
|
- ✅ Dependencies installed
|
|
- ✅ CLI functional
|
|
- ✅ Examples working
|
|
- ✅ Backward compatible
|
|
- ⚠️ YNAB integration requires valid API token
|
|
|
|
---
|
|
|
|
## Next Steps (Optional)
|
|
|
|
1. **Database Integration** - Store transactions in SQLite/PostgreSQL
|
|
2. **Matching Algorithm** - Auto-match WF↔YNAB transactions by amount/date
|
|
3. **Web Dashboard** - Flask/FastAPI interface for viewing reports
|
|
4. **Scheduling** - Celery/APScheduler for periodic reconciliation
|
|
5. **Export Formats** - CSV, Excel, PDF report generation
|
|
|