first commit
This commit is contained in:
269
PROJECT_SUMMARY.md
Normal file
269
PROJECT_SUMMARY.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# FargoImport Project - Complete Module Overview
|
||||
|
||||
## Project Structure
|
||||
|
||||
This project provides utilities for extracting Wells Fargo transaction data and integrating it with YNAB (You Need A Budget) for financial reconciliation.
|
||||
|
||||
## Core Modules
|
||||
|
||||
### 1. **wf_debit_extractor.py** - Wells Fargo Data Extraction
|
||||
Extracts proprietary Wells Fargo JSON payloads from HAR-style JSON files.
|
||||
|
||||
**Classes:**
|
||||
- `WellsFargoPayloadExtractor` - Main extraction engine
|
||||
- `extract_from_file(path)` - Load and parse HAR JSON file
|
||||
- `extract_from_data(har_data)` - Parse raw HAR data
|
||||
- `get_objects()` - Access parsed objects
|
||||
- Properties: `objects`, `errors`
|
||||
|
||||
**Functions:**
|
||||
- `extract_wf_objects(har_data)` - Backward-compatible functional API
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
from wf_debit_extractor import WellsFargoPayloadExtractor
|
||||
|
||||
extractor = WellsFargoPayloadExtractor()
|
||||
payloads = extractor.extract_from_file("posted.dat")
|
||||
print(len(payloads)) # Number of extracted payloads
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. **transaction_processor.py** - Wells Fargo Transaction Processing
|
||||
Transforms raw Wells Fargo payloads into structured transaction data.
|
||||
|
||||
**Classes:**
|
||||
- `TransactionProcessor` - Transaction extraction and transformation
|
||||
- `extract_transactions(extracted_objects)` - Extract from payloads
|
||||
|
||||
**Data Structure:**
|
||||
- `ProcessedTransaction` TypedDict with fields:
|
||||
- `id` - Transaction ID
|
||||
- `transaction_amount_cents` - Amount as integer cents
|
||||
- `transaction_date_timestamp` - Milliseconds since epoch
|
||||
- `transaction_date_datetime` - Python datetime object
|
||||
- `post_date_timestamp` - Milliseconds since epoch
|
||||
- `post_date_datetime` - Python datetime object
|
||||
- `transaction_description` - Transaction description
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
from transaction_processor import TransactionProcessor
|
||||
from wf_debit_extractor import WellsFargoPayloadExtractor
|
||||
|
||||
extractor = WellsFargoPayloadExtractor()
|
||||
payloads = extractor.extract_from_file("posted.dat")
|
||||
|
||||
processor = TransactionProcessor()
|
||||
transactions = processor.extract_transactions(payloads)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. **ynab_client.py** - YNAB Integration
|
||||
Authenticates with YNAB API and fetches transactions using official SDK.
|
||||
|
||||
**Classes:**
|
||||
- `YNABClient` - YNAB API wrapper
|
||||
- `__init__(config_path)` - Initialize from YAML config
|
||||
- `get_transactions_for_days(days)` - Fetch transactions for past N days
|
||||
- `get_account_balance(account_id)` - Get account balance
|
||||
|
||||
**Data Structure:**
|
||||
- `YNABTransaction` TypedDict with fields:
|
||||
- `id` - Transaction ID
|
||||
- `date` - Date (YYYY-MM-DD format)
|
||||
- `payee_name` - Payee name
|
||||
- `category_name` - Category name
|
||||
- `amount_milliCurrency` - Amount in milliCurrency units
|
||||
- `memo` - Transaction memo
|
||||
- `account_id` - YNAB account ID
|
||||
- `account_name` - Account name
|
||||
- `cleared` - Cleared status
|
||||
|
||||
**Configuration (ynab.yml):**
|
||||
```yaml
|
||||
ynab:
|
||||
- token: "your_api_token"
|
||||
plan_id: "your_budget_id"
|
||||
accounts:
|
||||
- name: "Credit Card"
|
||||
id: "account_id"
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
from ynab_client import YNABClient
|
||||
|
||||
client = YNABClient("ynab.yml")
|
||||
transactions = client.get_transactions_for_days(days=30)
|
||||
balance = client.get_account_balance("account_id")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. **reconcile.py** - Reconciliation Integration
|
||||
Combines Wells Fargo and YNAB transactions for comparison and reporting.
|
||||
|
||||
**Functions:**
|
||||
- `reconcile_transactions()` - Fetch both data sources and create report
|
||||
- `main()` - CLI interface with summary and JSON output
|
||||
|
||||
**Data Structure:**
|
||||
- `ReconciliationReport` TypedDict with:
|
||||
- `timestamp` - Report timestamp
|
||||
- `wellsfargo_transactions` - List of WF transactions
|
||||
- `ynab_transactions` - List of YNAB transactions
|
||||
- `wellsfargo_total` - Total WF amount
|
||||
- `ynab_total` - Total YNAB amount
|
||||
- `transaction_count_wf` - Count of WF transactions
|
||||
- `transaction_count_ynab` - Count of YNAB transactions
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python reconcile.py
|
||||
```
|
||||
|
||||
Output: `reconciliation_report.json` with full comparison
|
||||
|
||||
---
|
||||
|
||||
## Example Scripts
|
||||
|
||||
### **example_usage.py** - Wells Fargo Transaction Example
|
||||
Demonstrates extraction and processing of Wells Fargo data.
|
||||
|
||||
**Output:**
|
||||
- Console display of first 5 transactions
|
||||
- Saves all transactions to `processed_transactions.json`
|
||||
|
||||
**Run:**
|
||||
```bash
|
||||
python example_usage.py
|
||||
```
|
||||
|
||||
### **ynab_example.py** - YNAB Transaction Example
|
||||
Demonstrates fetching and displaying YNAB transactions.
|
||||
|
||||
**Output:**
|
||||
- Console display of first 10 transactions
|
||||
- Saves all transactions to `ynab_transactions.json`
|
||||
|
||||
**Run:**
|
||||
```bash
|
||||
python ynab_example.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Interface
|
||||
|
||||
### **main.py** - Command-line Entry Point
|
||||
Thin wrapper around extraction functionality.
|
||||
|
||||
**Arguments:**
|
||||
- `--input` - Input HAR file (default: posted.dat)
|
||||
- `--output` - Output JSON file (optional)
|
||||
- `--show-errors` - Display parse errors
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Extract to console
|
||||
python main.py --input posted.dat
|
||||
|
||||
# Extract to file
|
||||
python main.py --input posted.dat --output extracted.json
|
||||
|
||||
# Show parsing errors
|
||||
python main.py --input posted.dat --show-errors
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
posted.dat (HAR JSON)
|
||||
↓
|
||||
wf_debit_extractor.py (Extract proprietary payloads)
|
||||
↓
|
||||
Extracted payload objects
|
||||
↓
|
||||
transaction_processor.py (Transform to structured format)
|
||||
↓
|
||||
Processed transaction list
|
||||
↓
|
||||
├─→ example_usage.py (Save to processed_transactions.json)
|
||||
└─→ reconcile.py (Compare with YNAB)
|
||||
↓
|
||||
ynab_client.py (Fetch from YNAB API)
|
||||
↓
|
||||
reconciliation_report.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **ynab** (v2.1.0) - Official YNAB Python SDK
|
||||
- **pyyaml** (v6.0.3) - YAML configuration parsing
|
||||
|
||||
**Install:**
|
||||
```bash
|
||||
pip install ynab pyyaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Manifest
|
||||
|
||||
| File | Purpose | Lines |
|
||||
|------|---------|-------|
|
||||
| `wf_debit_extractor.py` | Core extraction logic | 91 |
|
||||
| `transaction_processor.py` | Transaction transformation | 70 |
|
||||
| `ynab_client.py` | YNAB API integration | 150 |
|
||||
| `reconcile.py` | Reconciliation orchestration | 117 |
|
||||
| `main.py` | CLI wrapper | ~40 |
|
||||
| `example_usage.py` | WF usage example | ~60 |
|
||||
| `ynab_example.py` | YNAB usage example | ~60 |
|
||||
| `ynab.yml` | YNAB configuration | 6 |
|
||||
| `README.md` | Project documentation | ~100 |
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
✓ Extracts Wells Fargo proprietary JSON from HAR files
|
||||
✓ Transforms transactions with amount normalization
|
||||
✓ Converts millisecond timestamps to Python datetime objects
|
||||
✓ Integrates with YNAB API for budget tracking
|
||||
✓ Supports date range filtering (past X days)
|
||||
✓ Generates reconciliation reports
|
||||
✓ Error handling and reporting
|
||||
✓ Type-safe with TypedDict definitions
|
||||
✓ Configuration via YAML
|
||||
✓ Modular, reusable architecture
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ All modules created and tested
|
||||
- ✅ No static errors
|
||||
- ✅ Dependencies installed
|
||||
- ✅ CLI functional
|
||||
- ✅ Examples working
|
||||
- ✅ Backward compatible
|
||||
- ⚠️ YNAB integration requires valid API token
|
||||
|
||||
---
|
||||
|
||||
## Next Steps (Optional)
|
||||
|
||||
1. **Database Integration** - Store transactions in SQLite/PostgreSQL
|
||||
2. **Matching Algorithm** - Auto-match WF↔YNAB transactions by amount/date
|
||||
3. **Web Dashboard** - Flask/FastAPI interface for viewing reports
|
||||
4. **Scheduling** - Celery/APScheduler for periodic reconciliation
|
||||
5. **Export Formats** - CSV, Excel, PDF report generation
|
||||
|
||||
Reference in New Issue
Block a user