feat: migrate backend persistence from TinyDB to MongoDB Atlas

- Implement lazy MongoDB client initialization in backend/db/mongo_client.py.
- Create Gunicorn configuration to ensure MongoDB client is initialized per worker.
- Refactor database access layer to support MongoDB with a new MongoLockedTable adapter.
- Add migration script to transfer existing TinyDB data to MongoDB, preserving idempotency.
- Update tracking event handling to ensure deterministic ordering with a monotonic sequence.
- Modify tests to use mongomock for MongoDB integration and ensure existing tests pass.
- Add integration test script to run tests against a local MongoDB Docker container.
- Document environment variables and migration process in specs/feat-database-migration.md.
This commit is contained in:
2026-07-28 01:10:52 -04:00
parent c9c1fa18a2
commit da7ed41938
17 changed files with 1364 additions and 62 deletions
+33 -2
View File
@@ -4,7 +4,7 @@ A family-friendly application for managing chores, tasks, and rewards for childr
## 🏗️ Architecture
- **Backend**: Flask (Python) with TinyDB for data persistence
- **Backend**: Flask (Python) with TinyDB or MongoDB for data persistence
- **Frontend**: Vue 3 (TypeScript) with real-time SSE updates
- **Deployment**: Docker with nginx reverse proxy
@@ -38,6 +38,37 @@ npm run dev
| `ACCOUNT_DELETION_THRESHOLD_HOURS` | Hours to wait before deleting marked accounts | 720 (30 days) |
| `DB_ENV` | Database environment (`prod` or `test`) | `prod` |
| `DATA_ENV` | Data directory environment (`prod` or `test`) | `prod` |
| `USE_MONGODB` | Use MongoDB instead of TinyDB (`true`/`false`) | `false` |
| `MONGO_URI` | MongoDB connection URI (required when `USE_MONGODB=true`) | — |
| `MONGO_DB_NAME` | MongoDB database name (optional) | Parsed from `MONGO_URI`, or `chore_db`/`chore_db_test`/`chore_db_e2e` based on `DB_ENV` |
### Database Backend
The application supports two persistence backends:
- **TinyDB** (default): JSON-file storage in `backend/data/db/` (or `backend/test_data/db/` for `test`/`e2e`). No extra configuration needed.
- **MongoDB**: Set `USE_MONGODB=true` and provide `MONGO_URI`. Useful for production deployments and managed database hosting (e.g., MongoDB Atlas).
#### Migrating from TinyDB to MongoDB
```bash
cd backend
# Dry run to preview what will be migrated
python -m scripts.migrate_to_mongodb --dry-run
# Run the migration (backs up TinyDB files first)
python -m scripts.migrate_to_mongodb
```
The migration script reads the existing TinyDB JSON files and inserts each record into the matching MongoDB collection, skipping records that already exist. Original TinyDB files are backed up to `backend/data/db/backups/<timestamp>/`.
#### Rolling Back
To revert to TinyDB, simply set `USE_MONGODB=false` (or unset it). The original JSON files remain in place.
#### Gunicorn / Docker
When running multiple Gunicorn workers, each worker must create its own MongoDB client after forking. This is handled automatically by `backend/gunicorn.conf.py`, which is loaded by `backend/Dockerfile` via `-c gunicorn.conf.py`.
### Account Deletion Scheduler
@@ -145,7 +176,7 @@ npm run test
├── backend/
│ ├── api/ # REST API endpoints
│ ├── config/ # Configuration files
│ ├── db/ # TinyDB setup
│ ├── db/ # TinyDB / MongoDB persistence layer
│ ├── events/ # SSE event system
│ ├── models/ # Data models
│ ├── tests/ # Backend tests