feat: migrate backend persistence from TinyDB to MongoDB Atlas
- Implement lazy MongoDB client initialization in backend/db/mongo_client.py. - Create Gunicorn configuration to ensure MongoDB client is initialized per worker. - Refactor database access layer to support MongoDB with a new MongoLockedTable adapter. - Add migration script to transfer existing TinyDB data to MongoDB, preserving idempotency. - Update tracking event handling to ensure deterministic ordering with a monotonic sequence. - Modify tests to use mongomock for MongoDB integration and ensure existing tests pass. - Add integration test script to run tests against a local MongoDB Docker container. - Document environment variables and migration process in specs/feat-database-migration.md.
This commit is contained in:
@@ -0,0 +1,164 @@
|
||||
# Feature: Migrate Backend Persistence from TinyDB to MongoDB Atlas
|
||||
|
||||
## Overview
|
||||
|
||||
**Goal:** Replace the TinyDB JSON-file persistence layer in the Flask backend with MongoDB Atlas while keeping the existing API surface, model definitions, and test infrastructure largely unchanged. Introduce a migration script that exports existing TinyDB data into MongoDB.
|
||||
|
||||
**User Story:** As a developer and operator, I want the application to run on a managed, scalable document database so that concurrent access, multi-worker deployments, and production hosting are reliable.
|
||||
|
||||
**Rules:**
|
||||
- Backend routes and response shapes must remain unchanged.
|
||||
- Python models in `backend/models/` keep their `from_dict()` / `to_dict()` interface.
|
||||
- Every mutation must continue to emit SSE events through the existing broadcaster.
|
||||
- The migration must be reversible (keep TinyDB files as a backup) and idempotent.
|
||||
- All existing tests must continue to pass; new MongoDB-backed tests must be added.
|
||||
|
||||
---
|
||||
|
||||
## Data Model Changes
|
||||
|
||||
### Backend Model
|
||||
- No changes to dataclass fields or `to_dict()` / `from_dict()` output.
|
||||
- The model's `id` field is stored as MongoDB's `_id` primary key. The original `id` field is removed from storage so documents do not contain duplicate values.
|
||||
- On read, `_id` is mapped back to `id` and the document is wrapped as a TinyDB-compatible `Document` with `doc_id` set.
|
||||
- `created_at` and `updated_at` remain epoch floats.
|
||||
|
||||
### Frontend Model
|
||||
- No changes. The API contract is preserved.
|
||||
|
||||
---
|
||||
|
||||
## Backend Implementation
|
||||
|
||||
### 1. Database client initialization
|
||||
- Create `backend/db/mongo_client.py`.
|
||||
- Do **not** create a `MongoClient` at module import; it is created lazily on first access.
|
||||
- `get_mongo_client()` returns a cached process-level singleton so tests, scripts, and Flask requests all share the same client.
|
||||
- For Gunicorn multi-worker deployments, `backend/gunicorn.conf.py` defines a `post_fork` hook that calls `init_mongo_client()` so each worker process gets its own client after forking.
|
||||
- Fail-fast settings for development:
|
||||
```python
|
||||
MongoClient(
|
||||
os.getenv("MONGO_URI"),
|
||||
serverSelectionTimeoutMS=5000,
|
||||
connectTimeoutMS=5000,
|
||||
maxPoolSize=20,
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Collection mapping
|
||||
- One MongoDB collection per existing TinyDB file:
|
||||
- `children` ← `children.json`
|
||||
- `tasks` ← `tasks.json`
|
||||
- `routines` ← `routines.json`
|
||||
- `routine_items` ← `routine_items.json`
|
||||
- `routine_schedules` ← `routine_schedules.json`
|
||||
- `routine_extensions` ← `routine_extensions.json`
|
||||
- `rewards` ← `rewards.json`
|
||||
- `images` ← `images.json`
|
||||
- `pending_rewards` ← `pending_rewards.json` (legacy)
|
||||
- `pending_confirmations` ← `pending_confirmations.json`
|
||||
- `users` ← `users.json`
|
||||
- `tracking_events` ← `tracking_events.json`
|
||||
- `child_overrides` ← `child_overrides.json`
|
||||
- `chore_schedules` ← `chore_schedules.json`
|
||||
- `task_extensions` ← `task_extensions.json`
|
||||
- `refresh_tokens` ← `refresh_tokens.json`
|
||||
- `push_subscriptions` ← `push_subscriptions.json`
|
||||
- `digest_action_tokens` ← `digest_action_tokens.json`
|
||||
|
||||
### 3. `LockedTable` MongoDB adapter
|
||||
- Refactor `backend/db/db.py` so that each exported `*_db` object continues to expose the same methods: `search(...)`, `get(...)`, `all()`, `insert(...)`, `insert_multiple(...)`, `update(...)`, `remove(...)`, `truncate()`.
|
||||
- Implement a new `MongoLockedTable` class that mirrors the existing `LockedTable` API but delegates to a MongoDB collection.
|
||||
- Keep `LockedTable` (TinyDB) as a fallback selectable by `USE_MONGODB=true|false` (default `false`).
|
||||
- Store the model `id` field as MongoDB `_id` on writes; do not keep a duplicate `id` field in storage.
|
||||
- Restore the model `id` field from `_id` on reads and return TinyDB-compatible `Document` objects with `doc_id` set so existing callers that rely on `doc_id` continue to work.
|
||||
- Support `update(fields, doc_ids=[...])` and callable updater functions (`update(lambda rec: ..., cond)`) to match TinyDB behavior.
|
||||
- Reuse function names where possible to minimize API file changes.
|
||||
|
||||
### 4. Indexes
|
||||
- Create indexes on application startup (`backend/main.py`) and in the pytest session fixture. Index creation is intentionally **not** done at module import time so the `MongoClient` remains lazily initialized.
|
||||
- Required indexes:
|
||||
- Unique primary key is provided by MongoDB `_id` (mapped from model `id`); no separate `id` index is created.
|
||||
- `user_id` — secondary on `children`, `tasks`, `routines`, `routine_items`, `rewards`, `images`, `tracking_events`, `pending_confirmations`, `chore_schedules`, `task_extensions`, `refresh_tokens`, `push_subscriptions`, `digest_action_tokens`.
|
||||
- `child_id` — secondary on `child_overrides`, `chore_schedules`, `task_extensions`, `pending_confirmations`, `tracking_events`.
|
||||
- `entity_id` + `entity_type` — compound secondary on `pending_confirmations`, `tracking_events`, `child_overrides`.
|
||||
- `token` — unique on `refresh_tokens` and `digest_action_tokens`.
|
||||
|
||||
### 5. Migration script
|
||||
- Create `backend/scripts/migrate_to_mongodb.py`.
|
||||
- Reads each TinyDB JSON file in `data/db/` (or `test_data/db/` when `DB_ENV=test`).
|
||||
- Inserts each `_default` record into the corresponding MongoDB collection, mapping TinyDB integer keys to the record's own `id` field.
|
||||
- Skips records that already exist by `id` (idempotent).
|
||||
- Backs up TinyDB files to `data/db/backups/<timestamp>/` before first run.
|
||||
- Prints a summary of migrated records per collection.
|
||||
- Usage:
|
||||
```bash
|
||||
cd backend
|
||||
python -m scripts.migrate_to_mongodb [--dry-run]
|
||||
```
|
||||
|
||||
### 6. Environment variables
|
||||
- `MONGO_URI` — required when `USE_MONGODB=true`.
|
||||
- `MONGO_DB_NAME` — optional database name; defaults parsed from `MONGO_URI` or falls back to `chore_db`/`chore_db_test`/`chore_db_e2e` based on `DB_ENV`/`DATA_ENV`.
|
||||
- `USE_MONGODB` — `true` to use MongoDB backend; `false` to keep TinyDB.
|
||||
|
||||
### 7. Default data initialization
|
||||
- `backend/db/default.py` (`initializeImages`, `createDefaultTasks`, `createDefaultRewards`) must work with the new MongoDB-backed `*_db` objects. No API changes.
|
||||
|
||||
### 8. Scheduler compatibility
|
||||
- Background schedulers (`account_deletion_scheduler`, `digest_scheduler`, etc.) read/write through the same `*_db` objects, so they require no changes once the adapter is in place.
|
||||
|
||||
### 9. Tracking event ordering
|
||||
- `db/tracking.py` adds a hidden monotonic `_seq` field to tracking events on insert and sorts by `(occurred_at, created_at, _seq)` descending. This guarantees deterministic ordering when consecutive events share the same timestamp (common in fast tests).
|
||||
|
||||
### 10. Gunicorn / Docker deployment
|
||||
- `backend/gunicorn.conf.py` provides the `post_fork` hook required by the wiki plan.
|
||||
- `backend/Dockerfile` now loads the config via `-c gunicorn.conf.py`.
|
||||
|
||||
---
|
||||
|
||||
## Backend Tests
|
||||
|
||||
- [x] Add `mongomock` to test dependencies and create `backend/tests/test_mongo_adapter.py` that verifies CRUD operations against `mongomock`.
|
||||
- [x] Update `backend/tests/conftest.py` to set `USE_MONGODB=true` and point `MONGO_URI` at `mongomock` for the default test run.
|
||||
- [x] Ensure every existing API test still passes with the MongoDB adapter by running `pytest tests/`.
|
||||
- [x] Add integration test script `backend/scripts/run_integration_tests.ps1` that starts a local MongoDB Docker container, runs a targeted pytest suite, and tears down the container.
|
||||
- [x] Add `frontend/.env.test` with MongoDB configuration and update Playwright webServer env to pass `USE_MONGODB`/`MONGO_URI` to the Flask backend; E2E database is reset via the existing `/auth/e2e-seed` endpoint.
|
||||
|
||||
---
|
||||
|
||||
## Frontend Implementation
|
||||
|
||||
- No frontend changes are required. The API contract remains unchanged.
|
||||
|
||||
## Frontend Tests
|
||||
|
||||
- [ ] Run `npx playwright test` and confirm E2E tests pass against the MongoDB-backed Flask backend with the isolated `chore_db_e2e` database.
|
||||
|
||||
---
|
||||
|
||||
## Future Considerations
|
||||
|
||||
- After a production burn-in period, remove the TinyDB fallback and `LockedTable` code.
|
||||
- Add database connection health-check endpoint.
|
||||
- Consider MongoDB transactions for multi-document operations (e.g., child deletion cascades).
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria (Definition of Done)
|
||||
|
||||
### Backend
|
||||
- [x] `backend/db/db.py` exports MongoDB-backed `*_db` objects when `USE_MONGODB=true`.
|
||||
- [x] `backend/db/mongo_client.py` implements lazy client initialization and a Gunicorn-compatible init hook.
|
||||
- [x] All existing API endpoints continue to work without route or response changes.
|
||||
- [x] `backend/scripts/migrate_to_mongodb.py` migrates existing TinyDB JSON files idempotently.
|
||||
- [x] Required unique and secondary indexes are created on application startup or migration.
|
||||
- [x] Unit tests pass with `mongomock` (`pytest tests/`).
|
||||
- [ ] Integration tests pass against a local Docker MongoDB container (requires a running Docker daemon; script verified to start/stop container when daemon is available).
|
||||
|
||||
### Frontend
|
||||
- [x] E2E setup and representative parent-mode smoke tests pass against the isolated `chore_db_e2e` mongomock database, with the DB reset via `/auth/e2e-seed`.
|
||||
|
||||
### Operations
|
||||
- [x] `MONGO_URI` and optional `MONGO_DB_NAME` environment variables are documented.
|
||||
- [x] Rollback instructions exist to switch back to TinyDB by setting `USE_MONGODB=false`.
|
||||
Reference in New Issue
Block a user