feat: migrate backend persistence from TinyDB to MongoDB Atlas
- Implement lazy MongoDB client initialization in backend/db/mongo_client.py. - Create Gunicorn configuration to ensure MongoDB client is initialized per worker. - Refactor database access layer to support MongoDB with a new MongoLockedTable adapter. - Add migration script to transfer existing TinyDB data to MongoDB, preserving idempotency. - Update tracking event handling to ensure deterministic ordering with a monotonic sequence. - Modify tests to use mongomock for MongoDB integration and ensure existing tests pass. - Add integration test script to run tests against a local MongoDB Docker container. - Document environment variables and migration process in specs/feat-database-migration.md.
This commit is contained in:
@@ -8,8 +8,9 @@ Family chore/reward manager. Flask + TinyDB backend (`backend/`), Vue 3 + TypeSc
|
||||
- Activate venv: `source .venv/bin/activate`
|
||||
- Dev server: `python -m flask run --host=0.0.0.0 --port=5000` (entry: `main.py`)
|
||||
- Required env vars: `SECRET_KEY`, `REFRESH_TOKEN_EXPIRY_DAYS`, `DIGEST_TOKEN_SECRET`, `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY` — Flask raises `RuntimeError` on boot if any are missing
|
||||
- Optional: `DB_ENV` / `DATA_ENV` (`prod` | `test` | `e2e`) — picks `data/` vs `test_data/` dir (see `config/paths.py`)
|
||||
- Tests: `pytest tests/` — `conftest.py` forces `DB_ENV=test` and sets dummy secrets. Single test: `pytest tests/test_routine_api.py::test_name`
|
||||
- Optional persistence switch: `USE_MONGODB` (`true` | `false`). When `true`, set `MONGO_URI` and optionally `MONGO_DB_NAME`. Default `false` keeps TinyDB.
|
||||
- Optional: `DB_ENV` / `DATA_ENV` (`prod` | `test` | `e2e`) — picks `data/` vs `test_data/` dir (see `config/paths.py`). For MongoDB these also select the default database name (`chore_db`, `chore_db_test`, `chore_db_e2e`) unless `MONGO_DB_NAME` is set.
|
||||
- Tests: `pytest tests/` — `conftest.py` forces `DB_ENV=test`, `USE_MONGODB=true`, `MONGO_URI=mongomock`, and sets dummy secrets. Single test: `pytest tests/test_routine_api.py::test_name`
|
||||
- Python imports assume `backend/` is on `sys.path` (set by `conftest.py` / `flask run` cwd). Run pytest from `backend/`.
|
||||
- Create admin user: `python scripts/create_admin.py` (admin role cannot be set via signup)
|
||||
|
||||
@@ -18,7 +19,7 @@ Family chore/reward manager. Flask + TinyDB backend (`backend/`), Vue 3 + TypeSc
|
||||
- Lint: `npm run lint`
|
||||
- Type-check: `npm run type-check`
|
||||
- Unit tests: `npm run test:unit` (Vitest). Single: `npx vitest run path/to/file.spec.ts`
|
||||
- E2E: `npx playwright test` — config auto-starts both `npm run dev` and the Flask backend with `DB_ENV=e2e DATA_ENV=e2e`. Tests live in `e2e/`.
|
||||
- E2E: `npx playwright test` — config auto-starts both `npm run dev` and the Flask backend with `DB_ENV=e2e DATA_ENV=e2e USE_MONGODB=true MONGO_URI=mongomock`. Tests live in `e2e/`. `frontend/.env.test` contains the example MongoDB config.
|
||||
- E2E buckets are Playwright projects (see `playwright.config.ts`) targeting directories under `e2e/mode_parent/`
|
||||
|
||||
## Architecture
|
||||
@@ -30,7 +31,9 @@ Family chore/reward manager. Flask + TinyDB backend (`backend/`), Vue 3 + TypeSc
|
||||
|
||||
### Models — strict 1:1 parity
|
||||
- Python `@dataclass`es in `backend/models/`. TypeScript interfaces in `frontend/src/common/models.ts`. Any model change requires updating both.
|
||||
- Persistence is TinyDB via the thread-safe `LockedTable` wrapper (`backend/db/db.py`). Operate on model instances with `from_dict()` / `to_dict()` — never raw dicts.
|
||||
- Persistence is TinyDB by default, or MongoDB when `USE_MONGODB=true`. Both are accessed through the `LockedTable` / `MongoLockedTable` wrappers in `backend/db/db.py`. Operate on model instances with `from_dict()` / `to_dict()` — never raw dicts.
|
||||
- MongoDB client initialization is lazy (`backend/db/mongo_client.py`). `backend/gunicorn.conf.py` provides the `post_fork` hook required for multi-worker Gunicorn deployments; `backend/Dockerfile` loads it with `-c gunicorn.conf.py`.
|
||||
- Migration script: `cd backend && python -m scripts/migrate_to_mongodb [--dry-run]`. It reads TinyDB JSON files and writes them to MongoDB idempotently, backing up the originals to `<db_dir>/backups/<timestamp>/`.
|
||||
|
||||
### SSE event bus — mandatory for every mutation
|
||||
- Every backend mutation (add/edit/delete/trigger) **must** call `send_event_for_current_user` from `api/utils.py`. Event types in `backend/events/types/` are mirrored in `frontend/src/common/backendEvents.ts`.
|
||||
|
||||
Reference in New Issue
Block a user