diff --git a/AGENTS.md b/AGENTS.md index 0f3d66d..89da2cc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,8 +8,9 @@ Family chore/reward manager. Flask + TinyDB backend (`backend/`), Vue 3 + TypeSc - Activate venv: `source .venv/bin/activate` - Dev server: `python -m flask run --host=0.0.0.0 --port=5000` (entry: `main.py`) - Required env vars: `SECRET_KEY`, `REFRESH_TOKEN_EXPIRY_DAYS`, `DIGEST_TOKEN_SECRET`, `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY` — Flask raises `RuntimeError` on boot if any are missing -- Optional: `DB_ENV` / `DATA_ENV` (`prod` | `test` | `e2e`) — picks `data/` vs `test_data/` dir (see `config/paths.py`) -- Tests: `pytest tests/` — `conftest.py` forces `DB_ENV=test` and sets dummy secrets. Single test: `pytest tests/test_routine_api.py::test_name` +- Optional persistence switch: `USE_MONGODB` (`true` | `false`). When `true`, set `MONGO_URI` and optionally `MONGO_DB_NAME`. Default `false` keeps TinyDB. +- Optional: `DB_ENV` / `DATA_ENV` (`prod` | `test` | `e2e`) — picks `data/` vs `test_data/` dir (see `config/paths.py`). For MongoDB these also select the default database name (`chore_db`, `chore_db_test`, `chore_db_e2e`) unless `MONGO_DB_NAME` is set. +- Tests: `pytest tests/` — `conftest.py` forces `DB_ENV=test`, `USE_MONGODB=true`, `MONGO_URI=mongomock`, and sets dummy secrets. Single test: `pytest tests/test_routine_api.py::test_name` - Python imports assume `backend/` is on `sys.path` (set by `conftest.py` / `flask run` cwd). Run pytest from `backend/`. - Create admin user: `python scripts/create_admin.py` (admin role cannot be set via signup) @@ -18,7 +19,7 @@ Family chore/reward manager. Flask + TinyDB backend (`backend/`), Vue 3 + TypeSc - Lint: `npm run lint` - Type-check: `npm run type-check` - Unit tests: `npm run test:unit` (Vitest). Single: `npx vitest run path/to/file.spec.ts` -- E2E: `npx playwright test` — config auto-starts both `npm run dev` and the Flask backend with `DB_ENV=e2e DATA_ENV=e2e`. Tests live in `e2e/`. +- E2E: `npx playwright test` — config auto-starts both `npm run dev` and the Flask backend with `DB_ENV=e2e DATA_ENV=e2e USE_MONGODB=true MONGO_URI=mongomock`. Tests live in `e2e/`. `frontend/.env.test` contains the example MongoDB config. - E2E buckets are Playwright projects (see `playwright.config.ts`) targeting directories under `e2e/mode_parent/` ## Architecture @@ -30,7 +31,9 @@ Family chore/reward manager. Flask + TinyDB backend (`backend/`), Vue 3 + TypeSc ### Models — strict 1:1 parity - Python `@dataclass`es in `backend/models/`. TypeScript interfaces in `frontend/src/common/models.ts`. Any model change requires updating both. -- Persistence is TinyDB via the thread-safe `LockedTable` wrapper (`backend/db/db.py`). Operate on model instances with `from_dict()` / `to_dict()` — never raw dicts. +- Persistence is TinyDB by default, or MongoDB when `USE_MONGODB=true`. Both are accessed through the `LockedTable` / `MongoLockedTable` wrappers in `backend/db/db.py`. Operate on model instances with `from_dict()` / `to_dict()` — never raw dicts. +- MongoDB client initialization is lazy (`backend/db/mongo_client.py`). `backend/gunicorn.conf.py` provides the `post_fork` hook required for multi-worker Gunicorn deployments; `backend/Dockerfile` loads it with `-c gunicorn.conf.py`. +- Migration script: `cd backend && python -m scripts/migrate_to_mongodb [--dry-run]`. It reads TinyDB JSON files and writes them to MongoDB idempotently, backing up the originals to `/backups//`. ### SSE event bus — mandatory for every mutation - Every backend mutation (add/edit/delete/trigger) **must** call `send_event_for_current_user` from `api/utils.py`. Event types in `backend/events/types/` are mirrored in `frontend/src/common/backendEvents.ts`. diff --git a/README.md b/README.md index a6098e5..b14c04d 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A family-friendly application for managing chores, tasks, and rewards for childr ## 🏗️ Architecture -- **Backend**: Flask (Python) with TinyDB for data persistence +- **Backend**: Flask (Python) with TinyDB or MongoDB for data persistence - **Frontend**: Vue 3 (TypeScript) with real-time SSE updates - **Deployment**: Docker with nginx reverse proxy @@ -38,6 +38,37 @@ npm run dev | `ACCOUNT_DELETION_THRESHOLD_HOURS` | Hours to wait before deleting marked accounts | 720 (30 days) | | `DB_ENV` | Database environment (`prod` or `test`) | `prod` | | `DATA_ENV` | Data directory environment (`prod` or `test`) | `prod` | +| `USE_MONGODB` | Use MongoDB instead of TinyDB (`true`/`false`) | `false` | +| `MONGO_URI` | MongoDB connection URI (required when `USE_MONGODB=true`) | — | +| `MONGO_DB_NAME` | MongoDB database name (optional) | Parsed from `MONGO_URI`, or `chore_db`/`chore_db_test`/`chore_db_e2e` based on `DB_ENV` | + +### Database Backend + +The application supports two persistence backends: + +- **TinyDB** (default): JSON-file storage in `backend/data/db/` (or `backend/test_data/db/` for `test`/`e2e`). No extra configuration needed. +- **MongoDB**: Set `USE_MONGODB=true` and provide `MONGO_URI`. Useful for production deployments and managed database hosting (e.g., MongoDB Atlas). + +#### Migrating from TinyDB to MongoDB + +```bash +cd backend +# Dry run to preview what will be migrated +python -m scripts.migrate_to_mongodb --dry-run + +# Run the migration (backs up TinyDB files first) +python -m scripts.migrate_to_mongodb +``` + +The migration script reads the existing TinyDB JSON files and inserts each record into the matching MongoDB collection, skipping records that already exist. Original TinyDB files are backed up to `backend/data/db/backups//`. + +#### Rolling Back + +To revert to TinyDB, simply set `USE_MONGODB=false` (or unset it). The original JSON files remain in place. + +#### Gunicorn / Docker + +When running multiple Gunicorn workers, each worker must create its own MongoDB client after forking. This is handled automatically by `backend/gunicorn.conf.py`, which is loaded by `backend/Dockerfile` via `-c gunicorn.conf.py`. ### Account Deletion Scheduler @@ -145,7 +176,7 @@ npm run test ├── backend/ │ ├── api/ # REST API endpoints │ ├── config/ # Configuration files -│ ├── db/ # TinyDB setup +│ ├── db/ # TinyDB / MongoDB persistence layer │ ├── events/ # SSE event system │ ├── models/ # Data models │ ├── tests/ # Backend tests diff --git a/backend/Dockerfile b/backend/Dockerfile index e17b1df..49cacad 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -15,4 +15,4 @@ ENV PYTHONIOENCODING=utf-8 VOLUME ["/app/data"] # Use Gunicorn instead of python main.py -CMD ["gunicorn", "--bind", "0.0.0.0:5000", "-k", "gevent", "--workers", "1", "--timeout", "120", "--access-logfile", "-", "--error-logfile", "-", "--log-level", "info", "main:app"] \ No newline at end of file +CMD ["gunicorn", "--bind", "0.0.0.0:5000", "-k", "gevent", "--workers", "1", "--timeout", "120", "--access-logfile", "-", "--error-logfile", "-", "--log-level", "info", "-c", "gunicorn.conf.py", "main:app"] \ No newline at end of file diff --git a/backend/api/auth_api.py b/backend/api/auth_api.py index 91f5abf..384883c 100644 --- a/backend/api/auth_api.py +++ b/backend/api/auth_api.py @@ -29,6 +29,8 @@ from db.db import ( users_db, refresh_tokens_db, child_db, task_db, reward_db, image_db, pending_reward_db, pending_confirmations_db, tracking_events_db, child_overrides_db, chore_schedules_db, task_extensions_db, + routine_db, routine_items_db, routine_schedules_db, routine_extensions_db, + push_subscriptions_db, digest_action_tokens_db, ) from db.default import initializeImages, createDefaultTasks, createDefaultRewards from api.utils import normalize_email @@ -623,6 +625,12 @@ def e2e_seed(): chore_schedules_db.truncate() task_extensions_db.truncate() refresh_tokens_db.truncate() + routine_db.truncate() + routine_items_db.truncate() + routine_schedules_db.truncate() + routine_extensions_db.truncate() + push_subscriptions_db.truncate() + digest_action_tokens_db.truncate() # Recreate only baseline defaults for e2e runs. initializeImages() diff --git a/backend/db/db.py b/backend/db/db.py index ff5ca72..3091dfb 100644 --- a/backend/db/db.py +++ b/backend/db/db.py @@ -1,13 +1,42 @@ # python import os -from config.paths import get_database_dir import threading +from config.paths import get_database_dir from tinydb import TinyDB +from tinydb.queries import QueryInstance +from db.mongo_client import get_mongo_client, get_mongo_db_name + +try: + from tinydb.table import Document +except ImportError: # pragma: no cover - tinydb version compatibility + from tinydb.database import Document + + +def _stable_clause_key(clause: dict) -> str: + """Return a stable string key for sorting MongoDB filter clauses.""" + import json + return json.dumps(clause, sort_keys=True, default=str) + + +try: + from pymongo import ASCENDING +except ImportError: # pragma: no cover - pymongo is a required dependency + ASCENDING = 1 + + +USE_MONGODB = os.environ.get('USE_MONGODB', 'false').lower() == 'true' +# Resolve the MongoDB database name once at module load so runtime changes to +# DB_ENV/DATA_ENV in tests do not switch databases mid-process. +_mongo_db_name = get_mongo_db_name() if USE_MONGODB else None base_dir = get_database_dir() os.makedirs(base_dir, exist_ok=True) +# --------------------------------------------------------------------------- +# TinyDB-backed table wrapper +# --------------------------------------------------------------------------- + class LockedTable: """ Thread-safe wrapper around a TinyDB table. All callable attribute access @@ -65,6 +94,361 @@ class LockedTable: with self._lock: return self._table.truncate() + def close(self): + with self._lock: + return self._table.close() + + +# --------------------------------------------------------------------------- +# TinyDB Query -> MongoDB filter translator +# --------------------------------------------------------------------------- + +_MONGO_OP_MAP = { + '==': '$eq', + '!=': '$ne', + '<': '$lt', + '<=': '$lte', + '>': '$gt', + '>=': '$gte', +} + +_NEGATED_OPS = { + '$eq': '$ne', + '$ne': '$eq', + '$lt': '$gte', + '$lte': '$gt', + '$gt': '$lte', + '$gte': '$lt', +} + + +def _field_to_mongo(field_path: tuple) -> str: + """Map a TinyDB field path to a MongoDB field name. + + The model ``id`` field is stored as the MongoDB ``_id`` field, so queries + on ``id`` are translated to queries on ``_id``. + """ + if len(field_path) == 1: + return '_id' if field_path[0] == 'id' else field_path[0] + return '.'.join('_id' if p == 'id' else p for p in field_path) + + +def _negate_condition(cond: dict) -> dict | None: + """Return a MongoDB condition that negates a single-field condition.""" + if len(cond) != 1: + return None + field, inner = next(iter(cond.items())) + if not isinstance(inner, dict) or len(inner) != 1: + return None + op, value = next(iter(inner.items())) + if op in _NEGATED_OPS: + return {field: {_NEGATED_OPS[op]: value}} + return None + + +def _hash_to_mongo_filter(query_hash) -> dict | None: + """Translate a TinyDB query hash tuple to a MongoDB filter document. + + Returns ``None`` when the query construct cannot be expressed as a native + MongoDB filter, signalling that the caller should fall back to in-memory + TinyDB evaluation. + """ + if not isinstance(query_hash, tuple) or len(query_hash) == 0: + return None + + op = query_hash[0] + + if op in _MONGO_OP_MAP: + field = _field_to_mongo(query_hash[1]) + value = query_hash[2] + return {field: {_MONGO_OP_MAP[op]: value}} + + if op == 'exists': + field = _field_to_mongo(query_hash[1]) + return {field: {'$exists': True}} + + if op == 'one_of': + field = _field_to_mongo(query_hash[1]) + return {field: {'$in': list(query_hash[2])}} + + if op == 'any': + field = _field_to_mongo(query_hash[1]) + return {field: {'$in': list(query_hash[2])}} + + if op == 'all': + field = _field_to_mongo(query_hash[1]) + return {field: {'$all': list(query_hash[2])}} + + if op == 'matches': + field = _field_to_mongo(query_hash[1]) + # TinyDB matches() anchors the regex at the start of the string. + return {field: {'$regex': f'^{query_hash[2]}'}} + + if op == 'search': + field = _field_to_mongo(query_hash[1]) + return {field: {'$regex': query_hash[2]}} + + if op == 'and': + merged: dict = {} + for sub_hash in query_hash[1]: + sub = _hash_to_mongo_filter(sub_hash) + if sub is None: + return None + for field, inner in sub.items(): + if field in merged: + if isinstance(merged[field], dict) and isinstance(inner, dict): + merged[field].update(inner) + elif isinstance(merged[field], list) and isinstance(inner, list): + merged[field].extend(inner) + else: + return None + else: + merged[field] = ( + dict(inner) if isinstance(inner, dict) + else list(inner) if isinstance(inner, list) + else inner + ) + return merged + + if op == 'or': + clauses = [_hash_to_mongo_filter(sub_hash) for sub_hash in query_hash[1]] + if any(c is None for c in clauses): + return None + return {'$or': sorted(clauses, key=_stable_clause_key)} + + if op == 'not': + inner = _hash_to_mongo_filter(query_hash[1]) + if inner is None: + return None + negated = _negate_condition(inner) + if negated is not None: + return negated + return None + + # Unsupported operation (test, fragment, noop, etc.) -> fall back. + return None + + +def _query_to_mongo_filter(query) -> dict | None: + """Translate a TinyDB QueryInstance to a MongoDB filter, if possible.""" + if isinstance(query, dict): + return query + if isinstance(query, QueryInstance): + return _hash_to_mongo_filter(query._hash) + return None + + +def _evaluate_in_memory(docs, query) -> list: + """Evaluate a TinyDB query against an in-memory list of documents.""" + if callable(query): + return [doc for doc in docs if query(doc)] + return docs + + +# --------------------------------------------------------------------------- +# MongoDB-backed table wrapper +# --------------------------------------------------------------------------- + +class MongoLockedTable: + """Drop-in replacement for ``LockedTable`` that delegates to MongoDB. + + The adapter preserves the existing table API while mapping the model + ``id`` field to MongoDB's ``_id`` field on reads and writes. + """ + + def __init__(self, collection_name: str): + self.collection_name = collection_name + + def _collection(self): + client = get_mongo_client() + return client[_mongo_db_name][self.collection_name] + + @staticmethod + def _doc_to_mongo(doc: dict) -> dict: + """Store a copy of ``doc`` with ``id`` promoted to MongoDB ``_id``. + + This avoids storing both ``_id`` and ``id`` with identical values. + The original ``id`` field is removed from the stored document. + """ + if doc is None: + return None + d = dict(doc) + if 'id' in d: + d['_id'] = d.pop('id') + return d + + @staticmethod + def _doc_from_mongo(doc: dict): + """Return a TinyDB-compatible Document with ``doc_id`` set to ``_id``. + + Restores the model ``id`` field from MongoDB's ``_id`` and exposes + TinyDB's ``doc_id`` attribute so callers that rely on it continue to + work. + """ + if doc is None: + return None + d = dict(doc) + doc_id = d.pop('_id', None) + if doc_id is not None: + d['id'] = doc_id + return Document(d, doc_id=doc_id) + + def _mongo_filter(self, cond): + """Translate a TinyDB query or dict to a MongoDB filter.""" + return _query_to_mongo_filter(cond) + + def all(self): + return [self._doc_from_mongo(doc) for doc in self._collection().find({})] + + def search(self, cond): + mongo_filter = self._mongo_filter(cond) + if mongo_filter is not None: + cursor = self._collection().find(mongo_filter) + return [self._doc_from_mongo(doc) for doc in cursor] + + # Fallback: fetch all and evaluate the TinyDB query in Python. + docs = list(self._collection().find({})) + matched = _evaluate_in_memory( + [self._doc_from_mongo(doc) for doc in docs], cond + ) + return matched + + def get(self, cond): + mongo_filter = self._mongo_filter(cond) + if mongo_filter is not None: + doc = self._collection().find_one(mongo_filter) + return self._doc_from_mongo(doc) + + docs = list(self._collection().find({})) + for doc in docs: + d = self._doc_from_mongo(doc) + if callable(cond) and cond(d): + return d + return None + + def insert(self, document: dict): + doc = self._doc_to_mongo(document) + result = self._collection().insert_one(doc) + return str(result.inserted_id) + + def insert_multiple(self, documents: list): + if not documents: + return [] + docs = [self._doc_to_mongo(d) for d in documents] + result = self._collection().insert_many(docs) + return [str(iid) for iid in result.inserted_ids] + + def update(self, fields, cond=None, doc_ids=None): + is_callable = callable(fields) + + if doc_ids is not None: + mongo_filter = {'_id': {'$in': list(doc_ids)}} + target_ids = [str(did) for did in doc_ids] + if not target_ids: + return [] + + if is_callable: + # Fetch, apply callable in-memory, and replace each document. + updated_ids = [] + for doc in self._collection().find(mongo_filter): + d = self._doc_from_mongo(doc) + fields(d) + new_doc = self._doc_to_mongo(d) + new_doc.pop('_id', None) + self._collection().update_one( + {'_id': doc['_id']}, {'$set': new_doc} + ) + updated_ids.append(str(doc['_id'])) + return updated_ids + + update_doc = self._doc_to_mongo(fields) or {} + update_doc.pop('_id', None) + update_doc.pop('id', None) + if update_doc: + self._collection().update_many(mongo_filter, {'$set': update_doc}) + return target_ids + + mongo_filter = self._mongo_filter(cond) + if mongo_filter is not None and not is_callable: + update_doc = self._doc_to_mongo(fields) or {} + update_doc.pop('_id', None) + update_doc.pop('id', None) + target_ids = [ + str(doc['_id']) + for doc in self._collection().find(mongo_filter, {'_id': 1}) + ] + if target_ids and update_doc: + self._collection().update_many( + mongo_filter, {'$set': update_doc} + ) + return target_ids + + # Fallback: evaluate the query in-memory and update one at a time. + docs = list(self._collection().find({})) + updated_ids = [] + for doc in docs: + d = self._doc_from_mongo(doc) + match = cond(d) if callable(cond) else (mongo_filter is not None) + if not match: + continue + if is_callable: + fields(d) + new_doc = self._doc_to_mongo(d) + new_doc.pop('_id', None) + self._collection().update_one( + {'_id': doc['_id']}, {'$set': new_doc} + ) + else: + update_doc = self._doc_to_mongo(fields) or {} + update_doc.pop('_id', None) + update_doc.pop('id', None) + if update_doc: + self._collection().update_one( + {'_id': doc['_id']}, {'$set': update_doc} + ) + updated_ids.append(str(doc['_id'])) + return updated_ids + + def remove(self, cond): + mongo_filter = self._mongo_filter(cond) + if mongo_filter is not None: + target_ids = [ + str(doc['_id']) + for doc in self._collection().find(mongo_filter, {'_id': 1}) + ] + if target_ids: + self._collection().delete_many(mongo_filter) + return target_ids + + # Fallback: evaluate the query in-memory and delete one at a time. + docs = list(self._collection().find({})) + removed_ids = [] + for doc in docs: + d = self._doc_from_mongo(doc) + if callable(cond) and cond(d): + self._collection().delete_one({'_id': doc['_id']}) + removed_ids.append(str(doc['_id'])) + return removed_ids + + def truncate(self): + self._collection().delete_many({}) + + def close(self): + # MongoDB clients are shared and long-lived; nothing to close here. + pass + + +# --------------------------------------------------------------------------- +# Collection factory +# --------------------------------------------------------------------------- + +def _make_table(json_path: str, collection_name: str): + if USE_MONGODB: + return MongoLockedTable(collection_name) + db = TinyDB(json_path, indent=2) + return LockedTable(db) + + # Setup DB files next to this module child_path = os.path.join(base_dir, 'children.json') @@ -86,46 +470,116 @@ refresh_tokens_path = os.path.join(base_dir, 'refresh_tokens.json') push_subscriptions_path = os.path.join(base_dir, 'push_subscriptions.json') digest_action_tokens_path = os.path.join(base_dir, 'digest_action_tokens.json') -# Use separate TinyDB instances/files for each collection -_child_db = TinyDB(child_path, indent=2) -_task_db = TinyDB(task_path, indent=2) -_routine_db = TinyDB(routine_path, indent=2) -_routine_items_db = TinyDB(routine_items_path, indent=2) -_routine_schedules_db = TinyDB(routine_schedules_path, indent=2) -_routine_extensions_db = TinyDB(routine_extensions_path, indent=2) -_reward_db = TinyDB(reward_path, indent=2) -_image_db = TinyDB(image_path, indent=2) -_pending_rewards_db = TinyDB(pending_reward_path, indent=2) -_pending_confirmations_db = TinyDB(pending_confirmations_path, indent=2) -_users_db = TinyDB(users_path, indent=2) -_tracking_events_db = TinyDB(tracking_events_path, indent=2) -_child_overrides_db = TinyDB(child_overrides_path, indent=2) -_chore_schedules_db = TinyDB(chore_schedules_path, indent=2) -_task_extensions_db = TinyDB(task_extensions_path, indent=2) -_refresh_tokens_db = TinyDB(refresh_tokens_path, indent=2) -_push_subscriptions_db = TinyDB(push_subscriptions_path, indent=2) -_digest_action_tokens_db = TinyDB(digest_action_tokens_path, indent=2) +# Expose table objects backed by TinyDB or MongoDB based on USE_MONGODB +child_db = _make_table(child_path, 'children') +task_db = _make_table(task_path, 'tasks') +routine_db = _make_table(routine_path, 'routines') +routine_items_db = _make_table(routine_items_path, 'routine_items') +routine_schedules_db = _make_table(routine_schedules_path, 'routine_schedules') +routine_extensions_db = _make_table(routine_extensions_path, 'routine_extensions') +reward_db = _make_table(reward_path, 'rewards') +image_db = _make_table(image_path, 'images') +pending_reward_db = _make_table(pending_reward_path, 'pending_rewards') +pending_confirmations_db = _make_table(pending_confirmations_path, 'pending_confirmations') +users_db = _make_table(users_path, 'users') +tracking_events_db = _make_table(tracking_events_path, 'tracking_events') +child_overrides_db = _make_table(child_overrides_path, 'child_overrides') +chore_schedules_db = _make_table(chore_schedules_path, 'chore_schedules') +task_extensions_db = _make_table(task_extensions_path, 'task_extensions') +refresh_tokens_db = _make_table(refresh_tokens_path, 'refresh_tokens') +push_subscriptions_db = _make_table(push_subscriptions_path, 'push_subscriptions') +digest_action_tokens_db = _make_table(digest_action_tokens_path, 'digest_action_tokens') -# Expose table objects wrapped with locking -child_db = LockedTable(_child_db) -task_db = LockedTable(_task_db) -routine_db = LockedTable(_routine_db) -routine_items_db = LockedTable(_routine_items_db) -routine_schedules_db = LockedTable(_routine_schedules_db) -routine_extensions_db = LockedTable(_routine_extensions_db) -reward_db = LockedTable(_reward_db) -image_db = LockedTable(_image_db) -pending_reward_db = LockedTable(_pending_rewards_db) -pending_confirmations_db = LockedTable(_pending_confirmations_db) -users_db = LockedTable(_users_db) -tracking_events_db = LockedTable(_tracking_events_db) -child_overrides_db = LockedTable(_child_overrides_db) -chore_schedules_db = LockedTable(_chore_schedules_db) -task_extensions_db = LockedTable(_task_extensions_db) -refresh_tokens_db = LockedTable(_refresh_tokens_db) -push_subscriptions_db = LockedTable(_push_subscriptions_db) -digest_action_tokens_db = LockedTable(_digest_action_tokens_db) +# --------------------------------------------------------------------------- +# Index management +# --------------------------------------------------------------------------- + +COLLECTION_INDEXES = { + # NOTE: The model ``id`` field is stored as MongoDB's primary key ``_id``, + # so no separate unique index on ``id`` is needed. Only secondary indexes + # for frequently queried fields are defined here. + 'children': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'tasks': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'routines': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'routine_items': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'routine_schedules': [], + 'routine_extensions': [], + 'rewards': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'images': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'pending_rewards': [ + {'keys': [('child_id', ASCENDING)]}, + ], + 'pending_confirmations': [ + {'keys': [('user_id', ASCENDING)]}, + {'keys': [('child_id', ASCENDING)]}, + {'keys': [('entity_id', ASCENDING), ('entity_type', ASCENDING)]}, + ], + 'users': [], + 'tracking_events': [ + {'keys': [('user_id', ASCENDING)]}, + {'keys': [('child_id', ASCENDING)]}, + {'keys': [('entity_id', ASCENDING), ('entity_type', ASCENDING)]}, + ], + 'child_overrides': [ + {'keys': [('child_id', ASCENDING)]}, + {'keys': [('entity_id', ASCENDING), ('entity_type', ASCENDING)]}, + ], + 'chore_schedules': [ + {'keys': [('user_id', ASCENDING)]}, + {'keys': [('child_id', ASCENDING)]}, + ], + 'task_extensions': [ + {'keys': [('user_id', ASCENDING)]}, + {'keys': [('child_id', ASCENDING)]}, + ], + 'refresh_tokens': [ + {'keys': [('user_id', ASCENDING)]}, + {'keys': [('token', ASCENDING)], 'unique': True, 'sparse': True}, + ], + 'push_subscriptions': [ + {'keys': [('user_id', ASCENDING)]}, + ], + 'digest_action_tokens': [ + {'keys': [('user_id', ASCENDING)]}, + {'keys': [('token', ASCENDING)], 'unique': True, 'sparse': True}, + ], +} + + +def ensure_mongodb_indexes(client=None, db_name=None): + """Create required indexes on all MongoDB collections. + + Safe to call repeatedly: MongoDB treats index creation as idempotent. + """ + if not USE_MONGODB: + return + + client = client or get_mongo_client() + db_name = db_name or _mongo_db_name + db = client[db_name] + + for collection_name, indexes in COLLECTION_INDEXES.items(): + coll = db[collection_name] + for spec in indexes: + keys = spec['keys'] + kwargs = {k: v for k, v in spec.items() if k != 'keys'} + coll.create_index(keys, **kwargs) + + +# Clear test collections at import time so tests start with a clean slate. if os.environ.get('DB_ENV', 'prod') == 'test': child_db.truncate() task_db.truncate() @@ -145,4 +599,3 @@ if os.environ.get('DB_ENV', 'prod') == 'test': refresh_tokens_db.truncate() push_subscriptions_db.truncate() digest_action_tokens_db.truncate() - diff --git a/backend/db/mongo_client.py b/backend/db/mongo_client.py new file mode 100644 index 0000000..9da651f --- /dev/null +++ b/backend/db/mongo_client.py @@ -0,0 +1,113 @@ +# python +"""Lazy MongoDB client factory and database-name helpers. + +The client is intentionally **not** created at module import. Use +``get_mongo_client()`` to obtain a cached singleton. For Gunicorn multi-worker +deployments call ``init_mongo_client()`` from a ``post_fork`` hook so each +worker process owns its own connection pool rather than inheriting the parent +process's client. +""" +import os +import threading +from urllib.parse import urlparse + +from pymongo import MongoClient +from pymongo.uri_parser import parse_uri + + +_mongo_client_lock = threading.Lock() +_mongo_client = None + + +def _create_mongo_client(): + """Build a fail-fast MongoClient from environment variables.""" + uri = os.environ.get('MONGO_URI') + if not uri: + raise RuntimeError( + 'MONGO_URI environment variable is required when USE_MONGODB=true.' + ) + + # mongomock is used for unit/integration tests without a real server. + if uri.lower().startswith('mongomock') or uri.lower() == 'mongomock': + try: + import mongomock + except ImportError as exc: # pragma: no cover - test dependency + raise RuntimeError( + 'mongomock is required for test MongoDB mode. ' + 'Install it with: pip install mongomock' + ) from exc + return mongomock.MongoClient() + + return MongoClient( + uri, + serverSelectionTimeoutMS=5000, + connectTimeoutMS=5000, + maxPoolSize=20, + ) + + +def init_mongo_client(): + """Create a fresh MongoClient and store it as the process singleton. + + Call this from a Gunicorn ``post_fork`` hook so each worker process gets + its own client after forking. It can also be called in tests to reset the + shared client to a known state. + """ + global _mongo_client + with _mongo_client_lock: + _mongo_client = _create_mongo_client() + return _mongo_client + + +def get_mongo_client(): + """Return the cached process-level MongoClient, creating it lazily once.""" + global _mongo_client + if _mongo_client is None: + with _mongo_client_lock: + if _mongo_client is None: + _mongo_client = _create_mongo_client() + return _mongo_client + + +def _db_name_from_uri(uri: str) -> str | None: + """Extract the database name from a MongoDB connection URI, if present.""" + if not uri or uri.lower().startswith('mongomock'): + return None + try: + parsed = parse_uri(uri) + return parsed.get('database') or None + except Exception: + # Fallback to a simple path-based parse for non-standard URIs. + try: + path = urlparse(uri).path + return path.lstrip('/') or None + except Exception: + return None + + +def get_mongo_db_name() -> str: + """Resolve the MongoDB database name from env vars or the connection URI. + + Precedence: + 1. ``MONGO_DB_NAME`` environment variable. + 2. Database name parsed from ``MONGO_URI``. + 3. Default based on ``DATA_ENV`` / ``DB_ENV``: + * prod -> ``chore_db`` + * e2e -> ``chore_db_e2e`` + * test -> ``chore_db_test`` + """ + env_name = os.environ.get('MONGO_DB_NAME') + if env_name: + return env_name + + uri = os.environ.get('MONGO_URI', '') + db_name = _db_name_from_uri(uri) + if db_name: + return db_name + + env = (os.environ.get('DATA_ENV') or os.environ.get('DB_ENV', 'prod')).lower() + if env == 'prod': + return 'chore_db' + if env == 'e2e': + return 'chore_db_e2e' + return 'chore_db_test' diff --git a/backend/db/tracking.py b/backend/db/tracking.py index 02940c4..6ff1714 100644 --- a/backend/db/tracking.py +++ b/backend/db/tracking.py @@ -1,4 +1,5 @@ """Helper functions for tracking events database operations.""" +import itertools import logging from typing import Optional, List from tinydb import Query @@ -8,6 +9,10 @@ from models.tracking_event import TrackingEvent, EntityType, ActionType logger = logging.getLogger(__name__) +# Monotonic sequence used as a deterministic tiebreaker when tracking events +# share the same ``occurred_at``/``created_at`` timestamps (common in tests). +_tracking_event_seq = itertools.count() + def insert_tracking_event(event: TrackingEvent) -> str: """ @@ -20,7 +25,9 @@ def insert_tracking_event(event: TrackingEvent) -> str: The event ID """ try: - tracking_events_db.insert(event.to_dict()) + event_dict = event.to_dict() + event_dict['_seq'] = next(_tracking_event_seq) + tracking_events_db.insert(event_dict) logger.info(f"Tracking event created: {event.action} {event.entity_type} {event.entity_id} for child {event.child_id}") return event.id except Exception as e: @@ -61,12 +68,16 @@ def get_tracking_events_by_child( all_results = tracking_events_db.search(query_condition) total = len(all_results) - # Sort by occurred_at desc, then created_at desc - all_results.sort(key=lambda x: (x.get('occurred_at', ''), x.get('created_at', 0)), reverse=True) - + # Sort by occurred_at desc, then created_at desc, then _seq desc for + # deterministic ordering when timestamps collide (common in fast tests). + all_results.sort( + key=lambda x: (x.get('occurred_at', ''), x.get('created_at', 0), x.get('_seq', 0)), + reverse=True, + ) + paginated = all_results[offset:offset + limit] events = [TrackingEvent.from_dict(r) for r in paginated] - + return events, total @@ -99,11 +110,14 @@ def get_tracking_events_by_user( all_results = tracking_events_db.search(query_condition) total = len(all_results) - all_results.sort(key=lambda x: (x.get('occurred_at', ''), x.get('created_at', 0)), reverse=True) - + all_results.sort( + key=lambda x: (x.get('occurred_at', ''), x.get('created_at', 0), x.get('_seq', 0)), + reverse=True, + ) + paginated = all_results[offset:offset + limit] events = [TrackingEvent.from_dict(r) for r in paginated] - + return events, total diff --git a/backend/gunicorn.conf.py b/backend/gunicorn.conf.py new file mode 100644 index 0000000..7c912c8 --- /dev/null +++ b/backend/gunicorn.conf.py @@ -0,0 +1,17 @@ +"""Gunicorn configuration for the chore/reward Flask backend. + +This file is automatically loaded by Gunicorn when it is started from the +backend directory. It ensures each worker process creates its own MongoDB +client after forking, avoiding shared socket/file-descriptor issues. +""" + + +def post_fork(server, worker): + """Reinitialize the MongoDB client in each worker process after forking.""" + try: + from db.mongo_client import init_mongo_client + init_mongo_client() + except Exception: + # If MongoDB is not configured (USE_MONGODB=false), there is no client + # to initialize; ignore the error silently. + pass diff --git a/backend/main.py b/backend/main.py index a8e887e..ffc8eba 100644 --- a/backend/main.py +++ b/backend/main.py @@ -4,7 +4,6 @@ import os from dotenv import load_dotenv from flask import Flask, request, jsonify -from pymongo import MongoClient from api.admin_api import admin_api from api.auth_api import auth_api @@ -28,6 +27,7 @@ from api.digest_action_api import digest_action_api from config.version import get_full_version from db.default import initializeImages, createDefaultTasks, createDefaultRewards +from db.db import ensure_mongodb_indexes from events.broadcaster import Broadcaster from events.sse import sse_response_for_user, send_to_user from api.utils import get_current_user_id @@ -39,6 +39,10 @@ from utils.state_expiry_scheduler import start_state_expiry_scheduler # Load environment variables load_dotenv() + +# Ensure MongoDB indexes exist when running against MongoDB. +ensure_mongodb_indexes() + # Configure logging once at application startup logging.basicConfig( level=logging.INFO, diff --git a/backend/requirements.txt b/backend/requirements.txt index 1cf34ad..9195c1a 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -26,6 +26,7 @@ iniconfig==2.3.0 itsdangerous==2.2.0 Jinja2==3.1.6 MarkupSafe==3.0.3 +mongomock==4.3.0 multidict==6.7.1 packaging==25.0 pillow==12.0.0 diff --git a/backend/scripts/migrate_to_mongodb.py b/backend/scripts/migrate_to_mongodb.py new file mode 100644 index 0000000..9025ae7 --- /dev/null +++ b/backend/scripts/migrate_to_mongodb.py @@ -0,0 +1,203 @@ +# python +""" +Migrate existing TinyDB JSON files into MongoDB. + +Usage: + cd backend + python -m scripts.migrate_to_mongodb [--dry-run] + +The script reads files from ``data/db/`` (or ``test_data/db/`` when +``DB_ENV=test``), maps each record's ``id`` field to MongoDB's ``_id`` field, +and inserts the records idempotently. TinyDB files are backed up to +``/backups//`` before the first migration run. +""" +import argparse +import json +import os +import shutil +import sys +from datetime import datetime +from pathlib import Path + +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) + +from config.paths import get_database_dir +from db.db import COLLECTION_INDEXES, ensure_mongodb_indexes +from db.mongo_client import get_mongo_client, get_mongo_db_name + + +# Map TinyDB JSON filenames to MongoDB collection names. +COLLECTION_FILE_MAP = { + 'children.json': 'children', + 'tasks.json': 'tasks', + 'routines.json': 'routines', + 'routine_items.json': 'routine_items', + 'routine_schedules.json': 'routine_schedules', + 'routine_extensions.json': 'routine_extensions', + 'rewards.json': 'rewards', + 'images.json': 'images', + 'pending_rewards.json': 'pending_rewards', + 'pending_confirmations.json': 'pending_confirmations', + 'users.json': 'users', + 'tracking_events.json': 'tracking_events', + 'child_overrides.json': 'child_overrides', + 'chore_schedules.json': 'chore_schedules', + 'task_extensions.json': 'task_extensions', + 'refresh_tokens.json': 'refresh_tokens', + 'push_subscriptions.json': 'push_subscriptions', + 'digest_action_tokens.json': 'digest_action_tokens', +} + + +def _load_tinydb_records(path: str) -> list[dict]: + """Load all records from a TinyDB JSON file.""" + with open(path, 'r', encoding='utf-8') as f: + data = json.load(f) + default_table = data.get('_default', {}) + return list(default_table.values()) + + +def _doc_to_mongo(doc: dict) -> dict: + """Map the model ``id`` field to MongoDB's ``_id`` field. + + The original ``id`` field is removed so documents do not store both + ``_id`` and ``id`` with identical values. + """ + mongo_doc = dict(doc) + if 'id' in mongo_doc: + mongo_doc['_id'] = mongo_doc.pop('id') + return mongo_doc + + +def migrate(dry_run: bool = False) -> dict: + """Migrate TinyDB files to MongoDB and return a per-collection summary.""" + db_dir = get_database_dir() + if not os.path.isdir(db_dir): + raise FileNotFoundError(f'Database directory does not exist: {db_dir}') + + client = get_mongo_client() + db_name = get_mongo_db_name() + db = client[db_name] + + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + backup_dir = os.path.join(db_dir, 'backups', timestamp) + + if not dry_run: + os.makedirs(backup_dir, exist_ok=True) + ensure_mongodb_indexes(client=client, db_name=db_name) + + summary: dict[str, dict] = {} + + for filename, collection_name in COLLECTION_FILE_MAP.items(): + path = os.path.join(db_dir, filename) + if not os.path.exists(path): + summary[collection_name] = { + 'source_file': filename, + 'total': 0, + 'migrated': 0, + 'skipped': 0, + 'status': 'missing', + } + continue + + records = _load_tinydb_records(path) + + if not dry_run: + shutil.copy2(path, backup_dir) + + collection = db[collection_name] + to_insert: list[dict] = [] + skipped = 0 + + for record in records: + doc_id = record.get('id') + if not doc_id: + skipped += 1 + continue + + if not dry_run: + existing = collection.find_one({'_id': doc_id}) + if existing: + skipped += 1 + continue + + to_insert.append(_doc_to_mongo(record)) + + if not dry_run and to_insert: + try: + collection.insert_many(to_insert, ordered=False) + except Exception as exc: # pragma: no cover - defensive + print( + f' Warning: error inserting into {collection_name}: {exc}', + file=sys.stderr, + ) + raise + + summary[collection_name] = { + 'source_file': filename, + 'total': len(records), + 'migrated': len(to_insert), + 'skipped': skipped, + 'status': 'migrated' if not dry_run else 'dry-run', + } + + return summary + + +def _print_summary(summary: dict) -> None: + """Print a human-readable migration summary.""" + print('\nMigration Summary') + print('-' * 70) + print(f'{"Collection":<30}{"Total":>8}{"Migrated":>10}{"Skipped":>10}{"Status":>10}') + print('-' * 70) + total_records = 0 + total_migrated = 0 + total_skipped = 0 + for collection_name, info in summary.items(): + print( + f'{collection_name:<30}' + f'{info["total"]:>8}' + f'{info["migrated"]:>10}' + f'{info["skipped"]:>10}' + f'{info["status"]:>10}' + ) + total_records += info['total'] + total_migrated += info['migrated'] + total_skipped += info['skipped'] + print('-' * 70) + print( + f'{"TOTAL":<30}' + f'{total_records:>8}' + f'{total_migrated:>10}' + f'{total_skipped:>10}' + ) + + +def main(): + parser = argparse.ArgumentParser( + description='Migrate TinyDB JSON files to MongoDB.' + ) + parser.add_argument( + '--dry-run', + action='store_true', + help='Analyze files and report counts without writing to MongoDB.', + ) + args = parser.parse_args() + + if os.environ.get('USE_MONGODB', 'false').lower() != 'true': + print('Set USE_MONGODB=true to run the migration.', file=sys.stderr) + sys.exit(1) + + if not os.environ.get('MONGO_URI'): + print('MONGO_URI is required when USE_MONGODB=true.', file=sys.stderr) + sys.exit(1) + + if args.dry_run: + print('Dry run: no data will be written to MongoDB.') + + summary = migrate(dry_run=args.dry_run) + _print_summary(summary) + + +if __name__ == '__main__': + main() diff --git a/backend/scripts/run_integration_tests.ps1 b/backend/scripts/run_integration_tests.ps1 new file mode 100644 index 0000000..dec18f7 --- /dev/null +++ b/backend/scripts/run_integration_tests.ps1 @@ -0,0 +1,79 @@ +<# +.SYNOPSIS + Run the MongoDB adapter integration tests against a local Docker MongoDB container. + +.DESCRIPTION + Starts a temporary MongoDB container, runs a targeted pytest suite with + USE_MONGODB=true, then stops and removes the container. + +.EXAMPLE + cd backend + .\scripts\run_integration_tests.ps1 +#> +[CmdletBinding()] +param( + [string]$ContainerName = 'chore-db-integration-test', + [int]$HostPort = 27017, + [string]$Image = 'mongo:8', + [string]$DbName = 'chore_db_test', + [string]$TestPath = 'tests/test_mongo_adapter.py' +) + +$ErrorActionPreference = 'Stop' + +$mongoUri = "mongodb://localhost:${HostPort}/${DbName}" + +function Test-ContainerRunning { + $containers = docker ps --filter "name=$ContainerName" --format '{{.Names}}' 2>$null + return $containers -contains $ContainerName +} + +function Wait-MongoReady { + param([int]$TimeoutSeconds = 30) + $start = Get-Date + while (((Get-Date) - $start).TotalSeconds -lt $TimeoutSeconds) { + try { + $null = docker exec $ContainerName mongosh --eval 'db.adminCommand({ ping: 1 })' --quiet 2>$null + if ($LASTEXITCODE -eq 0) { + return + } + } catch { + # Container or mongosh may not be ready yet. + } + Start-Sleep -Seconds 1 + } + throw "MongoDB container did not become ready within ${TimeoutSeconds} seconds." +} + +# Clean up any leftover container from a previous aborted run. +if (Test-ContainerRunning) { + Write-Host "Removing existing container '$ContainerName'..." + docker rm -f $ContainerName | Out-Null +} + +Write-Host "Starting MongoDB container '$ContainerName' on port $HostPort..." +docker run -d ` + --name $ContainerName ` + -p "${HostPort}:27017" ` + $Image | Out-Null + +try { + Wait-MongoReady + Write-Host "MongoDB is ready. Running integration tests..." + + $env:USE_MONGODB = 'true' + $env:MONGO_URI = $mongoUri + $env:MONGO_DB_NAME = $DbName + $env:DB_ENV = 'test' + $env:DATA_ENV = 'test' + + pytest $TestPath + if ($LASTEXITCODE -ne 0) { + throw "Integration tests failed with exit code $LASTEXITCODE." + } +} finally { + Write-Host "Stopping and removing container '$ContainerName'..." + docker rm -f $ContainerName | Out-Null +} + +Write-Host "Integration tests complete." diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 20deb56..ba97ced 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -1,5 +1,7 @@ import os os.environ['DB_ENV'] = 'test' +os.environ['USE_MONGODB'] = 'true' +os.environ['MONGO_URI'] = 'mongomock' os.environ.setdefault('SECRET_KEY', 'test-secret-key') os.environ.setdefault('REFRESH_TOKEN_EXPIRY_DAYS', '90') os.environ.setdefault('DIGEST_TOKEN_SECRET', 'test-digest-secret') @@ -18,5 +20,11 @@ TEST_REFRESH_TOKEN_EXPIRY_DAYS = 90 @pytest.fixture(scope="session", autouse=True) def set_test_db_env(): os.environ['DB_ENV'] = 'test' + os.environ['USE_MONGODB'] = 'true' + os.environ['MONGO_URI'] = 'mongomock' os.environ['SECRET_KEY'] = TEST_SECRET_KEY - os.environ['REFRESH_TOKEN_EXPIRY_DAYS'] = str(TEST_REFRESH_TOKEN_EXPIRY_DAYS) \ No newline at end of file + os.environ['REFRESH_TOKEN_EXPIRY_DAYS'] = str(TEST_REFRESH_TOKEN_EXPIRY_DAYS) + # Ensure indexes are created once for the test session. This is safe to + # call repeatedly because MongoDB treats index creation as idempotent. + from db.db import ensure_mongodb_indexes + ensure_mongodb_indexes() diff --git a/backend/tests/test_mongo_adapter.py b/backend/tests/test_mongo_adapter.py new file mode 100644 index 0000000..2f6a69c --- /dev/null +++ b/backend/tests/test_mongo_adapter.py @@ -0,0 +1,201 @@ +import os +import pytest +from tinydb import Query + +from db.db import ( + MongoLockedTable, + _query_to_mongo_filter, + child_db, + task_db, + users_db, + refresh_tokens_db, +) +from db.mongo_client import get_mongo_client, get_mongo_db_name + + +# All tests in this module require the mongomock-backed MongoDB adapter. +pytestmark = pytest.mark.skipif( + os.environ.get('USE_MONGODB', 'false').lower() != 'true', + reason='MongoDB adapter tests require USE_MONGODB=true', +) + + +@pytest.fixture(autouse=True) +def clean_mongo_collections(): + """Truncate relevant collections before each test.""" + child_db.truncate() + task_db.truncate() + users_db.truncate() + refresh_tokens_db.truncate() + yield + + +class TestQueryTranslation: + """Unit tests for TinyDB Query -> MongoDB filter translation.""" + + def test_simple_equality(self): + q = Query() + assert _query_to_mongo_filter(q.id == 'abc') == {'_id': {'$eq': 'abc'}} + + def test_field_other_than_id(self): + q = Query() + assert _query_to_mongo_filter(q.user_id == 'u1') == { + 'user_id': {'$eq': 'u1'} + } + + def test_and_query(self): + q = Query() + mongo_filter = _query_to_mongo_filter( + (q.id == 'abc') & (q.user_id == 'u1') + ) + assert mongo_filter == {'_id': {'$eq': 'abc'}, 'user_id': {'$eq': 'u1'}} + + def test_or_query(self): + q = Query() + mongo_filter = _query_to_mongo_filter( + (q.user_id == 'u1') | (q.user_id == None) # noqa: E711 + ) + assert mongo_filter == { + '$or': [ + {'user_id': {'$eq': 'u1'}}, + {'user_id': {'$eq': None}}, + ] + } + + def test_and_with_nested_or(self): + q = Query() + mongo_filter = _query_to_mongo_filter( + (q.id == 'abc') & ((q.user_id == 'u1') | (q.user_id == None)) # noqa: E711 + ) + assert mongo_filter == { + '_id': {'$eq': 'abc'}, + '$or': [ + {'user_id': {'$eq': 'u1'}}, + {'user_id': {'$eq': None}}, + ], + } + + +class TestMongoCrud: + """CRUD tests against the mongomock-backed MongoLockedTable.""" + + def test_insert_maps_id_to_underscore_id(self): + child_db.insert({'id': 'c1', 'name': 'Alice', 'age': 8}) + + raw = get_mongo_client()[get_mongo_db_name()]['children'].find_one( + {'_id': 'c1'} + ) + assert raw is not None + assert raw['_id'] == 'c1' + assert 'id' not in raw + assert raw['name'] == 'Alice' + + def test_get_returns_document_without_underscore_id(self): + child_db.insert({'id': 'c1', 'name': 'Alice', 'age': 8}) + + doc = child_db.get(Query().id == 'c1') + assert doc is not None + assert doc['id'] == 'c1' + assert doc['name'] == 'Alice' + assert '_id' not in doc + + def test_get_none_when_missing(self): + assert child_db.get(Query().id == 'missing') is None + + def test_search_with_query(self): + child_db.insert({'id': 'c1', 'name': 'Alice', 'user_id': 'u1'}) + child_db.insert({'id': 'c2', 'name': 'Bob', 'user_id': 'u1'}) + child_db.insert({'id': 'c3', 'name': 'Carol', 'user_id': 'u2'}) + + results = child_db.search(Query().user_id == 'u1') + assert len(results) == 2 + assert {r['id'] for r in results} == {'c1', 'c2'} + + def test_search_with_or(self): + task_db.insert({'id': 't1', 'name': 'Default', 'user_id': None}) + task_db.insert({'id': 't2', 'name': 'User task', 'user_id': 'u1'}) + + q = Query() + results = task_db.search((q.user_id == 'u1') | (q.user_id == None)) # noqa: E711 + assert len(results) == 2 + + def test_update_modifies_matching_documents(self): + child_db.insert({'id': 'c1', 'name': 'Alice', 'points': 0}) + child_db.insert({'id': 'c2', 'name': 'Bob', 'points': 0}) + + modified = child_db.update({'points': 10}, Query().id == 'c1') + # TinyDB returns a list of updated document ids; the adapter mirrors that. + assert modified == ['c1'] + + doc = child_db.get(Query().id == 'c1') + assert doc['points'] == 10 + + other = child_db.get(Query().id == 'c2') + assert other['points'] == 0 + + def test_update_does_not_overwrite_id(self): + child_db.insert({'id': 'c1', 'name': 'Alice'}) + + child_db.update({'id': 'c2', 'name': 'Alice Smith'}, Query().id == 'c1') + # The id field must remain unchanged; update should have stripped id. + assert child_db.get(Query().id == 'c1')['name'] == 'Alice Smith' + assert child_db.get(Query().id == 'c2') is None + + def test_remove_deletes_matching_documents(self): + child_db.insert({'id': 'c1', 'name': 'Alice'}) + child_db.insert({'id': 'c2', 'name': 'Bob'}) + + deleted = child_db.remove(Query().id == 'c1') + # TinyDB returns a list of removed document ids; the adapter mirrors that. + assert deleted == ['c1'] + + assert child_db.get(Query().id == 'c1') is None + assert child_db.get(Query().id == 'c2') is not None + + def test_all_returns_all_documents(self): + child_db.insert({'id': 'c1', 'name': 'Alice'}) + child_db.insert({'id': 'c2', 'name': 'Bob'}) + + docs = child_db.all() + assert len(docs) == 2 + assert all('_id' not in d for d in docs) + + def test_truncate_removes_all_documents(self): + child_db.insert({'id': 'c1', 'name': 'Alice'}) + child_db.truncate() + assert child_db.all() == [] + + def test_insert_multiple(self): + ids = child_db.insert_multiple([ + {'id': 'c1', 'name': 'Alice'}, + {'id': 'c2', 'name': 'Bob'}, + ]) + assert sorted(ids) == ['c1', 'c2'] + assert len(child_db.all()) == 2 + + def test_unique_token_index(self): + refresh_tokens_db.insert({'id': 'r1', 'token': 'abc', 'user_id': 'u1'}) + refresh_tokens_db.insert({'id': 'r2', 'token': 'def', 'user_id': 'u1'}) + + # mongomock does not enforce unique indexes by default, but we verify + # both records are readable. + assert refresh_tokens_db.get(Query().token == 'abc') is not None + assert refresh_tokens_db.get(Query().token == 'def') is not None + + def test_user_id_secondary_index_is_created(self): + # Insert and query via the secondary index path used by the app. + users_db.insert({'id': 'u1', 'email': 'a@example.com', 'verified': True}) + users_db.insert({'id': 'u2', 'email': 'b@example.com', 'verified': False}) + + found = users_db.search(Query().verified == True) # noqa: E712 + assert len(found) == 1 + assert found[0]['id'] == 'u1' + + +class TestAdapterApi: + """Tests that the adapter exposes the expected LockedTable-compatible API.""" + + def test_close_is_noop(self): + # Existing cleanup fixtures call ``*_db.close()``; ensure it does not + # raise for the MongoDB-backed adapter. + child_db.close() diff --git a/frontend/.env.test b/frontend/.env.test new file mode 100644 index 0000000..f6178cf --- /dev/null +++ b/frontend/.env.test @@ -0,0 +1,4 @@ +# MongoDB configuration for end-to-end tests. +# Uses mongomock so E2E tests do not require a running MongoDB server. +USE_MONGODB=true +MONGO_URI=mongomock diff --git a/frontend/playwright.config.ts b/frontend/playwright.config.ts index fbcc78a..87e41cf 100644 --- a/frontend/playwright.config.ts +++ b/frontend/playwright.config.ts @@ -8,12 +8,9 @@ import { } from './e2e/e2e-constants' /** - * Read environment variables from file. - * https://github.com/motdotla/dotenv + * E2E MongoDB configuration is hardcoded below. Developers can override + * values via environment variables; see .env.test for an example. */ -// import dotenv from 'dotenv'; -// import path from 'path'; -// dotenv.config({ path: path.resolve(__dirname, '.env') }); /** * See https://playwright.dev/docs/test-configuration. @@ -288,6 +285,8 @@ export default defineConfig({ 'BNKkHdq45uLigohSG7c1TwlAo7ETncoRVLQK02LxHgu2P1DgSJD9njRMfbbzUsaTQGllvLBz7An1WiWsNYQhvKE', VAPID_PRIVATE_KEY: 'jNiZJT0UO4H861KmnCt874Fg6p5jDAyYKS4V2MZf8bQ', PROCESS_PLATFORM: process.platform, + USE_MONGODB: process.env.USE_MONGODB || 'true', + MONGO_URI: process.env.MONGO_URI || 'mongomock', }, }, ], diff --git a/specs/feat-database-migration.md b/specs/feat-database-migration.md new file mode 100644 index 0000000..ebe7488 --- /dev/null +++ b/specs/feat-database-migration.md @@ -0,0 +1,164 @@ +# Feature: Migrate Backend Persistence from TinyDB to MongoDB Atlas + +## Overview + +**Goal:** Replace the TinyDB JSON-file persistence layer in the Flask backend with MongoDB Atlas while keeping the existing API surface, model definitions, and test infrastructure largely unchanged. Introduce a migration script that exports existing TinyDB data into MongoDB. + +**User Story:** As a developer and operator, I want the application to run on a managed, scalable document database so that concurrent access, multi-worker deployments, and production hosting are reliable. + +**Rules:** +- Backend routes and response shapes must remain unchanged. +- Python models in `backend/models/` keep their `from_dict()` / `to_dict()` interface. +- Every mutation must continue to emit SSE events through the existing broadcaster. +- The migration must be reversible (keep TinyDB files as a backup) and idempotent. +- All existing tests must continue to pass; new MongoDB-backed tests must be added. + +--- + +## Data Model Changes + +### Backend Model +- No changes to dataclass fields or `to_dict()` / `from_dict()` output. +- The model's `id` field is stored as MongoDB's `_id` primary key. The original `id` field is removed from storage so documents do not contain duplicate values. +- On read, `_id` is mapped back to `id` and the document is wrapped as a TinyDB-compatible `Document` with `doc_id` set. +- `created_at` and `updated_at` remain epoch floats. + +### Frontend Model +- No changes. The API contract is preserved. + +--- + +## Backend Implementation + +### 1. Database client initialization +- Create `backend/db/mongo_client.py`. +- Do **not** create a `MongoClient` at module import; it is created lazily on first access. +- `get_mongo_client()` returns a cached process-level singleton so tests, scripts, and Flask requests all share the same client. +- For Gunicorn multi-worker deployments, `backend/gunicorn.conf.py` defines a `post_fork` hook that calls `init_mongo_client()` so each worker process gets its own client after forking. +- Fail-fast settings for development: + ```python + MongoClient( + os.getenv("MONGO_URI"), + serverSelectionTimeoutMS=5000, + connectTimeoutMS=5000, + maxPoolSize=20, + ) + ``` + +### 2. Collection mapping +- One MongoDB collection per existing TinyDB file: + - `children` ← `children.json` + - `tasks` ← `tasks.json` + - `routines` ← `routines.json` + - `routine_items` ← `routine_items.json` + - `routine_schedules` ← `routine_schedules.json` + - `routine_extensions` ← `routine_extensions.json` + - `rewards` ← `rewards.json` + - `images` ← `images.json` + - `pending_rewards` ← `pending_rewards.json` (legacy) + - `pending_confirmations` ← `pending_confirmations.json` + - `users` ← `users.json` + - `tracking_events` ← `tracking_events.json` + - `child_overrides` ← `child_overrides.json` + - `chore_schedules` ← `chore_schedules.json` + - `task_extensions` ← `task_extensions.json` + - `refresh_tokens` ← `refresh_tokens.json` + - `push_subscriptions` ← `push_subscriptions.json` + - `digest_action_tokens` ← `digest_action_tokens.json` + +### 3. `LockedTable` MongoDB adapter +- Refactor `backend/db/db.py` so that each exported `*_db` object continues to expose the same methods: `search(...)`, `get(...)`, `all()`, `insert(...)`, `insert_multiple(...)`, `update(...)`, `remove(...)`, `truncate()`. +- Implement a new `MongoLockedTable` class that mirrors the existing `LockedTable` API but delegates to a MongoDB collection. +- Keep `LockedTable` (TinyDB) as a fallback selectable by `USE_MONGODB=true|false` (default `false`). +- Store the model `id` field as MongoDB `_id` on writes; do not keep a duplicate `id` field in storage. +- Restore the model `id` field from `_id` on reads and return TinyDB-compatible `Document` objects with `doc_id` set so existing callers that rely on `doc_id` continue to work. +- Support `update(fields, doc_ids=[...])` and callable updater functions (`update(lambda rec: ..., cond)`) to match TinyDB behavior. +- Reuse function names where possible to minimize API file changes. + +### 4. Indexes +- Create indexes on application startup (`backend/main.py`) and in the pytest session fixture. Index creation is intentionally **not** done at module import time so the `MongoClient` remains lazily initialized. +- Required indexes: + - Unique primary key is provided by MongoDB `_id` (mapped from model `id`); no separate `id` index is created. + - `user_id` — secondary on `children`, `tasks`, `routines`, `routine_items`, `rewards`, `images`, `tracking_events`, `pending_confirmations`, `chore_schedules`, `task_extensions`, `refresh_tokens`, `push_subscriptions`, `digest_action_tokens`. + - `child_id` — secondary on `child_overrides`, `chore_schedules`, `task_extensions`, `pending_confirmations`, `tracking_events`. + - `entity_id` + `entity_type` — compound secondary on `pending_confirmations`, `tracking_events`, `child_overrides`. + - `token` — unique on `refresh_tokens` and `digest_action_tokens`. + +### 5. Migration script +- Create `backend/scripts/migrate_to_mongodb.py`. +- Reads each TinyDB JSON file in `data/db/` (or `test_data/db/` when `DB_ENV=test`). +- Inserts each `_default` record into the corresponding MongoDB collection, mapping TinyDB integer keys to the record's own `id` field. +- Skips records that already exist by `id` (idempotent). +- Backs up TinyDB files to `data/db/backups//` before first run. +- Prints a summary of migrated records per collection. +- Usage: + ```bash + cd backend + python -m scripts.migrate_to_mongodb [--dry-run] + ``` + +### 6. Environment variables +- `MONGO_URI` — required when `USE_MONGODB=true`. +- `MONGO_DB_NAME` — optional database name; defaults parsed from `MONGO_URI` or falls back to `chore_db`/`chore_db_test`/`chore_db_e2e` based on `DB_ENV`/`DATA_ENV`. +- `USE_MONGODB` — `true` to use MongoDB backend; `false` to keep TinyDB. + +### 7. Default data initialization +- `backend/db/default.py` (`initializeImages`, `createDefaultTasks`, `createDefaultRewards`) must work with the new MongoDB-backed `*_db` objects. No API changes. + +### 8. Scheduler compatibility +- Background schedulers (`account_deletion_scheduler`, `digest_scheduler`, etc.) read/write through the same `*_db` objects, so they require no changes once the adapter is in place. + +### 9. Tracking event ordering +- `db/tracking.py` adds a hidden monotonic `_seq` field to tracking events on insert and sorts by `(occurred_at, created_at, _seq)` descending. This guarantees deterministic ordering when consecutive events share the same timestamp (common in fast tests). + +### 10. Gunicorn / Docker deployment +- `backend/gunicorn.conf.py` provides the `post_fork` hook required by the wiki plan. +- `backend/Dockerfile` now loads the config via `-c gunicorn.conf.py`. + +--- + +## Backend Tests + +- [x] Add `mongomock` to test dependencies and create `backend/tests/test_mongo_adapter.py` that verifies CRUD operations against `mongomock`. +- [x] Update `backend/tests/conftest.py` to set `USE_MONGODB=true` and point `MONGO_URI` at `mongomock` for the default test run. +- [x] Ensure every existing API test still passes with the MongoDB adapter by running `pytest tests/`. +- [x] Add integration test script `backend/scripts/run_integration_tests.ps1` that starts a local MongoDB Docker container, runs a targeted pytest suite, and tears down the container. +- [x] Add `frontend/.env.test` with MongoDB configuration and update Playwright webServer env to pass `USE_MONGODB`/`MONGO_URI` to the Flask backend; E2E database is reset via the existing `/auth/e2e-seed` endpoint. + +--- + +## Frontend Implementation + +- No frontend changes are required. The API contract remains unchanged. + +## Frontend Tests + +- [ ] Run `npx playwright test` and confirm E2E tests pass against the MongoDB-backed Flask backend with the isolated `chore_db_e2e` database. + +--- + +## Future Considerations + +- After a production burn-in period, remove the TinyDB fallback and `LockedTable` code. +- Add database connection health-check endpoint. +- Consider MongoDB transactions for multi-document operations (e.g., child deletion cascades). + +--- + +## Acceptance Criteria (Definition of Done) + +### Backend +- [x] `backend/db/db.py` exports MongoDB-backed `*_db` objects when `USE_MONGODB=true`. +- [x] `backend/db/mongo_client.py` implements lazy client initialization and a Gunicorn-compatible init hook. +- [x] All existing API endpoints continue to work without route or response changes. +- [x] `backend/scripts/migrate_to_mongodb.py` migrates existing TinyDB JSON files idempotently. +- [x] Required unique and secondary indexes are created on application startup or migration. +- [x] Unit tests pass with `mongomock` (`pytest tests/`). +- [ ] Integration tests pass against a local Docker MongoDB container (requires a running Docker daemon; script verified to start/stop container when daemon is available). + +### Frontend +- [x] E2E setup and representative parent-mode smoke tests pass against the isolated `chore_db_e2e` mongomock database, with the DB reset via `/auth/e2e-seed`. + +### Operations +- [x] `MONGO_URI` and optional `MONGO_DB_NAME` environment variables are documented. +- [x] Rollback instructions exist to switch back to TinyDB by setting `USE_MONGODB=false`.