# repo_core

**Modular Python repository framework with multi-backend support.**

`repo_core` provides a unified abstraction layer over PostgreSQL, Elasticsearch, and Apache Doris. It combines domain-driven design with built-in validation, schema serialization, lifecycle hooks, and Alembic-powered migrations — so your service layer never touches raw queries.

## Features

- **Multi-backend** — PostgreSQL (CRUD + raw SQL), Elasticsearch (search + index mgmt), Apache Doris (warehouse/OLAP)
- **Auto-wiring** — Repositories auto-create `db_core` clients from environment variables; no manual wiring needed
- **Domain-driven** — One folder per entity: model, schema, validator, repository
- **Pydantic v2 models** — `DomainModel` base with `from_db`/`to_db`/`from_search` serialization and backend-specific field filters
- **Request/Response/Patch schemas** — Separate concerns for input validation, output serialization, and partial updates
- **Lifecycle hooks** — `@hook("before_create")`, `@hook("after_save")`, etc. with async `HookRegistry`
- **Validator pipeline** — Chainable validators (`RequiredValidator`, `MaxLengthValidator`, custom)
- **PostgreSQL migrations** — Fluent ``MigrationBuilder`` DSL, version files in ``migrations/versions/postgres/``, ``python -m repo_core.migrations upgrade``
- **Doris warehouse migrations** — Same builder with dialect support, partitioning, properties, ``python -m repo_core.migrations doris-upgrade``
- **Async context manager** — `async with repo:` auto-connects/disconnects all backends

## Installation

```bash
# From the repo_core directory
pip install -e .
```

### Dependencies

- **Python** ≥ 3.11
- **pydantic** ≥ 2.0 (with email & timezone extras)
- **SQLAlchemy** ≥ 2.0 (with asyncio)
- **db_core** — local sibling library providing `PostgresClient`, `ElasticsearchClient`

## Quick Start

```python
from repo_core import BaseRepository, DomainModel, hook

# 1. Define your domain model
class User(DomainModel):
    name: str
    email: str

# 2. Create a repository — auto-wired from environment variables
class UserRepository(BaseRepository[User]):
    TABLE = "users"

    @hook("before_create")
    async def normalize_email(self, data: dict) -> None:
        data["email"] = data["email"].lower().strip()

# 3. Use — no manual client wiring needed
repo = UserRepository(model=User)

async with repo:
    user = await repo.insert_one("users", {"name": "Alice", "email": "ALICE@EXAMPLE.COM"})
    print(user.email)  # alice@example.com

# Explicit override for tests
from db_core import create_postgres_client, QueryBuilder

db = create_postgres_client(url="postgresql+asyncpg://user:pass@localhost/mydb")
repo = UserRepository(model=User, db=QueryBuilder(db, ""))
```

## Architecture

```text
repo_core/
├── __init__.py              # Public API re-exports
├── base/                    # All framework base classes
│   ├── __init__.py
│   ├── repository.py        # BaseRepository[T] — auto-wires db_core clients
│   ├── models.py            # DomainModel (Pydantic v2 base)
│   ├── schemas.py           # BaseRequestSchema, BaseResponseSchema, BasePatchSchema
│   └── validators.py        # BaseValidator, ValidatorPipeline, built-in validators
├── events/                  # HookRegistry + @hook decorator
│   ├── __init__.py
│   └── hooks.py
├── migrations/              # MigrationBuilder-based migration system
│   ├── builder.py           # MigrationBuilder fluent DSL
│   ├── runner.py            # Migration + MigrationRunner (schema_migrations table)
│   ├── discovery.py         # Auto-discovers version files, sorts by filename
│   ├── registry.py          # Thin wrapper — delegates to discovery
│   ├── __main__.py          # CLI: upgrade, downgrade, status, generate, doris-*
│   └── versions/
│       ├── postgres/        # PostgreSQL version files (0001_*, 0002_*, …)
│       └── doris/           # Doris warehouse version files (0001_*, 0002_*, …)

## Core Concepts

### BaseRepository

The central class that wires backends together via `db_core` clients. It's generic over your domain model type and auto-wires from environment variables:

```python
class ArticleRepository(BaseRepository[Article]):
    TABLE = "articles"
    _backends = {"db"}  # PostgreSQL only — the default
```

**DB CRUD methods**: `find_by_id`, `first`, `find`, `all`, `count`, `exists`, `insert_one`, `insert_many`, `save`, `update`, `update_by_id`, `remove`, `remove_by_id`

**Search methods**: `search_find`, `search_first`, `search_get`, `search_count`

**Warehouse methods**: `warehouse_find`, `warehouse_first`

**Scoping helpers**: `table(name)` → DB `QueryBuilder`, `search_table(index)` → search `QueryBuilder`, `warehouse_table(name)` → Doris `QueryBuilder`

**`db_*` aliases** exist for all DB methods (`db_find`, `db_insert_one`, `db_remove_by_id`) — use when subclass methods could shadow base names.

### Auto-Wiring

Repositories automatically create `db_core` clients from environment variables via lazy singletons. No manual `QueryBuilder` or `DatabaseBackend` needed:

```python
# Auto-wired — reads DATABASE_URL from env
repo = TelegramAccountRepository(model=TelegramAccount)

async with repo:
    accounts = await repo.all("telegram_accounts")

# Explicit override for tests
from db_core import QueryBuilder
repo = MyRepo(model=MyModel, db=mock_query_builder)
```

Declare which backends you need via `_backends`:

```python
class MyRepo(BaseRepository[MyModel]):
    TABLE = "my_table"
    _backends = {"db"}                     # PostgreSQL only (default)
    # _backends = {"db", "search"}         # PG + Elasticsearch
    # _backends = {"db", "doris"}          # PG + Doris warehouse
    # _backends = {"db", "search", "doris"} # all three
```

### Lifecycle Hooks

```python
from repo_core import hook

class ArticleRepository(BaseRepository[Article]):
    @hook("before_create")
    async def validate_title(self, data: dict) -> None:
        if not data.get("title"):
            raise ValueError("Title is required")

    @hook("after_create")
    async def index_in_search(self, instance: Article) -> None:
        await self.search_create("articles", instance.to_search())
```

**Standard events**: `before_create`, `after_create`, `before_update`, `after_update`, `before_save`, `after_save`, `before_delete`, `after_delete`, `before_find`, `after_find`

### Validators

```python
from repo_core import ValidatorPipeline, RequiredValidator, MaxLengthValidator

pipeline = ValidatorPipeline(
    RequiredValidator(field="title"),
    MaxLengthValidator(255, field="title"),
)
clean_title = await pipeline.validate(raw_title)
```

### Models & Serialization

```python
from repo_core import DomainModel

class Article(DomainModel):
    title: str
    body: str | None = None
    published: bool = False

# From database row
article = Article.from_db({"id": 1, "title": "Hello", "body": "..."})

# To database dict (full)
row = article.to_db()

# Partial update (only set fields)
partial = article.to_db_partial()
```

### Schemas

```python
from repo_core import BaseRequestSchema, BaseResponseSchema, BasePatchSchema

class CreateArticleRequest(BaseRequestSchema):
    title: str
    body: str | None = None

class ArticleResponse(BaseResponseSchema):
    id: int
    title: str
    body: str | None
    published: bool

class PatchArticleRequest(BasePatchSchema):
    title: str | None = None
    body: str | None = None
```

### Migrations

Fluent ``MigrationBuilder`` DSL — no Alembic, no model introspection, no hand-written DDL.

**How it works:**

1. Define schema explicitly with ``MigrationBuilder("users").uuid("id")...create_table()``
2. Add it to a version file in ``migrations/versions/postgres/`` (or ``migrations/versions/doris/`` for warehouse tables)
3. The ``MigrationRunner`` reads the ``schema_migrations`` table, applies only pending migrations

```python
from repo_core.migrations.builder import MigrationBuilder
from repo_core.migrations.runner import Migration

# Create a table
migration = Migration(
    version="article_0001",
    description="Create articles table",
    up=(
        MigrationBuilder("articles")
        .bigint("id", primary_key=True, nullable=False)
        .text("title", nullable=False)
        .text("body", nullable=True)
        .boolean("published", nullable=False, default=False)
        .timestamp("created_at", nullable=False, default="now()")
        .timestamp("updated_at", nullable=False, default="now()")
        .create_table()
    ),
    down=MigrationBuilder("articles").drop_table(),
)

# Alter a table
migration = Migration(
    version="article_0002",
    description="Add subtitle to articles",
    up=(
        MigrationBuilder("articles")
        .add_column("subtitle", "TEXT", nullable=True)
        .alter_table()
    ),
    down=(
        MigrationBuilder("articles")
        .drop_column("subtitle")
        .alter_table()
    ),
)

# Doris warehouse table with partitioning
migration = Migration(
    version="analytics_doris_0001",
    description="Create analytics warehouse table",
    dialect="doris",
    up=(
        MigrationBuilder("analytics", dialect="doris")
        .bigint("event_id", nullable=False)
        .date("event_date", nullable=False)
        .text("event_type", nullable=False)
        .doris_unique_key(["event_id"])
        .doris_partition_by_range("event_date")
        .doris_distributed_by("event_id", buckets=16)
        .doris_properties({"replication_num": "1"})
        .create_table()
    ),
    down=MigrationBuilder("analytics", dialect="doris").drop_table(),
)
```

**Builder column types:**

| Method | PostgreSQL | Doris |
|---|---|---|
| ``.uuid("col")`` | ``UUID`` | ``VARCHAR(36)`` |
| ``.text("col")`` | ``TEXT`` | ``TEXT`` |
| ``.bigint("col")`` | ``BIGINT`` | ``BIGINT`` |
| ``.integer("col")`` | ``INTEGER`` | ``INT`` |
| ``.boolean("col")`` | ``BOOLEAN`` | ``TINYINT`` |
| ``.float_("col")`` | ``DOUBLE PRECISION`` | ``DOUBLE`` |
| ``.float_short("col")`` | ``REAL`` | ``FLOAT`` |
| ``.jsonb("col")`` | ``JSONB`` | ``JSON`` |
| ``.timestamp("col")`` | ``TIMESTAMPTZ`` | ``DATETIME`` |
| ``.date("col")`` | ``DATE`` | ``DATE`` |
| ``.varchar("col", n)`` | ``VARCHAR(n)`` | ``VARCHAR(n)`` |

**CLI reference:**

```bash
# PostgreSQL
python -m repo_core.migrations status              # show applied/pending
python -m repo_core.migrations upgrade             # apply all pending
python -m repo_core.migrations upgrade --file 0002 # apply through file 0002 only
python -m repo_core.migrations downgrade           # roll back last migration
python -m repo_core.migrations generate -m "desc"  # scaffold new version file

# Doris (warehouse)
python -m repo_core.migrations doris-status
python -m repo_core.migrations doris-upgrade
python -m repo_core.migrations doris-downgrade
```

## Adding a New Domain

1. Create the domain folder under `domains/<name>/`
2. Implement `model.py`, `schema.py`, `validator.py` using the base classes
3. Optionally create a `repository.py` with custom hooks
4. Export all public names from the domain's `__init__.py`
5. Add a ``Migration`` in a new or existing version file under ``migrations/versions/postgres/`` (or ``migrations/versions/doris/`` for warehouse tables)

## Domains

| Domain                   | Models                           | Table(s)                               |
| ------------------------ | -------------------------------- | -------------------------------------- |
| `article`                | `Article`                        | `articles`                             |
| `authentication`         | `ApiKey`, `AuthUser`, `AuthRole` | `api_keys`, `auth_users`, `auth_roles` |
| `entity`                 | `Entity`                         | `entities`                             |
| `instagram/account`      | `InstagramAccount`               | `instagram_accounts`                   |
| `proxy`                  | `Proxy`, `ProxyLease`            | `proxies`, `proxy_leases`              |
| `telegram/account`       | `TelegramAccount`                | `telegram_accounts`                    |
| `telegram/message`       | `TelegramMessage`                | `telegram_messages`                    |
| `telegram/enriched_message` | `EnrichedTelegramMessage`     | `enriched_telegram_messages`           |
| `x/account`              | `XAccount`                       | `x_accounts`                           |

## Testing

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# With coverage
pytest --cov=repo_core tests/
```

Tests use mock `db_core` clients via `pytest-mock`. No real database needed. Fixtures are in `tests/conftest.py`.

Test suite: 20 test files covering repository CRUD, lifecycle hooks, validators, schemas, models, events, domain filters, Alembic metadata, migration registry, introspection, Doris type mapping, and SQLAlchemy generation.

## Relationship to db_core

`repo_core` **injects and wraps** `db_core` — it does not duplicate database logic. The dependency chain is:

```text
sam-backend / tele-crawler / persistence (consumers)
         │
         ▼
    repo_core          ← domain models, BaseRepository, schemas, validators, hooks
         │
         ▼
     db_core           ← PostgresClient, ElasticsearchClient, QueryBuilder, connection pooling
         │
         ▼
SQLAlchemy / elasticsearch-py / asyncpg   ← actual wire protocols
```

- **`db_core.PostgresClient`** — connection pooling, raw SQL execution, CRUD operations
- **`db_core.ElasticsearchClient`** — search indexing, document queries, index management
- **`db_core.DorisClient`** — OLAP queries via MySQL protocol
- **`db_core.QueryBuilder`** — fluent query chain; wraps a client + target table/index
- **`db_core.schema`** — `TableDef`, `ColumnDef`, `IndexDef`, `ForeignKeyDef`, `table_to_sqlalchemy()` for migration DDL generation

`repo_core` adds the repository pattern, domain models, validation, hooks, and migrations on top of `db_core`'s client wrappers. Services never create raw SQLAlchemy sessions — repositories auto-wire `db_core` clients from environment variables.

### Client Injection Pattern

```python
from repo_core.domains.telegram.account.repository import TelegramAccountRepository
from repo_core.domains.telegram.account.model import TelegramAccount

# Auto-wired from environment variables — no manual QueryBuilder needed
repo = TelegramAccountRepository(model=TelegramAccount)

async with repo:
    accounts = await repo.all("telegram_accounts", limit=20)

# For tests or custom URLs, pass explicit QueryBuilder:
from db_core import create_postgres_client, QueryBuilder

db = create_postgres_client(url="postgresql+asyncpg://user:pass@localhost/mydb")
repo = TelegramAccountRepository(model=TelegramAccount, db=QueryBuilder(db, ""))
```

## Downstream Service Integration

Three downstream services consume `repo_core` in the Dream monorepo:

| Service            | Path                                    | Status        | Pattern                                  |
| ------------------ | --------------------------------------- | ------------- | ---------------------------------------- |
| **sam-backend**    | `crawlers/social-account-manager/sam-backend/` | ✅ Integrated | FastAPI + auto-wired repos               |
| **persistence**    | `workers/persistence/`                  | ✅ Integrated | Kafka consumer + auto-wired repos        |
| **tele-crawler**   | `crawlers/telegram/tele-crawler/`       | 🟡 Partial    | HTTP client to sam-backend, some DB work |

### sam-backend Integration Pattern (Reference)

sam-backend is the canonical example of how a FastAPI service integrates with `repo_core`. It follows a strict **3-layer architecture**:

```text
FastAPI Route  →  Service (dataclass)  →  Repository (BaseRepository)  →  db_core.QueryBuilder
```

**Dependency wiring** (`dependencies.py`):

```python
from repo_core.domains.telegram.account.repository import TelegramAccountRepository
from repo_core.domains.telegram.account.model import TelegramAccount

def get_telegram_repository() -> TelegramAccountRepository:
    """Auto-wired from env vars — no manual QueryBuilder needed."""
    return TelegramAccountRepository(model=TelegramAccount)

def get_account_service(
    repo: Annotated[TelegramAccountRepository, Depends(get_telegram_repository)],
) -> AccountService:
    return AccountService(repo=repo)
```

**Service layer** (`service.py`):

```python
@dataclass
class AccountService:
    """Pure business logic — no HTTP, no SQL, no connection management."""
    repo: TelegramAccountRepository

    async def create_account(self, data: dict) -> Any:
        data.setdefault("account_metadata", {})
        return await self.repo.insert_one("telegram_accounts", data)
```

**Route layer** (`routes.py`):

```python
@router.post("/", response_model=AccountResponse)
async def create_account(
    body: AccountCreateRequest,
    service: AccountServiceDep,
):
    return await service.create_account(body.model_dump())
```

**Key rules**: `dependencies.py` is where repositories are instantiated. Services are `@dataclass` with no framework dependency. Routes validate with repo_core schemas and delegate to services. Repositories auto-wire `db_core` clients — no manual `QueryBuilder` construction needed.

## License

Proprietary — part of the Dream platform.
