Metadata-Version: 2.4
Name: summerizer
Version: 0.0.2
Summary: Extractive text summarization with MMR over semantic embeddings and TF-IDF, multilingual with first-class Persian support.
Keywords: summarization,nlp,extractive,mmr,tfidf,embeddings,persian,farsi
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: shekar<2,>=1.7
Provides-Extra: embeddings
Requires-Dist: sentence-transformers<7,>=6.1; extra == "embeddings"
Provides-Extra: laya
Requires-Dist: laya; extra == "laya"
Provides-Extra: all
Requires-Dist: sentence-transformers<7,>=6.1; extra == "all"
Requires-Dist: laya; extra == "all"

# summerizer

Extractive text summarization, Persian (Farsi) NLP, and keywords — as a library
and a CLI.

- **`summarize`** — the most important sentences of a text
- **`extract_keywords`** — the document's key terms and phrases
- **`normalize`** — Shekar normalization of Persian text
- **`spell_check`, `is_toxic`, `is_informal`, `get_sentiment`** — Persian analysis
- MMR selection over semantic embeddings + TF-IDF
- [Shekar](https://github.com/amirivojdan/shekar) is the **only required** dependency

The import name is **`summarizer`**; the console command is **`summarizer`**.

---

## Install

```bash
pip install .                 # required only: Shekar
pip install ".[embeddings]"   # + local semantic embeddings (adds torch)
pip install ".[all]"          # + embeddings and Laya
```

Shekar brings onnxruntime, sentencepiece, flashtext, numpy, tqdm and filelock —
no torch, no model download at install time. `sentence-transformers` is optional;
without it the summarizer uses a built-in `hashing` embedder. Python **3.11+**.

```bash
pip install --only-binary=:all: .   # force wheels, skip source builds
uv pip install .                    # fastest resolver
```

---

## Quick start

```python
from summarizer import summarize, extract_keywords, normalize

sentences = summarize(text, sentences=3)          # -> list[str]
keywords  = extract_keywords(text, top_n=10)      # -> list[str]
clean     = normalize(text)                       # -> Persian-normalized str
```

```bash
summarizer --file article.txt --title "Headline" --sentences 3
cat article.txt | summarizer - --ratio 0.3 --array-only
```

---

## Methods

Every functional method accepts the document as `(text, title, description)` —
`title` and `description` are **optional** context. They also accept a `config=`
or individual keyword options.

### `summarize`

```python
summarize(
    text: str,
    *,
    title: str | None = None,
    description: str | None = None,
    config: SummarizerConfig | None = None,
    **options,
) -> list[str]
```

Returns the selected sentences as strings, in output order.

```python
summarize(text, sentences=3)
summarize(text, ratio=0.3, embedder="hashing")
summarize(article, title="Transit plan approved", description="Council vote", sentences=5)
summarize(text, config=SummarizerConfig(fact_weight=0.3))
```

`title`/`description` are not summarized; they are used as context for keyword
extraction, so sentences matching the headline/deck are favoured.

### `summarize_result`

Same arguments as `summarize`, but returns a full `SummaryResult`.

```python
from summarizer import summarize_result

result = summarize_result(text, sentences=3)
result.summary           # joined string
result.sentences         # list[str]
result.key_sentences     # list[KeySentence] (scores + ranks)
result.statistics        # dict: counts, compression, backends, keywords, ...
result.title, result.description
result.to_dict()         # JSON-ready
```

**`SummaryResult`**


| Field                  | Meaning                                                                                                                                |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `summary`              | selected sentences joined by a space                                                                                                   |
| `sentences`            | list of selected sentence strings                                                                                                      |
| `key_sentences`        | list of`KeySentence`                                                                                                                   |
| `title`, `description` | the optional context you passed                                                                                                        |
| `embedder`             | embedding backend used (e.g.`hashing`, `openai:…`)                                                                                    |
| `statistics`           | `total_sentences`, `selected_sentences`, `compression_ratio`, `keywords`, `centrality`, `suppressed_duplicates`, `elapsed_seconds`, … |
| `config`               | the resolved configuration as a dict                                                                                                   |

**`KeySentence`** — `index`, `text`, `score`, `rank`, `semantic_relevance`,
`lexical_relevance`, `keyword_relevance`, `fact_relevance`, `tfidf_weight`,
`num_terms`. `to_dict()` serialises all of them.

### `extract_keywords`

```python
extract_keywords(
    text: str,
    *,
    title: str | None = None,
    description: str | None = None,
    top_n: int = 10,
    model: str = "rake",       # "rake" | "textrank"
    max_length: int = 3,       # max phrase length in words
    config: SummarizerConfig | None = None,
    strict: bool = False,
) -> list[str]
```

```python
extract_keywords(text, top_n=10)
extract_keywords(text, model="textrank")
extract_keywords(article, title="Headline", description="Deck", top_n=15)
```

Uses Shekar's keyword extractor over `title + description + text`. Returns `[]`
if extraction is unavailable unless `strict=True`, which raises
`KeywordExtractionError`.

### `extract_keywords_result`

Returns a full `KeywordResult`.

```python
from summarizer import extract_keywords_result

kw = extract_keywords_result(text, top_n=10)
kw.keywords          # list[str]
kw.weights           # dict[str, float] (rank weights)
kw.backend           # "shekar"
kw.used              # bool
kw.sentence_scores(sentences)   # per-sentence coverage
kw.to_dict()
```

### `normalize`

```python
normalize(text: str, *, strict: bool = False) -> str
```

Runs Shekar's `Normalizer` (letters, digits, punctuation, spacing, ZWNJ,
compound words). **Only applies to Persian text**; non-Persian text is returned
unchanged. `strict=True` raises for non-Persian input or a missing Shekar install.

```python
normalize("كتاب من يكي است و در سال 1921 منتشر شد")
# 'کتاب من یکی است و در سال ۱۹۲۱ منتشر شد'
normalize("Hello world")   # 'Hello world'
```

### `spell_check`

```python
spell_check(text: str, *, strict: bool = False) -> str
```

Shekar `SpellChecker`; non-Persian input is returned unchanged.

```python
spell_check("سسلام بر ششما ددوست من")   # 'سلام بر شما دوست من'
```

### `is_toxic`

```python
is_toxic(text: str, *, strict: bool = False) -> bool
```

Shekar `OffensiveLanguageClassifier`; `True` when the label is `offensive`.

```python
is_toxic("تو خیلی احمق و بی‌شرفی!")   # True
is_toxic("این یک متن معمولی است.")     # False
```

### `is_informal`

```python
is_informal(text: str, *, strict: bool = False) -> bool
```

Shekar `InformalLanguageClassifier` (rule-based). Conservative by design.

```python
is_informal("می‌خوام برم خونه")        # True
is_informal("لطفا گزارش را ارسال فرمایید.")   # False
```

### `get_sentiment`

```python
get_sentiment(text: str, *, strict: bool = False) -> dict
```

Shekar `SentimentClassifier`; returns `{"label": "positive"|"negative", "confidence": float}`. Non-Persian input yields `{"label": None, "confidence": 0.0}`.

```python
get_sentiment("سریال قصه‌های مجید عالی بود!")
# {'label': 'positive', 'confidence': 0.985}
```

### camelCase aliases

`spellCheck`, `isToxic`, `isInformal`, `getSentiment` are exact aliases of the
snake_case functions above.

### Persian analysis gating

`normalize`, `spell_check`, `is_toxic`, `is_informal` and `get_sentiment` are
**Persian-only**, lazy and fail-safe: non-Persian (or blank) input yields a
neutral result, and any Shekar error degrades gracefully. Use `strict=True` to
raise instead (`SummarizerError` for non-Persian, `PersianNLPNotAvailableError`
when Shekar is missing).

---

## Classes

### `Summarizer`

```python
from summarizer import Summarizer, SummarizerConfig

summarizer = Summarizer(SummarizerConfig(num_sentences=3), verbose=False)
result = summarizer.summarize(text, title="Headline", description="Deck")
result = summarizer.summarize_for(text, sentences=5, ratio=None, title=None, description=None)
sentences = summarizer.split_sentences(text)   # list[Sentence]
summarizer.embedder_name
```

Reuse one `Summarizer` for many documents so the embedding model loads once.

### `SummarizerConfig`

All tunables in one dataclass (validated on construction, JSON-serialisable).
Selected fields:


| Field                                                      | Default                      | Meaning                                                |
| ------------------------------------------------------------ | ------------------------------ | -------------------------------------------------------- |
| `num_sentences`                                            | `None`                       | fixed count;`None` = dynamic                           |
| `ratio`                                                    | `None`                       | fraction to keep (overrides`num_sentences`)            |
| `lead_sentences`                                           | `1`                          | always include the first N sentences                   |
| `min_sentences` / `max_sentences`                          | `3` / `20`                   | dynamic count bounds                                   |
| `dynamic_max_ratio`                                        | `0.45`                       | cap dynamic k for short docs                           |
| `lambda_`                                                  | `0.7`                        | MMR relevance vs diversity                             |
| `relevance_alpha`                                          | `0.6`                        | semantic vs lexical in relevance                       |
| `redundancy_beta`                                          | `0.6`                        | semantic vs lexical in redundancy                      |
| `position_bonus` / `position_decay`                        | `0.0` / `0.5`                | favour earlier sentences                               |
| `centrality`                                               | `hybrid`                     | `centroid` / `textrank` / `hybrid`                     |
| `dedupe_threshold`                                         | `0.55`                       | collapse near-duplicates to the richest member         |
| `lexical_weight_mode`                                      | `mean_tfidf`                 | `mean_tfidf` / `sum_tfidf` / `l2_norm`                 |
| `use_stopwords` / `min_token_length`                       | `True` / `2`                 | tokenizer                                              |
| `filter_sentences` / `min_filter_chars`                    | `True` / `12`                | drop non-prose fragments                               |
| `max_sentence_chars` / `split_long_sentences`              | `250` / `True`               | clause splitting                                       |
| `persian_nlp`                                              | `auto`                       | `auto` / `shekar` / `builtin`                          |
| `keyword_extraction`                                       | `auto`                       | `auto` / `on` / `off`                                  |
| `keyword_weight` / `keyword_model`                         | `0.3` / `rake`               | keyword boost                                          |
| `keyword_top_n` / `keyword_max_length`                     | `10` / `3`                   | keyword extractor                                      |
| `fact_weight`                                              | `0.3`                        | numbers/units/dates/events boost                       |
| `strip_attribution`                                        | `True`                       | strip dateline prefixes                                |
| `embedder`                                                 | `auto`                       | `auto` / `hashing` / `sentence-transformer` / `openai` |
| `embedding_model` / `embedding_prompt` / `embedding_dtype` | `None` / `None` / `auto`     | embedding backend                                      |
| `api_base` / `api_key` / `api_timeout` / `api_retries`     | `None` / `None` / `30` / `3` | OpenAI-compatible endpoint                             |
| `batch_size` / `trust_remote_code`                         | `64` / `False`               | embedding loading                                      |
| `output_order`                                             | `original`                   | `original` / `rank` / `score`                          |

**Option aliases** accepted by `summarize`/`extract_keywords` and friends:
`sentences`→`num_sentences`, `alpha`→`relevance_alpha`, `beta`→`redundancy_beta`,
`model`→`embedding_model`, `keywords`→`keyword_extraction`, `order`→`output_order`.
Any other `SummarizerConfig` field can be passed by its own name.

Passing both `config=` and individual options raises `ConfigError`; unknown
options raise `TypeError`.

### `KeywordExtractor`

```python
from summarizer import KeywordExtractor

extractor = KeywordExtractor(model="rake", top_n=10, max_length=3)
result = extractor.extract(text)          # KeywordResult
result = extractor.extract(text, apply=False)   # inert result
```

### `PersianNLP`

```python
from summarizer import PersianNLP, looks_persian, persian_nlp_available

looks_persian(text)          # bool: is the text Persian?
persian_nlp_available()      # bool: is Shekar importable?

nlp = PersianNLP(required=True)
nlp.normalize(text)          # Shekar-normalized text
nlp.split_sentences(text)    # list[str] | None
nlp.used, nlp.is_available, nlp.describe()
```

### `PersianAnalyzer`

The lazy holder behind the analysis methods (raw Shekar results).

```python
from summarizer import PersianAnalyzer

a = PersianAnalyzer()
a.correct(text)     # str | None
a.offensive(text)   # ("neutral"|"offensive", prob) | None
a.informal(text)    # ("informal", 0|1) | None
a.sentiment(text)   # ("positive"|"negative", prob) | None
```

### `LayaSentenceScorer` (optional)

Per-sentence usefulness scoring with [Laya](https://github.com/he-jev/laya)
(requires the `laya` extra).

```python
from summarizer import LayaSentenceScorer, laya_available

scorer = LayaSentenceScorer(subfolder="multilingual")
scorer.score(sentences, title="Headline")   # list[float | None]  (P(true))
scorer.score_table(sentences, title=None)   # [{"index","score","text"}]
```

### Utilities

```python
from summarizer import clean_sentences, textrank, CENTRALITY_MODES
from summarizer import load_dotenv, embedding_endpoint

clean_sentences(raw_sentences, min_chars=12)   # drop tables/URLs/headings
textrank(similarity_fn, n)                      # PageRank scores
load_dotenv(".env")                             # load env vars from a file
embedding_endpoint()                            # QWEN_ENDPOINT, if configured
```

### Exceptions

All derive from `SummarizerError`:


| Exception                                     | Raised when                                 |
| ----------------------------------------------- | --------------------------------------------- |
| `ConfigError`                                 | invalid configuration value                 |
| `EmptyInputError`                             | empty text / no usable sentences            |
| `EmbedderError` / `EmbedderNotAvailableError` | embedding backend failure                   |
| `PersianNLPNotAvailableError`                 | Shekar required but unavailable             |
| `KeywordExtractionError`                      | keyword extraction required but unavailable |

---

## Embedding backends


| `embedder`             | Requires        | Notes                                                   |
| ------------------------ | ----------------- | --------------------------------------------------------- |
| `auto` (default)       | —              | endpoint if configured, else local model, else`hashing` |
| `hashing`              | nothing         | deterministic, lexical/morphological similarity         |
| `sentence-transformer` | `.[embeddings]` | local model, Qwen3-Embedding-0.6B by default            |
| `openai`               | an endpoint     | any OpenAI-compatible`/embeddings` server               |

### Endpoint from `.env`

Set an endpoint to skip the local model; otherwise the model is downloaded and
loaded in-process (no server is started).


| `.env` key      | Aliases                                               | Meaning                                  |
| ----------------- | ------------------------------------------------------- | ------------------------------------------ |
| `QWEN_ENDPOINT` | `SUMMARIZER_EMBEDDING_ENDPOINT`, `EMBEDDING_ENDPOINT` | base URL, e.g.`http://127.0.0.1:8000/v1` |
| `QWEN_MODEL`    | `SUMMARIZER_EMBEDDING_MODEL`                          | model id                                 |
| `QWEN_API_KEY`  | `SUMMARIZER_EMBEDDING_API_KEY`                        | bearer token (optional for localhost)    |

```bash
# .env
QWEN_ENDPOINT=http://127.0.0.1:8000/v1
QWEN_MODEL=Qwen/Qwen3-Embedding-0.6B
```

---

## Command-line interface

The CLI summarizes text (the other methods are library-only).

```bash
summarizer --text "One. Two. Three. Four." --sentences 2
summarizer --file article.txt --title "Headline" --description "Deck" --sentences 3
cat article.txt | summarizer - --ratio 0.3 --array-only
summarizer --file article.txt --embedder hashing --keywords off
summarizer --file article.txt --usefulness -v    # + Laya per-sentence scores
```

Selected flags:


| Flag                                    | Default               | Meaning                       |
| ----------------------------------------- | ----------------------- | ------------------------------- |
| `--text` / `--file` / `-` / `--stdin`   | —                    | input source                  |
| `--title`, `--description`              | —                    | optional context              |
| `-n, --sentences`, `--ratio`            | dynamic               | how many sentences            |
| `--lead-sentences`                      | 1                     | always keep the first N       |
| `--lambda` / `--alpha` / `--beta`       | 0.7 / 0.6 / 0.6       | MMR trade-offs                |
| `--centrality`                          | `hybrid`              | semantic signal               |
| `--dedupe-threshold`                    | 0.55                  | near-duplicate collapse       |
| `--embedder` / `--model`                | auto / qwen3-0.6b     | embeddings                    |
| `--keywords` / `--keyword-weight`       | auto / 0.3            | Shekar keyword boost          |
| `--fact-weight`                         | 0.30                  | numbers/units/dates boost     |
| `--persian-nlp`                         | auto                  | `auto` / `shekar` / `builtin` |
| `--format` / `--array-only` / `--order` | json / off / original | output                        |
| `-o, --output`                          | stdout                | write to a file               |
| `--usefulness`                          | off                   | Laya per-sentence scores      |

Run `summarizer --help` for everything. Equivalent to `python -m summarizer …`.

---

## Notes

- **Persian-only methods** (`normalize`, `spell_check`, `is_toxic`,
  `is_informal`, `get_sentiment`) leave non-Persian text untouched; pass
  `strict=True` to raise instead.
- **Reuse a `Summarizer`** across documents so the embedding model loads once.
- See [`algorithm.md`](algorithm.md) for the exact selection formulas, and
  [`plan.md`](plan.md) for the design rationale; `benchmarks/` holds evaluation
  scripts.

## Tests

```bash
python -m unittest discover -s tests -t .
# real Shekar model calls (downloads models on first run):
SUMMARIZER_TEST_SHEKAR=1 python -m unittest tests.test_analysis
```
