extractive-summarizer

Extractive text summarization, Persian (Farsi) NLP, and keywords — as a library and a CLI.

• summarize — the most important sentences of a text
• extract_keywords — the document's key terms and phrases
• normalize — Shekar normalization of Persian text
• spell_check, is_toxic, is_informal, get_sentiment — Persian analysis
• MMR selection over semantic embeddings + TF-IDF
• Shekar is the only required dependency

The import name is summarizer; the console command is summarizer.

---

Install

pip install .                 # required only: Shekar
pip install ".[embeddings]"   # + local semantic embeddings (adds torch)
pip install ".[all]"          # + embeddings and Laya

Shekar brings onnxruntime, sentencepiece, flashtext, numpy, tqdm and filelock — no torch, no model download at install time. sentence-transformers is optional; without it the summarizer uses a built-in hashing embedder. Python 3.11+.

pip install --only-binary=:all: .   # force wheels, skip source builds
uv pip install .                    # fastest resolver

---

Quick start

from summarizer import summarize, extract_keywords, normalize

sentences = summarize(text, sentences=3)          # -> list[str]
keywords  = extract_keywords(text, top_n=10)      # -> list[str]
clean     = normalize(text)                       # -> Persian-normalized str

summarizer --file article.txt --title "Headline" --sentences 3
cat article.txt | summarizer - --ratio 0.3 --array-only

---

Methods

Every functional method accepts the document as (text, title, description) — title and description are optional context. They also accept a config= or individual keyword options.

summarize

summarize(
    text: str,
    *,
    title: str | None = None,
    description: str | None = None,
    config: SummarizerConfig | None = None,
    **options,
) -> list[str]

Returns the selected sentences as strings, in output order.

summarize(text, sentences=3)
summarize(text, ratio=0.3, embedder="hashing")
summarize(article, title="Transit plan approved", description="Council vote", sentences=5)
summarize(text, config=SummarizerConfig(fact_weight=0.3))

title/description are not summarized; they are used as context for keyword extraction, so sentences matching the headline/deck are favoured.

summarize_result

Same arguments as summarize, but returns a full SummaryResult.

from summarizer import summarize_result

result = summarize_result(text, sentences=3)
result.summary           # joined string
result.sentences         # list[str]
result.key_sentences     # list[KeySentence] (scores + ranks)
result.statistics        # dict: counts, compression, backends, keywords, ...
result.title, result.description
result.to_dict()         # JSON-ready

SummaryResult

Field	Meaning
summary	selected sentences joined by a space
sentences	list of selected sentence strings
key_sentences	list ofKeySentence
title, description	the optional context you passed
embedder	embedding backend used (e.g.hashing, openai:…)
statistics	total_sentences, selected_sentences, compression_ratio, keywords, centrality, suppressed_duplicates, elapsed_seconds, …
config	the resolved configuration as a dict

KeySentence — index, text, score, rank, semantic_relevance, lexical_relevance, keyword_relevance, fact_relevance, tfidf_weight, num_terms. to_dict() serialises all of them.

extract_keywords

extract_keywords(
    text: str,
    *,
    title: str | None = None,
    description: str | None = None,
    top_n: int = 10,
    model: str = "rake",       # "rake" | "textrank"
    max_length: int = 3,       # max phrase length in words
    config: SummarizerConfig | None = None,
    strict: bool = False,
) -> list[str]

extract_keywords(text, top_n=10)
extract_keywords(text, model="textrank")
extract_keywords(article, title="Headline", description="Deck", top_n=15)

Uses Shekar's keyword extractor over title + description + text. Returns [] if extraction is unavailable unless strict=True, which raises KeywordExtractionError.

extract_keywords_result

Returns a full KeywordResult.

from summarizer import extract_keywords_result

kw = extract_keywords_result(text, top_n=10)
kw.keywords          # list[str]
kw.weights           # dict[str, float] (rank weights)
kw.backend           # "shekar"
kw.used              # bool
kw.sentence_scores(sentences)   # per-sentence coverage
kw.to_dict()

normalize

normalize(text: str, *, strict: bool = False) -> str

Runs Shekar's Normalizer (letters, digits, punctuation, spacing, ZWNJ, compound words). Only applies to Persian text; non-Persian text is returned unchanged. strict=True raises for non-Persian input or a missing Shekar install.

normalize("كتاب من يكي است و در سال 1921 منتشر شد")
# 'کتاب من یکی است و در سال ۱۹۲۱ منتشر شد'
normalize("Hello world")   # 'Hello world'

spell_check

spell_check(text: str, *, strict: bool = False) -> str

Shekar SpellChecker; non-Persian input is returned unchanged.

spell_check("سسلام بر ششما ددوست من")   # 'سلام بر شما دوست من'

is_toxic

is_toxic(text: str, *, strict: bool = False) -> bool

Shekar OffensiveLanguageClassifier; True when the label is offensive.

is_toxic("تو خیلی احمق و بی‌شرفی!")   # True
is_toxic("این یک متن معمولی است.")     # False

is_informal

is_informal(text: str, *, strict: bool = False) -> bool

Shekar InformalLanguageClassifier (rule-based). Conservative by design.

is_informal("می‌خوام برم خونه")        # True
is_informal("لطفا گزارش را ارسال فرمایید.")   # False

get_sentiment

get_sentiment(text: str, *, strict: bool = False) -> dict

Shekar SentimentClassifier; returns {"label": "positive"|"negative", "confidence": float}. Non-Persian input yields {"label": None, "confidence": 0.0}.

get_sentiment("سریال قصه‌های مجید عالی بود!")
# {'label': 'positive', 'confidence': 0.985}

camelCase aliases

spellCheck, isToxic, isInformal, getSentiment are exact aliases of the snake_case functions above.

Persian analysis gating

normalize, spell_check, is_toxic, is_informal and get_sentiment are Persian-only, lazy and fail-safe: non-Persian (or blank) input yields a neutral result, and any Shekar error degrades gracefully. Use strict=True to raise instead (SummarizerError for non-Persian, PersianNLPNotAvailableError when Shekar is missing).

---

Classes

Summarizer

from summarizer import Summarizer, SummarizerConfig

summarizer = Summarizer(SummarizerConfig(num_sentences=3), verbose=False)
result = summarizer.summarize(text, title="Headline", description="Deck")
result = summarizer.summarize_for(text, sentences=5, ratio=None, title=None, description=None)
sentences = summarizer.split_sentences(text)   # list[Sentence]
summarizer.embedder_name

Reuse one Summarizer for many documents so the embedding model loads once.

SummarizerConfig

All tunables in one dataclass (validated on construction, JSON-serialisable). Selected fields:

Field	Default	Meaning
num_sentences	None	fixed count;None = dynamic
ratio	None	fraction to keep (overridesnum_sentences)
lead_sentences	1	always include the first N sentences
min_sentences / max_sentences	3 / 20	dynamic count bounds
dynamic_max_ratio	0.45	cap dynamic k for short docs
lambda_	0.7	MMR relevance vs diversity
relevance_alpha	0.6	semantic vs lexical in relevance
redundancy_beta	0.6	semantic vs lexical in redundancy
position_bonus / position_decay	0.0 / 0.5	favour earlier sentences
centrality	hybrid	centroid / textrank / hybrid
dedupe_threshold	0.55	collapse near-duplicates to the richest member
lexical_weight_mode	mean_tfidf	mean_tfidf / sum_tfidf / l2_norm
use_stopwords / min_token_length	True / 2	tokenizer
filter_sentences / min_filter_chars	True / 12	drop non-prose fragments
max_sentence_chars / split_long_sentences	250 / True	clause splitting
persian_nlp	auto	auto / shekar / builtin
keyword_extraction	auto	auto / on / off
keyword_weight / keyword_model	0.3 / rake	keyword boost
keyword_top_n / keyword_max_length	10 / 3	keyword extractor
fact_weight	0.3	numbers/units/dates/events boost
strip_attribution	True	strip dateline prefixes
embedder	auto	auto / hashing / sentence-transformer / openai
embedding_model / embedding_prompt / embedding_dtype	None / None / auto	embedding backend
api_base / api_key / api_timeout / api_retries	None / None / 30 / 3	OpenAI-compatible endpoint
batch_size / trust_remote_code	64 / False	embedding loading
output_order	original	original / rank / score

Option aliases accepted by summarize/extract_keywords and friends: sentences→num_sentences, alpha→relevance_alpha, beta→redundancy_beta, model→embedding_model, keywords→keyword_extraction, order→output_order. Any other SummarizerConfig field can be passed by its own name.

Passing both config= and individual options raises ConfigError; unknown options raise TypeError.

KeywordExtractor

from summarizer import KeywordExtractor

extractor = KeywordExtractor(model="rake", top_n=10, max_length=3)
result = extractor.extract(text)          # KeywordResult
result = extractor.extract(text, apply=False)   # inert result

PersianNLP

from summarizer import PersianNLP, looks_persian, persian_nlp_available

looks_persian(text)          # bool: is the text Persian?
persian_nlp_available()      # bool: is Shekar importable?

nlp = PersianNLP(required=True)
nlp.normalize(text)          # Shekar-normalized text
nlp.split_sentences(text)    # list[str] | None
nlp.used, nlp.is_available, nlp.describe()

PersianAnalyzer

The lazy holder behind the analysis methods (raw Shekar results).

from summarizer import PersianAnalyzer

a = PersianAnalyzer()
a.correct(text)     # str | None
a.offensive(text)   # ("neutral"|"offensive", prob) | None
a.informal(text)    # ("informal", 0|1) | None
a.sentiment(text)   # ("positive"|"negative", prob) | None

LayaSentenceScorer (optional)

Per-sentence usefulness scoring with Laya (requires the laya extra).

from summarizer import LayaSentenceScorer, laya_available

scorer = LayaSentenceScorer(subfolder="multilingual")
scorer.score(sentences, title="Headline")   # list[float | None]  (P(true))
scorer.score_table(sentences, title=None)   # [{"index","score","text"}]

Utilities

from summarizer import clean_sentences, textrank, CENTRALITY_MODES
from summarizer import load_dotenv, embedding_endpoint

clean_sentences(raw_sentences, min_chars=12)   # drop tables/URLs/headings
textrank(similarity_fn, n)                      # PageRank scores
load_dotenv(".env")                             # load env vars from a file
embedding_endpoint()                            # QWEN_ENDPOINT, if configured

Exceptions

All derive from SummarizerError:

Exception	Raised when
ConfigError	invalid configuration value
EmptyInputError	empty text / no usable sentences
EmbedderError / EmbedderNotAvailableError	embedding backend failure
PersianNLPNotAvailableError	Shekar required but unavailable
KeywordExtractionError	keyword extraction required but unavailable

---

Embedding backends

embedder	Requires	Notes
auto (default)	—	endpoint if configured, else local model, elsehashing
hashing	nothing	deterministic, lexical/morphological similarity
sentence-transformer	.[embeddings]	local model, Qwen3-Embedding-0.6B by default
openai	an endpoint	any OpenAI-compatible/embeddings server

Endpoint from .env

Set an endpoint to skip the local model; otherwise the model is downloaded and loaded in-process (no server is started).

.env key	Aliases	Meaning
QWEN_ENDPOINT	SUMMARIZER_EMBEDDING_ENDPOINT, EMBEDDING_ENDPOINT	base URL, e.g.http://127.0.0.1:8000/v1
QWEN_MODEL	SUMMARIZER_EMBEDDING_MODEL	model id
QWEN_API_KEY	SUMMARIZER_EMBEDDING_API_KEY	bearer token (optional for localhost)

# .env
QWEN_ENDPOINT=http://127.0.0.1:8000/v1
QWEN_MODEL=Qwen/Qwen3-Embedding-0.6B

---

Command-line interface

The CLI summarizes text (the other methods are library-only).

summarizer --text "One. Two. Three. Four." --sentences 2
summarizer --file article.txt --title "Headline" --description "Deck" --sentences 3
cat article.txt | summarizer - --ratio 0.3 --array-only
summarizer --file article.txt --embedder hashing --keywords off
summarizer --file article.txt --usefulness -v    # + Laya per-sentence scores

Selected flags:

Flag	Default	Meaning
--text / --file / - / --stdin	—	input source
--title, --description	—	optional context
-n, --sentences, --ratio	dynamic	how many sentences
--lead-sentences	1	always keep the first N
--lambda / --alpha / --beta	0.7 / 0.6 / 0.6	MMR trade-offs
--centrality	hybrid	semantic signal
--dedupe-threshold	0.55	near-duplicate collapse
--embedder / --model	auto / qwen3-0.6b	embeddings
--keywords / --keyword-weight	auto / 0.3	Shekar keyword boost
--fact-weight	0.30	numbers/units/dates boost
--persian-nlp	auto	auto / shekar / builtin
--format / --array-only / --order	json / off / original	output
-o, --output	stdout	write to a file
--usefulness	off	Laya per-sentence scores

Run summarizer --help for everything. Equivalent to python -m summarizer ….

---

Notes

• Persian-only methods (normalize, spell_check, is_toxic, is_informal, get_sentiment) leave non-Persian text untouched; pass strict=True to raise instead.
• Reuse a Summarizer across documents so the embedding model loads once.
• See algorithm.md for the exact selection formulas, and plan.md for the design rationale; benchmarks/ holds evaluation scripts.

Tests

python -m unittest discover -s tests -t .
# real Shekar model calls (downloads models on first run):
SUMMARIZER_TEST_SHEKAR=1 python -m unittest tests.test_analysis