chunkipy.text_chunker.fixed_size.fixed_size_text_chunker

Classes

FixedSizeTextChunker([chunk_size, ...])

Chunk text into fixed-size slices using the configured size estimator.

class chunkipy.text_chunker.fixed_size.fixed_size_text_chunker.FixedSizeTextChunker(chunk_size=None, size_estimator=None, overlap_ratio=0.0)[source]

Bases: BaseOverlapTextChunker

Chunk text into fixed-size slices using the configured size estimator.

Each segment emitted by size_estimator.segment is treated as a unit of size 1 during chunk assembly.

Parameters:

chunk_size (int)
size_estimator (BaseSizeEstimator)
overlap_ratio (float)

split_text(text)[source]

Split the provided text into smaller parts based on size estimator. Size Estimator is used to cut the text into segments and every segment has size equal to 1.

Parameters:: text (str) – The text to be split.
Yields:: Generator [TextPart, None, None] – A generator yielding TextPart objects, each containing a piece of text and its estimated size.
Return type:: Generator[TextPart, None, None]