chunkipy.text_chunker.recursive.recursive_text_chunker

Classes

RecursiveTextChunker([chunk_size, ...])

Chunk text by recursively applying increasingly fine-grained splitters.

class chunkipy.text_chunker.recursive.recursive_text_chunker.RecursiveTextChunker(chunk_size=None, size_estimator=None, overlap_ratio=0.0, text_splitters=None)

Bases: BaseOverlapTextChunker

Chunk text by recursively applying increasingly fine-grained splitters.

The chunker tries each splitter in order until a text part fits within the configured chunk_size. Custom splitters are attempted before the default fallback splitters.

Parameters:

• chunk_size (int)

• size_estimator (BaseSizeEstimator)

• overlap_ratio (float)

• text_splitters (List[BaseTextSplitter])

split_text(text)

Split the provided text into smaller parts based on the configured text splitters and chunk size. This method uses a recursive approach to apply different text splitters until the text fits properly within the chunk size (based on the size estimator).

Parameters:

text (str) – The text to be split.

Yields:

Generator [TextPart, None, None] – A generator yielding TextPart objects, each containing a piece of text and its estimated size.

Return type:

Generator[TextPart, None, None]