Sitelet https://github.com/deepset-ai/haystack/issues/12281
Skip to content

RecursiveDocumentSplitter with overlap emits chunks not present in the source #12281

Description

@LK-maker-007

Describe the bug

RecursiveDocumentSplitter with split_overlap > 0 produces chunks that contain text not in the source document. It happens whenever the text needs more than one separator level to split, which includes the default separators. _apply_overlap runs at every level of the internal recursion, so an already-overlapped chunk gets the overlap prepended again, duplicating and rearranging text.

This is separate from the already-fixed overlap issues #11767 (overlap ignored when no separator matches) and #11710 (split_idx_start offset); here the chunk content itself is wrong.

To Reproduce

from haystack import Document
from haystack.components.preprocessors.recursive_splitter import RecursiveDocumentSplitter

text = "Overview\nThis module handles ingestion and preprocessing of documents.\n\nDetails\nIt splits text into chunks for embedding."
splitter = RecursiveDocumentSplitter(split_length=50, split_overlap=10, split_unit="char")
splitter.warm_up()
for c in splitter.run([Document(content=text)])["documents"]:
    print(c.content in text, repr(c.content))

3 of the 6 chunks are not in the source:

False "Overview\nOverview\nThis module handles ingestion an"
False "gestion and preprocnd preprocessing rocessing of d"

Expected behavior

Every chunk should be text from the source document. Overlap should be applied once, not at each recursion level.

System:

  • Haystack version: 3.1.0rc0 (main)

I'm planning to open a PR that applies the overlap once, at the outermost level, so the recursive calls return non-overlapped chunks. If you'd prefer a different approach, let me know.

Metadata

Metadata

Assignees

Labels

P2Medium priority, add to the next sprint if no P1 available

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions