Sitelet https://github.com/deepset-ai/haystack/issues/12330
Skip to content

PythonCodeSplitter with strip_docstrings=True emits invalid Python for docstring-only function/method/class bodies #12330

Description

@AmirF194

Describe the bug

PythonCodeSplitter with strip_docstrings=True emits a syntactically invalid chunk for any function, method, or class whose body consists solely of a docstring, for example a one-line custom exception class or a stub method. The docstring is the unit's only statement, so removing it leaves the def/class header with no body at all.

Error message

SyntaxError: expected an indented block after class definition on line 2

Expected behavior

A unit whose body is only a docstring should either keep the docstring in place or otherwise remain valid Python after strip_docstrings=True. It should never come out unparsable.

To Reproduce

from haystack import Document
from haystack.components.preprocessors import PythonCodeSplitter

source = '''
class MyError(Exception):
    """Custom error raised when the widget explodes."""
'''

splitter = PythonCodeSplitter(min_effective_lines=1, max_effective_lines=10, strip_docstrings=True)
result = splitter.run(documents=[Document(content=source)])

import ast
for chunk in result["documents"]:
    ast.parse(chunk.content)  # raises SyntaxError

Same result for a docstring-only top-level function and a docstring-only method.

Additional context

strip_docstrings=True is documented as a RAG-oriented option to shrink stored chunk content while still letting the docstring influence retrieval via meta_fields_to_embed=["docstrings"]. With it enabled, exactly the compact/simple declarations (one-line custom exceptions, stub/abstract methods) get corrupted into unparsable text before they are embedded or indexed.

System:

  • Haystack version: HEAD (0c4b23dbd)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions