Describe the bug
PythonCodeSplitter with strip_docstrings=True emits a syntactically invalid chunk for any function, method, or class whose body consists solely of a docstring, for example a one-line custom exception class or a stub method. The docstring is the unit's only statement, so removing it leaves the def/class header with no body at all.
Error message
SyntaxError: expected an indented block after class definition on line 2
Expected behavior
A unit whose body is only a docstring should either keep the docstring in place or otherwise remain valid Python after strip_docstrings=True. It should never come out unparsable.
To Reproduce
from haystack import Document
from haystack.components.preprocessors import PythonCodeSplitter
source = '''
class MyError(Exception):
"""Custom error raised when the widget explodes."""
'''
splitter = PythonCodeSplitter(min_effective_lines=1, max_effective_lines=10, strip_docstrings=True)
result = splitter.run(documents=[Document(content=source)])
import ast
for chunk in result["documents"]:
ast.parse(chunk.content) # raises SyntaxError
Same result for a docstring-only top-level function and a docstring-only method.
Additional context
strip_docstrings=True is documented as a RAG-oriented option to shrink stored chunk content while still letting the docstring influence retrieval via meta_fields_to_embed=["docstrings"]. With it enabled, exactly the compact/simple declarations (one-line custom exceptions, stub/abstract methods) get corrupted into unparsable text before they are embedded or indexed.
System:
- Haystack version: HEAD (
0c4b23dbd)
Describe the bug
PythonCodeSplitterwithstrip_docstrings=Trueemits a syntactically invalid chunk for any function, method, or class whose body consists solely of a docstring, for example a one-line custom exception class or a stub method. The docstring is the unit's only statement, so removing it leaves thedef/classheader with no body at all.Error message
Expected behavior
A unit whose body is only a docstring should either keep the docstring in place or otherwise remain valid Python after
strip_docstrings=True. It should never come out unparsable.To Reproduce
Same result for a docstring-only top-level function and a docstring-only method.
Additional context
strip_docstrings=Trueis documented as a RAG-oriented option to shrink stored chunk content while still letting the docstring influence retrieval viameta_fields_to_embed=["docstrings"]. With it enabled, exactly the compact/simple declarations (one-line custom exceptions, stub/abstract methods) get corrupted into unparsable text before they are embedded or indexed.System:
0c4b23dbd)