Repository navigation
Add remaining non-wrapped functions #767
Description
Activity
Question: Have you ever used or do you know of a tool to run queries over python / rust codebases?
It would be nice if we could generate a concrete report of what is not exposed.
No, but I did write a small script to check and this is what I see missing:
Missing attribute. Object name: datafusion, Attribute name: Catalog Missing attribute. Object name: datafusion, Attribute name: Database Missing attribute. Object name: datafusion, Attribute name: ExecutionPlan Missing attribute. Object name: datafusion, Attribute name: LogicalPlan Missing attribute. Object name: datafusion, Attribute name: RecordBatch Missing attribute. Object name: datafusion, Attribute name: RecordBatchStream Missing attribute. Object name: datafusion, Attribute name: Table Missing value in list. Object name: datafusion, Attribute name: __all__, Value: runtime Missing value in list. Object name: datafusion, Attribute name: __all__, Value: Catalog Missing value in list. Object name: datafusion, Attribute name: __all__, Value: Database Missing value in list. Object name: datafusion, Attribute name: __all__, Value: Table Missing value in list. Object name: datafusion, Attribute name: __all__, Value: AggregateUDF Missing value in list. Object name: datafusion, Attribute name: __all__, Value: LogicalPlan Missing value in list. Object name: datafusion, Attribute name: __all__, Value: ExecutionPlan Missing value in list. Object name: datafusion, Attribute name: __all__, Value: RecordBatch Missing value in list. Object name: datafusion, Attribute name: __all__, Value: RecordBatchStream Missing value in list. Object name: datafusion, Attribute name: __all__, Value: common Missing value in list. Object name: datafusion, Attribute name: __all__, Value: expr Missing value in list. Object name: datafusion, Attribute name: __all__, Value: functions Missing value in list. Object name: datafusion, Attribute name: __all__, Value: object_store Missing value in list. Object name: datafusion, Attribute name: __all__, Value: substrait Missing attribute. Object name: datafusion.common, Attribute name: DFSchema Missing attribute. Object name: datafusion.common, Attribute name: DataType Missing attribute. Object name: datafusion.common, Attribute name: DataTypeMap Missing attribute. Object name: datafusion.common, Attribute name: NullTreatment Missing attribute. Object name: datafusion.common, Attribute name: PythonType Missing attribute. Object name: datafusion.common, Attribute name: RexType Missing attribute. Object name: datafusion.common, Attribute name: SqlFunction Missing attribute. Object name: datafusion.common, Attribute name: SqlSchema Missing attribute. Object name: datafusion.common, Attribute name: SqlStatistics Missing attribute. Object name: datafusion.common, Attribute name: SqlTable Missing attribute. Object name: datafusion.common, Attribute name: SqlType Missing attribute. Object name: datafusion.common, Attribute name: SqlView Missing attribute. Object name: datafusion.common, Attribute name: __all__ Missing attribute. Object name: datafusion.expr, Attribute name: EmptyRelation Missing attribute. Object name: Expr, Attribute name: __radd__ Missing attribute. Object name: Expr, Attribute name: __rand__ Missing attribute. Object name: Expr, Attribute name: __rmod__ Missing attribute. Object name: Expr, Attribute name: __rmul__ Missing attribute. Object name: Expr, Attribute name: __ror__ Missing attribute. Object name: Expr, Attribute name: __rsub__ Missing attribute. Object name: Expr, Attribute name: __rtruediv__ Missing attribute. Object name: datafusion.expr, Attribute name: IsNull Missing attribute. Object name: datafusion.expr, Attribute name: Unnest Missing attribute. Object name: datafusion.expr, Attribute name: Window Missing attribute. Object name: datafusion.expr, Attribute name: __all__ Missing attribute. Object name: datafusion.functions, Attribute name: __all__ Missing attribute. Object name: datafusion.object_store, Attribute name: AmazonS3 Missing attribute. Object name: datafusion.object_store, Attribute name: GoogleCloud Missing attribute. Object name: datafusion.object_store, Attribute name: LocalFileSystem Missing attribute. Object name: datafusion.object_store, Attribute name: MicrosoftAzure Missing attribute. Object name: datafusion.object_store, Attribute name: __all__ Missing attribute. Object name: datafusion, Attribute name: runtime Missing attribute. Object name: datafusion.substrait, Attribute name: __all__Code to generate:
import datafusion import datafusion.functions import datafusion.object_store import datafusion.substrait def missing_exports(internal_obj, wrapped_obj): for attr in dir(internal_obj): if attr not in dir(wrapped_obj): print(f"Missing attribute. Object name: {wrapped_obj.__name__}, Attribute name: {attr}") continue internal_attr = getattr(internal_obj, attr) wrapped_attr = getattr(wrapped_obj, attr) if internal_attr is not None and wrapped_attr is None: print(f"Attribute exists but is None. Object name: {wrapped_obj.__name__}, Attribute name: {attr}") if attr in ["__self__", "__class__"]: continue if isinstance(internal_attr, list): for val in internal_attr: if val not in wrapped_attr: print(f"Missing value in list. Object name: {wrapped_obj.__name__}, Attribute name: {attr}, Value: {val}") elif hasattr(internal_attr, '__dict__'): missing_exports(internal_attr, wrapped_attr) missing_exports(datafusion._internal, datafusion)I can work on adding these tomorrow morning and I can also add this code as a unit test.
FWIW I don't know if all of these need to be exported. It's probably worth looking through each one.
Just an update on this one. Upon running the script, this is the output:
Missing attribute. Object name: DataFrame, Attribute name: format_column_name Missing attribute. Object name: NullTreatment, Attribute name: __delattr__ Missing attribute. Object name: NullTreatment, Attribute name: __dir__ Missing attribute. Object name: NullTreatment, Attribute name: __eq__ Missing attribute. Object name: NullTreatment, Attribute name: __format__ Missing attribute. Object name: NullTreatment, Attribute name: __ge__ Missing attribute. Object name: NullTreatment, Attribute name: __getattribute__ Missing attribute. Object name: NullTreatment, Attribute name: __gt__ Missing attribute. Object name: NullTreatment, Attribute name: __hash__ Missing attribute. Object name: NullTreatment, Attribute name: __init__ Missing attribute. Object name: NullTreatment, Attribute name: __init_subclass__ Missing attribute. Object name: NullTreatment, Attribute name: __int__ Missing attribute. Object name: NullTreatment, Attribute name: __le__ Missing attribute. Object name: NullTreatment, Attribute name: __lt__ Missing attribute. Object name: NullTreatment, Attribute name: __ne__ Missing attribute. Object name: NullTreatment, Attribute name: __new__ Missing attribute. Object name: NullTreatment, Attribute name: __reduce__ Missing attribute. Object name: NullTreatment, Attribute name: __reduce_ex__ Missing attribute. Object name: NullTreatment, Attribute name: __repr__ Missing attribute. Object name: NullTreatment, Attribute name: __setattr__ Missing attribute. Object name: NullTreatment, Attribute name: __sizeof__ Missing attribute. Object name: NullTreatment, Attribute name: __str__ Missing attribute. Object name: NullTreatment, Attribute name: __subclasshook__
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We still have a few classes that do not yet have wrapper functions. Namely
datafusion.object_storeanddatafusion.common. Additionally indatafusion.substraitwe referenceLogicalPlanthat is not exposed.Also worth doing is reviewing the excellent PR #751 to see how it now fits in with the updated python wrappers.
Describe the solution you'd like
Add missing wrappers and validate namespace corrections
Describe alternatives you've considered
None
Additional context
This is follow on work to #750