Cannot convert Expr to InList

Describe the bug
When a SQL query contains a InList Expr, I can't get the InList object through Expr.to_variant().

To Reproduce

from datafusion import SessionContext
from datafusion.expr import Filter


def traverse_logical_plan(plan):
    cur_node = plan.to_variant()
    if isinstance(cur_node, Filter):
        predicate = cur_node.predicate().to_variant()
    if hasattr(plan, 'inputs'):
        for input_plan in plan.inputs():
            traverse_logical_plan(input_plan)


if __name__ == "__main__":
    ctx = SessionContext()
    data = {'id': [1, 2, 3], 'name': ['Alice', 'Bob', 'Charlie']}
    ctx.from_pydict(data, name='table1')
    query = "SELECT * FROM table1 t1 WHERE t1.name IN ('dfa', 'ad', 'dfre', 'vsa')"
    logical_plan = ctx.sql(query).optimized_logical_plan()
    traverse_logical_plan(logical_plan)

It produces,

  File "minimal_example.py", line 8, in traverse_logical_plan
    predicate = cur_node.predicate().to_variant()
RuntimeError: "Cannot convert this Expr to a Python object: InList(InList { expr: Column(Column { relation: Some(Bare { table: \"table1\" }), name: \"name\" }), list: [Literal(Utf8(\"dfa\")), Literal(Utf8(\"ad\")), Literal(Utf8(\"dfre\")), Literal(Utf8(\"vsa\"))], negated: false })"

Is there any other method to get the InList object?

I use datafusion 39.0.0 in Python 3.10.11.

Just for my own education, what are the use cases for invoking .to_variant()? I came across these when I was working on python wrapper functions and I don't know how people are using these objects. Can you help me understand why we would want to do this traversal? I'm going to tackle exposing all of those variants in #767 and the better I can understand the user needs, the better I can make sure I'm getting them covered. Thank you!

Just for my own education, what are the use cases for invoking .to_variant()? I came across these when I was working on python wrapper functions and I don't know how people are using these objects. Can you help me understand why we would want to do this traversal? I'm going to tackle exposing all of those variants in #767 and the better I can understand the user needs, the better I can make sure I'm getting them covered. Thank you!

Thanks for your support. I would like to implement a custom query optimizer based on the LogicalPlan generated by Datafusion. It needs to obtain expressions within the LogicalPlan (such as filter and join conditions), and then process them differently according to the type of expression (such as In, Like, Cast, etc.). However, implementing it with string matching is very difficult because many edge cases need to be considered. Therefore, I hope to use to_variant() to conveniently perform type checks and extract specific fields of different types of expressions.

Navigation Menu

Repository navigation

Cannot convert Expr to InList #781

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cannot convert Expr to InList #781

Description

Activity

HuashiSCNU0303 commented on Jul 27, 2024

timsaucer commented on Aug 1, 2024

HuashiSCNU0303 commented on Aug 2, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions