Repository navigation
fix(examples): make csv-read-options self-contained and add output to silent examples - #1789
Open
Mola-maker wants to merge 1 commit into
Open
Mola-maker wants to merge 1 commit into
Mola-maker wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Partially addresses #1728: this PR covers the two example fixes described in the issue. The CI job proposed there is intentionally left out, because the issue asks for the shape of that job (including its skip matrix) to be agreed before writing it.
Rationale for this change
See #1728: no CI job runs the top-level
examples/*.pyscripts, socsv-read-options.pyhas been crashing on a missingdata.csv, and nine examples produce no visible output at all, which makes them read as if they do nothing.What changes are included in this PR?
examples/csv-read-options.py: the example is now self-contained. It writes its own small CSV file, plus a gzipped copy, into a temporary directory at the top and reads from those paths, instead of expecting adata.csv/data.csv.gzthat does not exist in the repository.assertstatements now show their result in the terminal, with the asserts kept:export.pyprints each exported form,import.pyshows each created DataFrame,python-udaf.py,python-udf.pyandquery-pyarrow-data.pyshow their result DataFrame,sql-using-python-udaf.pyandsql-using-python-udf.pyshow the SQL result,sql-to-pandas.pyprints the pandas DataFrame before plotting it, andsubstrait.pyprints the logical plan recovered from the round trip.Testing: all ten modified examples were run against the published
datafusion54.1.0 wheel. The originalcsv-read-options.pyfails with exit code 1 (missingdata.csv); the fixed version and all nine output examples exit 0 and print their results.sql-to-pandas.pywas verified with a small generatedyellow_tripdata_2021-01.parquetfixture (the real file is a manual download, as the issue notes) and produced both the printed aggregate table andchart.png.substrait.pywas run from the repository root with thetestingsubmodule checked out.ruff format --checkpasses on all ten files.Are there any user-facing changes?
No API changes. Only the
examples/scripts change: one is fixed so it runs at all, and nine now print or show their results when run.