Sitelet https://web.archive.org/web/20220321010457/https://github.com/apache/spark/tree/master/python
Skip to content
master
Switch branches/tags
spark/python/
Go to file
spark/python/

Latest commit

@HyukjinKwon
HyukjinKwon [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
97335ea Mar 18, 2022
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5 
### What changes were proposed in this pull request?

This PR is a retry of #35871 with bumping up the version to 0.10.9.5.
It was reverted because of Python 3.10 is broken, and Python 3.10 was not officially supported in Py4J.

In Py4J 0.10.9.5, the issue was fixed (py4j/py4j#475), and it added Python 3.10 support officially with CI set up (py4j/py4j#477).

### Why are the changes needed?

See #35871

### Does this PR introduce _any_ user-facing change?

See #35871

### How was this patch tested?

Py4J sets up Python 3.10 CI now, and I manually tested PySpark with Python 3.10 with this patch:

```bash
./bin/pyspark
```

```
import py4j
py4j.__version__
spark.range(10).show()
```

```
Using Python version 3.10.0 (default, Mar  3 2022 03:57:21)
Spark context Web UI available at http://172.30.5.50:4040
Spark context available as 'sc' (master = local[*], app id = local-1647571387534).
SparkSession available as 'spark'.
>>> import py4j
>>> py4j.__version__
'0.10.9.5'
>>> spark.range(10).show()
+---+
| id|
+---+
...
```

Closes #35907 from HyukjinKwon/SPARK-38563-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
97335ea

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
. .
docs
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
Mar 18, 2022
lib
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
Mar 18, 2022
pyspark
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
Mar 18, 2022
test_coverage
[SPARK-36092][INFRA][BUILD][PYTHON] Migrate to GitHub Actions with Co…
Aug 1, 2021
test_support
[SPARK-37632][PYTHON] Drop code that targets Python < 3.7
Dec 16, 2021
.coveragerc
[SPARK-7721][PYTHON][TESTS] Adds PySpark coverage generation script
Jan 22, 2018
.gitignore
[SPARK-3946] gitignore in /python includes wrong directory
Oct 14, 2014
MANIFEST.in
[SPARK-32714][PYTHON] Initial pyspark-stubs port
Sep 24, 2020
README.md
[SPARK-36474][PYTHON][DOCS] Mention 'pandas API on Spark' in Spark ov…
Aug 11, 2021
mypy.ini
[SPARK-38424][PYTHON] Warn unused casts and ignores
Mar 16, 2022
run-tests
[SPARK-29672][PYSPARK] update spark testing framework to use python3
Nov 14, 2019
run-tests-with-coverage
[SPARK-36092][INFRA][BUILD][PYTHON] Migrate to GitHub Actions with Co…
Aug 1, 2021
run-tests.py
[SPARK-37836][PYTHON][INFRA] Enable F841, E722, E305 and E226 for PEP…
Jan 7, 2022
setup.cfg
[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed
Nov 16, 2016
setup.py
[SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.5
Mar 18, 2022
Apache Spark Online Documentation Python Packaging Python Requirements

README.md

Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

https://spark.apache.org/

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project web page

Python Packaging

This README file only contains basic information related to pip installed PySpark. This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility). Using PySpark requires the Spark JARs, and if you are building this from source please see the builder instructions at "Building Spark".

The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to set up your own standalone Spark cluster. You can download the full version of Spark from the Apache Spark downloads page.

NOTE: If you are using this with a Spark standalone cluster you must ensure that the version (including minor version) matches or you may experience odd errors.

Python Requirements

At its core PySpark depends on Py4J, but some additional sub-packages have their own extra requirements for some features (including numpy, pandas, and pyarrow). See also Dependencies for production, and dev/requirements.txt for development.