Sitelet https://web.archive.org/web/20220321004721/https://github.com/apache/spark/tree/master/sql
Skip to content
master
Switch branches/tags
spark/sql/
Go to file
spark/sql/

Latest commit

@shardulm94 @srowen
shardulm94 and srowen [SPARK-38510][SQL] Retry ClassSymbol.selfType to work around cyclic r…
91614ff Mar 19, 2022
[SPARK-38510][SQL] Retry ClassSymbol.selfType to work around cyclic r… 
…eferences

### What changes were proposed in this pull request?
Retry `ClassSymbol.selfType` in case we receive a cyclic reference exception.

### Why are the changes needed?
`ClassSymbol.selfType` fails on certain classes like `HiveGenericUDF` due to cyclic reference. This issue is due to [bug#12190 in Scala](scala/bug#12190) which does not handle cyclic references in Java annotations correctly. The cyclic reference in this case comes from `InterfaceAudience` annotation which [annotates itself](https://github.com/apache/hadoop/blob/db8ae4b65448c506c9234641b2c1f9b8e894dc18/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceAudience.java#L45). This annotation class is present in the type hierarchy of `HiveGenericUDF`.

To workaround the issue, we can just retry the operation and it will succeed on the retry probably because the annotation is partially resolved from the previous attempt.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Added a unit test. This unit test prefers to be run a separate JVM as the error for cyclic references may not be thrown if the annotation class is previously loaded by some other test and so may be dependent on test execution order.

Closes #35852 from shardulm94/spark-38510.

Authored-by: Shardul Mahadik <smahadik@linkedin.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
91614ff

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
. .
catalyst
[SPARK-38510][SQL] Retry ClassSymbol.selfType to work around cyclic r…
Mar 19, 2022
core
[SPARK-38600][SQL] Include unit into the sql string of `TIMESTAMPAD…
Mar 18, 2022
hive-thriftserver
[SPARK-38572][BUILD] Setting version to 3.4.0-SNAPSHOT
Mar 17, 2022
hive
[SPARK-38510][SQL] Retry ClassSymbol.selfType to work around cyclic r…
Mar 19, 2022
README.md
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration op…
Feb 9, 2020
create-docs.sh
[SPARK-34010][SQL][DODCS] Use python3 instead of python in SQL docume…
Jan 5, 2021
gen-sql-api-docs.py
[SPARK-34747][SQL][DOCS] Add virtual operators to the built-in functi…
Mar 19, 2021
gen-sql-config-docs.py
[SPARK-36657][SQL] Update comment in 'gen-sql-config-docs.py'
Sep 3, 2021
gen-sql-functions-docs.py
[SPARK-31562][SQL] Update ExpressionDescription for substring, curren…
Apr 26, 2020
mkdocs.yml
[SPARK-30731] Update deprecated Mkdocs option
Feb 19, 2020

README.md

Spark SQL

This module provides support for executing relational queries expressed in either SQL or the DataFrame/Dataset API.

Spark SQL is broken up into four subprojects:

  • Catalyst (sql/catalyst) - An implementation-agnostic framework for manipulating trees of relational operators and expressions.
  • Execution (sql/core) - A query planner / execution engine for translating Catalyst's logical query plans into Spark RDDs. This component also includes a new public interface, SQLContext, that allows users to execute SQL or LINQ statements against existing RDDs and Parquet files.
  • Hive Support (sql/hive) - Includes extensions that allow users to write queries using a subset of HiveQL and access data from a Hive Metastore using Hive SerDes. There are also wrappers that allow users to run queries that include Hive UDFs, UDAFs, and UDTFs.
  • HiveServer and CLI support (sql/hive-thriftserver) - Includes support for the SQL CLI (bin/spark-sql) and a HiveServer2 (for JDBC/ODBC) compatible server.

Running ./sql/create-docs.sh generates SQL documentation for built-in functions under sql/site, and SQL configuration documentation that gets included as part of configuration.md in the main docs directory.