Sitelet https://web.archive.org/web/20211006133700/https://github.com/apache/hive/pull/2259
Skip to content

/ hive Public

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HIVE-25101: Remove HBase libraries from Hive distribution #2259

Merged
merged 2 commits into from Jun 23, 2021
Merged

HIVE-25101: Remove HBase libraries from Hive distribution #2259

merged 2 commits into from Jun 23, 2021

Conversation

@stoty
Copy link
Contributor

@stoty stoty commented May 11, 2021 •
edited

What changes were proposed in this pull request?

  • Mark all HBase dependencies as provided, so that they are not include in the distribution package
  • Add the hbase mapredcp jars to CLASSPATH in the hive startup script
  • remove SKIP_HBASECP option from llap service in hive script
  • Add new -k llap option to specify hbase mapredcp libraries
  • Change the code that adds the hbase dependencies to LLAP jobs automatically ti use -k instead
  • Let HiveSparkClientFactory work even if it cannot access the HBase libraries

Why are the changes needed?

Hive currently includes the HBase libraries in its distribution package, and also adds the HBase libraries separately
vi the 'hbase mapredcp' command in the startup script.

There are multiple problems with this:

  • It's redundant
  • The included HBase libraries are ancient, and cannot be easily updated
  • The hbase-shaded-mapredcp.jar added by hbase mapredcp is incompatible with the included unshaded libraries on an API level, and the ordering of the classes, and thus the effective api can be different between the local and the MR/Tez classpath

Does this PR introduce any user-facing change?

With this change, if the HBASE_HOME environment variable is not set, then the HBase libraries will not
be available in Hive. This is change from the current behaviour.
We are also removing some special case code that added the unshaded hbase libraries to the llap tar.gz, and use
hive.aux.jars.path instead.
Also, with this change there is no need to manually replace the HBase libraries in the Hive distribution.

How was this patch tested?

I have built a local pseudodistributed cluster, and manually tested that the hbase handler works with these changes.
I successfully tested with both the mr and tez engines.
I did not get llap fully working, but I tested that the generated Yarn service tar.gz includes the shaded hbase libraries and the hbase storage dirver.
AFAIK Hive does not have end-to-end tests that can test the generated package and start-up scripts.
@stoty stoty changed the title [WIP] HIVE-25101: Remove HBase libraries from HBase distribution HIVE-25101: Remove HBase libraries from Hive distribution May 12, 2021
@stoty
Copy link
Contributor Author

@stoty stoty commented May 18, 2021 •
edited

Here's what gets changed/removed from the distribution with the patch.

Apart from the obvious HBase libraries, there are some HBase dependencies.
The big one is Jersey 2.25. AFAICT, where Hive uses jersey, it includes the older com.sun.jersey artifacts from Hadoop, so this shouldn't be a problem.
I have added some missing explicit dependencies where the Hive source used undeclared dependencies for validation-api and commons-math3 .

diff  <(cd hbase;find . |sort) <(cd nohbase;find . |sort)
515d514
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/aopalliance-repackaged-2.5.0-b32.jar
574c573
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/disruptor-3.3.6.jar
---
> ./apache-hive-4.0.0-SNAPSHOT-bin/lib/disruptor-3.3.7.jar
578d576
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/findbugs-annotations-1.3.9-1.jar
584,602d581
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-client-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-common-2.0.0-alpha4-tests.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-common-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-hadoop-compat-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-hadoop2-compat-2.0.0-alpha4-tests.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-hadoop2-compat-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-http-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-mapreduce-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-metrics-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-metrics-api-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-prefix-tree-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-procedure-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-protocol-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-protocol-shaded-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-replication-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-server-2.0.0-alpha4.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-shaded-miscellaneous-1.0.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-shaded-netty-1.0.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hbase-shaded-protobuf-1.0.1.jar
641,643d619
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hk2-api-2.5.0-b32.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hk2-locator-2.5.0-b32.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/hk2-utils-2.5.0-b32.jar
645c621
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/htrace-core-3.2.0-incubating.jar
---
> ./apache-hive-4.0.0-SNAPSHOT-bin/lib/htrace-core-3.1.0-incubating.jar
671d646
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/javax.inject-2.5.0-b32.jar
680d654
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jcodings-1.0.18.jar
683,687d656
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-client-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-common-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-container-servlet-core-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-guava-2.25.1.jar
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-media-jaxb-2.25.1.jar
689d657
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/jersey-server-2.25.1.jar
720d687
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/joni-2.1.11.jar
777d743
< ./apache-hive-4.0.0-SNAPSHOT-bin/lib/osgi-resource-locator-1.0.1.jar

@github-actions github-actions bot requested a review from pgaref May 26, 2021
@stoty
Copy link
Contributor Author

@stoty stoty commented Jun 2, 2021 •
edited

This is the difference between the llap yarn archive contents with and and without the patch:

Removed:
Only in old/lib: commons-lang3-3.9.jar
Only in old/lib: hadoop-common-3.1.0.jar
Only in old/lib: hadoop-mapreduce-client-core-3.1.0.jar
Only in old/lib: hbase-client-2.0.0-alpha4.jar
Only in old/lib: hbase-common-2.0.0-alpha4.jar
Only in old/lib: hbase-hadoop-compat-2.0.0-alpha4.jar
Only in old/lib: hbase-hadoop2-compat-2.0.0-alpha4.jar
Only in old/lib: hbase-mapreduce-2.0.0-alpha4.jar
Only in old/lib: hbase-metrics-2.0.0-alpha4.jar
Only in old/lib: hbase-metrics-api-2.0.0-alpha4.jar
Only in old/lib: hbase-prefix-tree-2.0.0-alpha4.jar
Only in old/lib: hbase-protocol-2.0.0-alpha4.jar
Only in old/lib: hbase-protocol-shaded-2.0.0-alpha4.jar
Only in old/lib: hbase-server-2.0.0-alpha4.jar
Only in old/lib: hbase-shaded-miscellaneous-1.0.1.jar
Only in old/lib: hbase-shaded-netty-1.0.1.jar
Only in old/lib: hbase-shaded-protobuf-1.0.1.jar
Only in old/lib: htrace-core-3.1.0-incubating.jar
Only in old/lib: metrics-core-3.1.0.jar
Only in old/lib: zookeeper-3.5.5.jar

Added:
Only in new/lib: audience-annotations-0.5.0.jar
Only in new/lib: commons-logging-1.2.jar
Only in new/lib: hbase-shaded-mapreduce-2.4.2.jar
Only in new/lib: htrace-core4-4.2.0-incubating.jar

@stoty
Copy link
Contributor Author

@stoty stoty commented Jun 9, 2021

I have noticed one more thing while testing this change.
The hive script changes will always overwrite the hbase.aux.jar.path configuration parameter.

Now a lot of other settings, like having and auxjars directory, or setting the HIVE_AUX_JARS_PATH will do the same,
but this change will overwrite the hbase.aux.jar.path set in hbase-site.xml pretty much every single time.

I'm not sure how much of a problem this is, but I wanted to give a heads-up.

I could explore reverting to using the HADOOP_CLASSPATH instead, though I have doubts if that actually works for the distributed operations.

@kgyrtkirk

@stoty
HIVE-25101: Remove HBase libraries from Hive distribution
697de58
@stoty
re-add hbase config dir to classpath
c04d7fb
@kgyrtkirk kgyrtkirk merged commit 68daf79 into apache:master Jun 23, 2021
2 checks passed
pvary added a commit to pvary/hive that referenced this issue Jun 24, 2021
@stoty @pvary
HIVE-25101: Remove HBase libraries from Hive distribution (apache#2259)…
011b8a1 
… (Istvan Toth reviewed by Zoltan Haindrich)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kgyrtkirk kgyrtkirk

@pgaref pgaref

Assignees
No one assigned
Labels
tests passed
Projects
None yet
Milestone
No milestone
2 participants
@stoty @kgyrtkirk