Skip to content

Bump hadoop-common to 3.3.4#3197

Closed
devinrsmith wants to merge 1 commit intodeephaven:mainfrom
devinrsmith:bump-hadoop-common
Closed

Bump hadoop-common to 3.3.4#3197
devinrsmith wants to merge 1 commit intodeephaven:mainfrom
devinrsmith:bump-hadoop-common

Conversation

@devinrsmith
Copy link
Copy Markdown
Member

https://hadoop.apache.org/docs/r3.3.4/

Two things of potential interest:

  • ARM Support
  • Adoption of lz4-java and snappy-java
    • For LZ4 and Snappy compression codec, Hadoop now moves to use lz4-java and snappy-java instead of requring the native libraries of these to be installed on the systems running Hadoop.

@devinrsmith
Copy link
Copy Markdown
Member Author

Will test on M1.

@niloc132
Copy link
Copy Markdown
Member

If we don't run out-of-band test, we don't see any actual compression happening that would exercise this. Please verify on x86 and arm, and see if we can run the brotli test on arm (just change the arch check in the extensions-parquet-table to always be true when running on arm).

I don't imagine that "ARM support" actually means arm support for brotli, but with it phrased so generally it is hard to tell.

@devinrsmith
Copy link
Copy Markdown
Member Author

Also running (x86-64) nightlies.

@devinrsmith
Copy link
Copy Markdown
Member Author

No dice on the brotli test:

java.lang.UnsatisfiedLinkError: Couldn't load native library 'brotli'. [LoaderResult: os.name="Mac OS X", os.arch="aarch64", os.version="12.5.1", java.vm.name="OpenJDK 64-Bit Server VM", java.vm.version="11.0.17+8", java.vm.vendor="Eclipse Adoptium", alreadyLoaded="null", loadedFromSystemLibraryPath="false", nativeLibName="libbrotli.dylib", temporaryLibFile="/var/folders/7z/khph2v4j2pzdf6rmc9gljvww0000gn/T/brotli11915672578012296863/libbrotli.dylib", libNameWithinClasspath="/lib/darwin-aarch64/libbrotli.dylib", usedThisClassloader="false", usedSystemClassloader="false", java.library.path="/Users/deephaven/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."]
	at org.meteogroup.jbrotli.libloader.BrotliLibraryLoader.loadBrotli(BrotliLibraryLoader.java:35)
	at org.apache.hadoop.io.compress.BrotliCodec.<init>(BrotliCodec.java:40)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780)
	... 63 more
./gradlew extensions-parquet-table:test
./gradlew extensions-parquet-table:testOutOfBand

work otherwise though.

@devinrsmith
Copy link
Copy Markdown
Member Author

@devinrsmith
Copy link
Copy Markdown
Member Author

This is out of date now, there is hadoop 3.3.5

@devinrsmith
Copy link
Copy Markdown
Member Author

See #6265

@github-actions github-actions Bot locked and limited conversation to collaborators Oct 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants