Skip to content

README.md

Latest commit

 

History

History
56 lines (41 loc) · 1.52 KB

README.md

File metadata and controls

56 lines (41 loc) · 1.52 KB

DataFusion-hdfs-native

Connecting DataFusion to HDFS through Native HDFS client (libhdfs3).

Setup libhdfs3

  1. Install libhdfs3

You can either install it via Conda

conda install -c conda-forge libhdfs3

or build it from source

# A specific version that could be compiled on osx for HDFS of 2.6.x version
git clone https://github.com/ClickHouse-Extras/libhdfs3.git
cd libhdfs3
git checkout 24b058c356794ef6cc2d31323dc9adf0386652ff

# then build it
mkdir build && cd build
../bootstrap --prefix=/usr/local
make
make install

Configuration

# client conf to use, env LIBHDFS3_CONF or hdfs-client.xml in working directory
export LIBHDFS3_CONF=/path/to/libhdfs3-hdfs-client.xml
⚡