Native Libraries Guide

Native Hadoop Library

Hadoop has native implementations of certain components for performance reasons and for non-availability of Java implementations. These components are available in a single, dynamically-linked native library called the native hadoop library. On the *nix platforms the library is named libhadoop.so.

Usage

It is fairly easy to use the native hadoop library:

1. Review the components.
2. Review the supported platforms.
3. Either download a hadoop release, which will include a pre-built version of the native hadoop library, or build your own version of the native hadoop library. Whether you download or build, the name for the library is the same: libhadoop.so
4. Install the compression codec development packages (>zlib-1.2, >gzip-1.2):
   • If you download the library, install one or more development packages - whichever compression codecs you want to use with your deployment.
   • If you build the library, it is mandatory to install both development packages.
5. Check the runtime log files.

Components

The native hadoop library includes various components:

• Compression Codecs (bzip2, lz4, snappy, zlib)
• Native IO utilities for HDFS Short-Circuit Local Reads and Centralized Cache Management in HDFS
• CRC32 checksum implementation

Supported Platforms

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:

• RHEL4/Fedora
• Ubuntu
• Gentoo

On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.

Build

The native hadoop library is written in ANSI C and is built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). This means it should be straight-forward to build the library on any platform with a standards-compliant C compiler and the GNU autotools-chain (see the supported platforms).

The packages you need to install on the target platform are:

• C compiler (e.g. GNU C Compiler)
• GNU Autools Chain: autoconf, automake, libtool
• zlib-development package (stable version >= 1.2.0)
• openssl-development package(e.g. libssl-dev)

Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:

   $ mvn package -Pdist,native -DskipTests -Dtar

You should see the newly-built library in:

   $ hadoop-dist/target/hadoop-2.7.1/lib/native

Please note the following:

• It is mandatory to install both the zlib and gzip development packages on the target platform in order to build the native hadoop library; however, for deployment it is sufficient to install just one package if you wish to use only one codec.
• It is necessary to have the correct 32/64 libraries for zlib, depending on the 32/64 bit jvm for the target platform, in order to build and deploy the native hadoop library.

 

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html