Merge pull request #1 from apache/trunk

sync

Author: 2899722744

.gitignore

@@ -24,4 +24,5 @@ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/tla/yarnregistry.t
 yarnregistry.pdf
 hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml
 hadoop-tools/hadoop-aws/src/test/resources/contract-test-options.xml
+hadoop-tools/hadoop-azure/src/test/resources/azure-auth-keys.xml
 patchprocess/

BUILDING.txt

@@ -11,7 +11,6 @@ Requirements:
 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
 * Zlib devel (if compiling native code)
 * openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
-* Jansson C XML parsing library (if compiling libwebhdfs)
 * Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
 * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
 * python (for releasedocs)
@@ -28,19 +27,19 @@ On Linux:
     $ ./start-build-env.sh
 
 On Mac:
-    First make sure Homebrew has been installed ( http://brew.sh/ )
-    $ brew install docker boot2docker
-    $ boot2docker init -m 4096
-    $ boot2docker start
-    $ $(boot2docker shellinit)
+    First make sure Virtualbox and docker toolbox are installed.
+    You can use docker toolbox as described in http://docs.docker.com/mac/step_one/.
+    $ docker-machine create --driver virtualbox \
+                            --virtualbox-memory "4096" hadoopdev
+    $ eval $(docker-machine env hadoopdev)
     $ ./start-build-env.sh
 
 The prompt which is then presented is located at a mounted version of the source tree
 and all required tools for testing and building have been installed and configured.
 
 Note that from within this docker environment you ONLY have access to the Hadoop source
 tree from where you started. So if you need to run
-    dev-support/test-patch.sh /path/to/my.patch
+    dev-support/bin/test-patch /path/to/my.patch
 then the patch must be placed inside the hadoop source tree.
 
 Known issues:
@@ -74,6 +73,8 @@ Optional packages:
 
 * Snappy compression
   $ sudo apt-get install snappy libsnappy-dev
+* Intel ISA-L library for erasure coding
+  Please refer to https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
 * Bzip2
   $ sudo apt-get install bzip2 libbz2-dev
 * Jansson (C Library for JSON)
@@ -148,7 +149,8 @@ Maven build goals:
     installed snappy using a package manager.
   * Use -Dbundle.snappy to copy the contents of the snappy.lib directory into
     the final tar file. This option requires that -Dsnappy.lib is also given,
-    and it ignores the -Dsnappy.prefix option.
+    and it ignores the -Dsnappy.prefix option. If -Dsnappy.lib isn't given, the
+    bundling and building will fail.
 
  OpenSSL build options:
 
@@ -169,7 +171,8 @@ Maven build goals:
     installed openssl using a package manager.
   * Use -Dbundle.openssl to copy the contents of the openssl.lib directory into
     the final tar file. This option requires that -Dopenssl.lib is also given,
-    and it ignores the -Dopenssl.prefix option.
+    and it ignores the -Dopenssl.prefix option. If -Dopenssl.lib isn't given, the
+    bundling and building will fail.
 
    Tests options:
 
@@ -178,6 +181,34 @@ Maven build goals:
   * -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....
   * -Dtest.exclude=<TESTCLASSNAME>
   * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java
+  * To run all native unit tests, use: mvn test -Pnative -Dtest=allNative
+  * To run a specific native unit test, use: mvn test -Pnative -Dtest=<test>
+  For example, to run test_bulk_crc32, you would use:
+  mvn test -Pnative -Dtest=test_bulk_crc32
+
+ Intel ISA-L build options:
+
+   Intel ISA-L is a erasure coding library that can be utilized by the native code.
+   It is currently an optional component, meaning that Hadoop can be built with
+   or without this dependency. Note the library is used via dynamic module. Please
+   reference the official site for the library details.
+   https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
+
+  * Use -Drequire.isal to fail the build if libisal.so is not found.
+    If this option is not specified and the isal library is missing,
+    we silently build a version of libhadoop.so that cannot make use of ISA-L and
+    the native raw erasure coders.
+    This option is recommended if you plan on making use of native raw erasure
+    coders and want to get more repeatable builds.
+  * Use -Disal.prefix to specify a nonstandard location for the libisal
+    library files. You do not need this option if you have installed ISA-L to the
+    system library path.
+  * Use -Disal.lib to specify a nonstandard location for the libisal library
+    files.
+  * Use -Dbundle.isal to copy the contents of the isal.lib directory into
+    the final tar file. This option requires that -Disal.lib is also given,
+    and it ignores the -Disal.prefix option. If -Disal.lib isn't given, the
+    bundling and building will fail.
 
 ----------------------------------------------------------------------------------
 Building components separately

dev-support/README.md

@@ -0,0 +1,57 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+This directory contains tools to help in the development and release of Apache Hadoop.
+
+* bin
+
+  * releasedocmaker
+
+    Build release notes for a given Hadoop project or subproject.  This is called from Maven when -Preleasedocs is used.  See BUILDING.txt for more information.
+
+  * shelldocs
+
+    Create documentation for the Unix Shell API.  This is called from Maven when -Pshelldocs is used.
+
+  * smart-apply-patch
+
+    Intelligently apply a patch file to a source tree.
+
+  * test-patch
+
+    Test a patch against a source tree.
+
+* create-release.sh
+
+  Helps REs create a release of Apache Hadoop for distribution.
+
+* determine-flaky-tests-hadoop.py
+
+  Given a jenkins test job, this script examines all runs of the job done within specified period of time (number of days prior to the execution time of this script), and reports all failed tests.
+
+* docker
+
+  Various helpers for the start-build-env.sh script, including the Dockerfile itself. See parent BUILDING.txt for more information.
+
+* findHangingTest.sh
+
+  Finds hanging test from Jenkins build output.
+
+
+Previously, the scripts test-patch.sh, smart-apply-patch.sh, releasedocmaker.py, and shelldocs.py were in this directory.  They have been moved to the Apache Yetus project (https://yetus.apache.org).  These scripts have been replaced with wrapper scripts located in the bin directory. Command line options are generally different than the previous versions that shipped with older versions of Apache Hadoop.
+
+The wrapper scripts will download, verify (if GPG is installed), and cache a local copy of Apache Yetus in the hadoop/patchprocess directory. The version that is used may be overridden by setting the HADOOP\_YETUS\_VERSION environment variable.  The cache directory may be overwritten by setting the HADOOP\_PATCHPROCESS directory.  If a local version of Apache Yetus is already installed, it may be used instead by setting the YETUS\_HOME environment variable to point to that directory.

dev-support/bin/releasedocmaker

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
+exec "${BINDIR}/yetus-wrapper" releasedocmaker "$@"

dev-support/bin/shelldocs

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
+exec "${BINDIR}/yetus-wrapper" shelldocs "$@"

dev-support/bin/smart-apply-patch

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
+exec "${BINDIR}/yetus-wrapper" smart-apply-patch --project=hadoop "$@"

dev-support/bin/test-patch

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
+exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop --skip-dir=dev-support "$@"