@@ -492,39 +492,66 @@ Building on CentOS 8
492492
493493----------------------------------------------------------------------------------
494494
495- Building on Windows
495+ Building on Windows 10
496496
497497----------------------------------------------------------------------------------
498498Requirements:
499499
500- * Windows System
500+ * Windows 10
501501* JDK 1.8
502- * Maven 3.0 or later
503- * Boost 1.72
504- * Protocol Buffers 3.7.1
505- * CMake 3.19 or newer
506- * Visual Studio 2010 Professional or Higher
507- * Windows SDK 8.1 (if building CPU rate control for the container executor)
508- * zlib headers (if building native code bindings for zlib)
502+ * Maven 3.0 or later (maven.apache.org)
503+ * Boost 1.72 (boost.org)
504+ * Protocol Buffers 3.7.1 (https://github.com/protocolbuffers/protobuf/releases)
505+ * CMake 3.19 or newer (cmake.org)
506+ * Visual Studio 2019 (visualstudio.com)
507+ * Windows SDK 8.1 (optional, if building CPU rate control for the container executor. Get this from
508+ http://msdn.microsoft.com/en-us/windows/bg162891.aspx)
509+ * Zlib (zlib.net, if building native code bindings for zlib)
510+ * Git (preferably, get this from https://git-scm.com/download/win since the package also contains
511+ Unix command-line tools that are needed during packaging).
512+ * Python (python.org, for generation of docs using 'mvn site')
509513* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
510- * Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
511- tools must be present on your PATH.
512- * Python ( for generation of docs using 'mvn site')
513514
514- Unix command-line tools are also included with the Windows Git package which
515- can be downloaded from http://git-scm.com/downloads
515+ ----------------------------------------------------------------------------------
516516
517- If using Visual Studio, it must be Professional level or higher.
518- Do not use Visual Studio Express. It does not support compiling for 64-bit,
519- which is problematic if running a 64-bit system.
517+ Building guidelines:
520518
521- The Windows SDK 8.1 is available to download at:
519+ Hadoop repository provides the Dockerfile for building Hadoop on Windows 10, located at
520+ dev-support/docker/Dockerfile_windows_10. It is highly recommended to use this and create the
521+ Docker image for building Hadoop on Windows 10, since you don't have to install anything else
522+ other than Docker and no additional steps are required in terms of aligning the environment with
523+ the necessary paths etc.
522524
523- http://msdn.microsoft.com/en-us/windows/bg162891.aspx
525+ However, if you still prefer taking the route of not using Docker, this Dockerfile_windows_10 will
526+ still be immensely useful as a raw guide for all the steps involved in creating the environment
527+ needed to build Hadoop on Windows 10.
524528
525- Cygwin is not required.
529+ Building using the Docker:
530+ We first need to build the Docker image for building Hadoop on Windows 10. Run this command from
531+ the root of the Hadoop repository.
532+ > docker build -t hadoop-windows-10-builder -f .\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
533+
534+ Start the container with the image that we just built.
535+ > docker run --rm -it hadoop-windows-10-builder
536+
537+ You can now clone the Hadoop repo inside this container and proceed with the build.
538+
539+ NOTE:
540+ While one may perceive the idea of mounting the locally cloned (on the host filesystem) Hadoop
541+ repository into the container (using the -v option), we have seen the build to fail owing to some
542+ files not being able to be located by Maven. Thus, we suggest cloning the Hadoop repository to a
543+ non-mounted folder inside the container and proceed with the build. When the build is completed,
544+ you may use the "docker cp" command to copy the built Hadoop tar.gz file from the docker container
545+ to the host filesystem. If you still would like to mount the Hadoop codebase, a workaround would
546+ be to copy the mounted Hadoop codebase into another folder (which doesn't point to a mount) in the
547+ container's filesystem and use this for building.
548+
549+ However, we noticed no build issues when the Maven repository from the host filesystem was mounted
550+ into the container. One may use this to greatly reduce the build time. Assuming that the Maven
551+ repository is located at D:\Maven\Repository in the host filesystem, one can use the following
552+ command to mount the same onto the default Maven repository location while launching the container.
553+ > docker run --rm -v D:\Maven\Repository:C:\Users\ContainerAdministrator\.m2\repository -it hadoop-windows-10-builder
526554
527- ----------------------------------------------------------------------------------
528555Building:
529556
530557Keep the source code tree in a short path to avoid running into problems related
@@ -540,6 +567,24 @@ configure the bit-ness of the build, and set several optional components.
540567Several tests require that the user must have the Create Symbolic Links
541568privilege.
542569
570+ To simplify the installation of Boost, Protocol buffers, OpenSSL and Zlib dependencies we can use
571+ vcpkg (https://github.com/Microsoft/vcpkg.git). Upon cloning the vcpkg repo, checkout the commit
572+ 7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d to get the required versions of the dependencies
573+ mentioned above.
574+ > git clone https://github.com/Microsoft/vcpkg.git
575+ > cd vcpkg
576+ > git checkout 7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d
577+ > .\bootstrap-vcpkg.bat
578+ > .\vcpkg.exe install boost:x64-windows
579+ > .\vcpkg.exe install protobuf:x64-windows
580+ > .\vcpkg.exe install openssl:x64-windows
581+ > .\vcpkg.exe install zlib:x64-windows
582+
583+ Set the following environment variables -
584+ (Assuming that vcpkg was checked out at C:\vcpkg)
585+ > set PROTOBUF_HOME=C:\vcpkg\installed\x64-windows
586+ > set MAVEN_OPTS=-Xmx2048M -Xss128M
587+
543588All Maven goals are the same as described above with the exception that
544589native code is built by enabling the 'native-win' Maven profile. -Pnative-win
545590is enabled by default when building on Windows since the native components
@@ -557,6 +602,24 @@ the zlib 1.2.7 source tree.
557602
558603http://www.zlib.net/
559604
605+
606+ Build command:
607+ The following command builds all the modules in the Hadoop project and generates the tar.gz file in
608+ hadoop-dist/target upon successful build. Run these commands from an
609+ "x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
610+ Windows start menu. If you're using the Docker image from Dockerfile_windows_10, you'll be
611+ logged into "x64 Native Tools Command Prompt for VS 2019" automatically when you start the
612+ container.
613+
614+ > set classpath=
615+ > set PROTOBUF_HOME=C:\vcpkg\installed\x64-windows
616+ > mvn clean package -Dhttps.protocols=TLSv1.2 -DskipTests -DskipDocs -Pnative-win,dist^
617+ -Drequire.openssl -Drequire.test.libhadoop -Pyarn-ui -Dshell-executable=C:\Git\bin\bash.exe^
618+ -Dtar -Dopenssl.prefix=C:\vcpkg\installed\x64-windows^
619+ -Dcmake.prefix.path=C:\vcpkg\installed\x64-windows^
620+ -Dwindows.cmake.toolchain.file=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -Dwindows.cmake.build.type=RelWithDebInfo^
621+ -Dwindows.build.hdfspp.dll=off -Dwindows.no.sasl=on -Duse.platformToolsetVersion=v142
622+
560623----------------------------------------------------------------------------------
561624Building distributions:
562625
0 commit comments