Windows Development and Testing¶
This documents what you need to build and test on Windows
- Windows server 2012
- Create an admin account for yourself
- On a virtual machine: take snapshots as you go along
- Install openssl 1.0.1i 64-bit
into a dir with no spaces,
C:\bin\openssl. Add its bin dir to your path. (this will also need the Visual C++ 2008 Redistributables (DLL libraries) installed.
- Ask for lots of RAM, CPU. You can vary CPU and limits as you go, and increase them for test runs/full screen work.
- 2D acceleration of video
- Critical: Enable PAE under "processor". this avoids random hangs.
In the server management app, go to "manage features", where you can add
- Telnet server: lets you telnet in.
- Maybe: desktop experience
Firewall setup is a mystery. You need to convince the OS that your Ether network is "private", then edit the firewall to open it up.
- Bitvise SSH server; add your private key to the account you create. (even with this installed, you will still be mostly working from the windows command prompt on the machine, as it has the env variables set up)
- Java 7+
- Python 2.7
- Everything Apache Hadoop's
BUILDING.TXTneeds. This includes the cygwin tools, protoc, python and more. If you cannot build Hadoop from the
Windows SDK Command Promptthen don't proceed.
- JEdit is a good text editor for developing and editing simple files.
- Chrome can be used a browser instead of IE if you want —but you should still view all web pages in IE to verify they look OK.
tcpviewtool is ideal for seeing what ports Hadoop services are listening on.
- Knowledge of the CMD.EXE command line
How to set
- In explorer, find "This PC"
- right mouse button onto "properties"
- then "advanced properties"
- then "Environment variables"
- then the user env vars ... scroll down to find path.
- copy the
Pathvalue, edit in jedit or similar, paste back in.
- Open a new command window.
- Follow the Hadoop
- install cygwin64 with it's
- Get the cygwin64 bin dir on the PATH (remember, ";" as separators)
- find where the
CLC/C++ compiler is and make sure it is on the PATH
- get the (native) CMake 2.8+ command on the path, not any cygwin one.
Things to install in cygwin
- having cygwin
cmakeon your path; it must be the native cmake
CYGWIN=nodosfilewarning PATH = cmake bin, cygwin bin, python bin, hadoop_home bin (see below)
C:\bin\cygwin64\bin;C:\bin\Python27\;C:\Windows\system32;C:\Windows; C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\; c:\java\jdk7\bin;C:\Program Files\Microsoft Windows Performance Toolkit\; C:\apps\maven\bin;C:\apps\Git\cmd;c:\bin; C:\Program Files (x86)\MSBuild\12.0\Bin; C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin; C:\bin\cmake-2.8\bin;C:\bin\hadoop\bin;C:\bin\OpenSSL\bin
Commands to test CLI setup¶
cmake --version openssl version python --version git --version tar --version java -version mvn --version msbuild /version cl /help
and for hadoop, once installed
winutils sysinfo hadoop version hadoop checknative
Slider's test for windows support¶
You can test for windows support with the slider-core test
mvn test -Dtest=TestWindowsSupport
This looks for the arguments and other valid settings
- Find the "Windows SDK 7.1 command prompt" using the search field in the windows 8 screen
- Run this, and "pin" it to the task bar. This will be the primary console for building hadoop.
- Right mouse button on the icon in the top-left corner to bring up the "defaults" dialog.
- Choose whatever colors and fonts you want
- In layout, select a wider window size than 80 (if you want)
- in "screen buffer", set the no. of lines to 999.
- close the window and re-open it to get the new settings
From a Windows SDK command prompt:
set HADOOP_VERSION=2.6.0 set HADOOP_VERSION=2.7.0-SNPSHOT
in the root directory of the hadoop repository
mvn clean mvn install package -Pdist -Pnative-win -Dtar -DskipTests -Dmaven.javadoc.skip=true
(use two commands as sometimes clean can fail with some recursive directory-in-use exception)
pushd hadoop-dist\target tar -xvzf hadoop-%HADOOP_VERSION%.tar.gz rm -rf c:\bin\hadoop cp -r hadoop-%HADOOP_VERSION% c:\bin\hadoop popd
Hadoop is now installed into
Set up the env variables by hand
set HADOOP_HOME=C:\bin\hadoop set PATH=C:\bin\hadoop\bin;%PATH%
Verify the version is as expected
Verify that the binaries are there
hadoop checknative -a
This should report the hadoop binaries, even if it warns about missing compression libraries