Hadoop from scratch notes: Preparing a minimal CentOs Linux Hyper-V image

Motivation

Hadoop on virtual machines? These posts are describing how to setup a hadoop homelab, to get in touch with hadoop. Yet, replace 'virtual' with 'dedicated physical', then you should by on your way to build a production cluster.

Hadoop is yet another good tool in the toolbox, when working with data. Now a days Hadoop is available as cloud service, but it can be pretty expensive, and specially if you just want to train and play with Hadoop. Some vendors as Cloudera offers a single node 'play' version of Hadoop, which is a great way to start. Yet the reason I'mt writing these notes, was that I did find Cloudera very closed and slow, and also I had to use any other Virtual Machine system than Hyper-V. Also it is not that hard to set up a Hadoop node or cluster from scratch.

Not that, I have anything against e.g. Virtual Box. Even thou I see all OS'es as my play grounds, I'm in a Microsoft period(due to my current work), and thereby my Windows box is best suited for virtualization. And it does already comes with Hyper-V, and it actually works good in Windows 10(Earlier versions did lock the CPU clock cycle, and thereby disabled speed-step). I like my machines light, so I would hate to have more than one system for virtualization.

What is the goal?

The goal is to prepare a virtual machine with a minimal version of Centos Linux. The reason, I have selected Centos OS is, it is supported by Microsoft, and it is Azure certified, and when it is Azure certified, it means it can work better with Hyper-V through Hyper-V Integration Services. I could have chosen Ubuntu (Azure's Hadoop cloud solution runs on Ubuntu), but I had a challenge with a very slow apt-get, and generally did find CentOS more light weight.

When we have a fully configured virtual machine, with CentOS and Hadoop, we are going to use it as a template, for creating more Hadoop nodes.

I prefer to setup Hyper-V with PowerShell, it is good fun and practice, and it more compact than images of the GUI. If you are familiar with the Hyper-V GUI, then you should have no trouble to figure out what to press.

Before we start, make sure Hyper-V is enabled, and get CentOS from here https://www.centos.org/download/, the minimal ISO should be sufficient(CentOS 7 is currently the latest version).

A virtual switch

If you don't have a virtual switch configured in Hyper-V, you have to configure one. You are going to use it for connecting you Hadoop nodes, the internet and you working machine together. Thou the internet is optionally. Creating a so-called external virtual switch called "Virtual Switch" (Yes, I know, the creative name is striking :-) ), is done by typing following PowerShell:

New-VMSwitch -Name "Virtual Switch" -NetAdapterName "Wi-Fi" -AllowManagementOS 1

As NetAdaptorName use "Wi-Fi" or "Ethernet", depending on which NIC provides internet.

The virtual machine and disk

Often the virtual machine and the disk interpreted as one, but,  a virtual machine consist of the "Machine" and "disk image" with the OS, and further, of some data disks". We are going to create the machine and OS disk in one go. 

New-VM -Name "Hadoop01" -MemoryStartupBytes 4GB -NewVHDPath D:\VMs\Hadoop01.vhdx -NewVHDSizeBytes 10GB -SwitchName "Virtual Switch"

Memory (4 GigaBytes) and disk (10 GigaBytes) sizes are dymanic by default, but the machine is only configured with 1 CPU. It can be upgraded with:

Set-VMProcessor -VMName Hadoop01 -Count 2

Make the virtual DVD point to the download CentOS image:

Set-VMDvdDrive -VMName Hadoop01 -Path D:\Downloads\CentOS-7-x86_64-Minimal-1511.iso

Let's go:

Start-VM Hadoop01

You have to connect to the virtual machine by the Hyper-V GUI-

Installing CentOS

Press Enter. I might take a while, before reaching next step

Select your preferred language

Check that the properties circled with yellow, are correct. That will make thing easier for you in generel. The properties circled with red, are critical, so make sure to read below how to set them.

Press 'Done', that is all

Turn on the network. Failing to do this, can require you to turn it on, after every reboot.

Set the root password. Create a user for good practice.

After installation and reboot. Log in, so we can get the IP address of our new machine, by typing the following command(ifconfig is not available on CentOS minimal)

ip addr

 Note the IP address, it can be found under Eth0: 

We are not going to use the Hyper-V viewer further. It can't copy-paste between guest and host, and the proper way to connect to a Linux/Unix server is via a SSH client. I recommend Putty (http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html), but the Git Bash is just as fine.

Type in the IP and press Open.

If using the Git Bash, you can write:

ssh <ip> -l <user>

Where <IP> is the noted IP and <user> is either root or the user created earliere.

Installing/Upgrading Microsoft Linux Integration Services(LIS)

We don't have much in the CentOS minimal, and Microsoft haven't made it easy to download the LIS package without a browser.

Fortunately, it is GNU licensed, so I have made a script to get it from my GitHub account, and to install it, together with wget.

curl -O https://raw.githubusercontent.com/ChristianHenrikReich/automation-scripts/master/centos-minimal/install-hyperv-essentials.sh

chmod 755 install-hyperv-essentials.sh

sudo ./install-hyperv-essentials.sh

When the script is done. The Virtual machine is fully Hyper-V prep'ed and ready to go. And can be used to other things than Hadoop also.

Next: How to install Hadoop on the image

http://programminghave.blogspot.com/2016/03/hadoop-from-scratch-notes-installing.html

Wrestling the Azure Storage REST API - Part 2

This post is about authorization HTTP header, used when making requests to Azure Storage API. There is some dependencies the previous part of this series, specially regarding the x-ms-date header field.

Authorization

The authorization field is expressed in this way:

Authorization="[SharedKey|SharedKeyLite] <AccountName>:<Signature>"

The authorization supports 2 schemes for calculating signatures, Shared Key or Shared Key Lite. The scheme you are using for authorization must be stated with either SharedKey or SharedKeyLite as the first thing in the header field.

The difference between the schemes are, Shared Key Lite is backward compatible with earlier versions of the Azure Storage API. I can't remember to have seen any example with Shared Key, I guess it is because it requires more effort to make it work.

Important!!! When using dates in authorization, these dates must be the same as x-ms-date, or else the authorisation will fail.

Shared key

Blob, Queue and File Storage signature is calculated one way, while Table Storage signature is calculated in another way. 

Blob, Queue and File Storage:

StringToSign = 

VERB + "\n" +
Content-Encoding + "\n" +
Content-Language + "\n" +
Content-Length + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Date + "\n" +
If-Modified-Since + "\n" +
If-Match + "\n" +
If-None-Match + "\n" +
If-Unmodified-Since + "\n" +
Range + "\n" +
CanonicalizedHeaders +
CanonicalizedResource;

Table Storage:

StringToSign = 

VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Date + "\n" + 

CanonicalizedResource;

Shared key Lite

Like Shared Keys, there is a difference in calculating the keys depending on what kind of storage is used:

Blob, Queue and File Storage:

StringToSign =
VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Date + "\n" +
CanonicalizedHeaders +
CanonicalizedResource;

Table Storage:

StringToSign = 

Date + "\n" +
CanonicalizedResource

When comparing the 2 schemes, it begins to make sense why most chose to use Shared Key Lite

Which parameter must be filled, depends heavily on context. E.g. Date, while it must be set in every Table request, there is some Blob requests where it must not be set.

Canonicalized Headers

Just take all the header starting with x-ms-, sort them and concatenate them separated by \n.

Exmaple(taken from the Azure Storage documentation):

 x-ms-date:Sun, 20 Sep 2009 20:36:40 GMT\nx-ms-meta-m1:v1\nx-ms-meta-m2:v2\n

Canonicalized Resources

Canonicalized resources is form the following way:

Canonicalized resource = /account/resource

Example:

For this request

GET https://myaccount.table.core.windows.net/Tables HTTP/1.1

The canonicalized resource will be /myaccount/Tables

Query parameters must not be included. Unless you make following request(taken from documentation):

GET https://<account-name>.table.core.windows.net/?restype=service&comp=properties HTTP/1.1

Here the canonicalized resource will be /myaccount/?comp=properties

Calculating the signature

Here is the Azure Storage REST API documentation pretty weak. 2 things it misses are, when using HMAC you need to supply a key and a message. In context of making requests to the Azure Storage REST API, the key is either the Primary og Secondary key, which can be obtain from the Azure portal. The message is the StringToSign defined earlier in this post.

Also, the Primary and Secondary key which is found on the Azure portal are base64 encoded, you need to decode them, in order to be able to use them.

So what the documentation states as

Signature=Base64(HMAC-SHA256(UTF8(StringToSign))) 

Is in reality

Signature=Base64(HMAC-SHA256(UTF8(Debase64(key)),UTF8(StringToSign))) 

Where key is either the Primary or Secondary key.

And this is all for Authorization.

Wrestling the Azure Storage REST API - Part 1

Motivation

With Azure SDKs for a wide variety of programming languages, why should anybody want to learn about the Azure Storage REST API? 

Maybe there is no SDK for your favourite language, which was my case. Maybe the official SDK do not support the latest API version, which could mean it is not possible to communicate with JSON in Table Storage. Maybe you are just courios.

This blog post is based on my work on GoHaveAzureStorage, and hopefully you will also gain from challenges I have had the Azure Storage.

Request break down 

A REST call looks like this:

GET https://myaccount.table.core.windows.net/Tables HTTP/1.1

This request is used to get all tables for an storage account. There is 2 mandatory http header fields, and an additional optional which I recommend, which you must send for every request to make it work. They are 

• x-ms-date       - time for the request.
• x-ms-version  - Which API version is the request targeting
• authorization  - Which is a security digest

The first 2 header will be explained in this post, while the Authorization will be explained in part 2.

The URL

First the easy part. It is possible to use either HTTP or HTTPS, else it is more or less straight forward.

The x-ms-date header field

This field is used by Azure for validation and authorization. A valid request must be maximum 15 minutes old, and it must not be dated in the future. It can be expressed as:

current time =< x-ms-date < current time - 15

Pro tip

As it is close to impossible to be complete time synchronized with Azure, it is recommend to substract a few minutes from current time, when sending request.

One important last ting, Azure only understands time in RFC1123 format and GMT +0

If you have: Thu, 12 Feb 2015 21:16:45 UTC in a +1 time zone

It must converted to: Thu, 12 Feb 2015 20:16:45 GMT

The x-ms-version header field (Pro tip)

This an optional field, but you should prefer to set it, or else you will hit an earlier version of the Azure Storage API. you might experience challenges with JSON in table storage or with Shared Access Keys if not using the latest version.

The versions are defined as date, The date which the API version is released. I'm not sure wether this a good solution, because I find dates hard to remember after a while compared to versions. So I have to look up once in a while here: https://msdn.microsoft.com/en-us/library/azure/dd894041.aspx