Parag Ray
29-Sep-2014
Introduction
Welcome to the readers!
This is the fourth part of the series of articles. Here we actually start setting up HDFS in a multi node cluster and have a hands on experience of the HDFS.
If you have not gone through already,You would necessarily go back to third part of the single node set up(see related readings) as this post will depend on the single node set up.
Please note the related readings and target audience section to get help to better follow the blog.
We are assuming the operating system to be Ubuntu 14.04.
Agenda
Related readings/other blogs 29-Sep-2014
Introduction
Welcome to the readers!
This is the fourth part of the series of articles. Here we actually start setting up HDFS in a multi node cluster and have a hands on experience of the HDFS.
If you have not gone through already,You would necessarily go back to third part of the single node set up(see related readings) as this post will depend on the single node set up.
Please note the related readings and target audience section to get help to better follow the blog.
We are assuming the operating system to be Ubuntu 14.04.
Agenda
- Target audience
- Related readings/other blogs
- Multi-node cluster setup steps
- This is an intermediate level discussion on Hadoop and related tools.
- Best suited for audience who are looking for introduction to this technology.
- Prior knowledge in java and Linux required.
- Intermediate level understanding of networking necessary.
Please refer MR setup first part in section 3 first.You would also like to look at Cloudera home page & Hadoop home page for further details.
Multi-node cluster setup steps
Multi-node set up requires that single node set up is done in each of the nodes. This can be done any time before step 3 Hadoop configuration below.
- Network connect
Connect the underlying network - Ssh connectivity
Setup ssh connection - HDFS configuration for multi-node.
A home network is demonstrated here.The set up is academic and not production quality. A full blown setup would set up comprise of all components including segment A to B below, but we are covering only segment B here with two nodes, one master and one slave.IP 4 network is shown here.
Ip address set up:
Navigate to Use system settings > network>wired connection and turn the Wired network on.
Set 'manual' 'ip4' config and provide ip 'address' and use same 'netmask' for all PCs in network. Network config details may come under 'Options' button
In case you are using other Linux , the set up may be different. Here is one example for oracle linux.
SSH connectivity
SSH connectivity is used for connectivity may be used across nodes.
This needs to be connectivity across nodes where data communication is expected. Two way connectivity between master and slave node is a must.
Self connectivity is also a must.
In the above diagram NN indicate name node and DN indicate datanode of course.
- Master and slaves talk via ssh connectivity.
- We need ssh without need to provide credential for each requests.
- The set up for ssh without password to provided is not ideal for security but
In this case for each transaction of hadoop, providing credential is not practical.
- There are other ways ,but for the purpose of this presentation, we shall stick to this set up
- Both master and slave nodes must open ssh connectivity to each other and
to themselves.
- It is assumed that for all these we are using the hadoop user. Sudo to respective user is also an option.
- First from master node check that ssh to the localhost happens without a passphrase:
- If passphrase is requested, execute the following commands:
(The part -f ~/.ssh/id_rsa is not mandatory as that is the default location and system will ask for this location if not provided)
- Copy public key to authorized_keys file
- On master node try to ssh again to localhost and if passphrase is still requested, do the following
chmod 600 $HOME/.ssh/authorized_keys
chown <<hadoop user>> $HOME/.ssh/authorized_keys
it will be a good idea to check if you can do
ssh localhost and also ssh <<nodename>>
- If after executing ssh localhost any error message is returned that indicates that ssh server is not installed you may like to try the following,-
- Use the following command to install ssh server as under,-
- Then can repeat the steps starting from trying ssh localhost
- Now public key needs to be copied to all of slave machine (only one here). Ssh should be installed in all slave machines. We do strongly recommend ssh understanding.
- Now login (ssh <<slave host name>>) to your slave machine. Password will be needed still.
- While on your slave machine ,issue the following command to append your master machine’s hadoop user’s public key to the slave machine’s authorized key store,
- Issue exit to come out of slave
- Now, from the master node try to ssh to slave.
- if passphrase is still requested, do the following after logging in with password to slave
chmod 600 $HOME/.ssh/authorized_keys
chown <<hadoop user>> $HOME/.ssh/authorized_keys
- Exit and log in again from your master node.
- if it still asks for password , log in using password, log out and issue
- Repeat for all Hadoop Cluster Nodes which now will point back to master.
/etc/hosts
should have all host names
parag@PRUBNode1:/etc$ cat hosts
10.192.0.2 PRUBNode2
10.192.0.1 PRUBNode1
127.0.0.1 localhost
there could be one problem that might happen out of hosts file.
to detect this problem run sudo netstat -ntlp on the master , it shows:
tcp6 0 0 127.0.0.1:9020 :::* LISTEN 32646/java
This 127.0.0.1 means that it is only listening to connection on 9020 which is from localhost , all the connection on 9020 from outside cannot be received.
this may be resolved by preventing entries that point local host for both actual ip and 127.0.0.1.
the entry shown above will not have this problem but the following will
10.192.0.1 localhost
10.192.0.2 PRUBNode2
10.192.0.1 PRUBNode1
127.0.0.1 localhost
there is likely to be problems if hostnames are missed out as well.
slaves
slaves file in the folder $HADOOP_HOME/conf ,should have all the node names that has data node running
we have proved the listing as below,
parag@PRUBNode1:~/work/hadoop-1.2.1/conf$ cat slaves
PRUBNode1
PRUBNode2
master
master file n the folder $HADOOP_HOME/conf should have all the node names that has secondarynamenode running
parag@PRUBNode1:~/work/hadoop-1.2.1/conf$ cat masters
PRUBNode1
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</n
ame>
<value>/home/parag/work/hdf/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/parag/work/hdf/data</value>
</property>
</configuration>
all folders should be generic and same across all nodes
core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://PRUBNode1:9020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/parag/work/hdf/tmp</value>
</property>
</configuration>
all folders should be generic and same across all nodes. we are now mentioning the nodename of master for fs.defaultt.name instead of local host as we expect it to be accessed from different nodes.
Starting HDFS
In the name node containing Node issue command
./start-dfs.sh
from $HADOOP_HOME/bin folder
it should start name node , data node and secondary name node in master and data node in slave.
if it is first time, name node format to be done.
Test set up
Test 1: Issue command jps at master
parag@PRUBNode1:~/work/hadoop-1.2.1/bin$ jps
9530 DataNode
9700 SecondaryNameNode
9357 NameNode
10252 Jps
Issue command jps at slave will show only DataNode
Test 2: Open administrative console at http://<<hostname of master>>:50070
for me it is http://PRUBNode1:50070
No comments:
Post a Comment
It will be by pleasure to respond to any of your queries, and i do welcome your suggestions for making the blogs better.