[hadoop]完全分布式集群安装过程详解 1. 用vmware workstation创建4个虚拟机，每个虚拟机都装上centos（版本：centos-6.3-x86_64），示意图如下： 2. 在所有结点上修改/etc/hosts，使彼此之间都能够用机器名解析ip 192.168.231.131 node01 192.168.231.132 n
[hadoop]完全分布式集群安装过程详解
1. 用vmware workstation创建4个虚拟机，每个虚拟机都装上centos（版本：centos-6.3-x86_64），示意图如下：
2. 在所有结点上修改/etc/hosts，使彼此之间都能够用机器名解析ip
192.168.231.131 node01
192.168.231.132 node02
192.168.231.133 node03
192.168.231.134 node04
3. 在所有结点上安装jdk
首先，把jdk安装包（jdk-6u38-linux-x64.bin）放到/usr/java
增加可执行权限：
[root@localhost java]# chmod a+xjdk-6u38-linux-x64.bin
[root@localhost java]# ls -lrt
total 70376
-rwxr-xr-x. 1 root root 72058033 jan 2907:21 jdk-6u38-linux-x64.bin
下面开始安装jdk：
[root@localhost java]# ./jdk-6u38-linux-x64.bin
更改/etc/profile，添加以下几行：
java_home=/usr/java/jdk1.6.0_38
jre_home=/usr/java/jdk1.6.0_38/jre/
classpath=.:$java_home/lib:$java_home/lib/tools.jar
path=$java_home/bin:$jre_home/bin:$path
测试是否安装成功：
[root@localhost java]# source /etc/profile
[root@localhost java]# java -version
java version 1.6.0_38
java(tm) se runtime environment (build1.6.0_38-b05)
java hotspot(tm) 64-bit server vm (build20.13-b02, mixed mode)
4. 添加hadoop用户
[root@node02 ~]# useradd hadoop -g root
[root@node02 ~]# passwd hadoop
changing password for user hadoop.
new password:
bad password: it is too short
bad password: is too simple
retype new password:
passwd: all authentication tokens updatedsuccessfully.
5. ssh 配置
注意：下面开始以hadoop用户操作
[hadoop@node01 ~]$ ssh-keygen -t rsa
generating public/private rsa key pair.
enter file in which to save the key(/home/hadoop/.ssh/id_rsa):
created directory '/home/hadoop/.ssh'.
enter passphrase (empty for no passphrase):
enter same passphrase again:
your identification has been saved in /home/hadoop/.ssh/id_rsa.
your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.
the key fingerprint is:
1d:03:8c:2f:99:95:98:c1:3d:8b:21:61:3e:a9:cb:bfhadoop@node01
the key's randomart image is:
+--[ rsa 2048]----+
| oo.b.. |
| o..* *. |
| +. b oo |
| ..= o. o |
| . .s . |
| . . |
| o |
| . |
| e. |
+-----------------+
[hadoop@node01 ~]$ cd .ssh
[hadoop@node01 .ssh]$ cp id_rsa.pubauthorized_keys
把所有结点的authorized_keys的内容都互相拷贝，这样就可以免密码ssh连入。
6. 安装hadoop
[hadoop@node01 ~]$ ls
hadoop-0.20.2 hadoop-0.20.2.tar.gz
[hadoop@node01 ~]$ tar xzvf./hadoop-0.20.2.tar.gz
7. 配置namenode (node01)
修改hadoop-env.sh
[hadoop@node01 conf]$ vi hadoop-env.sh
# the java implementation to use. required.
export java_home=/usr/java/jdk1.6.0_38
修改core-site.xml
[hadoop@node01 conf]$ vi core-site.xml
fs.default.name
hdfs://192.168.231.131:9000
修改hdfs-site.xml
[hadoop@node01 conf]$ vi hdfs-site.xml
dfs.data.dir
/home/hadoop/hadoop-0.20.2/data
dfs.replication
3
修改mapred-site.xml
[hadoop@node01 conf]$ vi mapred-site.xml
mapred.job.tracker
192.168.231.131:9001
修改masters和slaves文件，记录集群中各个结点
[hadoop@node01 conf]$ vi masters
node01
[hadoop@node01 conf]$ vi slaves
node02
node03
node04
向其它3个结点复制hadoop
[hadoop@node01 ~]$ scp -r ./hadoop-0.20.2node02:/home/hadoop
[hadoop@node01 ~]$ scp -r ./hadoop-0.20.2node03:/home/hadoop
[hadoop@node01 ~]$ scp -r ./hadoop-0.20.2node04:/home/hadoop
8. 在各个结点上配置hadoop环境变量
[hadoop@node01 ~]$ su - root
password:
[root@node01 ~]# vi /etc/profile
exporthadoop_install=/home/hadoop/hadoop-0.20.2
export path=$path:$hadoop_install/bin
9. 格式化hdfs
[hadoop@node01 bin]$ ./hadoop namenode-format
13/01/30 00:59:04 info namenode.namenode:startup_msg:
/************************************************************
startup_msg: starting namenode
startup_msg: host = node01/192.168.231.131
startup_msg: args = [-format]
startup_msg: version = 0.20.2
startup_msg: build =https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707;compiled by 'chrisdo' on fri feb 19 08:07:34 utc 2010
************************************************************/
13/01/30 00:59:04 info namenode.fsnamesystem:fsowner=hadoop,root
13/01/30 00:59:04 infonamenode.fsnamesystem: supergroup=supergroup
13/01/30 00:59:04 infonamenode.fsnamesystem: ispermissionenabled=true
13/01/30 00:59:04 info common.storage:image file of size 96 saved in 0 seconds.
13/01/30 00:59:04 info common.storage:storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
13/01/30 00:59:04 info namenode.namenode:shutdown_msg:
/************************************************************
shutdown_msg: shutting down namenode atnode01/192.168.231.131
************************************************************/
10.启动守护进程
注意，在启动守护进程之前，一定要先关闭防火墙（所有的结点都要），否则datanode启动失败。
[root@node04 ~]# /etc/init.d/iptables stop
iptables: flushing firewall rules: [ ok ]
iptables: setting chains to policy accept:filter [ ok ]
iptables: unloading modules: [ ok ]
最好设置开机就不启动防火墙：
[root@node01 ~]# vi /etc/sysconfig/selinux
selinux=disable
[hadoop@node01bin]$ ./start-all.sh
startingnamenode, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-node01.out
node03:starting datanode, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-node03.out
node02:starting datanode, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-node02.out
node04:starting datanode, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-node04.out
hadoop@node01'spassword:
node01:starting secondarynamenode, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-node01.out
startingjobtracker, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-node01.out
node03:starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-node03.out
node02:starting tasktracker, logging to/home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-node02.out
node04:starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-node04.out
检测守护进程启动情况：
master结点：
[hadoop@node01 jdk1.6.0_38]$ /usr/java/jdk1.6.0_38/bin/jps
3986 jps
3639 namenode
3785 secondarynamenode
3858 jobtracker
slave结点（以node02为例）：
[root@node02 ~]# /usr/java/jdk1.6.0_38/bin/jps
3254 tasktracker
3175 datanode
3382 jps