Hadoop——虚拟机，系统镜像，jdk以及命令的配置

Hadoop学习笔记

环境工具：

VMware 12 虚拟机

Ubuntu 16.04.3系统镜像

Jdk1.8

Hadoop3.0

操作步骤：

1.永久关闭防火墙 执行命令：sudo ufw disable

备注：

关闭命令：sudo ufw disable

启动命令：sudo ufw enable

查看状态命令：Sudo ufw status

Status:active 启动

Status:inactive 关闭

2.安装ssh 执行命令：apt-get install ssh

3.安装jdk1.8

通过winscp上传jdk-8u161-linux-x64.tar.gz

解压：tar -zxvf jdk-8u121-linux-x64.tar.gz

创建文件夹

cd /usr/lib

sudo mkdir jvm

移动文件

sudo mv /home/master/Downloads/jdk1.8.0_161 /usr/lib/jvm

cd /usr/lib/jvm

ls

4.配置jdk环境变量

安装编辑工具vim:sudo apt-get install vim

编辑：vim ~/.bashrc

在bashrc末尾添加如下4行代码：

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_161（相应的jdk版本号）

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib

export PATH=$JAVA_HOME/bin:$PATH

保存退出：ESC :wq

使bashrc文件生效：source ~/.bashrc

查看版本验证：java -version

5.免密登录

准备两台虚拟机，master和slave

修改主机名

sudo passwd root

su root

vim /etc/hostname master/slave

vim /etc/hosts

master 192.168.163.131

slave1 192.168.163.129

(1) 生成公钥和私钥(master)

ssh-keygen -t rsa 一直回车

(2) 将公钥放到authorized_keys里

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

(3) 测试公钥是否可用

ssh localhost

(4) 节点间互相免密登录(slave1)

ssh-keygen -t rsa

cd ~/.ssh

ls

scp id_rsa.pub master@master:/tmp/id_rsa.pub.d1

(5) Master

cd /tmp

ls

mv id_rsa.pub.d1 ~/.ssh

cd ~/.ssh

cat id_rsa.pub.d1 >> authorized_keys

测试：slave1: ssh master@master

(6) 通过scp把master的authorized_keys传给slave1

Master: scp authorized_keys slave1@slave1

Slave1: cd /tmp

ls

mv authorized_keys ~/.ssh/

(7) 测试

Master: ssh slave1@slave1

6.hadoop配置

（1）将hadoop-3.0.0.tar.gz拷贝到master服务器上，解压,在home下创建hadoop文件夹，在/home/hadoop目录下创建数据存放的文件夹，tmp、hdfs、hdfs/data、hdfs/name

（2）放到/home/hadoop文件夹下面

sudo mkdir /home/hadoop

mv hadoop-3.0.0 /home/hadoop/

（3）配置hadoop-env.sh

cd /home/hadoop/hadoop-3.0.0/etc

vim Hadoop-env.sh

末尾添加：export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_161

（4）core-site.xml文件配置

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://192.168.1.121:9000</value> //master IP

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/tmp</value>

</property>

</configuration>

（5）hdfs-site.xml配置

<configuration>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>/home/hadoop/hdfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/home/Hadoop/hdfs/data</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

</configuration>

（6）workers配置

master@master

slave1@slave1

（7）mapred-site.xml配置

先执行：cp mapred-site.xml.template mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.application.classpath</name>

<value>

/home/hadoop/hadoop-3.0.0/etc/hadoop,

/home/hadoop/hadoop-3.0.0/share/hadoop/common/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/common/lib/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/hdfs/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/hdfs/lib/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/mapreduce/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/mapreduce/lib/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/yarn/*,

/home/hadoop/hadoop-3.0.0/share/hadoop/yarn/lib/*

</value>

</property>

</configuration>

（8）yarn.site.xml

<configuration>

<!-- Site specific YARN configurationproperties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandle</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>192.168.163.131:8025</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>192.168.163.131:8030</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>192.168.163.131:8040</value>

</property>

</configuration>

（9）将配置好的hadoop文件发送到各个节点

Master:

cd /home

scp -r hadoop slave1@slave1:/tmp

slave1:

cd ~/tmp

sudo mv hadoop /home

7.测试

（1）.格式化namenode

bin/hdfs namenode –format

（2）. 启动dfs和yarn

sbin/start-dfs.sh

sbin/start-yarn.sh

（3）.打开页面http://192.168.163.131:8088及http://192.168.163.131:9870 查看是否成功