首页 > 学院 > 开发设计 > 正文

基于hadoop2.7.3搭建多机环境（YARN+HA）

2019-11-06 07:15:13

字体：大中小

来源：转载

供稿：网友

第一：环境说明

parallels desktopCentOS-6.5-x86_64-bin-DVD1.isojdk-7u79-linux-x64.tar.gzHadoop-2.7.3.tar.gz搭建四个节点的集群。他们的hostname分布为hadoopA，hadoopB，hadoopC，hadoopD。其中hadoopA的角色为Activity namnode。hadoopB的角色为standby namenode，datanode，journalnode。hadoopC的角色为datanode，journalnode。hadoopD的角色为datanode，journalnode。

第二：操作系统配置

赋予hadoop用户sudo权限[root@hadoopa hadoop]# visudo## Allow root to run any commands anywhereroot ALL=(ALL) ALLhadoop ALL=(ALL) ALL修改hostname[hadoop@hadoopa hadoop-2.7.3]$ cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.1.201 hadoopA192.168.1.202 hadoopB192.168.1.203 hadoopC192.168.1.204 hadoopD

第三：安装和配置jdk

分别在hadoopA，hadoopB，hadoopC，hadoopD四个节点安装jdk。

[hadoop@hadoopb ~]$ tar -zxvf jdk-7u79-linux-x64.tar.gz

修改jdk的名称

[hadoop@hadoopb ~]$ mv jdk1.7.0_79/ jdk1.7

第四：安装和配置hadoop

在hadoopA,hadoopB,hadoopC,hadoopD四个节点上解压hadoop[hadoop@hadoopb ~]$ tar -zxvf hadoop-2.7.3.tar.gz在hadoopA上配置hadoop-env.sh# The java implementation to use.export JAVA_HOME=/home/hadoop/jdk1.7在hadoopA上配置core-site.xml<configuration> <PRoperty> <name>fs.defaultFS</name> <value>hdfs://hadoopA:8020</value> </property></configuration>在hadoopA配置hdfs-site.xml<configuration><property> <name>dfs.nameservices</name> <value>hadoop-test</value> <description> Comma-separated list of nameservices. </description></property><property> <name>dfs.ha.namenodes.hadoop-test</name> <value>nn1,nn2</value> <description> The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). </description></property><property> <name>dfs.namenode.rpc-address.hadoop-test.nn1</name> <value>hadoopA:8020</value> <description> RPC address for nomenode1 of hadoop-test </description></property><property> <name>dfs.namenode.rpc-address.hadoop-test.nn2</name> <value>hadoopB:8020</value> <description> RPC address for nomenode2 of hadoop-test </description></property><property> <name>dfs.namenode.http-address.hadoop-test.nn1</name> <value>hadoopA:50070</value> <description> The address and the base port where the dfs namenode1 web ui will listen on. </description></property><property> <name>dfs.namenode.http-address.hadoop-test.nn2</name> <value>hadoopB:50070</value> <description> The address and the base port where the dfs namenode2 web ui will listen on. </description></property><property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/hdfs/name</value> <description>Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description></property><property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoopB:8485;hadoopC:8485;hadoopD:8485/hadoop-test</value> <description>A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. </description></property><property> <name>dfs.datanode.data.dir</name> <value>file:///home/hadoop/hdfs/data</value> <description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. </description></property><property> <name>dfs.ha.automatic-failover.enabled</name> <value>false</value> <description> Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. </description></property><property> <name>dfs.journalnode.edits.dir</name> <value>/home/hadoop/hdfs/journal/</value></property></configuration>在hadoopA配置mapred-site.xml<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>mapreduce.jobhistory.address</name><value>hadoopB:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>hadoopB:19888</value></property></configuration>在hadoopA配置yarn-site.xml<configuration>  <property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>hadoopA</value> </property> <property> <description>The address of the applications manager interface in the RM.</description> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> </property> <property> <description>The address of the scheduler interface.</description> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname}:8030</value> </property> <property> <description>The http address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property> <property> <description>The https adddress of the RM web application.</description> <name>yarn.resourcemanager.webapp.https.address</name> <value>${yarn.resourcemanager.hostname}:8090</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname}:8031</value> </property> <property> <description>The address of the RM admin interface.</description> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname}:8033</value> </property> <property> <description>The class to use as the resource scheduler.</description> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property> <property> <description>fair-scheduler conf location</description> <name>yarn.scheduler.fair.allocation.file</name> <value>/home/hadoop/hadoop-2.7.3/etc/hadoop/fairscheduler.xml</value> </property> <property> <description>List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. </description> <name>yarn.nodemanager.local-dirs</name> <value>/home/hadoop/yarn/local</value> </property> <property> <description>Whether to enable log aggregation</description> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <description>Where to aggregate logs to.</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/tmp/logs</value> </property> <property> <description>Amount of physical memory, in MB, that can be allocated for containers.</description> <name>yarn.nodemanager.resource.memory-mb</name> <value>8720</value> </property> <property> <description>Number of CPU cores that can be allocated for containers.</description> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>2</value> </property> <property> <description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configuration>在hadoopA配置fairscheduler.xml<allocations> <queue name="infrastructure"> <minResources>102400 mb, 50 vcores </minResources> <maxResources>153600 mb, 100 vcores </maxResources> <maxRunningApps>200</maxRunningApps> <minSharePreemptionTimeout>300</minSharePreemptionTimeout> <weight>1.0</weight> <aclSubmitApps>root,yarn,search,hdfs</aclSubmitApps> </queue> <queue name="tool"> <minResources>102400 mb, 30 vcores</minResources> <maxResources>153600 mb, 50 vcores</maxResources> </queue> <queue name="sentiment"> <minResources>102400 mb, 30 vcores</minResources> <maxResources>153600 mb, 50 vcores</maxResources> </queue></allocations>在hadoopA配置slaves文件[root@hadoopa hadoop]# cat slaveshadoopBhadoopChadoopD将hadoopA上hadoop的安装目录复制到其它[hadoop@hadoopa hadoop-2.7.3]$ scp etc/hadoop/* hadoopB://home/hadoop/hadoop-2.7.3/etc/hadoop/[hadoop@hadoopa hadoop-2.7.3]$ scp etc/hadoop/* hadoopC://home/hadoop/hadoop-2.7.3/etc/hadoop/[hadoop@hadoopa hadoop-2.7.3]$ scp etc/hadoop/* hadoopD://home/hadoop/hadoop-2.7.3/etc/hadoop/

第五：启动hadoop

在各个JournalNode节点上，输入以下命令启动journalnode服务[hadoop@hadoopb hadoop-2.7.3]$ sbin/hadoop-daemon.sh start journalnode[hadoop@hadoopc hadoop-2.7.3]$ sbin/hadoop-daemon.sh start journalnode[hadoop@hadoopd hadoop-2.7.3]$ sbin/hadoop-daemon.sh start journalnode在[nn1]上，对其进行格式化，并启动：[root@hadoopa hadoop-2.7.3]# bin/hdfs namenode -format[root@hadoopa hadoop-2.7.3]# sbin/hadoop-daemon.sh start namenode在[nn2]上，同步nn1的元数据信息[hadoop@hadoopb hadoop-2.7.3]$ bin/hdfs namenode -bootstrapStandby在[nn2]上，启动NameNode：[hadoop@hadoopb hadoop-2.7.3]$ sbin/hadoop-daemon.sh start namenode(经过以上四步操作，nn1和nn2均处理standby状态)在[nn1]上，将NameNode切换为Active[root@hadoopa hadoop-2.7.3]# bin/hdfs haadmin -transitionToActive nn1在[nn1]上，启动所有datanode[root@hadoopa hadoop-2.7.3]# sbin/hadoop-daemons.sh start datanode启动yarn：在[nn1]上，输入以下命令[root@hadoopa hadoop-2.7.3]# sbin/start-yarn.sh关闭Hadoop集群：在[nn1]上，输入以下命令[root@hadoopa hadoop-2.7.3]# sbin/stop-dfs.sh[root@hadoopa hadoop-2.7.3]# sbin/stop-yarn.sh

第六：验证hadoop

hadoopA输入命令[root@hadoopa jdk1.7]# /home/hadoop/jdk1.7/bin/jps10747 -- process information unavailable15583 Jps16576 -- process information unavailablehadoopB输入命令[hadoop@hadoopb hadoop-2.7.3]$ /home/hadoop/jdk1.7/bin/jps15709 NodeManager2405 JournalNode11551 NameNode12862 DataNode15398 JpshadoopC输入命令[hadoop@hadoopc ~]$ /home/hadoop/jdk1.7/bin/jps2388 JournalNode13091 Jps13553 DataNode15214 NodeManagerhadoopD输入命令[hadoop@hadoopd hadoop-2.7.3]$ /home/hadoop/jdk1.7/bin/jps13506 DataNode12675 Jps15334 NodeManager2570 JournalNode

打开浏览器输入以下地址：

http://192.168.1.201:50070/dfshealth.html#tab-overviewhttp://192.168.1.202:50070/dfshealth.html#tab-overviewhttp://192.168.1.201:8088/cluster/scheduler

第七：关闭hadoop

关闭Hadoop集群：在[nn1]上，输入以下命令[root@hadoopa hadoop-2.7.3]# sbin/stop-dfs.sh[root@hadoopa hadoop-2.7.3]# sbin/stop-yarn.sh

第八：特别说明

说明：步骤2：在[nn1]上，对其进行格式化，并启动： bin/hdfs namenode -fromal 步骤3：在[nn2]上，同步nn1的元数据信息 bin/hdfs namenode -bootstrapStandby

这两步操作，只是在第一次建立集群的时候才使用下次重启节点，是不需要操作这两步

上一篇：虚拟机性能监控与故障处理工具

下一篇：Codeforces Round #302 (Div. 2) D - Destroying Roads 图论，最短路

学习交流

索泰发布一款GTX 1070 Mini迷你版本:小机

索泰发布一款GTX 1070 Mini迷你版本:小机箱大爱...

热门图片

猜你喜欢的新闻

猜你喜欢的关注

新闻热点

荣耀总裁赵明乌镇演讲：荣耀首款5G手机V30下月发布

2019-10-23 09:17:05

搜狐张朝阳：回归媒体是搜狐重新崛起的关键

2019-10-21 09:20:02

华为轮值董事长郭平：虚拟技术创造现实价值

2019-10-21 09:00:12

滴滴英文服务上线两周年用户已超200万

2019-09-26 08:57:12

华为推出全球至快AI训练集群Atlas900

2019-09-25 08:46:36

马斯克：特斯拉正组建中国技术团队

2019-09-25 08:15:43

疑难解答

图片精选

网友关注