搭建自动切换HA的HDFS集群(比手工切换ha集群,多了zookeeper集群)
----------------------------------------------------------------------------------------------
zookeeper:hadoop2-1、hadoop2-2、hadoop2-3
namenode:hadoop2-1和hadoop2-2
datanode:hadoop2-3、hadoop2-4、hadoop2-5、hadoop2-6
journalnode:hadoop2-1、hadoop2-2、hadoop2-3
2.0 搭建zookeeper集群,并启动
2.0.1 在hadoop2-1上解压缩,重命名为zookeeper,把conf/zoo_sample.cfg重命名为conf/zoo.cfg
修改文件conf/zoo.cfg
(1)dataDir=/usr/local/zookeeper/data
(2)增加以下内容
server.1=hadoop2-1:2888:3888
server.2=hadoop2-2:2888:3888
server.3=hadoop2-3:2888:3888
创建目录mkdir zookeeper/data
写入文件echo 1 >> zookeeper/data/myid
复制zookeeper文件夹到hadoop2-2、hadoop2-3上
scp -rq zookeeper hadoop2-2:/usr/local
scp -rq zookeeper hadoop2-3:/usr/local
在hadoop2上执行命令echo 2 >> zookeeper/data/myid
在hadoop3上执行命令echo 3 >> zookeeper/data/myid
2.0.2 启动
在hadoop2-1、hadoop2-2、hadoop2-3上,分别执行命令zookeeper/bin/zkServer.sh start
2.0.3 验证
执行命令zookeeper/bin/zkCli.sh
进入后执行ls /
2.1 配置文件(hadoop-env.sh、core-site.xml、hdfs-site.xml、slaves)
2.1.1 hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.7.0-45
2.1.2 core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://cluster1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop2-1:2181,hadoop2-2:2181,hadoop2-3:2181</value>
</property>
2.1.3 hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>cluster1</value>
</property>
<property>
<name>dfs.ha.namenodes.cluster1</name>
<value>hadoop2-1,hadoop2-2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.cluster1.hadoop2-1</name>
<value>hadoop2-1:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.cluster1.hadoop2-1</name>
<value>hadoop2-1:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.cluster1.hadoop2-2</name>
<value>hadoop2-2:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.cluster1.hadoop2-2</name>
<value>hadoop2-2:50070</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled.cluster1</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop2-1:8485;hadoop2-2:8485;hadoop2:8485/cluster1</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/hadoop/tmp/journal</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.cluster1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
2.1.6 slaves
hadoop2-3
hadoop2-4
hadoop2-5
2.1.7 删除其他节点的hadoop文件夹,然后把hadoop2-1上的hadoop文件夹复制到其他节点
2.2 格式化zk集群
在hadoop2-1上执行hadoop/bin/hdfs zkfc -formatZK
2.3 启动journalnode集群
在hadoop2-1、hadoop2-2、hadoop2-3上分别执行hadoop/sbin/hadoop-daemon.sh start journalnode
2.4 格式化namenode、启动namenode
在hadoop2-1上执行hadoop/bin/hdfs namenode -format
在hadoop2-1上分别执行hadoop/sbin/hadoop-daemon.sh start namenode
在hadoop2-2上执行hadoop/bin/hdfs namenode -bootstrapStandby
在hadoop2-2上分别执行hadoop/sbin/hadoop-daemon.sh start namenode
2.5 启动datanode
在hadoop2-1上分别执行hadoop/sbin/hadoop-daemons.sh start datanode
2.6 启动ZKFC
总结:
自动切换比手工切换多出来的
(1)配置上core-site.xml增加了配置项ha.zookeeper.quorum;hdfs-site.xml中把dfs.ha.automatic-failover.enabled.cluster1改为true
(2)操作上格式化zk,执行命令bin/hdfs zkfc -formatZK;启动zkfc,执行命令sbin/hadoop-daemon.sh start zkfc
----------------------------------------------------------------------------------------------
zookeeper:hadoop2-1、hadoop2-2、hadoop2-3
namenode:hadoop2-1和hadoop2-2
datanode:hadoop2-3、hadoop2-4、hadoop2-5、hadoop2-6
journalnode:hadoop2-1、hadoop2-2、hadoop2-3
2.0 搭建zookeeper集群,并启动
2.0.1 在hadoop2-1上解压缩,重命名为zookeeper,把conf/zoo_sample.cfg重命名为conf/zoo.cfg
修改文件conf/zoo.cfg
(1)dataDir=/usr/local/zookeeper/data
(2)增加以下内容
server.1=hadoop2-1:2888:3888
server.2=hadoop2-2:2888:3888
server.3=hadoop2-3:2888:3888
创建目录mkdir zookeeper/data
写入文件echo 1 >> zookeeper/data/myid
复制zookeeper文件夹到hadoop2-2、hadoop2-3上
scp -rq zookeeper hadoop2-2:/usr/local
scp -rq zookeeper hadoop2-3:/usr/local
在hadoop2上执行命令echo 2 >> zookeeper/data/myid
在hadoop3上执行命令echo 3 >> zookeeper/data/myid
2.0.2 启动
在hadoop2-1、hadoop2-2、hadoop2-3上,分别执行命令zookeeper/bin/zkServer.sh start
2.0.3 验证
执行命令zookeeper/bin/zkCli.sh
进入后执行ls /
2.1.1 hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.7.0-45
2.1.2 core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://cluster1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop2-1:2181,hadoop2-2:2181,hadoop2-3:2181</value>
</property>
2.1.3 hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>cluster1</value>
</property>
<property>
<name>dfs.ha.namenodes.cluster1</name>
<value>hadoop2-1,hadoop2-2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.cluster1.hadoop2-1</name>
<value>hadoop2-1:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.cluster1.hadoop2-1</name>
<value>hadoop2-1:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.cluster1.hadoop2-2</name>
<value>hadoop2-2:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.cluster1.hadoop2-2</name>
<value>hadoop2-2:50070</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled.cluster1</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop2-1:8485;hadoop2-2:8485;hadoop2:8485/cluster1</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/hadoop/tmp/journal</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.cluster1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
2.1.6 slaves
hadoop2-3
hadoop2-4
hadoop2-5
hadoop2-6
2.1.7 删除其他节点的hadoop文件夹,然后把hadoop2-1上的hadoop文件夹复制到其他节点
2.2 格式化zk集群
在hadoop2-1上执行hadoop/bin/hdfs zkfc -formatZK
2.3 启动journalnode集群
在hadoop2-1、hadoop2-2、hadoop2-3上分别执行hadoop/sbin/hadoop-daemon.sh start journalnode
2.4 格式化namenode、启动namenode
在hadoop2-1上执行hadoop/bin/hdfs namenode -format
在hadoop2-1上分别执行hadoop/sbin/hadoop-daemon.sh start namenode
在hadoop2-2上执行hadoop/bin/hdfs namenode -bootstrapStandby
在hadoop2-2上分别执行hadoop/sbin/hadoop-daemon.sh start namenode
2.5 启动datanode
在hadoop2-1上分别执行hadoop/sbin/hadoop-daemons.sh start datanode
2.6 启动ZKFC
在hadoop2-1、hadoop2-2上 启动zkfc,执行命令hadoop/sbin/hadoop-daemon.sh start zkfc
总结:
自动切换比手工切换多出来的
(1)配置上core-site.xml增加了配置项ha.zookeeper.quorum;hdfs-site.xml中把dfs.ha.automatic-failover.enabled.cluster1改为true
(2)操作上格式化zk,执行命令bin/hdfs zkfc -formatZK;启动zkfc,执行命令sbin/hadoop-daemon.sh start zkfc
相关推荐
赠送jar包:hadoop-hdfs-2.5.1.jar; 赠送原API文档:hadoop-hdfs-2.5.1-javadoc.jar; 赠送源代码:hadoop-hdfs-2.5.1-sources.jar; 赠送Maven依赖信息文件:hadoop-hdfs-2.5.1.pom; 包含翻译后的API文档:hadoop...
赠送jar包:hadoop-hdfs-2.5.1.jar; 赠送原API文档:hadoop-hdfs-2.5.1-javadoc.jar; 赠送源代码:hadoop-hdfs-2.5.1-sources.jar; 赠送Maven依赖信息文件:hadoop-hdfs-2.5.1.pom; 包含翻译后的API文档:hadoop...
Hadoop 分布式集群搭建 Hadoop由Apache基金会开发的分布式系统基础架构,是利用集群对大量数据进行分布式处理和存储的软件框架。用户可以轻松地在Hadoop集群上开发和运行处理海量数据的应用程序。Hadoop有高可靠,...
hadoop2.5.1汇总:最新编译_32位、64位安装包、源码包、API下载及新特性等
Hadoop实战高手之路---从零开始”的第六讲Hadoop图文训练课程:使用HDFS命令行工具操作Hadoop分布式集群初体验
本文根据实际的操作,详细的讲解了hadoop2.7.0集群的搭建过程
Hadoop HA 集群搭建所需要的配置文件:core-site,hdfs-site,mapred-site,yarn-site四个xml文件和一个slaves文件
可参考 http://www.bjsxt.com/down/3191.html 安装视频。
Hadoop分布式集群搭建所需安装包资源 jdk-8u171-linux-x64.tar.gz hadoop-2.7.5.tar.gz
集群搭建:里面包括hadoop hive hbase spark mongo 等组建
hadoop伪集群搭建及eclipse插件配置,主要介绍伪集群下配置文件的配置及eclipse插件配置。
hadoop 3.2.4集群搭建
基于haodoop 集群搭建hbase集群。(2台机器的情况。)
hadoop2.5.1 eclipse插件 eclipse版本Version: Luna Release (4.4.0)
Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程 按照文档中的操作步骤,一步步操作就可以完全实现hadoop2.2.0版本的完全分布式集群搭建过程
超详细集群搭建教程(从虚拟机安装到集群搭建完成,集群搭建成功验证),每一个步骤都有,照做即可。 内有集群搭建所需安装包:hadoop2.2.0,zookeeper3.4.5
目录 hadoop2集群搭建视频1 hadoop2集群搭建视频2 hadoop2集群搭建视频3 hadoop2集群搭建视频4 hadoop2
搭建手工切换HA的HDFS集群,学习大数据hadoop搭建环境的可以看下,通俗易懂
Ubantu-hadoop2.6.0集群搭建,详细介绍ubuntu下的hadoop搭建过程
hdfs官方文档 Hadoop分布式文件系统:结构与设计.pdf