Centos7.6安装CDH6.3.2集群

IT
IT
IT
335
文章
1
评论
更多
 2020年4月3日15:07:21 评论 2,331 19629字阅读65分25秒

简述

身处大数据时代，作为一名挨踢怎能不了解大数据技术，在我们学习的过程中又不可能使用企业生产级的CDH平台。那么本文章就带领大家通过Cloudera Manager可视化安装一个属于自己的CDH平台，供自己学习与研究。话不多说，下面开始，现有3台机器（这3台都是虚拟机，具体虚拟机的搭建过程就不在本文详述了），IP及主机名如下:

172.16.6.129（hadoop101）、172.16.6.130（Hadoop102）、172.16.6.131（hadoop103）

关闭防火墙

  • 三台机器一起操作
[root@hadoop101 ~]# setenforce 0
[root@hadoop101 ~]# sed -i "s/enforcing/disabled/g" `grep enforcing -rl /etc/selinux/config`
[root@hadoop101 ~]# systemctl stop firewalld
[root@hadoop101 ~]#  systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
  • 添加hosts文件,三台机器一起操作
[root@hadoop103 ~]#
cat > /etc/hosts << EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.6.129  hadoop101
172.16.6.130  hadoop102
172.16.6.131  hadoop103
EOF
  • 配置免密登录,在主节点
[root@hadoop101 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6acA1ScxmT56m8Ul+soNTtPjo2qygh3aa/+pkc1VFAY root@hadoop101
The key's randomart image is:
+---[RSA 2048]----+
|        E++.     |
|       .o=       |
|      ..o o      |
|     .  o=. .    |
|    .  .S+ o     |
|  .  =.oo.o      |
| = .o +.==+      |
|o = ...*oBo.     |
| ..++*+.*oo.     |
+----[SHA256]-----+
  • 拷贝密钥
[root@hadoop101 ~]# ssh-copy-id hadoop101
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop101 (172.16.6.129)' can't be established.
ECDSA key fingerprint is SHA256:l2uhfQgcEiepniZ3fYWf/jNeEI4mTGH3DVyKwrmE24s.
ECDSA key fingerprint is MD5:da:d4:28:53:b9:84:74:23:7c:18:ae:39:6e:75:6b:50.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop101's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'hadoop101'"
and check to make sure that only the key(s) you wanted were added.

[root@hadoop101 ~]# ssh-copy-id hadoop102
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop102 (172.16.6.130)' can't be established.
ECDSA key fingerprint is SHA256:ZLXMnAm/9ANze4sa1Vx+cB4v4L7yhh71Q0J2I/XUTrQ.
ECDSA key fingerprint is MD5:d7:79:b2:ed:72:03:a8:57:5d:31:f5:db:c1:09:a3:54.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop102's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'hadoop102'"
and check to make sure that only the key(s) you wanted were added.

[root@hadoop101 ~]# ssh-copy-id hadoop103
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop103 (172.16.6.131)' can't be established.
ECDSA key fingerprint is SHA256:zDzmbNw/0M63cN9OiKncD4T60PwWKHJ6rpeUYMAbRL0.
ECDSA key fingerprint is MD5:9b:e2:46:41:98:a7:f0:02:6f:09:aa:e0:c6:33:7b:c9.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop103's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'hadoop103'"
and check to make sure that only the key(s) you wanted were added.
  • 更换yum源三台机器一起
[root@hadoop101 ~]# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
[root@hadoop101 ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
--2020-04-03 15:14:32--  http://mirrors.aliyun.com/repo/Centos-7.repo
正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 124.200.113.112, 124.193.244.228, 124.200.113.114, ...
正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|124.200.113.112|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：2523 (2.5K) [application/octet-stream]
正在保存至: “/etc/yum.repos.d/CentOS-Base.repo”

100%[===============================================================================================================>] 2,523       --.-K/s 用时 0s      

2020-04-03 15:14:32 (177 MB/s) - 已保存 “/etc/yum.repos.d/CentOS-Base.repo” [2523/2523])

[root@hadoop101 ~]# yum clean all
[root@hadoop101 ~]# yum makecache
  • 调整Linux系统参数,三台机器一起执行
# 1、修改swappiness，最大限度地使用物理内存，然后是swap交换分区
echo "vm.swappiness = 0 " >> /etc/sysctl.conf
sysctl -p
# 2、禁用hugepage透明大页
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
# 确保重启生效
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
# 3、修改用户可打开的最大文件描述符和最大进程数
ulimit -SHn 100000
ulimit -SHu 100000
# 确保重启生效
echo "* soft nofile 100000" >> /etc/security/limits.conf
echo "* hard nofile 100000" >> /etc/security/limits.conf
echo "* soft nproc 100000" >> /etc/security/limits.conf
echo "* hard nproc 100000" >> /etc/security/limits.conf

基础服务完善

  • 启动NTP服务，同步阿里云时间,三台一起操作。
[root@hadoop101 ~]# yum install -y ntp
[root@hadoop101 ~]# vim /etc/ntp.conf
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst
server ntp4.aliyun.com iburst
  • 安装jdk,三台机器一起操作
[root@hadoop101 src]# rpm -ivh jdk-8u241-linux-x64.rpm 
警告：jdk-8u241-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:jdk1.8-2000:1.8.0_241-fcs        ################################# [100%]
Unpacking JAR files...
	tools.jar...
	plugin.jar...
	javaws.jar...
	deploy.jar...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar...
[root@hadoop101 src]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_241-amd64
export JAVA_BIN=/usr/java/jdk1.8.0_241-amd64/bin
PATH=$JAVA_HOME/bin:$PATH
[root@hadoop101 src]# source /etc/profile
[root@hadoop101 src]# java -version
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)
  • MySQL安装（用来存储CDH各组件的元数据，只在hadoop101上安装）
[root@hadoop101 src]# yum -y remove mariadb-libs
[root@hadoop101 src]# yum -y install http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
[root@hadoop101 src]# yum -y install mysql mysql-devel mysql-server mysql-utilities


[root@hadoop101 src]# systemctl start mysqld.service
[root@hadoop101 src]# systemctl enable mysqld.service
[root@hadoop101 src]# /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n] y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist
 ... Failed!  Not critical, keep moving...
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!


All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!


Cleaning up...
[root@hadoop101 src]#
  • 初始化数据库，每个数据库对应不同的服务，具体可查看官网说明。
[root@hadoop101 src]# mysql -uroot -p123456


Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[root@hadoop101 src]#
  • 下载第三方依赖（需要联网），3台机器都需要执行：
[root@hadoop102 src]# yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb

CM下载与安装

  • 首先下载CM安装包和CDH离线包，下载地址如下：

CM安装包下载地址：archive.cloudera.com/cm

该压缩包中包含CM的各种RPM包和对应的GPG加密文件。

https://archive.cloudera.com/cdh6/6.3.2/parcels/

JDBC驱动下载地址：cdn.mysql.com//Download

  • 将下载的5个安装包上传至/usr/local/src/下
[root@hadoop101 src]# ll
总用量 3416144
-rw------- 1 root root 2082186246 4月   3 17:22 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r-- 1 root root         40 11月 12 21:43 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
-rw------- 1 root root 1411444147 4月   3 17:24 cm6.3.1-redhat7.tar.gz
-rw-r--r-- 1 root root      33887 11月 12 21:43 manifest.json
-rw------- 1 root root    4456335 4月   3 16:38 mysql-connector-java-5.1.48.tar.gz
  • 解压
[root@hadoop101 src]# tar -zxvf cm6.3.1-redhat7.tar.gz 
cm6.3.1/
cm6.3.1/RPM-GPG-KEY-cloudera
cm6.3.1/RPMS/
cm6.3.1/RPMS/x86_64/
cm6.3.1/RPMS/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
cm6.3.1/RPMS/x86_64/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
cm6.3.1/RPMS/x86_64/cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
cm6.3.1/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cm6.3.1/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
cm6.3.1/RPMS/x86_64/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
cm6.3.1/RPMS/noarch/
cm6.3.1/repodata/
cm6.3.1/repodata/3662f97de72fd44c017bb0e25cee3bc9398108c8efb745def12130a69df2ecb2-filelists.sqlite.bz2
cm6.3.1/repodata/repomd.xml
cm6.3.1/repodata/49e4d60647407a36819f1d8ed901258a13361749b742e3be9065025ad31feb8e-filelists.xml.gz
cm6.3.1/repodata/8afda99b921fd1538dd06355952719652654fc06b6cd14515437bda28376c03d-other.sqlite.bz2
cm6.3.1/repodata/43f3725f730ee7522712039982aa4befadae4db968c8d780c8eb15ae9872cd4d-primary.xml.gz
cm6.3.1/repodata/repomd.xml.asc
cm6.3.1/repodata/b9300879675bdbc300436c1131a910a535b8b5a5dc6f38e956d51769b6771a96-primary.sqlite.bz2
cm6.3.1/repodata/e28836e19e07f71480c4dad0f7a87a804dc93970ec5277ad95614e8ffcff0d58-other.xml.gz
cm6.3.1/repodata/repomd.xml.key
cm6.3.1/SRPMS/
  • 执行rpm进行cm服务安装，3台机器均安装cm-daemons和cm-agent，hadoop101额外多安装一个cm-server。
[root@hadoop101 x86_64]# pwd
/usr/local/src/cm6.3.1/RPMS/x86_64
[root@hadoop101 x86_64]# ls
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm    cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm       enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm  cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm  oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
[root@hadoop101 x86_64]# pwd
/usr/local/src/cm6.3.1/RPMS/x86_64
[root@hadoop101 x86_64]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
警告：cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-daemons-6.3.1-14################################# [100%]
[root@hadoop101 x86_64]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 
警告：cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-server-6.3.1-146################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.
[root@hadoop101 x86_64]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 
警告：cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

另外两个节点安装

[root@hadoop101 x86_64]# scp cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm hadoop102:/root/
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm                                                                  100% 1148MB 191.3MB/s   00:06    
[root@hadoop101 x86_64]# scp cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm hadoop103:/root/
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm                                                                  100% 1148MB 191.3MB/s   00:06    
[root@hadoop101 x86_64]# scp cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm hadoop102:/root/
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm                                                                    100%   10MB 171.9MB/s   00:00    
[root@hadoop101 x86_64]# scp cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm hadoop103:/root/
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm                                                                    100%   10MB 158.6MB/s   00:00    
[root@hadoop102 ~]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 
警告：cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-daemons-6.3.1-14################################# [100%]
[root@hadoop102 ~]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 
警告：cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

第二台安装

[root@hadoop103 ~]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 
警告：cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-daemons-6.3.1-14################################# [100%]
[root@hadoop103 ~]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 
警告：cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

修改3台主机的cm-agent配置，将文件/etc/cloudera-scm-agent/config.ini中的server_host修改为hadoop101

[root@hadoop101 ~]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101
[root@hadoop102 ~]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101
[root@hadoop103 ~]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop101

将CDH离线包及相关文件拷贝至/opt/cloudera/parcel-repo/目录中

[root@hadoop101 src]# pwd
/usr/local/src
[root@hadoop101 src]# ll
总用量 3590892
-rw------- 1 root root 2082186246 4月   3 17:22 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r-- 1 root root         40 11月 12 21:43 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
drwxrwxr-x 5 2001 2001         75 9月  25 2019 cm6.3.1
-rw------- 1 root root 1411444147 4月   3 17:24 cm6.3.1-redhat7.tar.gz
-rw------- 1 root root  178939634 4月   8 17:17 jdk-8u241-linux-x64.rpm
-rw-r--r-- 1 root root      33887 11月 12 21:43 manifest.json
-rw------- 1 root root    4456335 4月   3 16:38 mysql-connector-java-5.1.48.tar.gz
[root@hadoop101 src]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
[root@hadoop101 src]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
[root@hadoop101 src]# cp manifest.json /opt/cloudera/parcel-repo/
[root@hadoop101 src]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

配置mysql连接器，并重命名（因为要和agent配置文件中指定的默认值保持一致）

解压之前下载的JDBC驱动文件

[root@hadoop101 src]# pwd
/usr/local/src
[root@hadoop101 src]# tar -zxvf mysql-connector-java-5.1.48.tar.gz

将解压后目录中的连接器jar包拷贝至/usr/share/java/目录【如果该目录不存在创建即可】下，并重命名

[root@hadoop101 src]# mkdir -p /usr/share/java/
[root@hadoop101 src]# cp mysql-connector-java-5.1.48/mysql-connector-java-5.1.48.jar /usr/share/java/mysql-connector-java.jar

为CM配置相关数据库信息，修改/etc/cloudera-scm-server/db.properties文件，修改完内容如下。

[root@hadoop101 src]# vim /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop101
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=123456

启动CM服务

首先启动服务节点：hadoop101

[root@hadoop101 ~]# systemctl start cloudera-scm-server

再分别启动工作节点：hadoop101、hadoop102、hadoop103

[root@hadoop101 ~]# systemctl start cloudera-scm-agent
[root@hadoop102 ~]# systemctl start cloudera-scm-agent
[root@hadoop103 ~]# systemctl start cloudera-scm-agent

查看hadoop101的7180端口是否被监听判断server是否启动成功

[root@hadoop101 ~]# netstat -tnlp | grep 7180
tcp        0      0 0.0.0.0:7180            0.0.0.0:*               LISTEN      5230/java

如果结果返回7180端口被监听，则代表安装成功，之后访问hadoop101:7180【用户名和密码均为admin】。登录成功后按提示即可进行CDH相关服务安装。

页面安装

http://172.16.6.129:7180/cmf/login

Centos7.6安装CDH6.3.2集群

首次admin登录后，一路“Continue”、接受使用条款、选择免费版本进入集群安装页面。

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

配置集群名称、选择主机、选择存储库（就之前下载的离线包），然后需要一些时间将离线安装包安装到集群所有主机上（因为这个安装包只在hadoop101上存在，需要安装到所有主机），最后会进行集群的网络和主机检查。此处需要根据主机检查的结果进行系统调整，直到全部检查通过。同样也可以忽略部分风险，如：Psycopg2版本过低（此处不用PostgreSQL做为元数据存储介质，所以可以忽略）。

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

检查群集

Centos7.6安装CDH6.3.2集群

修复出现的问题(三台机器一起操作）

Centos7.6安装CDH6.3.2集群

[root@hadoop101 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop101 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop101 ~]# yum -y install epel-release
[root@hadoop101 ~]# yum install python-pip
[root@hadoop101 ~]# pip install --upgrade pip
[root@hadoop101 ~]# yum install python-devel
[root@hadoop101 ~]# yum -y install python-psycopg2 
[root@hadoop101 ~]# pip install psycopg2
[root@hadoop101 ~]# vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop101 ~]# chmod +x /etc/rc.d/rc.local

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

角色权限默认即可

Centos7.6安装CDH6.3.2集群

设置数据库

Centos7.6安装CDH6.3.2集群

审核保持默认

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

等待安装完成

Centos7.6安装CDH6.3.2集群

Centos7.6安装CDH6.3.2集群

由于我的机器配置都是2核4GB，配置太低了。建议大家配置高一些。

下载信息 CDH6.3安装包
网盘密码：登录可见
下载地址：登录可见
继续阅读
  • 我的QQ
  • QQ扫一扫
  • weinxin
  • 我的头条
  • 头条扫一扫
  • weinxin
CDH 最后更新：2020-4-9
IT
  • 本文由 发表于 2020年4月3日15:07:21
  • 除非特殊声明，本站文章均为原创，转载请务必保留本文链接