简述
身处大数据时代，作为一名挨踢怎能不了解大数据技术，在我们学习的过程中又不可能使用企业生产级的CDH平台。那么本文章就带领大家通过Cloudera Manager可视化安装一个属于自己的CDH平台，供自己学习与研究。话不多说，下面开始，现有3台机器（这3台都是虚拟机，具体虚拟机的搭建过程就不在本文详述了），IP及主机名如下:
172.16.6.129（hadoop101）、172.16.6.130（Hadoop102）、172.16.6.131（hadoop103）
关闭防火墙
- 三台机器一起操作
[root@hadoop101 ~]# setenforce 0 [root@hadoop101 ~]# sed -i "s/enforcing/disabled/g" `grep enforcing -rl /etc/selinux/config` [root@hadoop101 ~]# systemctl stop firewalld [root@hadoop101 ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
- 添加hosts文件,三台机器一起操作
[root@hadoop103 ~]# cat > /etc/hosts << EOF 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.6.129 hadoop101 172.16.6.130 hadoop102 172.16.6.131 hadoop103 EOF
- 配置免密登录,在主节点
[root@hadoop101 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:6acA1ScxmT56m8Ul+soNTtPjo2qygh3aa/+pkc1VFAY root@hadoop101 The key's randomart image is: +---[RSA 2048]----+ | E++. | | .o= | | ..o o | | . o=. . | | . .S+ o | | . =.oo.o | | = .o +.==+ | |o = ...*oBo. | | ..++*+.*oo. | +----[SHA256]-----+
- 拷贝密钥
[root@hadoop101 ~]# ssh-copy-id hadoop101 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop101 (172.16.6.129)' can't be established. ECDSA key fingerprint is SHA256:l2uhfQgcEiepniZ3fYWf/jNeEI4mTGH3DVyKwrmE24s. ECDSA key fingerprint is MD5:da:d4:28:53:b9:84:74:23:7c:18:ae:39:6e:75:6b:50. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop101's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'hadoop101'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101 ~]# ssh-copy-id hadoop102 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop102 (172.16.6.130)' can't be established. ECDSA key fingerprint is SHA256:ZLXMnAm/9ANze4sa1Vx+cB4v4L7yhh71Q0J2I/XUTrQ. ECDSA key fingerprint is MD5:d7:79:b2:ed:72:03:a8:57:5d:31:f5:db:c1:09:a3:54. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop102's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'hadoop102'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101 ~]# ssh-copy-id hadoop103 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop103 (172.16.6.131)' can't be established. ECDSA key fingerprint is SHA256:zDzmbNw/0M63cN9OiKncD4T60PwWKHJ6rpeUYMAbRL0. ECDSA key fingerprint is MD5:9b:e2:46:41:98:a7:f0:02:6f:09:aa:e0:c6:33:7b:c9. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop103's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'hadoop103'" and check to make sure that only the key(s) you wanted were added.
- 更换yum源三台机器一起
[root@hadoop101 ~]# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup [root@hadoop101 ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo --2020-04-03 15:14:32-- http://mirrors.aliyun.com/repo/Centos-7.repo 正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 124.200.113.112, 124.193.244.228, 124.200.113.114, ... 正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|124.200.113.112|:80... 已连接。 已发出 HTTP 请求，正在等待回应... 200 OK 长度：2523 (2.5K) [application/octet-stream] 正在保存至: “/etc/yum.repos.d/CentOS-Base.repo” 100%[===============================================================================================================>] 2,523 --.-K/s 用时 0s 2020-04-03 15:14:32 (177 MB/s) - 已保存 “/etc/yum.repos.d/CentOS-Base.repo” [2523/2523]) [root@hadoop101 ~]# yum clean all [root@hadoop101 ~]# yum makecache
- 调整Linux系统参数,三台机器一起执行
# 1、修改swappiness，最大限度地使用物理内存，然后是swap交换分区
echo "vm.swappiness = 0 " >> /etc/sysctl.conf
sysctl -p
# 2、禁用hugepage透明大页
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
# 确保重启生效
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
# 3、修改用户可打开的最大文件描述符和最大进程数
ulimit -SHn 100000
ulimit -SHu 100000
# 确保重启生效
echo "* soft nofile 100000" >> /etc/security/limits.conf
echo "* hard nofile 100000" >> /etc/security/limits.conf
echo "* soft nproc 100000" >> /etc/security/limits.conf
echo "* hard nproc 100000" >> /etc/security/limits.conf基础服务完善
- 启动NTP服务，同步阿里云时间,三台一起操作。
[root@hadoop101 ~]# yum install -y ntp [root@hadoop101 ~]# vim /etc/ntp.conf server ntp2.aliyun.com iburst server ntp3.aliyun.com iburst server ntp4.aliyun.com iburst
- 安装jdk,三台机器一起操作
[root@hadoop101 src]# rpm -ivh jdk-8u241-linux-x64.rpm 警告：jdk-8u241-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:jdk1.8-2000:1.8.0_241-fcs ################################# [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... [root@hadoop101 src]# vim /etc/profile export JAVA_HOME=/usr/java/jdk1.8.0_241-amd64 export JAVA_BIN=/usr/java/jdk1.8.0_241-amd64/bin PATH=$JAVA_HOME/bin:$PATH [root@hadoop101 src]# source /etc/profile [root@hadoop101 src]# java -version java version "1.8.0_241" Java(TM) SE Runtime Environment (build 1.8.0_241-b07) Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)
- MySQL安装（用来存储CDH各组件的元数据，只在hadoop101上安装）
[root@hadoop101 src]# yum -y remove mariadb-libs
[root@hadoop101 src]# yum -y install http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
[root@hadoop101 src]# yum -y install mysql mysql-devel mysql-server mysql-utilities
[root@hadoop101 src]# systemctl start mysqld.service
[root@hadoop101 src]# systemctl enable mysqld.service
[root@hadoop101 src]# /usr/bin/mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MySQL to secure it, we'll need the current
password for the root user. If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.
Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] n
... skipping.
By default, MySQL comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] y
- Dropping test database...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist
... Failed! Not critical, keep moving...
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] y
... Success!
All done! If you've completed all of the above steps, your MySQL
installation should now be secure.
Thanks for using MySQL!
Cleaning up...
[root@hadoop101 src]#
- 初始化数据库，每个数据库对应不同的服务，具体可查看官网说明。
[root@hadoop101 src]# mysql -uroot -p123456 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> exit Bye [root@hadoop101 src]#
- 下载第三方依赖（需要联网），3台机器都需要执行：
[root@hadoop102 src]# yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb
CM下载与安装
- 首先下载CM安装包和CDH离线包，下载地址如下：
CM安装包下载地址：https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz
该压缩包中包含CM的各种RPM包和对应的GPG加密文件。
https://archive.cloudera.com/cdh6/6.3.2/parcels/
JDBC驱动下载地址：https://cdn.mysql.com//Download
- 将下载的5个安装包上传至/usr/local/src/下
[root@hadoop101 src]# ll 总用量 3416144 -rw------- 1 root root 2082186246 4月 3 17:22 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel -rw-r--r-- 1 root root 40 11月 12 21:43 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 -rw------- 1 root root 1411444147 4月 3 17:24 cm6.3.1-redhat7.tar.gz -rw-r--r-- 1 root root 33887 11月 12 21:43 manifest.json -rw------- 1 root root 4456335 4月 3 16:38 mysql-connector-java-5.1.48.tar.gz
- 解压
[root@hadoop101 src]# tar -zxvf cm6.3.1-redhat7.tar.gz cm6.3.1/ cm6.3.1/RPM-GPG-KEY-cloudera cm6.3.1/RPMS/ cm6.3.1/RPMS/x86_64/ cm6.3.1/RPMS/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cm6.3.1/RPMS/x86_64/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm cm6.3.1/RPMS/x86_64/cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm cm6.3.1/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm cm6.3.1/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm cm6.3.1/RPMS/x86_64/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm cm6.3.1/RPMS/noarch/ cm6.3.1/repodata/ cm6.3.1/repodata/3662f97de72fd44c017bb0e25cee3bc9398108c8efb745def12130a69df2ecb2-filelists.sqlite.bz2 cm6.3.1/repodata/repomd.xml cm6.3.1/repodata/49e4d60647407a36819f1d8ed901258a13361749b742e3be9065025ad31feb8e-filelists.xml.gz cm6.3.1/repodata/8afda99b921fd1538dd06355952719652654fc06b6cd14515437bda28376c03d-other.sqlite.bz2 cm6.3.1/repodata/43f3725f730ee7522712039982aa4befadae4db968c8d780c8eb15ae9872cd4d-primary.xml.gz cm6.3.1/repodata/repomd.xml.asc cm6.3.1/repodata/b9300879675bdbc300436c1131a910a535b8b5a5dc6f38e956d51769b6771a96-primary.sqlite.bz2 cm6.3.1/repodata/e28836e19e07f71480c4dad0f7a87a804dc93970ec5277ad95614e8ffcff0d58-other.xml.gz cm6.3.1/repodata/repomd.xml.key cm6.3.1/SRPMS/
- 执行rpm进行cm服务安装，3台机器均安装cm-daemons和cm-agent，hadoop101额外多安装一个cm-server。
[root@hadoop101 x86_64]# pwd /usr/local/src/cm6.3.1/RPMS/x86_64 [root@hadoop101 x86_64]# ls cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm [root@hadoop101 x86_64]# pwd /usr/local/src/cm6.3.1/RPMS/x86_64 [root@hadoop101 x86_64]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-daemons-6.3.1-14################################# [100%] [root@hadoop101 x86_64]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-server-6.3.1-146################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service. [root@hadoop101 x86_64]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-agent-6.3.1-1466################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service. Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
另外两个节点安装
[root@hadoop101 x86_64]# scp cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm hadoop102:/root/ cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 100% 1148MB 191.3MB/s 00:06 [root@hadoop101 x86_64]# scp cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm hadoop103:/root/ cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 100% 1148MB 191.3MB/s 00:06 [root@hadoop101 x86_64]# scp cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm hadoop102:/root/ cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm 100% 10MB 171.9MB/s 00:00 [root@hadoop101 x86_64]# scp cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm hadoop103:/root/ cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm 100% 10MB 158.6MB/s 00:00 [root@hadoop102 ~]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-daemons-6.3.1-14################################# [100%] [root@hadoop102 ~]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-agent-6.3.1-1466################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service. Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
第二台安装
[root@hadoop103 ~]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-daemons-6.3.1-14################################# [100%] [root@hadoop103 ~]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force 警告：cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY 准备中... ################################# [100%] 正在升级/安装... 1:cloudera-manager-agent-6.3.1-1466################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service. Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
修改3台主机的cm-agent配置，将文件/etc/cloudera-scm-agent/config.ini中的server_host修改为hadoop101
[root@hadoop101 ~]# vim /etc/cloudera-scm-agent/config.ini server_host=hadoop101 [root@hadoop102 ~]# vim /etc/cloudera-scm-agent/config.ini server_host=hadoop101 [root@hadoop103 ~]# vim /etc/cloudera-scm-agent/config.ini server_host=hadoop101
将CDH离线包及相关文件拷贝至/opt/cloudera/parcel-repo/目录中
[root@hadoop101 src]# pwd /usr/local/src [root@hadoop101 src]# ll 总用量 3590892 -rw------- 1 root root 2082186246 4月 3 17:22 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel -rw-r--r-- 1 root root 40 11月 12 21:43 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 drwxrwxr-x 5 2001 2001 75 9月 25 2019 cm6.3.1 -rw------- 1 root root 1411444147 4月 3 17:24 cm6.3.1-redhat7.tar.gz -rw------- 1 root root 178939634 4月 8 17:17 jdk-8u241-linux-x64.rpm -rw-r--r-- 1 root root 33887 11月 12 21:43 manifest.json -rw------- 1 root root 4456335 4月 3 16:38 mysql-connector-java-5.1.48.tar.gz [root@hadoop101 src]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha [root@hadoop101 src]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/ [root@hadoop101 src]# cp manifest.json /opt/cloudera/parcel-repo/ [root@hadoop101 src]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/
配置mysql连接器，并重命名（因为要和agent配置文件中指定的默认值保持一致）
解压之前下载的JDBC驱动文件
[root@hadoop101 src]# pwd /usr/local/src [root@hadoop101 src]# tar -zxvf mysql-connector-java-5.1.48.tar.gz
将解压后目录中的连接器jar包拷贝至/usr/share/java/目录【如果该目录不存在创建即可】下，并重命名
[root@hadoop101 src]# mkdir -p /usr/share/java/ [root@hadoop101 src]# cp mysql-connector-java-5.1.48/mysql-connector-java-5.1.48.jar /usr/share/java/mysql-connector-java.jar
为CM配置相关数据库信息，修改/etc/cloudera-scm-server/db.properties文件，修改完内容如下。
[root@hadoop101 src]# vim /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=hadoop101 com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.password=123456
启动CM服务
首先启动服务节点：hadoop101
[root@hadoop101 ~]# systemctl start cloudera-scm-server
再分别启动工作节点：hadoop101、hadoop102、hadoop103
[root@hadoop101 ~]# systemctl start cloudera-scm-agent [root@hadoop102 ~]# systemctl start cloudera-scm-agent [root@hadoop103 ~]# systemctl start cloudera-scm-agent
查看hadoop101的7180端口是否被监听判断server是否启动成功
[root@hadoop101 ~]# netstat -tnlp | grep 7180 tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 5230/java
如果结果返回7180端口被监听，则代表安装成功，之后访问http://hadoop101:7180【用户名和密码均为admin】。登录成功后按提示即可进行CDH相关服务安装。
页面安装
http://172.16.6.129:7180/cmf/login
首次admin登录后，一路“Continue”、接受使用条款、选择免费版本进入集群安装页面。
配置集群名称、选择主机、选择存储库（就之前下载的离线包），然后需要一些时间将离线安装包安装到集群所有主机上（因为这个安装包只在hadoop101上存在，需要安装到所有主机），最后会进行集群的网络和主机检查。此处需要根据主机检查的结果进行系统调整，直到全部检查通过。同样也可以忽略部分风险，如：Psycopg2版本过低（此处不用PostgreSQL做为元数据存储介质，所以可以忽略）。
检查群集
修复出现的问题(三台机器一起操作）
[root@hadoop101 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag [root@hadoop101 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@hadoop101 ~]# yum -y install epel-release [root@hadoop101 ~]# yum install python-pip [root@hadoop101 ~]# pip install --upgrade pip [root@hadoop101 ~]# yum install python-devel [root@hadoop101 ~]# yum -y install python-psycopg2 [root@hadoop101 ~]# pip install psycopg2 [root@hadoop101 ~]# vim /etc/rc.local echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@hadoop101 ~]# chmod +x /etc/rc.d/rc.local
角色权限默认即可
设置数据库
审核保持默认
等待安装完成
由于我的机器配置都是2核4GB，配置太低了。建议大家配置高一些。
继续阅读
- 我的QQ
- QQ扫一扫
-
- 我的头条
- 头条扫一扫
-
评论