首页 时间轴 标签 分类 文档 友链 关于我 简体中文

目录

CDH安装与部署

 收录于 大数据
   约 1714 字   预计阅读 4 分钟 

基础环境配置

安装环境信息

序号 服务器ID IP地址 操作系统
1 node01 10.0.0.7 CentOS 7.9
2 node02 10.0.0.14 CentOS 7.9
3 node03 10.0.0.22 CentOS 7.9

修改主机名

1
2
3

hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03

修改主机名映射

1
2
3

10.0.0.7 node01
10.0.0.14 node02
10.0.0.22 node03

关闭防火墙和Selinux

1
2
3
4
5
6
7

#关闭SELinux
setenforce 0
grep 'SELINUX=enforcing' /etc/selinux/config && sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

#关闭firewall
systemctl stop firewalld.service
systemctl disable firewalld.service

系统初始化参数调整 (设置swap和透明大页面)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

sysctl vm.swappiness=1
echo 'vm.swappiness=1' >> /etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

#设置集群的文件打开数
vim /etc/security/limits.d/20-nproc.conf
*         soft   nproc     409600
root      soft   nproc     unlimited

vim /etc/security/limits.conf
* hard nofile 655350
* soft nofile 655350

配置ssh免密登录

1
2
3
4
5
6
7
8
9

集群之间通过内网端口互相访问，配置ssh免密登录，注意开放外网端口需要配置iptables防火墙

#生成密码对 -所有节点
ssh-keygen -t rsa
   
#在每台节点分别执行以下命令
ssh-copy-id root@10.0.0.10
ssh-copy-id root@10.0.0.14
ssh-copy-id root@10.0.0.22

三台节点配置时钟同步 & 安装相关依赖包

1
2
3
4
5
6

# 安装
yum -y install gcc python-devel cyrus-sasl* chrony iotop yum-utils yum-plugin-downloadonly psmisc ntp ntpdate createrepo wget git rsync mod_ssl cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi libxslt krb5-devel cyrus-sasl-gssapi cyrus-sasl-devel libxml2-devel libxslt-devel openldap-devel python-devel python-simplejson sqlite-devel

# 时钟同步
systemctl start ntpd
systemctl enable ntpd

安装http服务

安装httpd服务

1、安装nginx

1
2

#安装httpd服务:
yum -y install nginx

2、添加配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#修改/etc/nginx/conf.d/clouera.conf
server  
{  
 listen 8000;  
 server_name _;  
 root /var/www/html;  
 autoindex on;  
 autoindex_exact_size off;  
 autoindex_localtime on;  
 charset utf-8;  
}

3、启动nginx

1
2
3

#启动httpd服务:
systemctl enable nginx
systemctl start nginx

配置Web服务器

1、解压压缩包

1
2
3

#上传本地cdh6.2.1.tar.gz软件包到服务器/usr/local/src/目录并解压
scp cdh6.2.1.tar.gz root@10.0.0.7:/usr/local/src
tar zxvf /usr/local/src/cdh6.2.1.tar.gz

2、配置rpm包

1

mv /usr/local/src/html/* /var/www/html/

3、验证浏览器是否可以正常访问

配置mysql

安装Mysql

1、查询默认安装包卸载

1
2

rpm -qa |grep mariadb
yum -y remove mariadb-libs

2、安装mysql

1
2
3

wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm && \
  rpm -ivh mysql57-community-release-el7-11.noarch.rpm && \
  yum install -y mysql-community-server mysql-community-devel

3、启动mysql

1
2

systemctl enable mysqld
systemctl start mysqld

4、初始化mysql账号密码

1
2

grep 'temporary password' /var/log/mysqld.log
mysql_secure_installation

5、创建对应的表和库

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'xxxxx';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE cm DEFAULT CHARACTER SET utf8;
CREATE USER 'cm'@'%' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE am DEFAULT CHARACTER SET utf8;
CREATE USER 'am'@'%' IDENTIFIED BY 'xxxxxxx';
GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE rm DEFAULT CHARACTER SET utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE hue DEFAULT CHARACTER SET utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8;
CREATE USER 'sentry'@'%' IDENTIFIED BY 'xxxxxxx';
GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE nav_ms DEFAULT CHARACTER SET utf8;
CREATE USER 'nav_ms'@'%' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%';
FLUSH PRIVILEGES;

CREATE DATABASE nav_as DEFAULT CHARACTER SET utf8;
CREATE USER 'nav_as'@'%' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%';
FLUSH PRIVILEGES;

6、安装mysql驱动

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# 创建目录：
mkdir -p /usr/share/java/
# 上传本地jar包到服务器的指定目录中：
scp mysql-connector-java-5.1.47.jar root@10.0.0.7:/usr/share/java/
#进入目录：
cd /usr/share/java
# 修改权限：
chmod 777 mysql-connector-java-5.1.47.jar
# 创建软连接：
ln -s mysql-connector-java-5.1.47.jar mysql-connector-java.jar

Cloudera Manager安装

配置本地yum源

1、下载CM6.2的安装包

1

地址为: https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/RPMS/x86_64

2、下载CDH6.2的安装包

1

地址为: https://archive.cloudera.com/cdh6/6.2.1/parcels

3、执行createrepo命令生成rpm元数据

4、制作Cloudera Manager的yum源

1
2
3
4
5
6
7

cat >> /etc/yum.repos.d/cm.repo <<EOF
[cmrepo]
name = cm_repo
baseurl = http://10.0.0.7:8000/cm6.2.1
enable = true
gpgcheck = false
EOF

5、更新yun源

1

yum repolist

配置JDK

1、安装jdk

1

yum -y install oracle-j2sdk1.8-1.8.0+update181-1.x86_64

2、配置jdk环境变量

1
2
3
4
5
6
7
8

cat >> /etc/profile <<EOF
#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$PATH:$JAVA_HOME/bin
EOF

# 激活
source /etc/profile

3、查看jdk版本

1
2
3

java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

安装Cloudera Manager Server

1、初始化系统磁盘

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# 创建pv卷
pvcreate /dev/vdb
# 创建vg卷组
vgcreate apps-vg /dev/vdb
# 创建lv卷组
lvcreate -L 2T -n apps-lv apps-vg
# 格式化lv
mkfs.ext4 /dev/apps-vg/apps-lv
# 创建目录
mkdir /apps
# 挂载
mount /dev/apps-vg/apps-lv /apps
# 查看磁盘uuid
blkid
/dev/manager/apps-vg-apps-lv: UUID="0d567c75-6fbc-41e0-98ef-df78bcd74433" TYPE="ext4"
# 配置开机启动
cat >> /etc/fstab <<EOF
UUID=0d567c75-6fbc-41e0-98ef-df78bcd74433 /data ext4 defaults 0 0
EOF

2、通过yum源安装Cloudera Manger Server

1
2
3

# 先创建软连接
mkdir -pv /apps/cloudera
ln -s /apps/clouera /opt/clouera

2、初始化数据库，在master节点上执行如下命令：

1

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db_host db_name db_user db_password

3、启动Cloudera Manager Server

1
2
3
4

systemctl start cloudera-scm-server
systemctl status cloudera-scm-server -l
# 观察日志是否启动成功
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

4、检查端口是否监听，开放外网端口号为7180

1

netstat -lntp |grep 7180

访问CM, 登录地址 http://localhost:7180 默认账号密码为： admin/admin https://tc.ctq6.cn/tc/cdh-0.png

CDH 安装

集群安装向导

1、admin/admin 登录cm https://tc.ctq6.cn/tc/cdh-1.png

2、同意license协议，点击继续 https://tc.ctq6.cn/tc/cdh-3.png

3、选择60天试用，点击继续 https://tc.ctq6.cn/tc/cdh-4.png

4、点击继续 https://tc.ctq6.cn/tc/cdh-5.png

5、点击继续，输入集群名称，可以默认试用"Cluster 1" https://tc.ctq6.cn/tc/cdh-6.png

6、输入主机IP或者名称，点击搜索找到主机后点击继续 https://tc.ctq6.cn/tc/cdh-7.png

7、选择自定义存储库，输入cm的http地址 http://10.0.0.14:8000/cm6.2.1 https://tc.ctq6.cn/tc/cdh-8.png

8、使用Parcel建议，点击“更多选项”,点击“-”删除远程 Parcel 存储库 URL其它所有地址，输入http://10.0.0.14:8000/cdh6.2.1，点击“保存更改” https://tc.ctq6.cn/tc/cdh-9.png

https://tc.ctq6.cn/tc/cdh-10.png

9、点击继续，进入下一步安装jdk https://tc.ctq6.cn/tc/cdh-11.png

10、点击继续，进入下一步配置ssh账号密码，选择所有主机接受相同私钥

1
2

# 查看主节点私钥信息，复制粘贴
cat ~/.ssh/id_rsa

https://tc.ctq6.cn/tc/cdh-12.png

11.点击继续，进入下一步，安装Cloudera Manager相关到各个节点

更新于 2022-03-12
阅读原始文档
 CDH
返回 | 主页