导航：首页 > 互联网科技 >

Cloudera Manager大数据平台搭建

发表于：2025-01-23 作者：千家信息网编辑

千家信息网最后更新 2025年01月23日，在企业里快速搭建大数据平台除了Ambria外还可以用Cloudera Manager，这两种工具相对于纯手工搭建确实便捷很多，但是有利也有弊，相对于新手来说就不便于掌握内部原理，不好排错等,而纯手工搭

千家信息网最后更新 2025年01月23日Cloudera Manager大数据平台搭建

    在企业里快速搭建大数据平台除了Ambria外还可以用Cloudera Manager，这两种工具相对于纯手工搭建确实便捷很多，但是有利也有弊，相对于新手来说就不便于掌握内部原理，不好排错等,而纯手工搭建，出错较多，不容易成功。

一. 规划：
192.168.3.201 server/agent mysql
192.168.3.202 agent namenode
192.168.3.203 agent namenode（备） resourcemanager
192.168.3.204 agent resourcemanager （备）
192.168.3.205 agent

二. 环境准备

系统：Centos 6.5 64位

配置IP，hostname,
各个服务器IP配置：# vim /etc/sysconfig/network-script/ifcfg-eth0
service network restart
各个服务器名配置：# vim /etc/sysconfig/network

vim /etc/hosts

            192.168.3.201    rac201.com            192.168.3.202    rac202.com            192.168.3.203    rac203.com            192.168.3.204    rac204.com            192.168.3.205    rac205.com  3.检查DNS配置，确保能上网    # vim /etc/resolve.conf      #   ping www.baidu.com  4.关闭防火墙和禁用selinux     getenforce   查看selinux的状态并修改为disable状态    #  vim  /etc/sysconfig/selinux                 SELINUX=disable            重启生效  5. 禁用和关闭防火墙      临时禁用  #  iptables -F                    #  setenforce 0            或设置开机不启动                #   service  iptables stop                #   chkconfig iptables off  6. 所有服务器卸载自带的JDK，安装新版本，比如1.7或1.8版本的     # rpm  -qa | grep java     # rpm -e --nodeps tzdata-java-2012j-1.el6.noarch java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64     # mkdir -p /opt/modules         # tar zxf jdk-8u141-linux-x64.gz -C /opt/modules     7. 对所有服务器配置JAVA环境变量     # vi /etc/profile     export JAVA_HOME=/opt/modules/jdk1.8.0_141     export PATH=$PATH:$JAVA_HOME/bin     # source /etc/profile  8. 拷贝配置到其他服务器     # scp -r  /opt/modules/jdk1.8.0_141  root@rac202.com:/opt/modules/     # scp -r  /opt/modules/jdk1.8.0_141  root@rac203.com:/opt/modules/     # scp -r  /opt/modules/jdk1.8.0_141  root@rac204.com:/opt/modules/     # scp -r  /opt/modules/jdk1.8.0_141  root@rac205.com:/opt/modules/     检查：  java -version  9. 配置服务器之间免密钥登录     9.1  #  ssh-keygen          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac201.com          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac202.com          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac203.com          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac204.com          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac205.com     9.2  # ssh rac01.com          # exit            #不要忘记连接后要退出          # ssh rac02.com          # exit          # ssh rac03.com          # exit          # ssh rac04.com          # exit          # ssh rac05.com          # exit 10. 服务器时间同步    10.1 启动ntpd进程服务        # service ntpd start        # chkconfig ntpd on        # chkconfig --list | grep ntpd          10.2 修改时间服务器的配置        # vim /etc/ntp.conf        打开注释，修改成本服务器的网段        # restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap                       注释掉以下几行，可能是三行也可能是五行        #server 0.centos.pool.ntp.org        #server 1.centos.pool.ntp.org        #server 2.centos.pool.ntp.org        手工添加以下两行        server  127.127.1.0     # local clock        fudge   127.127.1.0 s           启动并且配置开机时启动        # service ntpd start && chkconfig ntpd on && chkconfig --list | grep ntpd11. 对所有服务器设置能打开的文件最大数量、进程最大数，能使用的内存等     # vi /etc/security/limits.conf                     *               soft    nofile          32728        *               hard    nofile          1024567        *               soft    nproc           65535        *               hard    nproc           unlimited        *               soft    memlock         unlimited        *               hard    memlock         unlimited     复制limits.conf到其他服务器中     # scp /etc/security/limits.conf root@rac202.com:/etc/security/limits.conf     # scp /etc/security/limits.conf root@rac203.com:/etc/security/limits.conf     # scp /etc/security/limits.conf root@rac204.com:/etc/security/limits.conf     # scp /etc/security/limits.conf root@rac205.com:/etc/security/limits.conf     重启生效：  ulimit -a

三. Mysql数据库安装

在rac201服务器检查并卸载系统自动安装的Mysql版本,
rpm -qa | grep mysql
```
    # rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
```

采用yum源方式安装Mysql5.6, 先下载yum源文件,

wget http://repo.mysql.com/mysql57-community-release-el6-8.noarch.rpm

    # rpm -Uvh mysql57-community-release-el6-8.noarch.rpm    修改yum源文件：      vim /etc/yum.repos.d/mysql-community.repo      [mysql56-community]      enabled=1      gpgcheck=1      [mysql57-community]      enabled=0      gpgcheck=1      vim /etc/yum.repos.d/mysql-community-source.repo      [mysql56-community-source]      enabled=1         gpgcheck=1      [mysql-tools-preview-source]      name=MySQL Tools Preview - Source      baseurl=http://repo.mysql.com/yum/mysql-tools-preview/el/6/SRPMS      enabled=1         gpgcheck=1    用yum repolist mysql这个命令查看一下是否已经有mysql可安装文件      #yum repolist all | grep mysql    yum安装mysql, 一直yes下去    # yum -y install mysql-community-server    安装成功后，启动并初始化    # service mysqld start    # service mysqld status    mysql刚安装完，mysql的root用户的密码默认是空的，需要给root用户设置密码密码，执行下列命令进行安全设置    # mysql_secure_installation            输入当前用户：回车    输入root密码：123456    production environment.    Remove anonymous users? [Y/n] y     ... Success!    Normally, root should only be allowed to connect from 'localhost'.  This    ensures that someone cannot guess at the root password from the network.    Disallow root login remotely? [Y/n] n     ... skipping.    By default, MySQL comes with a database named 'test' that anyone can    access.  This is also intended only for testing, and should be removed    before moving into a production environment.    Remove test database and access to it? [Y/n] n     ... skipping.    Reloading the privilege tables will ensure that all changes made so far    will take effect immediately.    Reload privilege tables now? [Y/n] y     ... Success!

四. Cloudera Manager安装

在每台服务器上安装第三方依赖包，或者在Centos系统安装时安装好开发包
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
提前下载好cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
在rac201服务器上创建CM的解压目录
mkdir -p /opt/cloudera-manager
tar -zxf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz -C /opt/cloudera-manager
在rac201服务器上修改修改agent配置
vi /opt/cloudera-manager/cm-5.3.6/etc/cloudera-scm-agent/config.ini
Hostname of the CM server.
```
server_host=rac201.com server_port=7182 
```

拷贝解压后的CM文件夹到其他服务器

scp -r /opt/cloudera-manager/ root@rac202.com:/opt/

# scp -r /opt/cloudera-manager/ root@rac203.com:/opt/# scp -r /opt/cloudera-manager/ root@rac204.com:/opt/# scp -r /opt/cloudera-manager/ root@rac205.com:/opt/

在所有服务器上创建CM的用户
useradd --system --home=/opt/cloudera-manager/cm-5.3.6/run/cloudera-scm-server --no-create-home --shell=/bin/false cloudera-scm
```
# cat /etc/passwd | grep cloudera-scm
```

设置CM数据库的权限

mysql -uroot -p123456

mysql> grant all privileges on *.*  to 'temp'@'%' identified by 'temp' with grant option;mysql> grant all privileges on *.* to 'temp'@'rac01.com' identified by 'temp' with grant option;mysql> flush privileges;mysql> use  mysql;mysql> select host ,user from user;     进入rac201服务器以下文件夹，执行脚本命令，事先要将mysql-connector-java.jar放入/usr/share/java文件夹中# cd /opt/cloudera-manager/cm-5.3.6/share/cmf/schema# ./scm_prepare_database.sh mysql -h rac01.com -utemp -ptemp --scm-host rac01.com scm scm scm

提前下载好CDH源文件
CDH-5.3.6-1.cdh6.3.6.p0.11-el6.parcel
CDH-5.3.6-1.cdh6.3.6.p0.11-el6.parcel.sha
在rac201服务器上创建以下目录

mkdir -p /opt/cloudera/parcel-repo

# mv CDH-5.3.6-1.cdh6.3.6.p0.11-el6.parcel* /opt/cloudera/parcel-repo/在所有服务器上创建# mkdir -p /opt/cloudera/parcels        更改parcel目录的所有人在rac201服务器上 # chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/在所有服务器上 # chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels

启动CM的进程
在rac201服务器上
cd /opt/cloudera-manager/cm-5.3.6/etc/init.d
```
 # ./cloudera-scm-server  start 
```
在所有服务器上启动agent进程
cd /opt/cloudera-manager/cm-5.3.6/etc/init.d/
```
 # ./cloudera-scm-agent start 
```
CM的服务器登录界面
http://rac01.com:7180
登录账号和密码 admin admin