centos 6.8下 CDH5.13.3安装与配置（详细部署）
centos 6.8下 CDH5.13.3安装与配置（详细部署）
一、安装前准备
1.1、下载地址
wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz
http://archive.cloudera.com/cdh6/parcels/5.13.3/
wget http://archive.cloudera.com/cdh6/parcels/5.13.3/manifest.json
wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel
wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1
1.2、防火墙关闭
/etc/init.d/iptables stop
chkconfig iptables off
1.3、设置selinux
vi /etc/selinux/config
1.4、设置时间同步ntp(每台机器)
yum -y install ntpdate
crontab -e
*/5 * * * * ntpdate 0.rhel.pool.ntp.org
保存退出
vi /etc/hosts（hadoop01增加）
注意：Cloudera-Manager台需要增加127.0.0.1 hadoop01
127.0.0.1 hadoop01
192.168.1.201 hadoop01
192.168.1.202 hadoop02
192.168.1.203 hadoop03
vi /etc/hosts（其他两台hadoo02、hadoop03）
192.168.1.201 hadoop01
192.168.1.202 hadoop02
192.168.1.203 hadoop03
二、依赖包安装（每台机器）
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi openssh-clients
三、jdk安装（每台机器）
tar zxvf jdk-8u111-linux-x64.gz
mv jdk1.8.0_111 /usr/local/jdk
配置环境变量
vi /etc/profile
##jdk
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
让环境变量生产
source /etc/profile
四、ssh免密码登录配置
ssh-keygen -t rsa
ssh-copy-id 192.168.1.201
ssh-copy-id 192.168.1.202
ssh-copy-id 192.168.1.202
五、安装mysql(在haoop01上安装)
5.1、安装依赖包
yum -y install libaio*
yum -y install libnuma*
5.2、创建mysql用户
useradd mysql
5.3、创建目录
mkdir -p /data/app
mkdir /data/app/mysql3306
mkdir /data/app/undolog
5.4、解压文件
tar zxvf mysql-5.6.41-linux-glibc2.12-x86_64.tar.gz
mv mysql-5.6.41-linux-glibc2.12-x86_64 /data/app/mysql5.6
mkdir /data/app/mysql5.6/logs
5.5、配置my.cnf
vi /data/app/mysql5.6/my.cnf
[client]
socket=/data/app/mysql5.6/mysqld.sock
port=3306
[mysql]
prompt=\\u@\\d \\r:\\m:\\s>
no-auto-rehash
default-character-set=utf8
[mysqld_safe]
log-error=/data/app/mysql3306/mysqld.error
[mysqld]
socket=/data/app/mysql5.6/mysqld.sock
pid-file=/data/app/mysql5.6/mysqld.pid
basedir=/data/app/mysql5.6
datadir=/data/app/mysql3306
port=3306
server-id=1201
character-set-server=utf8
skip-external-locking
#skip-name-resolve
max_connections=1000
max_connect_errors=1000
max_allowed_packet=128M
table_definition_cache=500
table_open_cache=500
sort_buffer_size = 1M
tmp_table_size = 64M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_type=0
query_cache_size=0
lower_case_table_names = 1
log_bin_trust_function_creators = 1
#################slow log####################
slow-query_log=1
slow-query_log_file=/data/app/mysql5.6/logs/mysql.slow
long_query_time=2
####################binlog######################
log-bin=mysql-bin
binlog-format=ROW
expire_logs_days=10
log-slave-updates=1
sync_binlog=0
auto_increment_increment=2
auto_increment_offset=1
###
###
###
innodb_undo_directory=/data/app/undolog
innodb_undo_logs = 128
innodb_undo_tablespaces = 3
################INNODB################
transaction-isolation=READ-COMMITTED
innodb_buffer_pool_size=2G
innodb_flush_log_at_trx_commit=2
innodb_strict_mode=1
innodb_flush_method=O_DIRECT
innodb_file_format=Barracuda
innodb_log_files_in_group=3
innodb_file_per_table=1
innodb_io_capacity=500
innodb_support_xa=1
[mysqldump]
quick
max_allowed_packet=128M
[mysqlhotcopy]
interactive_timeout = 28400
5.6、增加权限
chown -R mysql:mysql /data/app
5.7、配置环境变量
vi /etc/profile
##mysql
export MYSQL_HOME=/data/app/mysql5.6
export MY_BASEDIR_VERSION=/data/app/mysql5.6
export PATH=/data/app/mysql5.6/bin:$PATH
export LD_LIBRARY_PATH=/data/app/mysql5.6/lib
环境生效
source /etc/profile
5.8、初始化数据库
/data/app/mysql5.6/scripts/mysql_install_db --user=mysql --basedir=/data/app/mysql5.6 --datadir=/data/app/mysql3306
5.9、配置启动文件
cp /data/app/mysql5.6/support-files/mysql.server /etc/init.d/mysqld
vi /etc/init.d/mysqld
basedir=/data/app/mysql5.6
datadir=/data/app/mysql3306
保存退出
chmod +x /etc/init.d/mysqld
chkconfig mysqld on
启动
/etc/init.d/mysqld start
5.10、初始化密码
/data/app/mysql5.6/bin/mysqladmin -u root password 'yb2018#08'
5.11、设置远程权限
# mysql -uroot -pyb2018#08
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.41-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
root@(none) 10:28:44>grant all on *.* to 'scm'@'localhost' identified by 'temp' with grant option;
Query OK, 0 rows affected (0.00 sec)
root@(none) 10:28:58>grant all on *.* to 'cdh'@'%' identified by 'cdh#2018' with grant option;
Query OK, 0 rows affected (0.00 sec)
root@(none) 10:29:08>create database hive;
Query OK, 1 row affected (0.00 sec)
root@(none) 10:29:16>create database ooz;
Query OK, 1 row affected (0.00 sec)
root@(none) 10:29:27>create database hue;
Query OK, 1 row affected (0.00 sec)
root@(none) 10:29:36>flush privileges;
Query OK, 0 rows affected (0.00 sec)
六、安装Cloudera-Manager
6.1、解压包到指定目录，所有服务器都要
mkdir -p /data/cloudera-manager
tar zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz -C /data/cloudera-manager
6.2创建cloudera-scm用户(所有节点)
useradd -r -d /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm
id cloudera-scm
6.3、配置从节点cloudera-manger-agent指向主节点服务器（hadoop02,hadoop03配置）
vi /data/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini
将server_host改为CMS所在的主机名即hadoop01
6.4、主节点中创建parcel-repo仓库目录（这个只在主节点hadoop01上需要）
mkdir -p /data/cloudera/parcel-repo
chown cloudera-scm:cloudera-scm /data/cloudera/parcel-repo
cp CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 manifest.json /data/cloudera/parcel-repo/
cd /data/cloudera/parcel-repo/
mv CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha
注意：其中CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 后缀要把1去掉
6.5、所有节点创建parcels目录（所有节点）
mkdir -p /data/cloudera/parcels
chown cloudera-scm:cloudera-scm /data/cloudera/parcels
解释：Clouder-Manager将CDHs从主节点的/data/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/data/cloudera/parcels目录中
6.6、初始脚本配置数据库scm_prepare_database.sh(在主节点上)
/data/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uscm -ptemp scm scm temp
如果报错，操作
mkdir /usr/share/java
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
上图说明成功
成功把记住把#127.0.0.1 localhost hadoop01 注释掉，不然后面就会有问题
vi /etc/hosts
#127.0.0.1 hadoop01
192.168.1.201 hadoop01
192.168.1.202 hadoop02
192.168.1.203 hadoop03
6.7、启动主节点cloudera-scm-server(hadoop01操作)
cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
chmod +x /etc/init.d/cloudera-scm-server
启动
/etc/init.d/cloudera-scm-server start
如果报File not found: /usr/sbin/cmf-server错误
vi /etc/init.d/cloudera-scm-server
找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default
启动
/etc/init.d/cloudera-scm-server start
6.8、启动cloudera-scm-agent所有节点(hadoop02,hadoop03操作)
mkdir /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-agent
cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
chmod +x /etc/init.d/cloudera-scm-agent
/etc/init.d/cloudera-scm-agent start
vi /etc/init.d/cloudera-scm-agent
找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default
启动
/etc/init.d/cloudera-scm-agent start
6.9、登录用户admin 密码admin
http://192.168.1.201:7180
继续
选择免费，继续
继续
选择当前管理主机选择hadoop02,hadoop03,继续
点击更多选项配置
修改好后，保存修改
重启后才后出下如下图
/etc/init.d/cloudera-scm-server restart
重启好后才能如上图继续，因为默认/opt下安装的
继续
每台执行以下
echo 10 >/proc/sys/vm/swappiness
echo never >/sys/kernel/mm/transparent_hugepage/defrag
echo never >/sys/kernel/mm/transparent_hugepage/enabled
以图根据自己需求选择不一样点继续
以上Datanode修改两台
把连接主机名，数据库名称，用户名，密码输入进去，点击测试连接都Succesful成功后，点击继续
根据自己修改点击继续
继续
安装完成
