kubeadm使用外部etcd集群tls部署kubernetes
环境：ubuntu 16.04.2
cpu 4 内存 8G
内核4.4.0-119
ip地址：192.168.0.62
192.168.0.63
192.168.0.64
etcd版本： 3.2.12
kubernetes版本：1.11.5
一、部署etcd集群(需要sudo或者root权限)
1生成证书及etcd的二进制文件包，工具下载地址
wget -O /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget -O /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x /bin/cfssl*
wget https://github.com/etcd-io/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
tar xf etcd-v3.2.18-linux-amd64.tar.gz
cp etcd-v3.2.18-linux-amd64/etcd* /usr/bin/
2.ca-config.json配置文件，修改过期时间为10年（红色部分）
内容如下：
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"etcd": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"client auth",
"server auth"
]
}
}
}
}
3.ca-csr.json配置文件如下：
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shanghai",
"L": "shanghai",
"O": "etcd",
"OU": "System"
}
]
}
4.etcd集群的etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.0.62",
"192.168.0.63",
"192.168.0.64"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shanghai",
"L": "shanghai",
"O": "etcd",
"OU": "System"
}
]
}
4.生成证书并自签名
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd
复制pem文件到你指定的目录，3台主机都要复制的
不建议使用目录/etc/kubernetes/pki/etcd
mkdir -p /etc/etcdCA
cp *.pem /etc/etcdCA
5.etcd的配置文件如下，红色部分自行更改。复制配置文件为/etc/default/etcd
ETCD_NAME=test-node62
ETCD_DATA_DIR="/var/lib/etcd/"
ETCD_LISTEN_PEER_URLS="https://192.168.0.62:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.62:2379,https://127.0.0.1:4001"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.62:2380"
ETCD_INITIAL_CLUSTER="test-node62=https://192.168.0.62:2380,test-node63=https://192.168.0.63:2380,test-node64=https://192.168.0.64:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-sdn"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.62:2379"
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/etc/etcdCA/ca.pem"
ETCD_CERT_FILE="/etc/etcdCA/etcd.pem"
ETCD_KEY_FILE="/etc/etcdCA/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/etc/etcdCA/ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcdCA/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcdCA/etcd-key.pem"
6.创建用户和服务并授权
useradd etcd
chmod 755 /etc/etcdCA/*
echo '[Unit]
Description=etcd - highly-available key value store
Documentation=https://github.com/coreos/etcd
Documentation=man:etcd
After=network.target
Wants=network-online.target
[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=on-abnormal
#RestartSec=10s
#LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd3.service' >/lib/systemd/system/etcd.service
7.启动服务
systemctl start etcd
8.检查集群状态
export ETCDCTL_API=3etcdctl \ --cacert=/etc/etcdCA/ca.pem \ --cert=/etc/etcdCA/etcd.pem \ --key=/etc/etcdCA/etcd-key.pem \ --endpoints=192.168.0.62:2379,192.168.0.63:2379,192.168.0.64:2379 \ endpoint health
看到下图就ok了
二、部署kubernetes
安装docker-ce （18.06.3）
sudo apt-get updatesudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-commoncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -sudo apt-key fingerprint 0EBFCD88sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable"sudo apt-get updatesudo apt-get install docker-ce=18.06.3~ce~3-0~ubuntu
2.安装kubernetes包
apt-get update && apt-get install -y apt-transport-httpscurl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat </etc/apt/sources.list.d/kubernetes.listdeb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial mainEOFapt-get install -y kubelet=1.11.5-00 kubeadm=1.11.5-00 kubectl=1.11.5-00
3.使用配置文件进行安装，配置文件（kubeadm-config.yaml）如下
apiVersion: kubeadm.k8s.io/v1alpha1kind: MasterConfigurationnetworking: podSubnet: 172.16.0.0/16 serviceSubnet: 10.96.0.0/12etcd: endpoints: - https://192.168.0.62:2379 - https://192.168.0.63:2379 - https://192.168.0.64:2379 caFile: /etc/etcdCA/ca.pem certFile: /etc/etcdCA/etcd.pem keyFile: /etc/etcdCA/etcd-key.pemkubernetesVersion: v1.11.5kubeProxy: config: mode: "ipvs"
4.加载需要的kubernetes镜像
A="kube-proxy-amd64:v1.11.5kube-apiserver-amd64:v1.11.5kube-controller-manager-amd64:v1.11.5kube-scheduler-amd64::v1.11.5pause:3.1"for i in $A;do docker pull mirrorgooglecontainers/$i docker tag mirrorgooglecontainers/$i k8s.gcr.io/$idonedocker pull coredns/coredns:1.1.3docker tag coredns/coredns:1.1.3 k8s.gcr.io/coredns:1.1.3
5.安装master,出现下图就master的安装好了
kubeadm init --config /path/kubeadm-config.yaml
6.授权客户端访问
mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
7.安装客户端（请先执行1.安装docker-ce 2.kubernetes包和4.加载需要的kubernetes镜像）
执行master生成后的kubeadm jion ，需要root或者sudo权限
如上图是：
kubeadm join 192.168.0.62:6443 --token 4msj6v.plj3rcsq89c4y4mn --discovery-token-ca-cert-hash sha256:7fb655510bc0af2dda7e401a45932709c473b0f33acef0794924b54715512bbc
三、安装calico插件
wget https://github.com/projectcalico/calico/releases/download/v2.6.12/release-v2.6.12.tgztar xf release-v2.6.12.tgzcd release-v2.6.12/k8s-manifests/hostedsed -i 's?http://127.0.0.1:2379?https://192.168.0.62:2379,https://192.168.0.63:2379,https://192.168.0.64:2379?g' calico.yamlcat /etc/etcdCA/etcd-key.pem|base64 -w 0 > ETCD-KEYcat /etc/etcdCA/ca.pem|base64 -w 0 > ETCD-CAcat /etc/etcdCA/etcd.pem|base64 -w 0 > ETCD-CERTsed -i "s?# etcd-key: null?etcd-key: $(cat ETCD-KEY)?g" calico.yamlsed -i "s?# etcd-ca: null?etcd-ca: $(cat ETCD-CA)?g" calico.yamlsed -i "s?# etcd-cert: null?etcd-cert: $(cat ETCD-CERT)?g" calico.yamlsed -i 's?etcd_ca: ""?etcd_ca: "/calico-secrets/etcd-ca"?g' calico.yamlsed -i 's?etcd_cert: ""?etcd_cert: "/calico-secrets/etcd-cert"?g' calico.yamlsed -i 's?etcd_key: ""?etcd_key: "/calico-secrets/etcd-key"?g' calico.yamlkubectl apply -f calico.yamlkubectl apply -f rbac-kdd.yaml
四、查看状态
至此k8s的基础部分完成
补充calico 3.10部分
wget https://github.com/projectcalico/calico/releases/download/v3.10.2/release-v3.10.2.tgztar xf release-v3.10.2.tgzcd release-v3.10.2/k8s-manifestssed -i 's?http://: ?https://192.168.0.62:2379,https://192.168.0.63:2379,https://192.168.0.64:2379?g' calico-etcd.yamlcat /etc/etcdCA/etcd-key.pem|base64 -w 0 > ETCD-KEYcat /etc/etcdCA/ca.pem|base64 -w 0 > ETCD-CAcat /etc/etcdCA/etcd.pem|base64 -w 0 > ETCD-CERTsed -i "s?# etcd-key: null?etcd-key: $(cat ETCD-KEY)?g" calico-etcd.yamlsed -i "s?# etcd-ca: null?etcd-ca: $(cat ETCD-CA)?g" calico-etcd.yamlsed -i "s?# etcd-cert: null?etcd-cert: $(cat ETCD-CERT)?g" calico-etcd.yamlsed -i 's?etcd_ca: ""?etcd_ca: "/calico-secrets/etcd-ca"?g' calico-etcd.yamlsed -i 's?etcd_cert: ""?etcd_cert: "/calico-secrets/etcd-cert"?g' calico-etcd.yamlsed -i 's?etcd_key: ""?etcd_key: "/calico-secrets/etcd-key"?g' calico-etcd.yamlsed -i 's?192.168.0.0/16?172.16.0.0/16?g' calico-etcd.yamlkubectl apply -f calico-etcd.yaml
注意：加密的etcd集群和明文的etcd集群不能通用
