在k8s中安装jenkins并实现动态生成 jenkins slave
安装jenkins
1、创建一个命名空间
$ kubectl create namespace kube-ops
2、为jenkins创建pvc（也可以使用存储类创建）
apiVersion: v1kind: PersistentVolumemetadata: name: opspvspec: capacity: storage: 2Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete nfs: server: 192.168.1.244 path: /data/k8s---kind: PersistentVolumeClaimapiVersion: v1metadata: name: opspvc namespace: kube-opsspec: accessModes: - ReadWriteMany resources: requests: storage: 2Gi3、创建jenkins需要的rbac权限
apiVersion: v1kind: ServiceAccountmetadata: name: jenkins2 namespace: kube-ops---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1beta1metadata: name: jenkins2rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"]---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata: name: jenkins2 namespace: kube-opsroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins2subjects: - kind: ServiceAccount name: jenkins2 namespace: kube-ops也可以为ServiceAccoun绑定一个系统现有的 cluster-admin 集群角色权限
4、创建jenkins pod
$ docker pull docker.io/jenkins/jenkins:lts
$ docker pull cnych/jenkins:jnlp6
---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: jenkins2 namespace: kube-opsspec: template: metadata: labels: app: jenkins2 spec: terminationGracePeriodSeconds: 10 serviceAccount: jenkins2 containers: - name: jenkins image: docker.io/jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 500m memory: 512Mi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 volumeMounts: - name: jenkinshome subPath: jenkins2 mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: opspvc---apiVersion: v1kind: Servicemetadata: name: jenkins2 namespace: kube-ops labels: app: jenkins2spec: selector: app: jenkins2 type: NodePort ports: - name: web port: 8080 targetPort: web nodePort: 30003 - name: agent port: 50000 targetPort: agent$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常启动
$ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops
$ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops
5、在nfs服务器上修改jenkins持久目录的权限并重新创建jenkins pod
$ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上
$ kubectl delete -f jenkins2.yaml
$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0
为什么是1000？
上述镜像的Dockerfile文件中定义的是：user=jenkins group=jenkins uid=1000 gid=1000
Dockerfile文件的地址：
https://github.com/jenkinsci/docker/blob/master/Dockerfile
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
ARG JENKINS_HOME=/var/jenkins_home
$ kubectl get svc -n kube-ops
jenkins2 NodePort 10.105.121.176
http://192.168.1.243:30003
初始密码在nfs服务器上
$ cat /data/k8s/jenkins2/secrets/initialAdminPassword
在jenkins上创建kubernetes云
enkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除
这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。
1、安装插件
安装kubernetes plugin， 点击 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin
2、增加kubernetes云
点击 Manage Jenkins -> Configure System -> (拖到最下方)Add a new cloud -> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息----连接测试
name：kubernetes
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Kubernetes 命名空间：kube-ops
Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc）
3、添加pod模板
添加pod模板----Kubernetes Pod Template
名称：jnlp
命名空间：kube-ops
标签列表：dongyali-jnlp
4、添加容器模板
添加容器----Container Template
名称：jnlp
Docker 镜像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的镜像名字去掉6）
工作目录：/home/jenkins/agent
运行的命令：清空
命令参数：清空
5、添加两个卷
添加卷----Host Path Volume
主机路径：/var/run/docker.sock
挂载路径：/var/run/docker.sock
主机路径：/root/.kube
挂载路径：/root/.kube
6、可能需要配置ServiceAccount
$ kubectl get sa -n kube-ops
jenkins2 1 14h
点击添加卷下面的高级----Service Account----jenkins2
7、用shell测试 Kubernetes 动态生成 jenkins slave
新建任务----名字----自由风格
通用----勾选限制项目的运行节点----标签表达式：dongyali-jnlp
构建----执行shell----输入如下内容----保存立即构建
echo "测试 Kubernetes 动态生成 jenkins slave"echo "==============docker in docker==========="docker infoecho "=============kubectl============="kubectl get pods观察 Kubernetes 集群中 Pod 的变化：
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h59m
jnlp-tl1km 1/1 Running 0 44s
当任务运行完毕，jnlp这个slave pod就会自动消失。
8、用pipeline测试 Kubernetes 动态生成 jenkins slave
新建任务----名字----流水线
在流水线脚本中输入如下内容----保存立即构建
node('dongyali-jnlp') { stage('Clone') { echo "1.Clone Stage" } stage('Test') { echo "2.Test Stage" } stage('Build') { echo "3.Build Stage" } stage('Deploy') { echo "4. Deploy Stage" }}
