Kubernetes中Node异常时Pod状态是怎样的

这篇文章主要讲解了"Kubernetes中Node异常时Pod状态是怎样的"，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着小编的思路慢慢深入，一起来研究和学习"Kubernetes中Node异常时Pod状态是怎样的"吧！

Kubelet进程异常，Pod状态变化

一个节点上运行着pod前提下，这个时候把kubelet进程停掉。里面的pod会被干掉吗？会在其他节点recreate吗？

结论：

（1）Node状态变为NotReady （2）Pod 5分钟之内状态无变化，5分钟之后的状态变化：Daemonset的Pod状态变为Nodelost，Deployment、Statefulset和Static Pod的状态先变为NodeLost，然后马上变为Unknown。Deployment的pod会recreate，但是Deployment如果是node selector停掉kubelet的node，则recreate的pod会一直处于Pending的状态。Static Pod和Statefulset的Pod会一直处于Unknown状态。

Kubelet恢复，Pod行为

如果kubelet 10分钟后又起来了，node和pod会怎样？

结论：

（1）Node状态变为Ready。 （2）Daemonset的pod不会recreate，旧pod状态直接变为Running。 （3）Deployment的则是将kubelet进程停止的Node删除（原因可能是因为旧Pod状态在集群中有变化，但是Pod状态在变化时发现集群中Deployment的Pod实例数已经够了，所以对旧Pod做了删除处理） （4）Statefulset的Pod会重新recreate。 （5）Staic Pod没有重启，但是Pod的运行时间会在kubelet起来的时候置为0。

在kubelet停止后，statefulset的pod会变成nodelost，接着就变成unknown，但是不会重启，然后等kubelet起来后，statefulset的pod才会recreate。

还有一个就是Static Pod在kubelet重启以后应该没有重启，但是集群中查询Static Pod的状态时，Static Pod的运行时间变了

StatefulSet Pod为何在Node异常时没有Recreate

Node down后，StatefulSet Pods並沒有重建，為什麼？

我们在node controller中发现，除了daemonset pods外，都会调用delete pod api删除pod。

但并不是调用了delete pod api就会从apiserver/etcd中删除pod object，仅仅是设置pod 的deletionTimestamp，标记该pod要被删除。真正删除Pod的行为是kubelet，kubelet grace terminate该pod后去真正删除pod object。这个时候statefulset controller 发现某个replica缺失就会去recreate这个pod。

但此时由于kubelet挂了，无法与master通信，导致Pod Object一直无法从etcd中删除。如果能成功删除Pod Object，就可以在其他Node重建Pod。

另外，要注意，statefulset只会针对isFailed Pod，（但现在Pods是Unkown状态）才会去delete Pod。

// delete and recreate failed pods                if isFailed(replicas[I]) {                        ssc.recorder.Eventf(set, v1.EventTypeWarning, "RecreatingFailedPod",                                "StatefulSetPlus %s/%s is recreating failed Pod %s",                                set.Namespace,                                set.Name,                                replicas[I].Name)                        if err := ssc.podControl.DeleteStatefulPlusPod(set, replicas[I]); err != nil {                                return &status, err                        }                        if getPodRevision(replicas[I]) == currentRevision.Name {                                status.CurrentReplicas-                        }                        if getPodRevision(replicas[I]) == updateRevision.Name {                                status.UpdatedReplicas-                        }                        status.Replicas-                        replicas[I] = newVersionedStatefulSetPlusPod(                                currentSet,                                updateSet,                                currentRevision.Name,                                updateRevision.Name,                                i)                }

优化StatefulSet Pod的行为

所以针对node异常的情况，有状态应用(Non-Quorum)的保障，应该补充以下行为：

• 监测node的网络、kubelet进程、操作系统等是否异常，区别对待。

• 比如，如果是网络异常，Pod无法正常提供服务，那么需要kubectl delete pod -f -grace-period=0进行强制从etcd中删除该pod。

• 强制删除后，statefulset controller就会自动触发在其他Node上recreate pod。

亦或者，更粗暴的方法，就是放弃GracePeriodSeconds，StatefulSet Pod GracePeriodSeconds为nil或者0，则就会直接从etcd中删除该object。

// BeforeDelete tests whether the object can be gracefully deleted.// If graceful is set, the object should be gracefully deleted.  If gracefulPending// is set, the object has already been gracefully deleted (and the provided grace// period is longer than the time to deletion). An error is returned if the// condition cannot be checked or the gracePeriodSeconds is invalid. The options// argument may be updated with default values if graceful is true. Second place// where we set deletionTimestamp is pkg/registry/generic/registry/store.go.// This function is responsible for setting deletionTimestamp during gracefulDeletion,// other one for cascading deletions.func BeforeDelete(strategy RESTDeleteStrategy, ctx context.Context, obj runtime.Object, options *metav1.DeleteOptions) (graceful, gracefulPending bool, err error) {        objectMeta, gvk, kerr := objectMetaAndKind(strategy, obj)        if kerr != nil {                return false, false, kerr        }        if errs := validation.ValidateDeleteOptions(options); len(errs) > 0 {                return false, false, errors.NewInvalid(schema.GroupKind{Group: metav1.GroupName, Kind: "DeleteOptions"}, "", errs)        }        // Checking the Preconditions here to fail early. They'll be enforced later on when we actually do the deletion, too.        if options.Preconditions != nil && options.Preconditions.UID != nil && *options.Preconditions.UID != objectMeta.GetUID() {                return false, false, errors.NewConflict(schema.GroupResource{Group: gvk.Group, Resource: gvk.Kind}, objectMeta.GetName(), fmt.Errorf("the UID in the precondition (%s) does not match the UID in record (%s). The object might have been deleted and then recreated", *options.Preconditions.UID, objectMeta.GetUID()))        }        gracefulStrategy, ok := strategy.(RESTGracefulDeleteStrategy)        if !ok {                // If we're not deleting gracefully there's no point in updating Generation, as we won't update                // the obcject before deleting it.                return false, false, nil        }        // if the object is already being deleted, no need to update generation.        if objectMeta.GetDeletionTimestamp() != nil {                // if we are already being deleted, we may only shorten the deletion grace period                // this means the object was gracefully deleted previously but deletionGracePeriodSeconds was not set,                // so we force deletion immediately                // IMPORTANT:                // The deletion operation happens in two phases.                // 1. Update to set DeletionGracePeriodSeconds and DeletionTimestamp                // 2. Delete the object from storage.                // If the update succeeds, but the delete fails (network error, internal storage error, etc.),                // a resource was previously left in a state that was non-recoverable.  We                // check if the existing stored resource has a grace period as 0 and if so                // attempt to delete immediately in order to recover from this scenario.                if objectMeta.GetDeletionGracePeriodSeconds() == nil || *objectMeta.GetDeletionGracePeriodSeconds() == 0 {                        return false, false, nil                }                ...        }        ...}       

感谢各位的阅读，以上就是"Kubernetes中Node异常时Pod状态是怎样的"的内容了，经过本文的学习后，相信大家对Kubernetes中Node异常时Pod状态是怎样的这一问题有了更深刻的体会，具体使用情况还需要大家实践验证。这里是，小编将为大家推送更多相关知识点的文章，欢迎关注！