KEP-5823: Pod-Level Checkpoint/Restore #5851

rst0git · 2026-01-29T14:34:54Z

One-line PR description: Enable support for Pod-level checkpoint and restore.

Issue link: Pod-Level Checkpoint/Restore #5823

Other comments: Related to Forensic Container Checkpointing #2008 and Checkpointing API #5091

k8s-ci-robot · 2026-01-29T14:34:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2026-01-29T14:35:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rst0git
Once this PR has been reviewed and has the lgtm label, please assign dchen1107 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fals · 2026-01-29T16:50:04Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+allows to transparently capture the state of these workloads and to resume execution
+from the most recent snapshot in the case of failures.
+
+[CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads](https://arxiv.org/abs/2502.16631)


@rst0git can we somehow make the checkpointing and restore tool more generic. I can see the word CRIU few times on the document and as other members asked last meeting, we should consider a proposal that would also enable gVisor, kata-containers or any other qemu/KVM based runtime to profit of this implementation on Kubernetes, as different tools might not use CRIU to C/R.

For GPU other runtimes also use cuda-checkpoint as for example gVisor, but no Criu involved as they implement everything on runsc instead.

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

haircommander · 2026-01-29T17:28:27Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+
+### Non-Goals
+
+* The initial Pod-level checkpoint and restore implementation is limited to a subset of Pod resources. The coverage of additional Pod and cluster resources is out of scope for this proposal.


can you enumerate what is out of scope here?

haircommander · 2026-01-29T17:29:23Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+* Behavior:
+  * Recreates Pod and namespaces
+  * Restores container runtime state
+  * Reconciles networking and storage


this may be tricker than this bullet implies, may need to be stateless and not hold pod IP to start.

haircommander · 2026-01-29T17:29:31Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+  * Checkpoint identifier
+  * Operation status
+
+#### RestorePod


is restore in scope for this KEP

haircommander · 2026-01-29T17:32:18Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+* Scheduling constraints and security contexts
+
+#### Container Runtime State
+* CRIU-generated checkpoint images for each container


@fals what does this look like for other non CRIU checkpointing options?

gVisor does not bump in same format as CRIU and QEMU/kvm based are even less aligned as it just snapshots the whole microVM which has all pods and containers inside.

i don't think we need to specifically say what the runtime does here. I would like to find a way that we can phrase this KEP in a way that is implementation agnostic, while still maintaining shared pieces (if we pass an option down, we need to find a way that option can be passed to the various implementations, if relevant)

I would say we should phrase that the

Checkpoint can be generated by different tools depending on the runtime used by the pod. The same tool should be used to restore as there's no backwards compatibility between them. The high level CRD holding information about the checkpointed pod MUST contain details about the tool used during checkpoint and it will be passed downstream from API server to CRI during restoration. The solution must be tool agnostic and any additional param needed by specific tools can be passed during C/R using command-line arguments from API.

haircommander · 2026-01-29T17:32:54Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+* Container-specific security contexts and capabilities
+
+#### Shared Pod Resources
+* Pod network namespace state, including Pod IP address where feasible


we'll need to loop in sig networking here for pod IP saving, I think we should do that as a follow-on

haircommander · 2026-01-29T17:33:36Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+* Pod network namespace state, including Pod IP address where feasible
+
+Initial support focuses on metadata and container runtime state. Additional resources such as
+shared memory, EmptyDir volumes, and other volume types will be added in future iterations.


I'd even go so far as to say a checkpointed pod should be stateless for the first iteration, and then we address stateful pods in a follow-on kep

haircommander · 2026-01-29T17:37:44Z

keps/sig-node/5823-pod-level-checkpoint-restore/README.md

+* Restoring
+
+#### Allowed Transitions
+* Running → Checkpointing → Running


"running" is not a pod state though, there's a lot of pod states and the state machine is pretty complex. can we checkpoint if a pod is being resized in place? how about if the init containers are still running? I think we need to muscle through this a bit more

dfeigin-nv · 2026-02-02T08:04:37Z

keps/sig-node/5823-pod-level-checkpoint-restore/kep.yaml

+  - "@haircommander"
+owning-sig: sig-node
+participating-sigs:
+  - sig-node


Aren't this also participating sigs?

sig-storage

sig-api-machinery

keps/sig-node/5823-pod-level-checkpoint-restore/kep.yaml

wendy-ha18 · 2026-02-03T14:34:48Z

Hi @rst0git , I'm Wendy from SIG Node KEP Wrangler and this KEP is at risk for PRR deadline at the moment, are you still aiming for this KEP in v1.36? The PRR deadline is approaching tomorrow (Wednesday 4th February 2026 (AoE) / Thursday 5th February 2026, 12:00 UTC).

This is checklist we need to meet to be able to pass PRR, do you think we can land these requirements before deadline or you need more time for it? (considering for exception request in advance).

PR open or merged with the KEP's PRR questionnaire filled out. - PENDING

PR open or merged with kep.yaml updated with the stage, latest-milestone, and milestone struct filled out. - PENDING

PR open or merged with a PRR approval file with the PRR approver listed for the stage the KEP is targeting.- PENDING

rst0git · 2026-02-03T14:45:00Z

@wendy-ha18 Thank you for your message! I've updated the pull request with the KEP's PRR questionnaire and kep.yaml filled out.

wendy-ha18 · 2026-02-03T15:33:37Z

Thank @rst0git , one thing left to ensure we can safely pass PRR deadline is I'm not really sure who is PRR reviewer (and PRR shadow) for this KEP.

I have asked in #prod-readiness channel in Slack here. Please feel free to follow up further with it when you have time.

Co-authored-by: Adrian Reber <areber@redhat.com> Co-authored-by: Dan Feigin <dfeigin@nvidia.com> Signed-off-by: Radostin Stoyanov <radostin.stoyanov@eng.ox.ac.uk>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 29, 2026

k8s-ci-robot requested review from dchen1107 and derekwaynecarr January 29, 2026 14:35

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 29, 2026

fals reviewed Jan 29, 2026

View reviewed changes

haircommander reviewed Jan 29, 2026

View reviewed changes

keps/sig-node/5823-pod-level-checkpoint-restore/README.md Outdated Show resolved Hide resolved

haircommander reviewed Jan 29, 2026

View reviewed changes

Rajalakshmi-Girish mentioned this pull request Jan 31, 2026

Pod-Level Checkpoint/Restore #5823

Open

4 tasks

dfeigin-nv reviewed Feb 2, 2026

View reviewed changes

keps/sig-node/5823-pod-level-checkpoint-restore/kep.yaml Show resolved Hide resolved

dfeigin-nv reviewed Feb 2, 2026

View reviewed changes

keps/sig-node/5823-pod-level-checkpoint-restore/kep.yaml Outdated Show resolved Hide resolved

rst0git force-pushed the pod-checkpoint-restore branch 2 times, most recently from 25d89d1 to 155019a Compare February 3, 2026 14:40

rst0git marked this pull request as ready for review February 3, 2026 14:43

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 3, 2026

rst0git force-pushed the pod-checkpoint-restore branch 2 times, most recently from 34973b6 to 194e916 Compare February 3, 2026 14:49

rst0git force-pushed the pod-checkpoint-restore branch from 194e916 to c69e12b Compare February 4, 2026 04:05

KEP-5823: Pod-Level Checkpoint/Restore

92c6615

Co-authored-by: Adrian Reber <areber@redhat.com> Co-authored-by: Dan Feigin <dfeigin@nvidia.com> Signed-off-by: Radostin Stoyanov <radostin.stoyanov@eng.ox.ac.uk>

rst0git force-pushed the pod-checkpoint-restore branch from c69e12b to 92c6615 Compare February 4, 2026 05:04


		### Non-Goals

		* The initial Pod-level checkpoint and restore implementation is limited to a subset of Pod resources. The coverage of additional Pod and cluster resources is out of scope for this proposal.

KEP-5823: Pod-Level Checkpoint/Restore #5851

Are you sure you want to change the base?

KEP-5823: Pod-Level Checkpoint/Restore #5851

Conversation

rst0git commented Jan 29, 2026

Uh oh!

k8s-ci-robot commented Jan 29, 2026

Uh oh!

k8s-ci-robot commented Jan 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fals Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wendy-ha18 commented Feb 3, 2026

Uh oh!

rst0git commented Feb 3, 2026

Uh oh!

wendy-ha18 commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fals Jan 30, 2026 •

edited

Loading