Introduce synctest-based framework for deterministic testing #9099

GaetanoMar96 · 2026-01-22T13:09:29Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR introduces a new testing framework for Cluster Autoscaler based on synctest.

The new framework utilizes a "bubble" environment and virtual time to:

Ensure Determinism: Background goroutines are managed within the synctest bubble, preventing leaks.
Accelerate Test Execution: Full scale-up/scale-down lifecycles that normally take minutes of simulated time now execute in milliseconds of real time.
Structured Lifecycle Management: Provides a TestContext and Builder pattern for easier setup of complex cluster states.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

The PR is structured into three logical commits to provide a clear path from architecture to implementation:

Refactor for Dependency Injection (Builder Pattern): I introduced a Builder Pattern in the main initialization logic. This allows us to inject fakes (KubeClient, CloudProvider, PodObservers) during setup. This decoupling is essential for moving away from heavyweight integration tests toward lightweight, controlled simulations.
Lifecycle & Leak Management: To support synctest's requirement for a clean "bubble" exit, I refactored internal components to properly accept and propagate context.Context. This ensures that background goroutines observe shutdown signals immediately, preventing the "durable sleep" leaks common in components that previously relied on static timers.
Framework & Lifecycle Tests: This commit introduces the TestContext and TestBuilder. It includes a full-lifecycle regression test (Scale-up -> Stabilization -> Scale-down) that demonstrates the framework's ability to orchestrate complex state transitions deterministically using the encapsulated state of our fake K8s and Cloud Provider implementations.

Usage: See staticautoscaler_test.go for an example of the new RunTest and ctx.Step pattern.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

linux-foundation-easycla · 2026-01-22T13:09:37Z

The committers listed above are authorized under a signed CLA.

✅ login: GaetanoMar96 / name: Gaetano Maranzino (02e1ac6, 1c02f08, 58e27de)

k8s-ci-robot · 2026-01-22T13:09:38Z

Welcome @GaetanoMar96!

It looks like this is your first PR to kubernetes/autoscaler 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/autoscaler has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-01-22T13:09:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: GaetanoMar96
Once this PR has been reviewed and has the lgtm label, please assign towca for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

cluster-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-01-22T13:09:39Z

Hi @GaetanoMar96. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

GaetanoMar96 · 2026-01-23T10:05:13Z

/assign @kawych @mtrqq

cluster-autoscaler/builder/autoscaler.go

kawych · 2026-01-22T17:13:38Z

cluster-autoscaler/builder/autoscaler.go

+	var nodeInfoComparator nodegroupset.NodeInfoComparator
+	if len(autoscalingOptions.BalancingLabels) > 0 {
+		nodeInfoComparator = nodegroupset.CreateLabelNodeInfoComparator(autoscalingOptions.BalancingLabels)
+	} else {


Can we pull it outside, to main?

I just noticed that the core package calls the NewCloudProvider() -_-

My proposal of target state:

move that branch outside, to main

move initialization from core to main as well:

autoscaler/cluster-autoscaler/core/autoscaler.go

Line 139 in 872d31f

opts.CloudProvider = cloudBuilder.NewCloudProvider(opts, informerFactory)

in the core repo (and in the builder) just error out if CloudProvider is nil

split the cloudprovider/, the interface should be clearly separated from provider-specific stuff (including cloudprovider/builder)

Add some form of dependency enforcement, e.g. unit test or play with dependency visibility: at minimum, the core repo cannot have dependency on any of the cloud-specific providers.

I think 4 and 5 would require some larger discussion, but in this CL I think 1-3 are achievable.

Cloud provider depends on informer factory and core autoscaler options, which are initialized only later. If we move all this code in main then in test config we should handle all of them injecting them at build initialization. Are we sure we want to do this?

Right... that might be a bit unsafe, since we don't know 100% which options does the NewCloudProvider really depend on.

Happy case: we can just move it out and see if any of the tests fail. But I'm not too confident that we have sufficient test coverage to detect all issues.

Another solution would be to leave it as is. IMO in this case we should at least move this initialization out of the core package, e.g. to the builder package. And ideally add a comment in the code that a target state is initialization in main.go, but it wasn't done because we don't have ability to test it thoroughly.

I'd like to get some feedback from other CA maintainers @mtrqq @towca

And regarding the nodeInfoComparator initialization, I think it's better to leave it in main.go for now, since the code is different depending on cloud provider (also it's in the spirit of the TODO below to eventually migrate it somewhere in the CloudProvider package).

Migrated the nodeInfoComparator. I will leave the conversation unresolved for now waiting from CA maintainers feedback

cluster-autoscaler/test/integration/framework.go

cluster-autoscaler/builder/bigunittest/staticautoscaler_test.go

cluster-autoscaler/builder/bigunittest/framework.go

cluster-autoscaler/test/integration/framework.go

cluster-autoscaler/builder/bigunittest/framework.go

cluster-autoscaler/builder/bigunittest/testconfig.go

cluster-autoscaler/cloudprovider/test/fake_cloud_provider.go

cluster-autoscaler/builder/autoscaler.go

kawych · 2026-02-02T09:14:48Z

cluster-autoscaler/builder/autoscaler.go

+	var nodeInfoComparator nodegroupset.NodeInfoComparator
+	if len(autoscalingOptions.BalancingLabels) > 0 {
+		nodeInfoComparator = nodegroupset.CreateLabelNodeInfoComparator(autoscalingOptions.BalancingLabels)
+	} else {


And regarding the nodeInfoComparator initialization, I think it's better to leave it in main.go for now, since the code is different depending on cloud provider (also it's in the spirit of the TODO below to eventually migrate it somewhere in the CloudProvider package).

cluster-autoscaler/builder/clients.go

cluster-autoscaler/builder/autoscaler_test.go

cluster-autoscaler/utils/kubernetes/fake/kubernetes.go

cluster-autoscaler/test/integration/framework.go

cluster-autoscaler/cloudprovider/builder/cloud_provider_builder.go

cluster-autoscaler/builder/autoscaler.go

cluster-autoscaler/utils/fake/kubernetes.go

cluster-autoscaler/test/integration/config.go

mtrqq · 2026-02-03T13:27:42Z

cluster-autoscaler/test/integration/inmemory/staticautoscaler_test.go

+	testConfig := integration.NewConfig()
+
+	integration.RunTest(t, testConfig, func(ctx *integration.TestContext) {
+		ctx.BuildAutoscaler()


Is there any reason to keep it explicit and not executed within the RunTest? I would still keep this function public if there will be any test which needs to perform manual configuration

This is a good point and i will wait for Karol review as well. For context ideally we should make the autoscaler object explicit but then i was not able to find a proper way to make it work.

cluster-autoscaler/cloudprovider/test/fake_cloud_provider.go

k8s-ci-robot · 2026-02-04T02:03:31Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/feature Categorizes issue or PR as related to a new feature. labels Jan 22, 2026

k8s-ci-robot added do-not-merge/needs-area area/cluster-autoscaler labels Jan 22, 2026

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 22, 2026

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jan 22, 2026

k8s-ci-robot requested review from elmiko and vadasambar January 22, 2026 13:09

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed do-not-merge/needs-area cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 22, 2026

GaetanoMar96 marked this pull request as draft January 22, 2026 13:20

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 22, 2026

GaetanoMar96 force-pushed the synctest-framework branch 5 times, most recently from 35ac017 to eb540b2 Compare January 23, 2026 09:59

GaetanoMar96 marked this pull request as ready for review January 23, 2026 10:01

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2026

k8s-ci-robot requested review from aleksandra-malinowska and feiskyer January 23, 2026 10:01

k8s-ci-robot assigned kawych and mtrqq Jan 23, 2026

kawych suggested changes Jan 23, 2026

View reviewed changes

GaetanoMar96 force-pushed the synctest-framework branch from eb540b2 to dac1406 Compare January 26, 2026 14:02

GaetanoMar96 commented Jan 26, 2026

View reviewed changes

cluster-autoscaler/cloudprovider/test/fake_cloud_provider.go Show resolved Hide resolved

kawych reviewed Feb 2, 2026

View reviewed changes

GaetanoMar96 force-pushed the synctest-framework branch 2 times, most recently from 1fd965e to 5180f1d Compare February 3, 2026 12:24

GaetanoMar96 requested a review from kawych February 3, 2026 12:37

mtrqq reviewed Feb 3, 2026

View reviewed changes

GaetanoMar96 added 3 commits February 3, 2026 15:02

Refactor main to allow dependency injection

02e1ac6

Allow synctest usage passing context.

1c02f08

Add a full cycle CA test.

58e27de

GaetanoMar96 force-pushed the synctest-framework branch 3 times, most recently from 5180f1d to 58e27de Compare February 3, 2026 15:05

GaetanoMar96 requested a review from mtrqq February 3, 2026 15:29

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 4, 2026

Introduce synctest-based framework for deterministic testing #9099

Are you sure you want to change the base?

Introduce synctest-based framework for deterministic testing #9099

Uh oh!

Conversation

GaetanoMar96 commented Jan 22, 2026

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

linux-foundation-easycla bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jan 22, 2026

Uh oh!

k8s-ci-robot commented Jan 22, 2026

Uh oh!

k8s-ci-robot commented Jan 22, 2026

Uh oh!

GaetanoMar96 commented Jan 23, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

k8s-ci-robot commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

linux-foundation-easycla bot commented Jan 22, 2026 •

edited

Loading