Skip to content

Add support to accept incomplete release files for S3 artifact upload#315

Open
NotTheEvilOne wants to merge 1 commit intomainfrom
feature/s3-artifact-support-with-unsupported-release-files
Open

Add support to accept incomplete release files for S3 artifact upload#315
NotTheEvilOne wants to merge 1 commit intomainfrom
feature/s3-artifact-support-with-unsupported-release-files

Conversation

@NotTheEvilOne
Copy link
Contributor

What this PR does / why we need it:
This PR adds support to accept incomplete release files for S3 artifact upload. It falls back to the features directory to parse for remaining metadata.

@NotTheEvilOne NotTheEvilOne self-assigned this Feb 12, 2026
@NotTheEvilOne NotTheEvilOne force-pushed the feature/s3-artifact-support-with-unsupported-release-files branch 3 times, most recently from f7af29a to 70a33ba Compare February 12, 2026 18:39
@NotTheEvilOne NotTheEvilOne requested a review from a team February 12, 2026 18:41
@NotTheEvilOne NotTheEvilOne added the enhancement New feature or request label Feb 12, 2026
@NotTheEvilOne NotTheEvilOne added this to the 2026-02 milestone Feb 12, 2026
@codecov
Copy link

codecov bot commented Feb 12, 2026

Codecov Report

❌ Patch coverage is 88.88889% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.01%. Comparing base (2d52501) to head (c3bd51b).

Files with missing lines Patch % Lines
src/gardenlinux/features/cname.py 88.88% 2 Missing ⚠️
src/gardenlinux/s3/s3_artifacts.py 87.50% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #315      +/-   ##
==========================================
+ Coverage   91.69%   92.01%   +0.31%     
==========================================
  Files          42       42              
  Lines        2132     2141       +9     
==========================================
+ Hits         1955     1970      +15     
+ Misses        177      171       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nkraetzschmar
Copy link
Contributor

This will once again cause the same problems as in gardenlinux/gardenlinux#3824.

i.e. when we publish from main with the current feature graph of main checked out, then this will wrongly parse the cname the way it would be interpreted under that graph. Not the way it was actually build.

Proof

The following was run with gl-s3 from this branch and ~/git/gardenlinux/gardenlinux checked out at main:

GL_ROOT_DIR=~/git/gardenlinux/gardenlinux gl-s3 --path /tmp/tmp.DDbtNkig6f --bucket abcxyz --dry-run upload-artifacts-to-bucket --artifact-name openstack-gardener_prod-amd64-1877.10-3d56ba62
platform: openstack
architecture: amd64
base_image: null
build_committish: 3d56ba627f7f471e64f36608ffbc84a441d309d9
build_timestamp: 2026-02-13 13:18:35.984429
logs: null
modifiers:
- log
- sap
- ssh
- _fwcfg
- _legacy
- _nopkg
- _prod
- _slim
- base
- server
- cloud
- multipath
- iscsi
- nvme
- gardener
- openstackCloud
- openstack
require_uefi: false
secureboot: false
published_image_metadata: null
s3_bucket: abcxyz
s3_key: meta/singles/openstack-gardener_prod-amd64-1877.10-3d56ba62
test_result: null
version: '1877.10'
paths:
[...]

Yet

cat /tmp/tmp.DDbtNkig6f/openstack-gardener_prod-amd64-1877.10-3d56ba62.release
ID=gardenlinux
ID_LIKE=debian
NAME="Garden Linux"
PRETTY_NAME="Garden Linux 1877.10"
IMAGE_VERSION=1877.10
VARIANT_ID=openstack-gardener_prod-amd64
HOME_URL="https://gardenlinux.io"
SUPPORT_URL="https://github.com/gardenlinux/gardenlinux"
BUG_REPORT_URL="https://github.com/gardenlinux/gardenlinux/issues"
GARDENLINUX_CNAME=openstack-gardener_prod-amd64-1877.10
GARDENLINUX_PLATFORM=openstack
GARDENLINUX_FEATURES=log,sap,ssh,_legacy,_nopkg,_prod,_slim,base,server,cloud,openstack,multipath,iscsi,nvme,gardener
GARDENLINUX_VERSION=1877.10
GARDENLINUX_COMMIT_ID=3d56ba62
GARDENLINUX_COMMIT_ID_LONG=3d56ba627f7f471e64f36608ffbc84a441d309d9

So the features set that was guessed and included in the singles meta data does NOT match what's actually in the image.

Also in this case there is no need to fall back to any features directory parsing since this release file has everything it needs. As per the proposed spec in https://github.com/gardenlinux/gardenlinux/pull/4268/changes#diff-68c407bf61f5bd289c500f7e5d74da2cad07997626126f020d5eda350cd30bc9R51-R61 it contains all fields except those explicitly marked "informational only, not used in publishing". So it should NOT require a features directory at all.

So instead of querying a features directory and incorrectly guessing the features from a cname that may have been build from a completely unrelated feature graph, we should just drop the following requirements when loading from a released file:

"GARDENLINUX_FEATURES_ELEMENTS",
"GARDENLINUX_FEATURES_FLAGS",
"GARDENLINUX_FEATURES_PLATFORMS",

And instead utilize the GARDENLINUX_PLATFORM field where applicable. Only if the absolutely required fields are missing from the release file should any fallback behaviour be used. And even then using a features directory is always dangerous, so the fallback should probably be to guess the platform as simply the first part of the cname, and validate that this platform is at least part of the GARDENLINUX_FEATURES_PLATFORMS iff that key exists (to support even older release file formats where no fix is inplace to set the GARDENLINUX_PLATFORM key yet).

Copy link
Contributor

@nkraetzschmar nkraetzschmar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment above

@NotTheEvilOne
Copy link
Contributor Author

The assumption to use main is not intended for this change. For metadata generation based on a specific features directory the correct commit needs to be checked out as well. This is only a fallback solution in case the release file is not usable under certain circumstances. The idea is to use the same features tree in this case than what the source of the release file (builder at the moment) used when creating the original one.

@NotTheEvilOne NotTheEvilOne force-pushed the feature/s3-artifact-support-with-unsupported-release-files branch from 70a33ba to 406cf43 Compare February 13, 2026 13:41
@NotTheEvilOne
Copy link
Contributor Author

NotTheEvilOne commented Feb 13, 2026

@nkraetzschmar: With the correct commit checked out you will receive:

platform: openstack
architecture: amd64
base_image: null
build_committish: 3d56ba627f7f471e64f36608ffbc84a441d309d9
build_timestamp: 2026-02-13 14:43:07.907084
logs: null
modifiers:
- log
- sap
- ssh
- _boot
- _nopkg
- _prod
- _slim
- base
- server
- cloud
- openstack
- gardener
require_uefi: false
secureboot: false
published_image_metadata: null
s3_bucket: abcxyz
s3_key: meta/singles/openstack-gardener_prod-amd64-1877.10-3d56ba62
test_result: null
version: '1877.10'
paths:
- name: openstack-gardener_prod-amd64-1877.10-3d56ba62.release
  s3_bucket_name: abcxyz
  s3_key: objects/openstack-gardener_prod-amd64-1877.10-3d56ba62/openstack-gardener_prod-amd64-1877.10-3d56ba62.release
  suffix: .release
  md5sum: 6f1e0f851e2fe23861ebc24ddaeae6f3
  sha256sum: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
gardenlinux_epoch: 1877

Furthermore this PR is able to handle the current state of 1592 (without an GARDENLINUX_PLATFORM in os_release) and provides the new one platform policy:

platform: frankenstein
architecture: amd64
base_image: null
build_committish: 3d56ba627f7f471e64f36608ffbc84a441d309d9
build_timestamp: 2026-02-13 14:51:24.220904
logs: null
modifiers:
- log
- sap
- ssh
- _boot
- _nopkg
- _prod
- _slim
- base
- server
- metal
- openstackbaremetal
- gardener
require_uefi: false
secureboot: false
published_image_metadata: null
s3_bucket: abcxyz
s3_key: meta/singles/openstackbaremetal-gardener_prod-amd64-1592.17-3d56ba62
test_result: null
version: '1592.17'
paths:
- name: openstackbaremetal-gardener_prod-amd64-1592.17-3d56ba62.release
  s3_bucket_name: abcxyz
  s3_key: objects/openstackbaremetal-gardener_prod-amd64-1592.17-3d56ba62/openstackbaremetal-gardener_prod-amd64-1592.17-3d56ba62.release
  suffix: .release
  md5sum: 8202dfdbaac51defc6e0a32590dc82f6
  sha256sum: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
gardenlinux_epoch: 1592

Note: release files are actually simulated for these tests.

@NotTheEvilOne NotTheEvilOne force-pushed the feature/s3-artifact-support-with-unsupported-release-files branch from 406cf43 to c3bd51b Compare February 13, 2026 13:56
@nkraetzschmar
Copy link
Contributor

Furthermore this PR is able to handle the current state of 1592 (without an GARDENLINUX_PLATFORM in os_release) and provides the new one platform policy:

For the last release of 1592 the release file looks as follows:

ID=gardenlinux
NAME="Garden Linux"
PRETTY_NAME="Garden Linux 1592.16"
HOME_URL="https://gardenlinux.io"
SUPPORT_URL="https://github.com/gardenlinux/gardenlinux"
BUG_REPORT_URL="https://github.com/gardenlinux/gardenlinux/issues"
GARDENLINUX_CNAME=openstack-gardener_prod-amd64-1592.16
GARDENLINUX_PLATFORM=openstack
GARDENLINUX_FEATURES=log,sap,ssh,_boot,_nopkg,_prod,_slim,base,server,cloud,openstack,gardener
GARDENLINUX_VERSION=1592.16
GARDENLINUX_COMMIT_ID=c5ed8155
GARDENLINUX_COMMIT_ID_LONG=c5ed81554b5f60a0d9a1cca34dd8fe96ded044d6

So I'm not sure what you mean by this.

@nkraetzschmar
Copy link
Contributor

With the correct commit checked out you will receive: [...]

For openstack, yeah, but for openstackbaremetal it either fails with

assert len(platforms) < 2

Or with GL_ALLOW_FRANKENSTEIN=1 it produces:

platform: frankenstein
architecture: amd64
base_image: null
build_committish: 3d56ba627f7f471e64f36608ffbc84a441d309d9
build_timestamp: 2026-02-13 16:24:25.883786
logs: null
modifiers:
- log
- sap
- ssh
- _legacy
- _nopkg
- _prod
- _slim
- base
- server
- metal
- openstackbaremetal
- multipath
- iscsi
- nvme
- gardener
require_uefi: false
secureboot: false
published_image_metadata: null
s3_bucket: abcxyz
s3_key: meta/singles/openstackbaremetal-gardener_prod-amd64-1877.10-3d56ba62
test_result: null
version: '1877.10'
[...]

Which is technically correct, yes, but since we do NOT want to backport gardenlinux/gardenlinux#4253 into previous releases as this would change cnames mid release, this output is undesired here.

That's why treating GARDENLINUX_PLATFORM from the release file should always be the preferred option as this allows the release branch to overload this value as a workaround to achieve compatibility (gardenlinux/gardenlinux@ca36ed1).

That way can can simultaneously have the nice new handling of there should only be one platform on main, while release branches continue to work and only require adjustments in the release branches themselves, not any compatibility hacks outside of them.

@nkraetzschmar
Copy link
Contributor

Please see #316 for the intended input output characteristics that the S3 upload should have and adjust this PR to match that behaviour just properly integrated in the CName classes etc.

Signed-off-by: Tobias Wolf <wolf@b1-systems.de>
On-behalf-of: SAP <tobias.wolf@sap.com>
@NotTheEvilOne NotTheEvilOne force-pushed the feature/s3-artifact-support-with-unsupported-release-files branch from c3bd51b to 453883f Compare February 13, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants