Skip to content

Add v2 pipeline to collect Ubuntu OSV advisories#2145

Open
keshav-space wants to merge 9 commits intomainfrom
ubuntu-v2
Open

Add v2 pipeline to collect Ubuntu OSV advisories#2145
keshav-space wants to merge 9 commits intomainfrom
ubuntu-v2

Conversation

@keshav-space
Copy link
Member

@keshav-space keshav-space commented Feb 4, 2026 •
edited
Loading

Fixes #2035 #2037

Pipeline execution log:

INFO 2026-02-03 15:51:10.317865 UTC Pipeline [UbuntuOSVImporterPipeline] starting
INFO 2026-02-03 15:51:10.318089 UTC Step [clone] starting
INFO 2026-02-03 15:51:10.318352 UTC Cloning `git+https://github.com/canonical/ubuntu-security-notices/`
INFO 2026-02-03 15:51:10.318545 UTC Step [clone] completed in 0 seconds
INFO 2026-02-03 15:51:10.318619 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-03 15:51:10.499071 UTC Collecting 46,339 advisories
INFO 2026-02-03 15:51:17.409898 UTC Progress: 1% (464/46339) ETA: 684 seconds (11.4 minutes)
INFO 2026-02-03 15:51:24.896234 UTC Progress: 2% (927/46339) ETA: 705 seconds (11.8 minutes)
INFO 2026-02-03 15:51:30.231741 UTC Progress: 3% (1391/46339) ETA: 638 seconds (10.6 minutes)
INFO 2026-02-03 15:51:36.001616 UTC Progress: 4% (1854/46339) ETA: 612 seconds (10.2 minutes)
INFO 2026-02-03 15:51:44.025643 UTC Progress: 5% (2317/46339) ETA: 637 seconds (10.6 minutes)
INFO 2026-02-03 15:51:54.134193 UTC Progress: 6% (2781/46339) ETA: 684 seconds (11.4 minutes)
INFO 2026-02-03 15:54:34.900498 UTC Progress: 7% (3244/46339) ETA: 2716 seconds (45.3 minutes)
INFO 2026-02-03 15:59:24.001256 UTC Progress: 8% (3708/46339) ETA: 5675 seconds (1.6 hours)
INFO 2026-02-03 16:04:11.273787 UTC Progress: 9% (4171/46339) ETA: 7894 seconds (2.2 hours)
INFO 2026-02-03 16:09:06.876277 UTC Progress: 10% (4634/46339) ETA: 9687 seconds (2.7 hours)
INFO 2026-02-03 16:13:50.818697 UTC Progress: 11% (5098/46339) ETA: 11006 seconds (3.1 hours)
INFO 2026-02-03 16:18:34.336616 UTC Progress: 12% (5561/46339) ETA: 12055 seconds (3.3 hours)
INFO 2026-02-03 16:23:12.947528 UTC Progress: 13% (6025/46339) ETA: 12866 seconds (3.6 hours)
INFO 2026-02-03 16:28:18.592989 UTC Progress: 14% (6488/46339) ETA: 13687 seconds (3.8 hours)
INFO 2026-02-03 16:33:33.065014 UTC Progress: 15% (6951/46339) ETA: 14408 seconds (4.0 hours)
INFO 2026-02-03 16:38:59.013301 UTC Progress: 16% (7415/46339) ETA: 15060 seconds (4.2 hours)
INFO 2026-02-03 16:44:12.505972 UTC Progress: 17% (7878/46339) ETA: 15536 seconds (4.3 hours)
INFO 2026-02-03 16:49:16.499352 UTC Progress: 18% (8342/46339) ETA: 15881 seconds (4.4 hours)
INFO 2026-02-03 16:52:41.812273 UTC Progress: 19% (8805/46339) ETA: 15737 seconds (4.4 hours)
INFO 2026-02-03 16:52:53.709613 UTC Progress: 20% (9268/46339) ETA: 14813 seconds (4.1 hours)
INFO 2026-02-03 16:53:09.063942 UTC Progress: 21% (9732/46339) ETA: 13989 seconds (3.9 hours)
INFO 2026-02-03 16:53:22.994311 UTC Progress: 22% (10195/46339) ETA: 13233 seconds (3.7 hours)
INFO 2026-02-03 16:53:40.116180 UTC Progress: 23% (10658/46339) ETA: 12553 seconds (3.5 hours)
INFO 2026-02-03 16:53:51.453995 UTC Progress: 24% (11122/46339) ETA: 11910 seconds (3.3 hours)
INFO 2026-02-03 16:54:12.319782 UTC Progress: 25% (11585/46339) ETA: 11345 seconds (3.2 hours)
INFO 2026-02-03 16:54:23.894274 UTC Progress: 26% (12049/46339) ETA: 10797 seconds (3.0 hours)
INFO 2026-02-03 16:54:41.632001 UTC Progress: 27% (12512/46339) ETA: 10304 seconds (2.9 hours)
INFO 2026-02-03 16:56:23.112247 UTC Progress: 28% (12975/46339) ETA: 10061 seconds (2.8 hours)
INFO 2026-02-03 16:57:59.846773 UTC Progress: 29% (13439/46339) ETA: 9816 seconds (2.7 hours)
INFO 2026-02-03 16:59:50.640471 UTC Progress: 30% (13902/46339) ETA: 9614 seconds (2.7 hours)
INFO 2026-02-03 17:01:34.073620 UTC Progress: 31% (14366/46339) ETA: 9401 seconds (2.6 hours)
INFO 2026-02-03 17:03:11.396320 UTC Progress: 32% (14829/46339) ETA: 9182 seconds (2.6 hours)
INFO 2026-02-03 17:05:24.952163 UTC Progress: 33% (15292/46339) ETA: 9044 seconds (2.5 hours)
INFO 2026-02-03 17:07:21.335567 UTC Progress: 34% (15756/46339) ETA: 8873 seconds (2.5 hours)
INFO 2026-02-03 17:09:07.363767 UTC Progress: 35% (16219/46339) ETA: 8686 seconds (2.4 hours)
INFO 2026-02-03 17:10:57.108999 UTC Progress: 36% (16683/46339) ETA: 8510 seconds (2.4 hours)
INFO 2026-02-03 17:11:06.863338 UTC Progress: 37% (17146/46339) ETA: 8167 seconds (2.3 hours)
INFO 2026-02-03 17:11:20.014969 UTC Progress: 38% (17609/46339) ETA: 7847 seconds (2.2 hours)
INFO 2026-02-03 17:11:30.387522 UTC Progress: 39% (18073/46339) ETA: 7539 seconds (2.1 hours)
INFO 2026-02-03 17:11:40.111271 UTC Progress: 40% (18536/46339) ETA: 7244 seconds (2.0 hours)
INFO 2026-02-03 17:11:54.772378 UTC Progress: 41% (18999/46339) ETA: 6971 seconds (1.9 hours)
INFO 2026-02-03 17:12:09.368122 UTC Progress: 42% (19463/46339) ETA: 6710 seconds (1.9 hours)
INFO 2026-02-03 17:12:17.220569 UTC Progress: 43% (19926/46339) ETA: 6451 seconds (1.8 hours)
INFO 2026-02-03 17:12:28.657453 UTC Progress: 44% (20390/46339) ETA: 6209 seconds (1.7 hours)
INFO 2026-02-03 17:19:51.169309 UTC Progress: 45% (20853/46339) ETA: 6503 seconds (1.8 hours)
INFO 2026-02-03 17:28:28.648404 UTC Progress: 46% (21316/46339) ETA: 6853 seconds (1.9 hours)
INFO 2026-02-03 17:36:31.272378 UTC Progress: 47% (21780/46339) ETA: 7128 seconds (2.0 hours)
INFO 2026-02-03 17:45:39.942258 UTC Progress: 48% (22243/46339) ETA: 7442 seconds (2.1 hours)
INFO 2026-02-03 17:54:10.381275 UTC Progress: 49% (22707/46339) ETA: 7681 seconds (2.1 hours)
INFO 2026-02-03 18:02:28.323134 UTC Progress: 50% (23170/46339) ETA: 7878 seconds (2.2 hours)
INFO 2026-02-03 18:10:28.111802 UTC Progress: 51% (23633/46339) ETA: 8030 seconds (2.2 hours)
INFO 2026-02-03 18:19:03.430928 UTC Progress: 52% (24097/46339) ETA: 8190 seconds (2.3 hours)
INFO 2026-02-03 18:26:41.562089 UTC Progress: 53% (24560/46339) ETA: 8275 seconds (2.3 hours)
INFO 2026-02-03 18:34:38.951487 UTC Progress: 54% (25024/46339) ETA: 8355 seconds (2.3 hours)
INFO 2026-02-03 18:42:16.264634 UTC Progress: 55% (25487/46339) ETA: 8399 seconds (2.3 hours)
INFO 2026-02-03 18:44:44.893079 UTC Progress: 56% (25950/46339) ETA: 8183 seconds (2.3 hours)
INFO 2026-02-03 18:45:12.987950 UTC Progress: 57% (26414/46339) ETA: 7878 seconds (2.2 hours)
INFO 2026-02-03 18:45:49.130561 UTC Progress: 58% (26877/46339) ETA: 7588 seconds (2.1 hours)
INFO 2026-02-03 18:46:25.361055 UTC Progress: 59% (27341/46339) ETA: 7307 seconds (2.0 hours)
INFO 2026-02-03 18:47:01.453947 UTC Progress: 60% (27804/46339) ETA: 7034 seconds (2.0 hours)
INFO 2026-02-03 18:47:27.908889 UTC Progress: 61% (28267/46339) ETA: 6763 seconds (1.9 hours)
INFO 2026-02-03 18:47:43.493060 UTC Progress: 62% (28731/46339) ETA: 6492 seconds (1.8 hours)
INFO 2026-02-03 18:47:55.432841 UTC Progress: 63% (29194/46339) ETA: 6228 seconds (1.7 hours)
INFO 2026-02-03 18:48:05.178982 UTC Progress: 64% (29657/46339) ETA: 5971 seconds (1.7 hours)
INFO 2026-02-03 18:48:15.750727 UTC Progress: 65% (30121/46339) ETA: 5721 seconds (1.6 hours)
INFO 2026-02-03 18:48:25.105322 UTC Progress: 66% (30584/46339) ETA: 5478 seconds (1.5 hours)
INFO 2026-02-03 18:49:17.411374 UTC Progress: 67% (31048/46339) ETA: 5264 seconds (1.5 hours)
INFO 2026-02-03 18:56:14.801560 UTC Progress: 68% (31511/46339) ETA: 5226 seconds (1.5 hours)
INFO 2026-02-03 19:04:13.839970 UTC Progress: 69% (31974/46339) ETA: 5204 seconds (1.4 hours)
INFO 2026-02-03 19:11:57.163147 UTC Progress: 70% (32438/46339) ETA: 5163 seconds (1.4 hours)
INFO 2026-02-03 19:19:41.975918 UTC Progress: 71% (32901/46339) ETA: 5110 seconds (1.4 hours)
INFO 2026-02-03 19:27:42.947061 UTC Progress: 72% (33365/46339) ETA: 5053 seconds (1.4 hours)
INFO 2026-02-03 19:35:13.255080 UTC Progress: 73% (33828/46339) ETA: 4972 seconds (1.4 hours)
INFO 2026-02-03 19:42:46.330886 UTC Progress: 74% (34291/46339) ETA: 4882 seconds (1.4 hours)
INFO 2026-02-03 19:50:18.621715 UTC Progress: 75% (34755/46339) ETA: 4783 seconds (1.3 hours)
INFO 2026-02-03 19:57:59.163571 UTC Progress: 76% (35218/46339) ETA: 4676 seconds (1.3 hours)
INFO 2026-02-03 20:05:58.505502 UTC Progress: 77% (35682/46339) ETA: 4567 seconds (1.3 hours)
INFO 2026-02-03 20:13:03.988063 UTC Progress: 78% (36145/46339) ETA: 4432 seconds (1.2 hours)
INFO 2026-02-03 20:21:29.923376 UTC Progress: 79% (36608/46339) ETA: 4311 seconds (1.2 hours)
INFO 2026-02-03 20:22:25.421577 UTC Progress: 80% (37072/46339) ETA: 4069 seconds (1.1 hours)
INFO 2026-02-03 20:28:15.856784 UTC Progress: 81% (37535/46339) ETA: 3900 seconds (1.1 hours)
INFO 2026-02-03 20:33:16.896340 UTC Progress: 82% (37998/46339) ETA: 3716 seconds (1.0 hours)
INFO 2026-02-03 20:38:38.609591 UTC Progress: 83% (38462/46339) ETA: 3533 seconds (58.9 minutes)
INFO 2026-02-03 20:44:08.685322 UTC Progress: 84% (38925/46339) ETA: 3348 seconds (55.8 minutes)
INFO 2026-02-03 20:49:31.896038 UTC Progress: 85% (39389/46339) ETA: 3159 seconds (52.6 minutes)
INFO 2026-02-03 20:54:54.061744 UTC Progress: 86% (39852/46339) ETA: 2967 seconds (49.5 minutes)
INFO 2026-02-03 21:00:08.925419 UTC Progress: 87% (40315/46339) ETA: 2770 seconds (46.2 minutes)
INFO 2026-02-03 21:05:43.614337 UTC Progress: 88% (40779/46339) ETA: 2574 seconds (42.9 minutes)
INFO 2026-02-03 21:11:16.798720 UTC Progress: 89% (41242/46339) ETA: 2374 seconds (39.6 minutes)
INFO 2026-02-03 21:16:42.724199 UTC Progress: 90% (41706/46339) ETA: 2170 seconds (36.2 minutes)
INFO 2026-02-03 21:16:51.863016 UTC Progress: 91% (42169/46339) ETA: 1933 seconds (32.2 minutes)
INFO 2026-02-03 21:16:59.045858 UTC Progress: 92% (42632/46339) ETA: 1700 seconds (28.3 minutes)
INFO 2026-02-03 21:17:06.462531 UTC Progress: 93% (43096/46339) ETA: 1472 seconds (24.5 minutes)
INFO 2026-02-03 21:17:40.801670 UTC Progress: 94% (43559/46339) ETA: 1250 seconds (20.8 minutes)
INFO 2026-02-03 21:18:17.203543 UTC Progress: 95% (44023/46339) ETA: 1033 seconds (17.2 minutes)
INFO 2026-02-03 21:18:48.873214 UTC Progress: 96% (44486/46339) ETA: 819 seconds (13.7 minutes)
INFO 2026-02-03 21:19:30.348559 UTC Progress: 97% (44949/46339) ETA: 609 seconds (10.2 minutes)
INFO 2026-02-03 21:20:16.971579 UTC Progress: 98% (45413/46339) ETA: 403 seconds (6.7 minutes)
INFO 2026-02-03 21:20:58.802185 UTC Progress: 99% (45876/46339) ETA: 200 seconds (3.3 minutes)
INFO 2026-02-03 21:21:30.354808 UTC Progress: 100% (46339/46339)
INFO 2026-02-03 21:21:30.432792 UTC Successfully collected 46,339 advisories
INFO 2026-02-03 21:21:30.432971 UTC Step [collect_and_store_advisories] completed in 19820 seconds (5.5 hours)
INFO 2026-02-03 21:21:30.433002 UTC Step [clean_downloads] starting
INFO 2026-02-03 21:21:30.433023 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-02-03 21:21:30.433042 UTC Pipeline completed in 19820 seconds (5.5 hours)

@keshav-space
Support Ubuntu priority scoring system
b8e47c2 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Parse and collect all supported scoring systems from osv
3810461 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Do not include version in AffectedPackage base PURL
e074872 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Support Ubuntu ecosystem in osv parser
c9f52de 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Create packages in bulk during AdvisoryV2 insertion
94e9da4 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Make pipeline progress step configurable
385088d 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add v2 pipeline to collect Ubuntu OSV advisories
2b0d198 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Test v2 ubuntu importer pipeline
47f732d 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space keshav-space linked an issue Feb 4, 2026 that may be closed by this pull request
VCIO-API-V2: Migrate Ubuntu USN importer to advisory V2 #2037
Open
@keshav-space
Collect upstream aliases in osv advisory
a057c98 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

VCIO-API-V2: Migrate Ubuntu USN importer to advisory V2 VCIO-API-V2: Migrate Ubuntu importer to advisory V2

1 participant

@keshav-space