Trivy missing vulnerabilies in CBL-Mariner 2.0 (Azure-Linux)

[apeschel]

Question

This seems similar to the following question:

#6846 (comment)

However, the documentation for CBL-Mariner suggests that Trivy does support detecting unfixed vulnerabilies:

https://trivy.dev/v0.60/docs/coverage/os/azure/

However, cross referencing grype and trivy show different results. Please see at the bottom for an example.

This is especially problematic on images built on mariner that include additional packages - many of the Mariner packages seem to be pretty old and have vulnerabilies, but Trivy doesn't report any of them. Python 3.9.19 and Busybox 1.35.0 seem to come up in a lot of images built on Mariner, but trivy doesn't catch them.

Does anyone know what is going on here?

Base image comparison:

> grype mcr.microsoft.com/cbl-mariner/distroless/base:2.0
 ✔ Loaded image                                                                                                                                                                                              mcr.microsoft.com/cbl-mariner/distroless/base:2.0
 ✔ Parsed image                                                                                                                                                                        sha256:55183599aebe41577d655f2bc7917b6ae474f729cb7dba095e7e56b08638bc3c
 ✔ Cataloged contents                                                                                                                                                                         51f94074031c7c0408ce80952d10af899bf47175471232368b9b5754786776b8
   ├── ✔ Packages                        [14 packages]
   ├── ✔ File digests                    [290 files]
   ├── ✔ File metadata                   [290 locations]
   └── ✔ Executables                     [292 executables]
 ✔ Scanned for vulnerabilities     [26 vulnerability matches]
   ├── by severity: 5 critical, 13 high, 8 medium, 0 low, 0 negligible
   └── by status:   26 fixed, 0 not-fixed, 0 ignored
NAME     INSTALLED  FIXED-IN         TYPE  VULNERABILITY   SEVERITY
glibc    2.35       0:2.35-1.cm2     rpm   CVE-2022-23218  Critical
glibc    2.35       0:2.35-1.cm2     rpm   CVE-2022-23219  Critical
glibc    2.35       0:2.35-1.cm2     rpm   CVE-2021-38604  High
glibc    2.35       0:2.35-7.cm2     rpm   CVE-2021-3998   High
glibc    2.35       0:2.35-1.cm2     rpm   CVE-2021-43396  High
glibc    2.35       0:2.35-5.cm2     rpm   CVE-2023-4911   High
glibc    2.35       0:2.35-6.cm2     rpm   CVE-2023-5156   High
glibc    2.35       0:2.35-7.cm2     rpm   CVE-2024-33599  High
glibc    2.35       0:2.35-7.cm2     rpm   CVE-2024-33600  High
glibc    2.35       0:2.35-7.cm2     rpm   CVE-2024-33601  High
glibc    2.35       0:2.35-7.cm2     rpm   CVE-2024-33602  High
glibc    2.35       0:2.35-6.cm2     rpm   CVE-2023-4806   Medium
openssl  1.1.1k     0:1.1.1k-11.cm2  rpm   CVE-2021-3711   Critical
openssl  1.1.1k     0:1.1.1k-15.cm2  rpm   CVE-2022-1292   Critical
openssl  1.1.1k     0:1.1.1k-17.cm2  rpm   CVE-2022-2068   Critical
openssl  1.1.1k     0:1.1.1k-11.cm2  rpm   CVE-2021-3712   High
openssl  1.1.1k     0:1.1.1k-12.cm2  rpm   CVE-2022-0778   High
openssl  1.1.1k     0:1.1.1k-21.cm2  rpm   CVE-2023-0286   High
openssl  1.1.1k     0:1.1.1k-31.cm2  rpm   CVE-2024-4741   High
openssl  1.1.1k     0:1.1.1k-13.cm2  rpm   CVE-2021-4160   Medium
openssl  1.1.1k     0:1.1.1k-20.cm2  rpm   CVE-2022-2097   Medium
openssl  1.1.1k     0:1.1.1k-23.cm2  rpm   CVE-2023-0465   Medium
openssl  1.1.1k     0:1.1.1k-23.cm2  rpm   CVE-2023-0466   Medium
openssl  1.1.1k     0:1.1.1k-25.cm2  rpm   CVE-2023-2650   Medium
openssl  1.1.1k     0:1.1.1k-28.cm2  rpm   CVE-2023-3446   Medium
openssl  1.1.1k     0:1.1.1k-26.cm2  rpm   CVE-2023-3817   Medium

vs trivy

> trivy image --scanners vuln mcr.microsoft.com/cbl-mariner/distroless/base:2.0
2025-03-26T13:59:56-07:00       INFO    [vuln] Vulnerability scanning is enabled
2025-03-26T13:59:57-07:00       INFO    Detected OS     family="cbl-mariner" version="2.0"
2025-03-26T13:59:57-07:00       INFO    [cbl-mariner] Detecting vulnerabilities...      os_version="2.0" pkg_num=12
2025-03-26T13:59:57-07:00       INFO    Number of language-specific files       num=0

Report Summary

┌─────────────────────────────────────────────────────────────────────┬─────────────┬─────────────────┐
│                               Target                                │    Type     │ Vulnerabilities │
├─────────────────────────────────────────────────────────────────────┼─────────────┼─────────────────┤
│ mcr.microsoft.com/cbl-mariner/distroless/base:2.0 (cbl-mariner 2.0) │ cbl-mariner │        0        │
└─────────────────────────────────────────────────────────────────────┴─────────────┴─────────────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

Target

Container Image

Scanner

Vulnerability

Output Format

Table

Mode

Standalone

Operating System

macOS Sequoia

Version

Version: 0.60.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2025-03-26 12:18:52.268822125 +0000 UTC
  NextUpdate: 2025-03-27 12:18:52.268821925 +0000 UTC
  DownloadedAt: 2025-03-26 17:26:30.968002 +0000 UTC
Check Bundle:
  Digest: sha256:ef2d9ad4fce0f933b20a662004d7e55bf200987c180e7f2cd531af631f408bb3
  DownloadedAt: 2024-08-16 18:47:48.269025 +0000 UTC

[apeschel]

I'm not fully clear here, but it may also be that the grype is having false positives, and fixes were back-ported to Mariner:

microsoft/azurelinux#1329

[apeschel]

It looks like there's a bit of both going on.

grype doesn't seem to do a good job of filtering out vulnerabilities that have had patches backported, leading to a bunch of false positives.

However, trivy seems to miss vulnerabilities as well. I have an image with busybox-1.35.0-12.cm2.x86_64 installed. This version of busybox definitely has vulnerabilities present, but trivy doesn't catch them.

There are fixes for these vulnerabilies in busybox-1.35.0-13.cm2.x86_64 - is trivy maybe not correctly identifying which package is installed?

[DmitriyLewen]

Hello @apeschel
Thanks for your interest to Trivy!

glibc 2.35 0:2.35-1.cm2 rpm CVE-2022-23218 Critical
glibc 2.35 0:2.35-1.cm2 rpm CVE-2022-23219 Critical
glibc 2.35 0:2.35-1.cm2 rpm CVE-2021-38604 High
glibc 2.35 0:2.35-7.cm2 rpm CVE-2021-3998 High
glibc 2.35 0:2.35-1.cm2 rpm CVE-2021-43396 High
glibc 2.35 0:2.35-5.cm2 rpm CVE-2023-4911 High
glibc 2.35 0:2.35-6.cm2 rpm CVE-2023-5156 High
glibc 2.35 0:2.35-7.cm2 rpm CVE-2024-33599 High
glibc 2.35 0:2.35-7.cm2 rpm CVE-2024-33600 High
glibc 2.35 0:2.35-7.cm2 rpm CVE-2024-33601 High
glibc 2.35 0:2.35-7.cm2 rpm CVE-2024-33602 High
glibc 2.35 0:2.35-6.cm2 rpm CVE-2023-4806 Medium

It looks like grype doesn't detect release part of version (or doesn't use it).
Trivy detects pkg:rpm/cbl-mariner/[email protected] for this image.
As you can see, this version doesn't contain these vulns.

I have an image with busybox-1.35.0-12.cm2.x86_64 installed. This version of busybox definitely has vulnerabilities present, but trivy doesn't catch them.

Hm.. Trivy should detect CVE-2022-48174 for this case.
You can check all found packages using -f json --list-all-pkgs flag.
Can you check that Trivy correctly detects version for busybox in your image?

[apeschel]

Here's an example image: mcr.microsoft.com/azure-cognitive-services/translator/text-translation:1.0.02927.655-amd64@sha256:636b115a9d53240e2be656408228e335cf9baed59e579bc7767e508f1464f341

It doesn't seem like trivy is finding the busybox package - not that the Azure-Linux team makes it easy.

The busybox package install isn't listed in /var/lib/rpmmanifest/container-manifest-* - which seems to be where trivy is getting most of the package information from.

The only place I can find in the filesystem that shows the installed busybox package version is in /lib/sysimage/tdnf/history.db, which shows the correct version of busybox-1.35.0-12.cm2.x86_64.

Based on looking at the file system layers with dive, it seems like the busybox package was installed with this step:

COPY /staging2/ / # buildkit

It seems like they have a multi-stage build, and I don't think the Dockerfile is available for inspection, so I can't see what they're doing exactly.

I checked the packages that are listed in /var/lib/rpmmanifest/container-manifest-*, and none of them seem to list busybox as a dependency. I'm not sure what is going on here - any thoughts?

[DmitriyLewen]

hard to figure out, don't have dockerfile of the image
but most likely you are right.

most packages are installed via rpm
but busybox was not installed, its binary file was just copied
Trivy supports only Go and Rust binaries - https://trivy.dev/latest/docs/coverage/language/#supported-languages
Trivy Premium supports more binaries - https://trivy.dev/latest/commercial/compare/