Verify cosign signatures of distroless base images

[dekkagaijin]

This'll ensure that the distroless base images were built on trusted infrastructure

$ docker run -it gcr.io/projectsigstore/cosign/ci/cosign@sha256:c581a4f0f6dd158220fa05d2351d8015f969b68de5c61d52c41478fae84e7064 -- verify -key https://raw.githubusercontent.com/GoogleContainerTools/distroless/main/cosign.pub gcr.io/distroless/base:latest
Unable to find image 'gcr.io/projectsigstore/cosign/ci/cosign@sha256:c581a4f0f6dd158220fa05d2351d8015f969b68de5c61d52c41478fae84e7064' locally
gcr.io/projectsigstore/cosign/ci/cosign@sha256:c581a4f0f6dd158220fa05d2351d8015f969b68de5c61d52c41478fae84e7064: Pulling from projectsigstore/cosign/ci/cosign
5dea5ec2316d: Already exists
bb771d6dc9a1: Already exists
9127c3610b7e: Already exists
72164b581b02: Already exists
6fe218878cac: Pull complete
Digest: sha256:c581a4f0f6dd158220fa05d2351d8015f969b68de5c61d52c41478fae84e7064
Status: Downloaded newer image for gcr.io/projectsigstore/cosign/ci/cosign@sha256:c581a4f0f6dd158220fa05d2351d8015f969b68de5c61d52c41478fae84e7064

Verification for gcr.io/distroless/base:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":""},"image":{"docker-manifest-digest":"sha256:6ec6da1888b18dd971802c2a58a76a7702902b4c9c1be28f38e75e871cedc2df"},"type":"cosign container signature"},"optional":null},{"critical":{"identity":{"docker-reference":""},"image":{"docker-manifest-digest":"sha256:6ec6da1888b18dd971802c2a58a76a7702902b4c9c1be28f38e75e871cedc2df"},"type":"cosign container signature"},"optional":null},{"critical":{"identity":{"docker-reference":""},"image":{"docker-manifest-digest":"sha256:6ec6da1888b18dd971802c2a58a76a7702902b4c9c1be28f38e75e871cedc2df"},"type":"cosign container signature"},"optional":null},{"critical":{"identity":{"docker-reference":""},"image":{"docker-manifest-digest":"sha256:6ec6da1888b18dd971802c2a58a76a7702902b4c9c1be28f38e75e871cedc2df"},"type":"cosign container signature"},"optional":null},{"critical":{"identity":{"docker-reference":""},"image":{"docker-manifest-digest":"sha256:6ec6da1888b18dd971802c2a58a76a7702902b4c9c1be28f38e75e871cedc2df"},"type":"cosign container signature"},"optional":null}]

/kind feature

NONE

[dekkagaijin]

/cc @justaugustus @dlorenc

[dlorenc]

This is my favorite PR.

[dlorenc]

/lgtm

[inferno-chromium]

Do we need to pin the public key url, right now tip-of-tree/main/master

[justaugustus]

Weeeeeeeeeeee!! 🎉
...but also:
/hold for discussion

cc: @kubernetes/release-engineering

[dekkagaijin]

Do we need to pin the public key url, right now tip-of-tree/main/master

That'd be one option, otherwise vendoring the public key in this repo would be a good option

[cpanato]

I said to Dan some time ago that we might want to start to publish the tagged image releases in another bucket that does not contain the ci on it to avoid confusion for the users.
But I know we all are working to make the cosign image a bit better before doing that. 😃

[dlorenc]

Ah, it's published in both places actually! Good catch. The releases are at gcr.io/projectsigstore/cosign:v0.3.1 as well.

[dekkagaijin]

done

[cpanato]

/priority important-soon

[saschagrunert]

Do we wanna pin this to an actual version/sha? If the main branch changes then the build will silently fail.

[dlorenc]

@priyawadhwa set this up intentionally - but she's out this week so I'll try to document the rationale in that repo.

The key rotation strategy for distroless is that we'll commit a new public key into the repo, then the images are built and signed with the corresponding private key. Since K8s takes distroless only from HEAD, the only appropriate key to check against is latest.

The branch name one is a good point though - is there a good way to access whatever the default branch is?

[dekkagaijin]

done :)

[cppforlife]

if i'm reading this right, verification runs against image with :latest tag and the later another process resolves :latest again. even though this is unlikely, there is a time gap where :latest could change to something different/malicious. are we not able to resolve image ref to a digest, verify it and use it in the following steps? since we are pretty early in the signing journey, it would be good to set a precedent for using exact assets that were verified as part of a build.

[dlorenc]

if i'm reading this right, verification runs against image with :latest tag and the later another process resolves :latest again. even though this is unlikely, there is a time gap where :latest could change to something different/malicious. are we not able to resolve image ref to a digest, verify it and use it in the following steps? since we are pretty early in the signing journey, it would be good to set a precedent for using exact assets that were verified as part of a build.

Yeah, that would be better IMO. See here for how Istio handles this (for distroless): https://github.com/istio/istio/blob/master/bin/update_deps.sh#L50

I can think of a few approaches:

• Do a docker pull before the build, and then get the pulled digest out of the daemon. Use that to check a signature, and leave ":latest" in the Dockerfile, relying on the daemon to do the right thing during build (and not pull again)
• Resolve, check the signature, and then template the tag into the Dockerfile itself. I made a POC here that does all of this except for the rewriting of the Dockerfile: WIP: Hypotenuse - a tool for extracting the base images out of a Dockerfile. sigstore/cosign#189
• Probably more!

[jonjohnsonjr]

Can we throw something like:

crane digest gcr.io/distroless/$_DISTROLESS_IMAGE:latest

In there to resolve the digest?

cc @imjasonh another use case for --full-ref flag or something.

I made a POC here that does all of this except for the rewriting of the Dockerfile: sigstore/cosign#189

This seems very similar to what we'd want to do, if not exactly what we'd want to do.

[dekkagaijin]

/test pull-release-image-go-runner

[dekkagaijin]

/test pull-release-image-go-runner

[cpanato]

/lgtm

[cpanato]

nit non blocking: maybe a new line

[dekkagaijin]

done

[cpanato]

maybe: gcr.io/projectsigstore/cosign:v0.3.1@sha256:c581a4f0f6dd158220fa05d2351d8015f969b68de5c61d52c41478fae84e7064

[dekkagaijin]

done :)

[puerco]

It's wonderful to see this is place!! Thanks @dekkagaijin!

/lgtm

[justaugustus]

One more time for paranoia's sake:
/test all

[dlorenc]

This is awesome!

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cpanato, dekkagaijin, saschagrunert

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justaugustus]

I did a git oops when pushing updates to @dekkagaijin's branch, which accidentally closed this.
Re-pushed with attribution here: #2016

[dekkagaijin]

Et tu, Stephen?

[justaugustus]

Et tu, Stephen?

Hehe, sometimes you get git and sometimes git gets you!

[justaugustus]

#2016 has merged.
Thank you again, @dekkagaijin!