These tests are failing because the runner migrated ownership to github,
and that somehow broke it even though it wasn't supposed to. They're not
responding to issues [other
than](https://github.com/actions/runner-images/issues/14100) "a CVE or
vulnerability" during the transition period (ends june 12). Let's just
turn these off for now. We weren't even running this at all in azure,
it's fine.
Related to https://github.com/certbot/certbot/issues/10581
Following up on #10631 and
https://github.com/certbot/certbot/pull/10622, this PR converts the
`nightly`
[pipeline](https://dev.azure.com/certbot/certbot/_build?definitionId=5)
from Azure to Github Actions.
`schedule` and `workflow_dispatch` triggers only work on merged
branches, not PRs. To see these tests running, I temporarily added a
`push` trigger in commit
[a2e9c43](https://github.com/certbot/certbot/pull/10634/commits/a2e9c4303e4a9549562aaa460f15787020f25d0d).
You can see the results of those tests
[here](https://github.com/certbot/certbot/actions/runs/25688414262).
I did not split each file into its own commit this time because I feel
like the general idea is clear. The relevant files in azure pipelines to
reference are:
- the deleted `.azure-pipelines/nightly.yml` -->
`.github/workflows/nightly.yml`
- `.azure-pipelines/templates/jobs/common-deploy-jobs.yml` -->
`.github/workflows/deploy_docker_images.yml` and
`.github/workflows/deploy_snaps.yml`
- `.azure-pipelines/templates/stages/changelog-stage.yml` -->
`.github/workflows/create_changelog.yml`
I chose to split `common-deploy-jobs` into `deploy_docker_images` and
`deploy_snaps`. This is because the docker arm32v6 build takes a long
time, but uploading to docker is quick, while the armhf snaps build
varies but is often quicker, but uploading the snaps can take some time.
By splitting them, we can specify the dependencies more precisely, and
hopefully shave some time off the total. Without the split, tests took
[53 minutes
total](https://github.com/certbot/certbot/actions/runs/25684264622).
After the split, tests took [33 minutes
total](https://github.com/certbot/certbot/actions/runs/25688414262)!
As before, the "nightly deploy stage" from azure has been omitted for
clarity.
`rerun.yml` did not exist before. There's not a great built-in way to
rerun individual jobs in github actions, which I wanted for the snap
builds specifically, since other timeouts can still happen. I could have
made an action or additional workflow and wrapped that in a script to
retry it, but I figured actually it's nicer to have the ability to rerun
anything. This is equivalent to clicking "rerun all failed jobs," which
I feel is usually what we want. Unfortunately, I am pretty sure that to
test it, the rerun script will need to be merged first, since it relies
on `workflow_dispatch`.
You can see that packages were successfully uploaded to
[dockerhub](https://hub.docker.com/r/certbot/certbot/tags) and the [snap
store](https://dashboard.snapcraft.io/stores/snaps/); looking at the
timestamps is probably the easiest way to confirm (about 11:45am
Monday).
Related to https://github.com/certbot/certbot/issues/10581
Following up on #10622, this PR converts the `full-test-suite`
[pipeline](https://dev.azure.com/certbot/certbot/_build?definitionId=4)
from Azure to Github Actions.
Nightly test changes for context not included in this PR are available
[here](https://github.com/certbot/certbot/compare/test-convert-full-pipeline...convert-all-pipelines).
Since this branch is named `test-convert-full-pipeline`, these tests
will show up in the checks section of this PR.
The major changes I made here are splitting the docker and snaps tests
for a better github actions UX, and removing the intermediate "stage"
file, since stages are not a concept in GHA. This means that we get the
nice dropdowns for the different categories on the left bar of the [test
run
page](https://github.com/certbot/certbot/actions/runs/25139155528/job/73684692548)
so it's easier to see each type of test. The very slight drawback is
that the four jobs listed in `.github/workflows/full_test_suite.yml` do
need to be duplicated in `nightly.yml`, but that's a reasonable tradeoff
to me.
Also, we now test our certbot and dns plugin snaps on all architectures
for the first time (using `dpkg --add-architecture` to run armhf tests
on an arm64 machine), which is very nice and in my opinion worth the
very slightly extra time and code.
In this PR, we build arm64 and amd64 snaps directly on github's runners.
armhf snaps are built using launchpad as before. This makes the workflow
file a little long. There are perhaps some micro-optimizations for code
deduplication I could make, like creating an action to install
dependencies based on the architecture, but I don't think it's super
worth it, especially since the dependencies vary enough that we'd still
need some code (for example, even between installing deps for certbot
and dns runs, we'd still need to additionally install `nginx-light`).
A very slight potential time improvement we could make here would be to
optionally depend on the different architectures before running their
respective tests. I'm not sure if this can be done without writing
different jobs, and since once those jobs start they run in parallel, it
didn't really seem worth looking into for me. I am of course open to
alternate points of view here and in general.
Another potential change to bring the two build strategies more in line
would be to stop using the python script to send off all the launchpad
builds, and instead put each in a separate, matrixed job like the github
jobs. We could even continue retrying the builds within each job. This
would mean that if one dns plugin build happens to fail three times, all
the builds wouldn't have to be retried. While I think that's not the
worst idea, I personally think that belongs in a separate PR, as this PR
is already quite long.
Speaking of the PR length, I can undo the changes made here to build
arm64 and amd64 snaps on github actions, to have a simpler
conversion-only PR to review. Some of the choices I made here,
particularly around UX, were based on the fact that the jobs would look
like this, so it might not be as clear why I made those choices, but if
it's easier to review it's no problem to put it back. I could also
remove the code that tests the other snaps since it's new, but I figured
it'd be nice to show that they are in fact being built correctly, since
otherwise the built snaps wouldn't be consumed anywhere.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Erica Portnoy