Migrate nightly test pipeline from azure pipelines to github actions (#10634)

Related to https://github.com/certbot/certbot/issues/10581

Following up on #10631 and
https://github.com/certbot/certbot/pull/10622, this PR converts the
`nightly`
[pipeline](https://dev.azure.com/certbot/certbot/_build?definitionId=5)
from Azure to Github Actions.

`schedule` and `workflow_dispatch` triggers only work on merged
branches, not PRs. To see these tests running, I temporarily added a
`push` trigger in commit
[a2e9c43](https://github.com/certbot/certbot/pull/10634/commits/a2e9c4303e4a9549562aaa460f15787020f25d0d).
You can see the results of those tests
[here](https://github.com/certbot/certbot/actions/runs/25688414262).

I did not split each file into its own commit this time because I feel
like the general idea is clear. The relevant files in azure pipelines to
reference are:
- the deleted `.azure-pipelines/nightly.yml` -->
`.github/workflows/nightly.yml`
- `.azure-pipelines/templates/jobs/common-deploy-jobs.yml` -->
`.github/workflows/deploy_docker_images.yml` and
`.github/workflows/deploy_snaps.yml`
- `.azure-pipelines/templates/stages/changelog-stage.yml` -->
`.github/workflows/create_changelog.yml`

I chose to split `common-deploy-jobs` into `deploy_docker_images` and
`deploy_snaps`. This is because the docker arm32v6 build takes a long
time, but uploading to docker is quick, while the armhf snaps build
varies but is often quicker, but uploading the snaps can take some time.
By splitting them, we can specify the dependencies more precisely, and
hopefully shave some time off the total. Without the split, tests took
[53 minutes
total](https://github.com/certbot/certbot/actions/runs/25684264622).
After the split, tests took [33 minutes
total](https://github.com/certbot/certbot/actions/runs/25688414262)!

As before, the "nightly deploy stage" from azure has been omitted for
clarity.

`rerun.yml` did not exist before. There's not a great built-in way to
rerun individual jobs in github actions, which I wanted for the snap
builds specifically, since other timeouts can still happen. I could have
made an action or additional workflow and wrapped that in a script to
retry it, but I figured actually it's nicer to have the ability to rerun
anything. This is equivalent to clicking "rerun all failed jobs," which
I feel is usually what we want. Unfortunately, I am pretty sure that to
test it, the rerun script will need to be merged first, since it relies
on `workflow_dispatch`.

You can see that packages were successfully uploaded to
[dockerhub](https://hub.docker.com/r/certbot/certbot/tags) and the [snap
store](https://dashboard.snapcraft.io/stores/snaps/); looking at the
timestamps is probably the easiest way to confirm (about 11:45am
Monday).

.github/workflows/create_changelog.yml

@@ -0,0 +1,27 @@
+name: Create changelog
+on:
+  workflow_call:
+permissions:
+  contents: read
+
+jobs:
+  prepare:
+    name: Changelog
+    runs-on: ubuntu-latest
+    steps:
+    # If we change the output filename from `release_notes.md`, it should also be changed in tools/create_github_release.py
+    - name: checkout
+      uses: actions/checkout@v6.0.2
+      with:
+        persist-credentials: false
+    - name: Prepare changelog
+      run: |-
+        CERTBOT_VERSION="$(cd certbot/src && python -c "import certbot; print(certbot.__version__)" && cd ~-)"
+        tools/extract_changelog.py "${CERTBOT_VERSION}" >> "${{ runner.temp }}/release_notes.md"
+      shell: bash
+    - name: Publish changelog
+      uses: actions/upload-artifact@v7.0.0
+      with:
+        # If we change the artifact's name, it should also be changed in tools/create_github_release.py
+        name: changelog
+        path: "${{ runner.temp }}/release_notes.md"

.github/workflows/deploy_docker_images.yml

@@ -0,0 +1,81 @@
+name: Deploy docker images
+on:
+  workflow_call:
+    inputs:
+      dockerTag:
+        required: true
+        description: 'tag to assign docker images'
+        type: string
+    secrets:
+      DOCKERHUB_TOKEN:
+        required: true
+
+permissions:
+  contents: read
+
+env:
+  DOCKER_TAG: "${{ inputs.dockerTag }}"
+
+jobs:
+  # The credentials used in the following jobs are for the shared
+  # certbotbot account on Docker Hub.
+  # They are located under the certbot organization settings,
+  # under Secrets and Variables -> Actions.
+  # DOCKERHUB_USERNAME is saved as a variable.
+  # DOCKERHUB_TOKEN is a secret, and it is a PAT created by
+  # following the instructions at
+  # https://docs.docker.com/security/access-tokens/
+  # with Read and Write permissions. The access token can be deleted
+  # on Docker Hub if these credentials need to be revoked.
+  # The password is a PAT following the advice given by
+  # https://github.com/docker/login-action?tab=readme-ov-file#docker-hub
+  publish_docker_by_arch:
+    name: Publish docker by arch
+    runs-on:
+      - 'ubuntu-24.04'
+    strategy:
+      fail-fast: false
+      matrix:
+        DOCKER_ARCH:
+        - arm32v6
+        - arm64v8
+        - amd64
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v6.0.2
+      with:
+        persist-credentials: false
+    - name: Retrieve Docker images
+      uses: actions/download-artifact@v8.0.1
+      with:
+        name: docker_${{ matrix.DOCKER_ARCH }}
+        path: "${{ github.workspace }}"
+    - name: Load Docker images
+      run: docker load --input ${{ github.workspace }}/images.tar
+      shell: bash
+    - name: Login to Docker Hub
+      uses: docker/login-action@v4.1.0
+      with:
+        username: "${{ vars.DOCKERHUB_USERNAME }}"
+        password: "${{ secrets.DOCKERHUB_TOKEN }}"
+    - name: Deploy the Docker images by architecture
+      run: tools/docker/deploy_images.sh "$DOCKER_TAG" ${{ matrix.DOCKER_ARCH }}
+      shell: bash
+  publish_docker_multiarch:
+    name: Publish docker multiarch
+    needs: publish_docker_by_arch
+    runs-on:
+      - 'ubuntu-24.04'
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v6.0.2
+      with:
+        persist-credentials: false
+    - name: Login to Docker Hub
+      uses: docker/login-action@v4.1.0
+      with:
+        username: "${{ vars.DOCKERHUB_USERNAME }}"
+        password: "${{ secrets.DOCKERHUB_TOKEN }}"
+    - name: Deploy the Docker multiarch manifests
+      run: tools/docker/deploy_manifests.sh "$DOCKER_TAG" all
+      shell: bash

.github/workflows/deploy_snaps.yml

@@ -0,0 +1,86 @@
+name: Deploy snaps
+on:
+  workflow_call:
+    inputs:
+      snapReleaseChannel:
+        description: 'snap channel to release to'
+        required: true
+        type: string
+    secrets:
+      SNAPCRAFTCFG:
+        required: true
+
+permissions:
+  contents: read
+
+env:
+  SNAP_RELEASE_CHANNEL: "${{ inputs.snapReleaseChannel }}"
+
+jobs:
+  # This job relies on credentials used to publish the Certbot snaps. This
+  # credential file was created by running:
+  #
+  #   snapcraft logout
+  #   snapcraft export-login --channels=beta,edge snapcraft.cfg
+  #     (provide the shared snapcraft credentials when prompted)
+  #
+  # Then the contents of the file were added as a secret in Github
+  # with the name SNAPCRAFTCFG under the Secrets and Variables -> Actions
+  # section of the settings for the certbot organization.
+  #
+  # Revoking these credentials can be done by changing the password of the
+  # account used to generate the credentials. See
+  # https://forum.snapcraft.io/t/revoking-exported-credentials/19031 for more
+  # info.
+  publish_snap:
+    name: Publish snap
+    if: ${{ inputs.snapReleaseChannel == 'edge' || inputs.snapReleaseChannel == 'beta' }}
+    runs-on:
+      - 'ubuntu-24.04'
+    strategy:
+      fail-fast: false
+      matrix:
+        SNAP_ARCH: [amd64, armhf, arm64]
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v6.0.2
+      with:
+        persist-credentials: false
+    - name: Install dependencies
+      run: |-
+        sudo apt-get update
+        sudo apt-get install -y --no-install-recommends snapd
+        sudo snap install --classic snapcraft
+      shell: bash
+    - name: Retrieve Certbot snaps
+      if: ${{ matrix.SNAP_ARCH == 'armhf' }}
+      uses: actions/download-artifact@v8.0.1
+      with:
+        name: snaps_${{ matrix.SNAP_ARCH }}
+        path: "${{ github.workspace }}/snap"
+    - name: Retrieve Certbot snaps
+      if: ${{ matrix.SNAP_ARCH != 'armhf' }}
+      uses: actions/download-artifact@v8.0.1
+      with:
+        pattern: snap-*-${{ matrix.SNAP_ARCH }}
+        merge-multiple: true
+        path: "${{ github.workspace }}/snap"
+    - name: Display structure of downloaded files
+      run: ls -R "${{ github.workspace }}/snap"
+    - name: Publish to Snap store
+      run: |-
+        export SNAPCRAFT_STORE_CREDENTIALS="${{ secrets.SNAPCRAFTCFG }}"
+        for SNAP_FILE in snap/*.snap; do
+          tools/retry.sh eval snapcraft upload --release="${SNAP_RELEASE_CHANNEL}" "${SNAP_FILE}"
+        done
+      shell: bash
+  publish_snap_invalid:
+    # Fail instead of silently skipping snap release
+    name: Fail on invalid snapReleaseChannel
+    if: ${{ inputs.snapReleaseChannel != 'edge' && inputs.snapReleaseChannel != 'beta' }}
+    runs-on:
+      - 'ubuntu-latest'
+    steps:
+    - name: Fail
+      run: exit 1
+      shell: bash

.github/workflows/nightly.yml

@@ -0,0 +1,120 @@
+# Nightly pipeline running each day for main.
+name: Nightly build
+on:
+  schedule:
+  - cron: 30 4 * * *
+  workflow_dispatch:
+
+jobs:
+  # While many of these jobs could be grouped in a separate workflow, the github actions UI
+  # is much nicer if they are instead listed explicitly here.
+  ###########################
+  ####    testing jobs    ###
+  ###########################
+  standard_tests_jobs:
+    name: Standard tests
+    permissions:
+      contents: read
+    uses: "./.github/workflows/standard_tests_jobs.yml"
+  extended_tests_jobs:
+    name: Extended tests
+    permissions:
+      contents: read
+    uses: "./.github/workflows/extended_tests_jobs.yml"
+    secrets:
+      AWS_TEST_FARM_PEM: "${{ secrets.AWS_TEST_FARM_PEM }}"
+      AWS_ACCESS_KEY_ID: "${{ secrets.AWS_ACCESS_KEY_ID }}"
+      AWS_SECRET_ACCESS_KEY: "${{ secrets.AWS_SECRET_ACCESS_KEY }}"
+  ###########################
+  ####   packaging jobs   ###
+  ###########################
+  docker_packaging_jobs:
+    name: Docker packaging
+    permissions:
+      contents: read
+    uses: "./.github/workflows/docker_packaging_jobs.yml"
+    with:
+      dockerTag: nightly
+  snap_packaging_jobs:
+    name: Snap packaging
+    permissions:
+      contents: read
+    uses: "./.github/workflows/snap_packaging_jobs.yml"
+    with:
+      snapBuildTimeout: 19800
+    secrets:
+      LAUNCHPAD_CREDENTIALS: "${{ secrets.LAUNCHPAD_CREDENTIALS }}"
+  create_changelog:
+    name: Create changelog
+    permissions:
+      contents: read
+    uses: "./.github/workflows/create_changelog.yml"
+  ############################
+  ####     deploy jobs     ###
+  ############################
+  docker_deploy_jobs:
+    name: Deploy docker images
+    permissions:
+      contents: read
+    needs:
+      - standard_tests_jobs
+      - extended_tests_jobs
+      - docker_packaging_jobs
+    uses: "./.github/workflows/deploy_docker_images.yml"
+    secrets:
+      DOCKERHUB_TOKEN: "${{ secrets.DOCKERHUB_TOKEN }}"
+    with:
+      dockerTag: nightly
+  snap_deploy_jobs:
+    name: Deploy snaps
+    permissions:
+      contents: read
+    needs:
+      - standard_tests_jobs
+      - extended_tests_jobs
+      - snap_packaging_jobs
+    uses: "./.github/workflows/deploy_snaps.yml"
+    secrets:
+      SNAPCRAFTCFG: "${{ secrets.SNAPCRAFTCFG }}"
+    with:
+      snapReleaseChannel: edge
+  ############################
+  ####      rerun job      ###
+  ############################
+  re-run:
+    name: Re-run
+    needs:
+      - standard_tests_jobs
+      - extended_tests_jobs
+      - docker_packaging_jobs
+      - snap_packaging_jobs
+      - create_changelog
+      - docker_deploy_jobs
+      - snap_deploy_jobs
+    if: failure() && fromJSON(github.run_attempt) < 3
+    permissions:
+      actions: write
+      checks: write
+    runs-on: ubuntu-latest
+    steps:
+      - env:
+          GH_REPO: ${{ github.repository }}
+          GH_TOKEN: ${{ github.token }}
+          GH_DEBUG: api
+        run: gh workflow run rerun.yml -F run_id=${{ github.run_id }}
+        shell: bash
+  ###########################
+  ####       notify       ###
+  ###########################
+  notify:
+    name: Notify
+    needs:
+      - re-run
+    # Returns true when any previous step of a job fails. If you have a chain of dependent
+    # jobs, failure() returns true if any ancestor job fails.
+    if: failure() && (needs.re-run.result == 'skipped' || needs.re-run.result == 'failure')
+    uses: "./.github/workflows/notify_nightly.yml"
+    permissions:
+      actions: read
+    secrets:
+      MATTERMOST_PUBLIC_CERTBOT_CHANNEL_WEBHOOK: "${{ secrets.MATTERMOST_PUBLIC_CERTBOT_CHANNEL_WEBHOOK }}"

.github/workflows/notify_nightly.yml

@@ -0,0 +1,59 @@
+name: Notify nightly failure
+
+on:
+  workflow_call:
+    secrets:
+      MATTERMOST_PUBLIC_CERTBOT_CHANNEL_WEBHOOK:
+        required: true
+
+permissions:
+  actions: read
+
+jobs:
+  notify_mattermost:
+    name: Notify mattermost
+    runs-on: ubuntu-latest
+    steps:
+    - name: Calculate duration
+      shell: bash
+      id: duration
+      env:
+        GH_TOKEN: ${{ github.token }}
+        GH_REPO: ${{ github.repository }}
+      run: |-
+        START=$(gh run view "$GITHUB_RUN_ID" --json startedAt --jq ".startedAt")
+        START_SECONDS=$(date -d"$START" +%s)
+        NOW_SECONDS=$(date +%s)
+        DURATION=$(date -d@"$((NOW_SECONDS - START_SECONDS))"  -u +%H:%M:%S)
+        echo "result = $DURATION"
+        echo "result=$DURATION" >> "$GITHUB_OUTPUT"
+    - name: Send to mattermost
+      uses: mattermost/action-mattermost-notify@b7d118e440bf2749cd18a4a8c88e7092e696257a
+      with:
+        MATTERMOST_WEBHOOK_URL: ${{ secrets.MATTERMOST_PUBLIC_CERTBOT_CHANNEL_WEBHOOK }}
+        PAYLOAD: |-
+          {
+            "attachments": [
+              {
+                "color": "danger",
+                "fields": [
+                  null,
+                  {
+                    "title": "Duration",
+                    "value": "${{ steps.duration.outputs.result }}",
+                    "short": true
+                  },
+                  {
+                    "title": "Workflow",
+                    "value": "nightly",
+                    "short": true
+                  }
+                ],
+                "pretext": "Run <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|${{ github.run_id }}> failed",
+                "mrkdwn_in": [
+                  "pretext"
+                ],
+                "fallback": "Run ${{ github.run_id }} failed https://github.com/certbot/actions/runs/${{ github.run_id }}"
+              }
+            ]
+          }

.github/workflows/rerun.yml

@@ -0,0 +1,27 @@
+# based on https://github.com/orgs/community/discussions/67654#discussioncomment-8038649
+name: Rerun
+on:
+  workflow_dispatch:
+    inputs:
+      run_id:
+        required: true
+        description: github.run_id of workflow to rerun
+permissions:
+  actions: write
+  checks: write
+
+env:
+  RUN_ID: "${{ inputs.run_id }}"
+
+jobs:
+  rerun:
+    runs-on: ubuntu-latest
+    steps:
+      - name: rerun "$RUN_ID"
+        env:
+          GH_REPO: ${{ github.repository }}
+          GH_TOKEN: ${{ github.token }}
+          GH_DEBUG: api
+        run: |
+          gh run watch "$RUN_ID" > /dev/null 2>&1
+          gh run rerun "$RUN_ID" --failed

.github/workflows/snap_packaging_jobs.yml

@@ -104,7 +104,7 @@ jobs:
         sudo apt-get update
         sudo apt-get install -y --no-install-recommends snapd
         sudo snap install --classic snapcraft
-    - uses: actions/setup-python@v5.0.0
+    - uses: actions/setup-python@v6.2.0
       with:
         python-version: '3.12'
     - name: Build snaps
@@ -147,7 +147,7 @@ jobs:
       uses: actions/checkout@v6.0.2
       with:
         persist-credentials: false
-    - uses: actions/setup-python@v5.0.0
+    - uses: actions/setup-python@v6.2.0
       with:
         python-version: '3.12'
     - name: Install armhf dependencies
@@ -221,7 +221,7 @@ jobs:
       run: |-
         sudo apt-get update
         sudo apt-get install -y --no-install-recommends snapd
-    - uses: actions/setup-python@v5.0.0
+    - uses: actions/setup-python@v6.2.0
       with:
         python-version: '3.12'
     - name: Retrieve Certbot snaps armhf