This PR automates the release process steps to:
- update the candidate branch with the contents of the release branch on
the repo in the subfolder
- create a PR merging the candidate branch into main
- create a new branch without the version bump. usually `1.2.x`, unless
it's a point release.
- if it's a point release, create a PR merging the new branch without
version bumps into the existing `1.2.x` branch.
This draws from steps 10, 12, and 13. Step 10 should still have the code
to push to pypi, since that's a different though, though I think that
could move here in the future as well.
My general design philosophy was "error out instead of letting the
script put git into a bad state," with the exception of PR creation
which seemed safe to skip and continue.
I've also added some flags, mostly to make testing this easier, but
could be useful for re-running the script as well. Unlike
`promote_snaps` and `generate_community_forum_post`,
`synchronize_github_repo` is *not* idempotent. I do not think it should
be, because of how git works. I think if branches already exist and the
user really did want to synchronize branches again, the user would want
to know that it can't be done automatically, and should instead be told
what to do to make it possible, or how to skip the whole thing. I don't
think we should, for example, go ahead and create a PR based on an old
version of a branch and just skip the pulling step, or automatically
delete a branch.
In `_create_and_push_branch_without_version_bump`, if you have created
the branch then fail after, you'll rerun and then get a message saying
to delete the branch. I think that's nicer than automatically deleting
it, in case you want to inspect it.
successful test run:
```
$ git switch create-pr
Switched to branch 'create-pr'
$ RELEASE_GPG_KEY=[test key] tools/release.sh 4.35.0 4.36.0
[release output]
$ tools/finish_release.py --test-version 4.35.0 --skip-snaps
Creating PR to merge candidate-4.35.0 into main...
PR location: https://github.com/certbot/certbot/pull/10714
Creating branch without version bump commit named 4.35.x...
Created.
Generating announcement text for community forum post
release not found
Generating announcement text failed.
$ git switch 4.35.x
Switched to branch '4.35.x'
$ RELEASE_GPG_KEY=[test key] tools/release.sh 4.35.1 4.36.0
[release output]
$ tools/finish_release.py --test-version 4.35.1 --skip-snaps
Creating PR to merge candidate-4.35.1 into main...
PR location: https://github.com/certbot/certbot/pull/10715
Creating branch without version bump commit named point-candidate-4.35.1...
Created.
Creating PR to merge point-candidate-4.35.1 into 4.35.x...
PR location: https://github.com/certbot/certbot/pull/10716
Generating announcement text for community forum post
release not found
Generating announcement text failed.
```
then here's some errors and their outputs --
trying to run `finish_release.py` again:
```
$ tools/finish_release.py --test-version 4.34.1 --skip-snaps
Creating PR to merge candidate-4.35.1 into main...
PR to merge release changes into main already exists...skipping creation. To create a new PR, delete the old one on GitHub.
PR location: https://github.com/certbot/certbot/pull/10715
Creating branch without version bump commit named point-candidate-4.35.1...
Error running `git branch point-candidate-4.35.1`
Branch point-candidate-4.35.1 already exists. Delete it using `git branch -D point-candidate-4.35.1`.
fatal: a branch named 'point-candidate-4.35.1' already exists
To skip pushing updated branches to GitHub and creating PRs, run this script with the `--skip-github-sync` flag.
Traceback (most recent call last):
File "/Users/erica/certbot/tools/finish_release.py", line 370, in <module>
main(sys.argv[1:])
~~~~^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 366, in main
synchronize_github_repo(version)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 329, in synchronize_github_repo
_create_and_push_branch_without_version_bump(version, branch_name)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 280, in _create_and_push_branch_without_version_bump
_run_silent_except_error(f'git branch {branch_name}'.split(), msg)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 209, in _run_silent_except_error
raise e
File "/Users/erica/certbot/tools/finish_release.py", line 201, in _run_silent_except_error
process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
File "/Users/erica/.pyenv/versions/3.14.3/lib/python3.14/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'branch', 'point-candidate-4.35.1']' returned non-zero exit status 128.
```
local changes to branch:
```
$ touch test_file.txt
$ git add -A
$ tools/finish_release.py --test-version 4.35.1 --skip-snaps
Error running `git diff --quiet HEAD`
You have uncommitted changes that will be deleted. Stash your changes before rerunning this script.
To skip pushing updated branches to GitHub and creating PRs, run this script with the `--skip-github-sync` flag.
Traceback (most recent call last):
File "/Users/erica/certbot/tools/finish_release.py", line 370, in <module>
main(sys.argv[1:])
~~~~^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 366, in main
synchronize_github_repo(version)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 315, in synchronize_github_repo
_run_silent_except_error('git diff --quiet HEAD'.split(), message)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 209, in _run_silent_except_error
raise e
File "/Users/erica/certbot/tools/finish_release.py", line 201, in _run_silent_except_error
process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
File "/Users/erica/.pyenv/versions/3.14.3/lib/python3.14/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'diff', '--quiet', 'HEAD']' returned non-zero exit status 1.
```
branch doesn't match the one on github (shows that stdout is now also
printed on error):
```
$ cd releases/le.4.35.1.89372/
$ git commit --amend # change the message
[candidate-4.35.1 8d34a67a4] Bump version to 4.36.0 new message
Date: Tue Jun 23 10:47:56 2026 -0700
20 files changed, 20 insertions(+), 20 deletions(-)
$ cd ../../
$ tools/finish_release.py --test-version 4.35.1 --skip-snaps
Error running `git push origin candidate-4.35.1`
To delete the branch on GitHub, run `git push origin --delete candidate-4.35.1`.
To https://github.com/certbot/certbot.git
! [rejected] candidate-4.35.1 -> candidate-4.35.1 (non-fast-forward)
error: failed to push some refs to 'https://github.com/certbot/certbot.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. If you want to integrate the remote changes,
hint: use 'git pull' before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
To skip pushing updated branches to GitHub and creating PRs, run this script with the `--skip-github-sync` flag.
Traceback (most recent call last):
File "/Users/erica/certbot/tools/finish_release.py", line 370, in <module>
main(sys.argv[1:])
~~~~^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 366, in main
synchronize_github_repo(version)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 317, in synchronize_github_repo
_sync_candidate_from_temp_to_origin(version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 247, in _sync_candidate_from_temp_to_origin
_run_silent_except_error(command_str.split(), message)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/erica/certbot/tools/finish_release.py", line 209, in _run_silent_except_error
raise e
File "/Users/erica/certbot/tools/finish_release.py", line 201, in _run_silent_except_error
process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
File "/Users/erica/.pyenv/versions/3.14.3/lib/python3.14/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'push', 'origin', 'candidate-4.35.1']' returned non-zero exit status 1.
```
I've hit basically all of the errors with text at some point during
testing, but can recreate them if you'd like.
Once this is merged, I'll update the release instructions and delete the
test branches and PRs.
---------
Co-authored-by: Will Greenberg <ifnspifn@gmail.com>
Co-authored-by: Will Greenberg <willg@eff.org>
I don't understand why this wasn't already in the release script, and
can see no reason to not add it. This doesn't do anything publicly or
permanently, it just fetches the branch from the release folder. If we
need to add the previous temp, it's right there in a `releases.*.bak`
folder.
This is step 8 of the release process, and should be deleted when this
is merged.
I did test this and it worked.
Fixes https://github.com/certbot/certbot/issues/10662
Launchpad is failing, so this should make testing this code pretty easy.
There is only one log per target, no matter how many arches, so we can
skip the per-arch code.
---------
Co-authored-by: Will Greenberg <willg@eff.org>
Item 1 of https://github.com/certbot/certbot/issues/10600
Once this is merged, the [release
instructions](https://github.com/EFForg/certbot-misc/wiki/The-Mystical-Release-Process)
should be updated to no longer say "Make sure Certbot's virtual
environment isn't activated."
Why in `release.sh` instead of `_release.sh`? This seemed to be the
"check the environment status" file.
`venv/bin/activate` does several things.
1. create `deactivate` shell function
2. create `_OLD_VIRTUAL_PATH` and modify `PATH` to prepend `venv/bin`
location. `_OLD_VIRTUAL_PATH` isn't exported.
3. unset `PYTHONHOME` and store the old `PYTHONHOME` in
`_OLD_VIRTUAL_PYTHONHOME`, again not exported.
4. export `VIRTUAL_ENV_PROMPT`
5. call `hash -r 2> /dev/null` for some sort of edge case
6. set `VIRTUAL_ENV`
7. change the prompt appearance (PS1)
1, 4, and 7 don't need to be undone. 2, 4, and 6 are managed here
manually.
3 is the hard one, since we don't have access to
`_OLD_VIRTUAL_PYTHONHOME`, and there's not a great way of grabbing it
from the shell. One thought I had was to modify `venv/bin/activate` in
`venv.py` so that it is exported. That's possible, but at least for me,
`PYTHONHOME` isn't set in the first place and so it doesn't seem worth
doing that. Given that this script only needs to run on a few people's
machines, I would say we can hold off on doing that if and until it
becomes necessary.
Added some prints and an `exit 0` after the relevant code in the script
to test:
```bash
$ RELEASE_GPG_KEY=dontmatter tools/release.sh 1.2.3 4.5.6
$ source venv/bin/activate
(venv) $ RELEASE_GPG_KEY=dontmatter tools/release.sh 1.2.3 4.5.6
Deactivating venv...
previous path:
/Users/erica/certbot/venv/bin:/opt/homebrew/opt/coreutils/libexec/gnubin:[rest of path omitted]
new path:
/opt/homebrew/opt/coreutils/libexec/gnubin:[rest of path omitted]
(venv) $ printenv PATH
/Users/erica/certbot/venv/bin:/opt/homebrew/opt/coreutils/libexec/gnubin:[rest of path omitted]
```
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
There's no way anyone is using these...
https://github.com/certbot/certbot/pull/9762 was just basing off of
readthedocs' defaults.https://github.com/certbot/certbot/pull/9762 was
just basing off of readthedocs' defaults.
Probably no one is using pdf either, but definitely no one is using
epubs. This is the more cautious version of
https://github.com/certbot/certbot/pull/10670
```
$ git grep epub
acme/docs/Makefile:.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
acme/docs/Makefile: @echo " epub to make an epub"
acme/docs/Makefile:epub:
acme/docs/Makefile: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
acme/docs/Makefile: @echo "Build finished. The epub file is in $(BUILDDIR)/epub."
acme/docs/make.bat: echo. epub to make an epub
acme/docs/make.bat:if "%1" == "epub" (
acme/docs/make.bat: %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
acme/docs/make.bat: echo.Build finished. The epub file is in %BUILDDIR%/epub.
certbot/docs/Makefile:.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
certbot/docs/Makefile: @echo " epub to make an epub"
certbot/docs/Makefile:epub:
certbot/docs/Makefile: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
certbot/docs/Makefile: @echo "Build finished. The epub file is in $(BUILDDIR)/epub."
certbot/docs/make.bat: echo. epub to make an epub
certbot/docs/make.bat:if "%1" == "epub" (
certbot/docs/make.bat: %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
certbot/docs/make.bat: echo.Build finished. The epub file is in %BUILDDIR%/epub.
```
Related to https://github.com/certbot/certbot/issues/10581
Following up on https://github.com/certbot/certbot/pull/10631,
https://github.com/certbot/certbot/pull/10622, and
https://github.com/certbot/certbot/pull/10634, this PR converts the
release
[pipeline](https://dev.azure.com/certbot/certbot/_build?definitionId=3)
from Azure to Github Actions.
While this is the last migration PR, I don't think we should close the
issue, as we're still using launchpad for armhf builds. I plan to
continue investigating that and at minimum write up my findings.
To test
[notifications](https://opensource.eff.org/eff-open-source/pl/nfgh6obi8tfqikn4ydp7dshakr)
and creating a [github
release](https://github.com/ohemorange/Things-that-are-gr9/releases/tag/v1.0.19),
I ran workflows that no-oped most the other jobs [in a test
repo](https://github.com/ohemorange/Things-that-are-gr9/actions/runs/25938082721)
(I've made minor changes to names and comments since then, but no code
changes). Everything else is basically the same as nightly, with
different tags. For the docker deployment, `${{ github.ref_name }}` is
the tag name, so `v1.2.3`.
Why not parametrize the tests a bit more, by putting an `env` at the top
with `dockerTag: ${{ github.ref_name }}` and `snapReleaseChannel: beta`?
Because the `env` context is [not
available](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts#context-availability)
to `with`; only `github, needs, strategy, matrix, inputs, vars`. We
could use `vars`, by creating `docker_tag_release` or whatever in the
[variable
section](https://github.com/certbot/certbot/settings/variables/actions)
of the repo settings by using the web interface, but that seems worse to
me than having it in the file but twice.
You will note that the contents of `release.yml` are very similar to
`nightly.yml`. While it would be nice to factor that out and reuse the
code, github actions would then flatten everything in the grouped code
together, making the results much harder to check. You can see what that
flattening would look like
[here](https://github.com/ohemorange/Things-that-are-gr9/actions/runs/25941524972)
(if we put them all in one workflow). Currently, it will look something
like
[this](https://github.com/certbot/certbot/actions/runs/25688414262),
which is much more readable.
We could split it into "stages" like we had in azure pipelines (probably
1. standard and extended tests (and changelog?), 2. snap package and
deploy (depend on tests), 3. docker package and deploy (depend on
tests), 4. github release (depend on snap package and deploy), 5-6.
notify (depend on github release)), but in addition to a minor slowdown
(currently github release only depends on snap and docker package, not
deploy, just so we're trying to do all the deployments simultaneously
and not partially in case of build failures), it would still be only
like three fewer jobs, since we'd still want all the info passing and
dependency relationships.
While it would be nice from a UX perspective to group the two
notification jobs together, you can't do that cleanly using the built-in
`if` key, and I don't think it's worth switching to a messier github
api-based version just to group them.
For mattermost notifications, we currently get the person to tag by
running `AUTHOR_NAME="$(git log -1 --pretty=format:'%an')"` and mapping
that to mattermost username. We could instead use `github.actor` to get
the github handle, and map that instead. I didn't bother since we
already have working, tested code, but can if a reviewer thinks it's
worth it.
---------
Co-authored-by: Will Greenberg <willg@eff.org>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Item 3 of https://github.com/certbot/certbot/issues/10600
If this is merged, the [release
instructions](https://github.com/EFForg/certbot-misc/wiki/The-Mystical-Release-Process)
should be updated to no longer manually create the branch.
I intentionally do not add error checking to `git switch -c` because I
think if the command fails, the script should fail, and we should
manually fix the error, which will probably be something like `fatal: a
branch named 'candidate-5.5.0' already exists` which is clear enough.
I do check if we're already on the intended branch, because we may want
to be able to rerun the release script without having to switch back to
main and delete the candidate branch each time.
I remove the check later on because I believe it is a holdover from a
previous version where it was possible for `RELEASE_BRANCH` to be not
equal to `candidate-$version`, but even in the existing code before the
other change, that should not be possible.
Item 5 of https://github.com/certbot/certbot/issues/10600
When this is merged, the setup section of the [release
process](https://github.com/EFForg/certbot-misc/wiki/The-Mystical-Release-Process)
should be modified.
- `gh` should be added to os packages for mac and debian
- A new step should be added: "Run `gh auth login` and log into a GitHub
account." Technically any account should work here.
The contents of the step saying to post to the community forum should be
shortened to say something like "copy the output from the terminal." We
could add the link to the client-dev tag here, but personally I think
it's easiest to just keep it in the release instructions.
These tests are failing because the runner migrated ownership to github,
and that somehow broke it even though it wasn't supposed to. They're not
responding to issues [other
than](https://github.com/actions/runner-images/issues/14100) "a CVE or
vulnerability" during the transition period (ends june 12). Let's just
turn these off for now. We weren't even running this at all in azure,
it's fine.
Fixes#10617
I restructured the conditionals to avoid too much nesting. This should
have the same effect, just with the additional check conditioned on all
the target snap files being available. Here is the logic I used for the
restructuring:
start
```python
dump_output = exit_code != 0 or failed_archs
if exit_code == 0 and not failed_archs:
# We expect to have all target snaps available, or something bad happened.
snaps_list = glob.glob(join(workspace, '*.snap'))
if not len(snaps_list) == len(archs):
print('Some of the expected snaps for a successful build are missing '
f'(current list: {snaps_list}).')
dump_output = True
else:
build_success = True
break
```
note that `(exit_code == 0 and not failed_archs) == not (exit_code != 0
or failed_archs) == not dump_output`
```python
dump_output = exit_code != 0 or failed_archs
if not dump_output:
# We expect to have all target snaps available, or something bad happened.
snaps_list = glob.glob(join(workspace, '*.snap'))
if not len(snaps_list) == len(archs):
print('Some of the expected snaps for a successful build are missing '
f'(current list: {snaps_list}).')
dump_output = True
else:
build_success = True
break
```
distribute the if
```python
dump_output = exit_code != 0 or failed_archs
snaps_list = glob.glob(join(workspace, '*.snap'))
if not dump_output and (not len(snaps_list) == len(archs)):
# We expect to have all target snaps available, or something bad happened.
print('Some of the expected snaps for a successful build are missing '
f'(current list: {snaps_list}).')
dump_output = True
if not dump_output and (len(snaps_list) == len(archs)): # redundant; if it were false, we would have changed dump_output right above this
build_success = True
break
```
remove redundant check
```python
dump_output = exit_code != 0 or failed_archs
snaps_list = glob.glob(join(workspace, '*.snap'))
if not dump_output and (not len(snaps_list) == len(archs)):
# We expect to have all target snaps available, or something bad happened.
print('Some of the expected snaps for a successful build are missing '
f'(current list: {snaps_list}).')
dump_output = True
if not dump_output:
build_success = True
break
```
As this shows, we can now add additional checks that only happen if we
think we're in danger of succeeding based on checks thus far, and simply
change `dump_output` to `True` if the additional check fails.
You can see the build step failing when the [file contains
html](https://github.com/certbot/certbot/compare/html-problem...refs/heads/test-html-problem-2)
at
https://github.com/certbot/certbot/actions/runs/26071811878/job/76654623490#step:5:42
Related to https://github.com/certbot/certbot/issues/10581
Following up on #10631 and
https://github.com/certbot/certbot/pull/10622, this PR converts the
`nightly`
[pipeline](https://dev.azure.com/certbot/certbot/_build?definitionId=5)
from Azure to Github Actions.
`schedule` and `workflow_dispatch` triggers only work on merged
branches, not PRs. To see these tests running, I temporarily added a
`push` trigger in commit
[a2e9c43](https://github.com/certbot/certbot/pull/10634/commits/a2e9c4303e4a9549562aaa460f15787020f25d0d).
You can see the results of those tests
[here](https://github.com/certbot/certbot/actions/runs/25688414262).
I did not split each file into its own commit this time because I feel
like the general idea is clear. The relevant files in azure pipelines to
reference are:
- the deleted `.azure-pipelines/nightly.yml` -->
`.github/workflows/nightly.yml`
- `.azure-pipelines/templates/jobs/common-deploy-jobs.yml` -->
`.github/workflows/deploy_docker_images.yml` and
`.github/workflows/deploy_snaps.yml`
- `.azure-pipelines/templates/stages/changelog-stage.yml` -->
`.github/workflows/create_changelog.yml`
I chose to split `common-deploy-jobs` into `deploy_docker_images` and
`deploy_snaps`. This is because the docker arm32v6 build takes a long
time, but uploading to docker is quick, while the armhf snaps build
varies but is often quicker, but uploading the snaps can take some time.
By splitting them, we can specify the dependencies more precisely, and
hopefully shave some time off the total. Without the split, tests took
[53 minutes
total](https://github.com/certbot/certbot/actions/runs/25684264622).
After the split, tests took [33 minutes
total](https://github.com/certbot/certbot/actions/runs/25688414262)!
As before, the "nightly deploy stage" from azure has been omitted for
clarity.
`rerun.yml` did not exist before. There's not a great built-in way to
rerun individual jobs in github actions, which I wanted for the snap
builds specifically, since other timeouts can still happen. I could have
made an action or additional workflow and wrapped that in a script to
retry it, but I figured actually it's nicer to have the ability to rerun
anything. This is equivalent to clicking "rerun all failed jobs," which
I feel is usually what we want. Unfortunately, I am pretty sure that to
test it, the rerun script will need to be merged first, since it relies
on `workflow_dispatch`.
You can see that packages were successfully uploaded to
[dockerhub](https://hub.docker.com/r/certbot/certbot/tags) and the [snap
store](https://dashboard.snapcraft.io/stores/snaps/); looking at the
timestamps is probably the easiest way to confirm (about 11:45am
Monday).
Related to https://github.com/certbot/certbot/issues/10581
Following up on #10622, this PR converts the `full-test-suite`
[pipeline](https://dev.azure.com/certbot/certbot/_build?definitionId=4)
from Azure to Github Actions.
Nightly test changes for context not included in this PR are available
[here](https://github.com/certbot/certbot/compare/test-convert-full-pipeline...convert-all-pipelines).
Since this branch is named `test-convert-full-pipeline`, these tests
will show up in the checks section of this PR.
The major changes I made here are splitting the docker and snaps tests
for a better github actions UX, and removing the intermediate "stage"
file, since stages are not a concept in GHA. This means that we get the
nice dropdowns for the different categories on the left bar of the [test
run
page](https://github.com/certbot/certbot/actions/runs/25139155528/job/73684692548)
so it's easier to see each type of test. The very slight drawback is
that the four jobs listed in `.github/workflows/full_test_suite.yml` do
need to be duplicated in `nightly.yml`, but that's a reasonable tradeoff
to me.
Also, we now test our certbot and dns plugin snaps on all architectures
for the first time (using `dpkg --add-architecture` to run armhf tests
on an arm64 machine), which is very nice and in my opinion worth the
very slightly extra time and code.
In this PR, we build arm64 and amd64 snaps directly on github's runners.
armhf snaps are built using launchpad as before. This makes the workflow
file a little long. There are perhaps some micro-optimizations for code
deduplication I could make, like creating an action to install
dependencies based on the architecture, but I don't think it's super
worth it, especially since the dependencies vary enough that we'd still
need some code (for example, even between installing deps for certbot
and dns runs, we'd still need to additionally install `nginx-light`).
A very slight potential time improvement we could make here would be to
optionally depend on the different architectures before running their
respective tests. I'm not sure if this can be done without writing
different jobs, and since once those jobs start they run in parallel, it
didn't really seem worth looking into for me. I am of course open to
alternate points of view here and in general.
Another potential change to bring the two build strategies more in line
would be to stop using the python script to send off all the launchpad
builds, and instead put each in a separate, matrixed job like the github
jobs. We could even continue retrying the builds within each job. This
would mean that if one dns plugin build happens to fail three times, all
the builds wouldn't have to be retried. While I think that's not the
worst idea, I personally think that belongs in a separate PR, as this PR
is already quite long.
Speaking of the PR length, I can undo the changes made here to build
arm64 and amd64 snaps on github actions, to have a simpler
conversion-only PR to review. Some of the choices I made here,
particularly around UX, were based on the fact that the jobs would look
like this, so it might not be as clear why I made those choices, but if
it's easier to review it's no problem to put it back. I could also
remove the code that tests the other snaps since it's new, but I figured
it'd be nice to show that they are in fact being built correctly, since
otherwise the built snaps wouldn't be consumed anywhere.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Related to https://github.com/certbot/certbot/issues/10581
This is the first step of migrating to github actions.
Nightly and full tests have been converted on branch
`convert-all-pipelines`; you can see additional changes to do those for
context
[here](https://github.com/certbot/certbot/compare/convert-pr-tests...convert-all-pipelines).
Some notes:
- All github workflows must be flat in the `.github/workflows/`
directory.
- Github actions doesn't have a concept of "stages." Instead, it
generates a dependency graph, which is kind of nice. You can see an
example of a more complicated one
[here](https://github.com/certbot/certbot/actions/runs/24580625688).
- I don't know why the actions in the left bar (under Actions tab -->
All workflows) are using the path instead of the listed name. I suspect
it has something to do with not being run on main. Once it's merged, if
the name doesn't change, we can delete previous runs and that will clear
the entry on the left.
- "permissions" is for the fine-grained github PAT. contents: read is
needed for the "checkout" action, which basically everything uses. it's
still best practice to define per-workflow. it can also be defined
per-job, but per-workflow seemed nicer to me.
[This](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#permissions)
is the best permissions explanation I've found; [some
actions](https://github.com/actions/checkout) mention what permissions
they need.
- For definitions of the keywords to `on`, see
[here](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows).
- Some of the potential inputs in tox steps are not used in this PR
because we're not running the AWS tests. It seemed messier to take them
out here and put them back later when the extended tests need them, but
I can do that on request.
We currently have a `main` [protection
rule](https://github.com/certbot/certbot/settings/branch_protection_rules/5466)
set that Azure pipelines PR test suite must pass before merging.
Obviously I don't want to turn that off before this PR is reviewed. In
github actions, it can only require a specific job to pass, though you
can have multiple. To address this, I've created a job that requires all
other jobs to pass, and that can be set at the required job. We probably
do not want to list every individual job, as that includes every job
generated by a matrix strategy. To find it in the protection rules page,
start typing "PR test suite success" and it will show up.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: Will Greenberg <willg@eff.org>
Fixes https://github.com/certbot/certbot/issues/10180.
So first of all, the core issue here is that [pyca deliberately
chose](https://github.com/pyca/cryptography/blob/ec80c1c2894320d30fd674ea2c6103d91b4e777e/src/cryptography/utils.py#L15-L18)
to override the default python functionality and make deprecation
warnings appear by default. This isn't common. If they'd actually used a
`DeprecationWarning`, it wouldn't have shown up to users, at least. That
being said, we should still try to catch it, as we do in fact want to
know about deprecation warnings for our own updates.
To do that, this PR searches upwards for a `pytest.ini` file from the
file's location. If found, it reads the warnings from the file, and
passes them using the `PYTHONWARNINGS` env variable. It also explicitly
sets warnings to `error` always in case we can't find the `pytest.ini`,
and ignores the subsequent unverified-https-on-localhost warning. It
also fixes a warning in our test nginx config that seemed reasonable to
address.
I tested this by adding a temporary warning, which I then removed, but
since it turned out there were two other warnings, that wasn't actually
necessary.
Options I considered and rejected:
- Switch from `atexit` to calling `main` directly. To do this, we'd have
to switch our `main` function to something like a try-finally. That's
complicated by the fact that we call `atexit` from other places in the
code. Also, `exc_info` isn't availabe in `finally` while it is in
`at_exit`, so it's not as versatile. But mostly if we wanted to do this,
we'd have to implement a custom atexit handler, basically, and that
seems worse than this option.
- Looking into pytest-forked. It's apparently buggy and not being
maintained. Not even sure this is what it's for anyway.
- Multiple
[-W](https://docs.python.org/3/using/cmdline.html#cmdoption-W) options
can be given instead of an env variable. The env version seemed cleaner.
- More closely mimicking [how pytest finds ini
files](https://docs.pytest.org/en/stable/reference/customize.html#finding-the-rootdir).
It seemed unnecessary to me.
Potential drawbacks:
- If we move or rename the `pytest.ini` file and for some reason don't
do a reasonable grep for `pytest.ini`, we will no longer catch any
additional `ignore`s in there. But imo we're likely to do that grep, and
also a missing ignore will then show up when we run the tests.
Fixes #10518.
`tools/pinning/current/repin.sh` is not run; only pytest version is
updated. This is because `pypinning` had a bunch of syntax changes that
seem simply but I believe should be in a separate PR, which I think
should be done after this to collect all repin changes.
As discussed further in #10518, these issues were caused by pytest's
internalization of pytest-subtest, which had several implementation
changes.
To fix these, we simply no longer use subtest in the failing tests. The
test in acme is now parametrized instead, and the tests in apache only
ever had a single parameter.
To use parametrization in the acme test, I converted `DNSTest` from
unittest to pytest style, which was pretty straightforward. The only
note there is that while it would be nice to make `ec_secp384r1_key` a
fixture, you [can't use fixtures in
parameters](https://github.com/pytest-dev/pytest/issues/349). You could
use requests, but that seemed less clear and messier, because then you'd
be checking the value of the parameter and only sometimes loading it.
Could also make it a global variable, but that didn't really seem
necessary, as it's only called twice. Happy to consider other options,
not strongly tied to this one, just seemed nicest to me.
https://github.com/certbot/certbot/pull/10146 was supposed to do this,
but because of multiple code paths, it did not. This PR simplifies the
code by creating a single code path.
In particular:
- `hooks.renew_hook()` is removed. There are now only calls to
`hooks.deploy_hook()`, which is called during certonly, run, and renew,
and runs both cli and directory hooks.
- `cli_config.renew_hook` is removed. Both `--renew-hook` (hidden option
kept for backwards compatibility purposes and `--deploy-hook` now set
`cli_config.deploy_hook`, which is used internally. When either or both
flags are used multiple times, the last value is kept, which is the
argparse default.
- references to running a "renew hook" internally are changed to "deploy
hook"
- To maintain downgrade compatibility, `deploy_hook` is written out to
renewal config files as `renew_hook`. This is achieved by translating to
and from `renew_hook` in `storage.py` and changing
`renewal.STR_CONFIG_ITEMS` to contain `deploy_hook`.
This results in the following behavior changes:
- Directory hooks are now run when getting a new cert using certonly/run
- If someone set a renew hook on the cli using `--renew-hook`, it would
previously not be run when getting a new (non-renewed) cert, but now
will be. But this option is hidden and should no longer be used anyway.
- When using `certbot reconfigure`, if someone sets `--renew-hook`
certbot will now also ask if someone would like to do a test run of the
new hook, whereas before it would only do so for `--deploy-hook`.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#10423.
In
https://github.com/certbot/certbot/pull/10409#issuecomment-3180214385,
we noted that a comment in
`certbot-ci/src/certbot_integration_tests/certbot_tests/test_main.py:test_renew_with_ec_keys`
says:
> since ecdsa is now default, the integration test is not actually
testing "When running non-interactively, if --key-type is unspecified
but the default value differs to the lineage key type, Certbot should
keep the lineage key type." as it says in the comment. To fix that, it
should be initially created with rsa, not ecdsa.
That is no longer accurate, since the default key type changed to
ecdsda.
This PR adds a new test that does what the comment specifies, and
updates the comment to reflect that the existing test no longer does
that.
Fixes https://github.com/certbot/certbot/issues/6180.
New output:
```
--deploy-hook DEPLOY_HOOK
Command to be run in a shell once for each successfully issued certificate, including on subsequent renewals. Unless --disable-hook-validation is
used, the command’s first word must be the absolute pathname of an executable or one found via the PATH environment variable. For this command, the
shell variable $RENEWED_LINEAGE will point to the config live subdirectory (for example, "/etc/letsencrypt/live/example.com") containing the new
certificates and keys; the shell variable $RENEWED_DOMAINS will contain a space-delimited list of renewed certificate domains (for example,
"example.com www.example.com") (default: None)
```
Pre and post hooks are still only shown in `renew` and `reconfigure`
help, though perhaps there is less confusion over those so it's not
necessary.
Several of these have been fixed, so let's update the requirement if
necessary and remove the warning catching.
`python-dateutil 2.9.0` was released Feb 29, 2024, so it's not widely
packaged in non-EOL major distros yet.
`pytest-cov 4.1.0` was released May 24, 2023.
Our pinned versions were already higher than these requirements.
Alternatively, we could just remove the warnings and not update the
minimum requirement, but I think it's nicer to note it in requirements
for anyone running our tests, like packagers.
We already require `poetry-plugin-export>=1.9.0`. `1.7.0` updated its
`requests-toolbelt` requirement to `>=1.0.0`, which is greater than the
minimum version needed to remove the warning.
It's a [drop-in
replacement](https://docs.astral.sh/uv/pip/compatibility/) that speeds
things up. I don't see any reason why not.
`--use-pep517` is [set by default](
https://docs.astral.sh/uv/pip/compatibility/#pep-517-build-isolation),
so we don't need it.
`--disable-pip-version-check` also does nothing on uv.
`uv` [uses
separate](https://docs.astral.sh/uv/pip/compatibility/#build-constraints)
`UV_BUILD_CONSTRAINT` and `UV_CONSTRAINT`. I just added it to both to do
the simplest thing here. We could split them.
We probably don't actually need to pipstrap pip anymore, I could take
that out.
What's happening with `parsedatetime` and `python-digitalocean` is that
they were always secretly wrong. Since `pip` compiles bytecode by
default, it was suppressing the errors. If you add the
`--compile-bytecode` flag to `uv`, it passes, but I don't think we
should do that. You can see the failure happen on main by passing
`--no-compile` to the pip args and running `certbot -r -e oldest`.
Now what I don't understand is that some places seem to say the `'\/'`
error from `parsedatetime` only started in python 3.12, whereas others
see it on earlier python. Perhaps pytest is vendorizing python or
something. Not too worried about that, needed to get updated anyway, and
it's an accurate oldest version based on our oldest OSes.
`python-digitalocean` is techincally newer than debian 11, but we've
made that decision before so it seems fine to me.
Part of https://github.com/certbot/certbot/issues/10403
We were never actually updating the versions in certbot-ci and letstest.
Not that it really matters, but let's do that there as well.
Final part of https://github.com/certbot/certbot/issues/10403
I tested running `tools/snap/generate_dnsplugins_snapcraft.sh
certbot-dns-dnsimple` and it put the correct description in to the
`snapcraft.yaml` file.
Part of https://github.com/certbot/certbot/issues/10403.
As far as I can tell, "stick it in setup.py" is the official way of
handling complex dependencies. But since the version is static, we have
a little more choice here than we had with `certbot/pyproject.toml`.
We could put the version in the respective `pyproject.toml`s and read it
directly from the toml file with something like
[this](https://stackoverflow.com/a/78082561). Or otherwise load and
parse that file. The benefit of doing it that way is that all
non-certbot versions would be canonically in the `pyproject.toml`, and
also if we wanted we could use that same toml parsing to change the
version at release time instead of `sed`. I actually suspect `acme`,
`certbot-ci`, and `certbot-compatibility-test` will be the only ones
where we can completely delete `setup.py`, as the others all have
lockstep dependencies. (side note - we just never update `certbot-ci`
version. it's still set at `0.32.0.dev0`. there's no way this matters
but just noting.) I chose to do it this way instead because it seems
cleaner since we have to keep `setup.py` around anyway, but I don't have
a strong preference.
Based on what I've read, there's not actually a clean way to grab and
insert the version number within the toml file. This is due to [design
decisions](https://github.com/toml-lang/toml/issues/77) by the toml
authors. The clean `all` extras specification that we used in
`certbot/pyproject.toml` [seems to be an
outlier](https://github.com/pypa/setuptools/discussions/3627#discussioncomment-6476654)
because it's pip handling the self-reference, not toml.
This was causing oldest tests to fail on my mac, which has an open file
limit of 256. Locks were being released at exit, but there were more
than 256 tests being run at once. Holding onto the file descriptor for
temporary files was making us keep the files open.
I also removed unnecessary setUps and tearDowns in subclasses so that
this could be fixed in only one spot.
If you wanted to do any testing locally, I was throwing this in places:
```
import errno, os, resource
open_file_handles = []
for fd in range(resource.getrlimit(resource.RLIMIT_NOFILE)[0]):
try: os.fstat(fd)
except OSError as e:
if e.errno == errno.EBADF: continue
open_file_handles.append(fd)
print(f'location description: {len(open_file_handles)}')
```
Alternative to https://github.com/certbot/certbot/pull/10408/ and
https://github.com/certbot/certbot/pull/10415/ that fixes production
code for account meta and puts autouse fixtures in certbot and acme
tests. Overrides all `time.sleep` calls while we're at it.
Fixes the production code where it's simple/clean, and fixes the tests
for HTTPServer-based code because we just don't have that many mac users
using standalone.
This PR is modeled on https://github.com/certbot/certbot/pull/10373/,
and is part of https://github.com/certbot/certbot/issues/10183.
relevant requirements:
use_tls13 >= 1.13.0
session_tix_off implemented: nginx >= 1.5.9 and openssl_version >=
1.0.2l
session tix off by default: >= 1.23.2
oldest non-deprecated major distros nginx versions:
debian 11 1.18.0
epel 8 1.23.1
ubuntu 22.04 1.18.0
Therefore, we can stop testing for use tls 13 and session tix off
allowed, with the same caveat as [in this
comment](https://github.com/certbot/certbot/pull/10373#issuecomment-3134101604).
While we could add a new split for configs that don't require session
tickets off to be set explicitly since it's the default, I don't think
it's worth doing now. I added a note in the comments about this.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This sets up a `pyproject.toml` file for certbot, initially generated
[using](https://hatch.pypa.io/latest/intro/#existing-project) `hatch new
--init` and modifying from there. Since we dynamically require acme of a
matching version, I kept that around in `setup.py` to do the simplest
thing in this PR.
Other possible (future) implementations:
- setuptools has a beta implementation to read from a
`requirements.txt`. we could generate one of those.
- we could just hardcode it and update at release time. I like not
having to keep the version up to date in various places but maybe it's
actually fine
- something something integration with poetry pinning?
I think the syntax for setting version dynamically in `pyproject.toml`
is much nicer than what we do in `setup.py`. It's a little silly to do
it there after we've bothered to calculate it, but I put it there in the
hopes of being able to remove it from `setup.py` someday/somehow.
It would be nice to access the version dynamically set in
`pyproject.toml` in `setup.py`, but I do not think it is likely to be
possible.
Here are some useful links regarding this migrations:
- [How to modernize a `setup.py` based
project?](https://packaging.python.org/en/latest/guides/modernize-setup-py-project/)
- [Writing your
`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/)
- [Configuring setuptools using `pyproject.toml` files](
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)
- [`pyproject.toml`
specification](https://packaging.python.org/en/latest/specifications/pyproject-toml/)
- [Platform compatibility
tags](https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/)
Alternative implementation for #7908.
In this PR:
- set up ruff in CI (add to `tox.ini`, mark dep in `certbot/setup.py`)
- add a `ruff.toml` that ignores particularly annoying errors. I think
line length isn't actually necessary to set with this workflow since
we're not checking it but putting it there for future usage.
- either fix or ignore the rest of the errors that come with the default
linting configuration. fixed errors are mostly unused variables. ignored
are usually where we're doing weird import things for a specific reason.
Part of #10183
> Option 4. Stop updating old files with security improvements. If
people want to be on old software they can but then they're not getting
the nice new things. We can either warn or not warn if we see people
using them, either on certbot install (what, who's installing new
certbot on these machines), new cert, cert renewal, or certbot update.
The second two would require code changes, I'm pretty sure. I don't
think we should warn too often because that's how we get people to
silence all output. This is a little weird because we don't usually keep
around deprecated things. We could also warn loudly and see if people
complain. Or do some sort of brownout.
This PR warns every time certbot is run. We could make it run less often
(only when a new config file is installed, probably), but that's a more
extensive code change, and honestly I think it's probably fine? But I
can change it.
Fixes https://github.com/certbot/certbot/issues/10342
When doing ARI checks in acme.renewal_time, we catch RequestException
and return a default value. That's so an unavailable ARI server doesn't
cause issues.
Before we get to acme.renewal_time, we have to create an ACME client,
and in the process fetch a directory. We should make the directory fetch
similarly resilient.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
https://github.com/certbot/certbot/pull/10297 modified
`letsencrypt-auto-source/letsencrypt-auto`. It should never be modified
except during a release, and we have a test to make sure of it. Old
scripts pull directly from that file on github, so let's put that back
asap.
Fixes https://github.com/certbot/certbot/issues/10259
This PR moves post-hook execution from `main.renew` to
`renewal.handle_renewal_request` so that failed and renewed domains
actually get passed into post-hook execution as promised, even when
failures happened.
I suspect the original PR was being overly cautious by putting the whole
thing into a try/finally so that post-hooks definitely happen, but
`handle_renewal_request` is already full of exception catching. I
understand the worry about executing a pre-hook and then failing to
execute its matching post-hook, but the code really is already
structured to make sure that that won't happen. And then when we added
`FAILED_DOMAINS` and `RENEWED_DOMAINS`, we both kept that
overly-cautious hooks execution location, but also kept the error so we
have a summary at the end...which meant that if failures happened, the
env vars were never set.
If we really want to keep the `hooks.run_saved_post_hooks` call on the
outside of everything in main, we can, but then we will have to do one
of the following:
- pass in the output lists to be filled out during execution. not my
favorite pattern
- throw the output lists in the error object or make a wrapper error,
not sure, haven't looked at `errors.py` too closely
- stop raising that final error where we report failures at the very
bottom. it's a little outdated maybe but I do like it and I think people
are used to it
- raise that error in main, returning the number of parse and renewal
failures. this is my favorite of the options, but I still like it less
than what I've implemented here.
Here's the integration/regression test failing on main:
https://dev.azure.com/certbot/certbot/_build/results?buildId=9237&view=logs&j=fca58cec-e7ce-563a-f36f-5c233894d750
You can see here that that branch just has the integration test without
the fix (and removing other tests for efficiency):
https://github.com/certbot/certbot/compare/main...test-fail-env-on-main
It's the default, but just to be clear, this should definitely have two
reviewers.
I added the exact same service hook we use for nightly failures for
release failures.
<img width="1347" alt="Screenshot 2025-06-11 at 10 32 18 AM"
src="https://github.com/user-attachments/assets/b4728d0b-212b-4ecb-84c6-0ed62715f0ff"
/>
Service hooks can be viewed here:
https://dev.azure.com/certbot/certbot/_settings/serviceHooks
Now there's no reason to keep around the manual notification stage, it
wasn't working in either case anyway. Since it's literally the same as
the nightly hook, I don't personally feel the need to test the release
branch but I can if the reviewer would like.
This is a feature people didn't have before and won't miss if it fails.
We can always raise it later, but let's reduce it for now to stop people
worrying about the big red warning.
This is one solution to https://github.com/certbot/certbot/issues/10327.
It won't test an ARI check during a dry run, since it will just avoid
the mismatch problem by checking for dry run first and returning before
checking ARI. This PR will make the big error (actually a warning, but
red and scary) go away though.
Erica Portnoy