Automate synchronizing github repo post-release (#10697)

This PR automates the release process steps to:
- update the candidate branch with the contents of the release branch on
the repo in the subfolder
- create a PR merging the candidate branch into main
- create a new branch without the version bump. usually `1.2.x`, unless
it's a point release.
- if it's a point release, create a PR merging the new branch without
version bumps into the existing `1.2.x` branch.

This draws from steps 10, 12, and 13. Step 10 should still have the code
to push to pypi, since that's a different though, though I think that
could move here in the future as well.

My general design philosophy was "error out instead of letting the
script put git into a bad state," with the exception of PR creation
which seemed safe to skip and continue.

I've also added some flags, mostly to make testing this easier, but
could be useful for re-running the script as well. Unlike
`promote_snaps` and `generate_community_forum_post`,
`synchronize_github_repo` is *not* idempotent. I do not think it should
be, because of how git works. I think if branches already exist and the
user really did want to synchronize branches again, the user would want
to know that it can't be done automatically, and should instead be told
what to do to make it possible, or how to skip the whole thing. I don't
think we should, for example, go ahead and create a PR based on an old
version of a branch and just skip the pulling step, or automatically
delete a branch.

In `_create_and_push_branch_without_version_bump`, if you have created
the branch then fail after, you'll rerun and then get a message saying
to delete the branch. I think that's nicer than automatically deleting
it, in case you want to inspect it.

successful test run:
```
$ git switch create-pr
Switched to branch 'create-pr'
$ RELEASE_GPG_KEY=[test key] tools/release.sh 4.35.0 4.36.0
[release output]
$ tools/finish_release.py --test-version 4.35.0 --skip-snaps
Creating PR to merge candidate-4.35.0 into main...
PR location: https://github.com/certbot/certbot/pull/10714
Creating branch without version bump commit named 4.35.x...
Created.
Generating announcement text for community forum post
release not found
Generating announcement text failed.
$ git switch 4.35.x
Switched to branch '4.35.x'
$ RELEASE_GPG_KEY=[test key] tools/release.sh 4.35.1 4.36.0
[release output]
$ tools/finish_release.py --test-version 4.35.1 --skip-snaps
Creating PR to merge candidate-4.35.1 into main...
PR location: https://github.com/certbot/certbot/pull/10715
Creating branch without version bump commit named point-candidate-4.35.1...
Created.
Creating PR to merge point-candidate-4.35.1 into 4.35.x...
PR location: https://github.com/certbot/certbot/pull/10716
Generating announcement text for community forum post
release not found
Generating announcement text failed.
```

then here's some errors and their outputs --

trying to run `finish_release.py` again:
```
$ tools/finish_release.py --test-version 4.34.1 --skip-snaps
Creating PR to merge candidate-4.35.1 into main...
PR to merge release changes into main already exists...skipping creation. To create a new PR, delete the old one on GitHub.
PR location: https://github.com/certbot/certbot/pull/10715
Creating branch without version bump commit named point-candidate-4.35.1...
Error running `git branch point-candidate-4.35.1`
Branch point-candidate-4.35.1 already exists. Delete it using `git branch -D point-candidate-4.35.1`.

fatal: a branch named 'point-candidate-4.35.1' already exists

To skip pushing updated branches to GitHub and creating PRs, run this script with the `--skip-github-sync` flag.
Traceback (most recent call last):
  File "/Users/erica/certbot/tools/finish_release.py", line 370, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 366, in main
    synchronize_github_repo(version)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 329, in synchronize_github_repo
    _create_and_push_branch_without_version_bump(version, branch_name)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 280, in _create_and_push_branch_without_version_bump
    _run_silent_except_error(f'git branch {branch_name}'.split(), msg)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 209, in _run_silent_except_error
    raise e
  File "/Users/erica/certbot/tools/finish_release.py", line 201, in _run_silent_except_error
    process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
  File "/Users/erica/.pyenv/versions/3.14.3/lib/python3.14/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'branch', 'point-candidate-4.35.1']' returned non-zero exit status 128.
```

local changes to branch:
```
$ touch test_file.txt
$ git add -A 
$ tools/finish_release.py --test-version 4.35.1 --skip-snaps
Error running `git diff --quiet HEAD`
You have uncommitted changes that will be deleted. Stash your changes before rerunning this script.


To skip pushing updated branches to GitHub and creating PRs, run this script with the `--skip-github-sync` flag.
Traceback (most recent call last):
  File "/Users/erica/certbot/tools/finish_release.py", line 370, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 366, in main
    synchronize_github_repo(version)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 315, in synchronize_github_repo
    _run_silent_except_error('git diff --quiet HEAD'.split(), message)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 209, in _run_silent_except_error
    raise e
  File "/Users/erica/certbot/tools/finish_release.py", line 201, in _run_silent_except_error
    process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
  File "/Users/erica/.pyenv/versions/3.14.3/lib/python3.14/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'diff', '--quiet', 'HEAD']' returned non-zero exit status 1.
```

branch doesn't match the one on github (shows that stdout is now also
printed on error):
```
$ cd releases/le.4.35.1.89372/
$ git commit --amend # change the message
[candidate-4.35.1 8d34a67a4] Bump version to 4.36.0 new message
 Date: Tue Jun 23 10:47:56 2026 -0700
 20 files changed, 20 insertions(+), 20 deletions(-)
$ cd ../../
$ tools/finish_release.py --test-version 4.35.1 --skip-snaps
Error running `git push origin candidate-4.35.1`
To delete the branch on GitHub, run `git push origin --delete candidate-4.35.1`.

To https://github.com/certbot/certbot.git
 ! [rejected]            candidate-4.35.1 -> candidate-4.35.1 (non-fast-forward)
error: failed to push some refs to 'https://github.com/certbot/certbot.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. If you want to integrate the remote changes,
hint: use 'git pull' before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

To skip pushing updated branches to GitHub and creating PRs, run this script with the `--skip-github-sync` flag.
Traceback (most recent call last):
  File "/Users/erica/certbot/tools/finish_release.py", line 370, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 366, in main
    synchronize_github_repo(version)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 317, in synchronize_github_repo
    _sync_candidate_from_temp_to_origin(version)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 247, in _sync_candidate_from_temp_to_origin
    _run_silent_except_error(command_str.split(), message)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erica/certbot/tools/finish_release.py", line 209, in _run_silent_except_error
    raise e
  File "/Users/erica/certbot/tools/finish_release.py", line 201, in _run_silent_except_error
    process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
  File "/Users/erica/.pyenv/versions/3.14.3/lib/python3.14/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'push', 'origin', 'candidate-4.35.1']' returned non-zero exit status 1.
```

I've hit basically all of the errors with text at some point during
testing, but can recreate them if you'd like.

Once this is merged, I'll update the release instructions and delete the
test branches and PRs.

---------

Co-authored-by: Will Greenberg <ifnspifn@gmail.com>
Co-authored-by: Will Greenberg <willg@eff.org>

tools/_release.sh

@@ -216,6 +216,7 @@ git commit -m "Remove built packages from git"
 
 if [ "$RELEASE_BRANCH" = candidate-"$version" ] ; then
     SetVersion "$nextversion".dev0
+    # If this message changes, it should also be changed in tools/finish_release.py
     git commit -m "Bump version to $nextversion"
 fi
 

tools/finish_release.py

@@ -1,24 +1,38 @@
 #!/usr/bin/env python
 """
-Post-release script to publish artifacts created from GitHub Actions.
+Post-release script to publish artifacts created from GitHub Actions, synchronize the repo
+  on Github, and print a formatted changelog for the latest release.
 
 This currently includes:
 
 * Moving snaps from the beta channel to the stable channel
+* Pushing the candidate branch to GitHub
+* Creating a minor release branch if it's not a point release
+* Opening PR(s) to merge the release changes back into `main` and/or the minor release branch
+* Printing a formmated changelog for the latest release for distribution
 
 Setup:
- - Install the snapcraft command line tool and log in to a privileged account.
+ - Install the `snapcraft` command line tool and log in to a privileged account.
    - https://snapcraft.io/docs/installing-snapcraft
    - Use the command `snapcraft login` to log in.
+ - Install the `gh` command line tool and log in to a privileged account.
+   - https://github.com/cli/cli#installation
+   - Use the command `gh auth login` to log in.
 
 Run:
 
-python tools/finish_release.py
+python tools/finish_release.py [--skip-snaps] [--skip-github-sync] [--test version <VERSION_NUM>]
 
 Testing:
 
 This script can be safely run between releases. When this is done, the script
-should execute successfully.
+should execute successfully, if it has not run before. Promoting snaps from beta
+and printing the changelog are idempotent, but synchronizing the github repo is not.
+
+To skip syncing with github, use the `--skip-github-sync` flag.
+To skip promoting snaps, use the `--skip-snaps` flag.
+To test with a version other than the latest release, use the `--test-version <VERSION_NUM>` flag
+  with a version number formatted as <A.B.C>.
 
 """
 
@@ -45,6 +59,8 @@ ALL_SNAPS = ['certbot'] + PLUGIN_SNAPS
 # This is the count of the architectures currently supported by our snaps used
 # for sanity checking.
 SNAP_ARCH_COUNT = 3
+SKIP_SYNC_MESSAGE = ('To skip pushing updated branches to GitHub and creating PRs, '
+                     'run this script with the `--skip-github-sync` flag.')
 
 
 def parse_args(args):
@@ -61,6 +77,12 @@ def parse_args(args):
     # Use the file's docstring for the help text and don't let argparse reformat it.
     parser = argparse.ArgumentParser(description=__doc__,
                                      formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument('--test-version', type=str, default=None,
+                        help='version in the form of 1.2.3, mainly for testing')
+    parser.add_argument('--skip-snaps', action='store_true',
+                        help='don\'t promote snaps; used for testing')
+    parser.add_argument('--skip-github-sync', action='store_true',
+                        help='don\'t synchronize branches to GitHub or create PRs')
     return parser.parse_args(args)
 
 
@@ -171,6 +193,145 @@ def fetch_version_number():
     assert len(version.split('.')) == 3
     return version
 
+
+def _run_silent_except_error(cmd: list[str], message: str = None) -> subprocess.CompletedProcess:
+    # For some reason, git prints a bunch of non-error output to stderr. Let's keep this script
+    # quiet by capturing it and only printing it if we hit an error.
+    try:
+        process = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
+    except subprocess.CalledProcessError as e:
+        print(f'Error running `{' '.join(cmd)}`')
+        if message is not None:
+            print(message)
+        print(e.output)
+        print(e.stderr)
+        print(SKIP_SYNC_MESSAGE)
+        raise e
+    else:
+        return process
+
+
+def _create_pr(title: str, body: str, description: str, other_opts: list[str] | None = None) -> str:
+    cmd = ['gh', 'pr', 'create', '--title', title, '--body', body]
+    if other_opts is not None:
+        cmd = cmd + other_opts
+    try:
+        proc = subprocess.run(cmd, check=True, universal_newlines=True, capture_output=True)
+    except subprocess.CalledProcessError as e:
+        if 'already exists' in e.stderr:
+            print(f'{description} already exists...skipping creation. '
+                   'To create a new PR, delete the old one on GitHub.')
+            # The error message looks like:
+            # a pull request for branch "candidate-4.24.0" into branch "main" already exists:
+            # https://github.com/certbot/certbot/pull/10698
+            last_e_word = e.stderr.split()[-1]
+            if 'https' in last_e_word:
+                output = last_e_word
+        else:
+            print(e.stderr)
+            print(SKIP_SYNC_MESSAGE)
+            raise e
+    else:
+        output = proc.stdout.rstrip()
+    return output
+
+
+def _sync_candidate_from_temp_to_origin(version: str) -> None:
+    command_str = f'git pull temp candidate-{version}'
+    message = ('To run successfully, stash any changes you\'ve made to this branch. '
+               'Do not attempt to merge and continue, as that will fail.')
+    _run_silent_except_error(command_str.split(), message)
+    command_str = f'git push origin candidate-{version}'
+    message = ('To delete the branch on GitHub, run '
+              f'`git push origin --delete candidate-{version}`.')
+    _run_silent_except_error(command_str.split(), message)
+
+
+def _create_release_pr_to_main(version: str) -> None:
+    print(f'Creating PR to merge candidate-{version} into main...')
+    title = f'update files from {version} release'
+    body = 'this PR only needs 1 review and should be merged, not squashed'
+    result = _create_pr(title, body, 'PR to merge release changes into main')
+    print(f'PR location: {result}')
+
+
+def _create_release_pr_to_minor_branch(
+        version: str,
+        branch_name:str,
+        point_x_branch_name: str) -> None:
+    print(f'Creating PR to merge {branch_name} into {point_x_branch_name}...')
+    title = f'update files from {version} release'
+    body = 'this PR only needs 1 review and should be merged, not squashed'
+    pr_opts = [ '--head', branch_name,
+                '--base', point_x_branch_name]
+    result = _create_pr(title, body, 'PR to merge release changes into .x branch', pr_opts)
+    print(f'PR location: {result}')
+
+
+def _create_and_push_branch_without_version_bump(version: str, branch_name: str) -> None:
+    # Usually a branch of form 1.2.x
+    # When it's a point release, it'll be any name, and then merged back into 1.2.x
+    print(f'Creating branch without version bump commit named {branch_name}...')
+    # Check if there are uncommited changes, since reset will blow them away
+
+    try:
+        msg = (f'Branch {branch_name} already exists. Delete it using '
+               f'`git branch -D {branch_name}`.')
+        _run_silent_except_error(f'git branch {branch_name}'.split(), msg)
+
+        _run_silent_except_error(f'git switch {branch_name}'.split())
+
+        # Make sure the last commit message is 'Bump version to {next version}'
+        output = _run_silent_except_error('git log -1 --pretty=%B'.split()).stdout
+        assert_msg = 'The most recent commit message should start with "Bump version to"'
+        assert output.startswith('Bump version to'), assert_msg
+
+        _run_silent_except_error('git reset --hard HEAD~1'.split())
+
+        msg = ('You shouldn\'t be trying to re-push a minor version branch. If you really want '
+               'to, go into GitHub, turn off branch deletion protection, and delete it there. ')
+        _run_silent_except_error(f'git push origin {branch_name}'.split(), msg)
+    finally:
+        # Switching to the current branch exits 0
+        _run_silent_except_error(f'git switch candidate-{version}'.split())
+    print('Created.')
+
+
+def _check_branch_matches_version(version: str) -> None:
+    # This function assumes we're on a branch like `candidate-1.2.0` or `candidate-1.2.3`
+    # where the number after candidate should be equal to the version number
+    process = _run_silent_except_error('git branch --show'.split())
+    current_branch = process.stdout.rstrip()
+    if current_branch != f'candidate-{version}':
+        print(f'Unexpected branch name found. The current branch should be candidate-{version}.')
+        print(SKIP_SYNC_MESSAGE)
+        sys.exit(1)
+
+
+def synchronize_github_repo(version: str):
+    _check_branch_matches_version(version)
+    message = ('You have uncommitted changes that will be deleted. '
+               'Stash your changes before rerunning this script.')
+    _run_silent_except_error('git diff --quiet HEAD'.split(), message)
+
+    _sync_candidate_from_temp_to_origin(version)
+    _create_release_pr_to_main(version)
+
+    # Check the last element of the version number to see if this is a point release
+    point_version = version.split('.')[-1]
+    point_release = point_version != '0'
+    point_x_branch_name = '.'.join(version.split('.')[:-1]) + '.x'
+    if not point_release:
+        branch_name = point_x_branch_name
+    else:
+        branch_name = f'point-candidate-{version}'
+
+    _create_and_push_branch_without_version_bump(version, branch_name)
+
+    if point_release:
+        _create_release_pr_to_minor_branch(version, branch_name, point_x_branch_name)
+
+
 def generate_community_forum_post(version: str):
     print('Generating announcement text for community forum post')
 
@@ -196,8 +357,13 @@ def generate_community_forum_post(version: str):
 
 def main(args):
     parsed_args = parse_args(args)
-    version = fetch_version_number()
-    promote_snaps(ALL_SNAPS, 'beta', version)
+    version = parsed_args.test_version
+    if not version:
+        version = fetch_version_number()
+    if not parsed_args.skip_snaps:
+        promote_snaps(ALL_SNAPS, 'beta', version)
+    if not parsed_args.skip_github_sync:
+        synchronize_github_repo(version)
     generate_community_forum_post(version)
 
 if __name__ == "__main__":