Fixes https://github.com/certbot/certbot/issues/6180.
New output:
```
--deploy-hook DEPLOY_HOOK
Command to be run in a shell once for each successfully issued certificate, including on subsequent renewals. Unless --disable-hook-validation is
used, the command’s first word must be the absolute pathname of an executable or one found via the PATH environment variable. For this command, the
shell variable $RENEWED_LINEAGE will point to the config live subdirectory (for example, "/etc/letsencrypt/live/example.com") containing the new
certificates and keys; the shell variable $RENEWED_DOMAINS will contain a space-delimited list of renewed certificate domains (for example,
"example.com www.example.com") (default: None)
```
Pre and post hooks are still only shown in `renew` and `reconfigure`
help, though perhaps there is less confusion over those so it's not
necessary.
this is the first part of the nginx refactoring work i wanted to do.
ohemorange, if this conflicts with your work on updating our nginx ssl
config, please feel free to either ignore this for now or point me to
your branch after merging this and i can fix up any merge conflicts
myself like i previously offered
the main thing this PR does is create a new choose_or_make_vhosts
function with the current choose_vhosts behavior and makes choose_vhosts
only return existing ssl vhosts which i think is the behavior we want
when setting up HSTS or OCSP stapling. [this is what we do in
apache](https://github.com/certbot/certbot/blob/867b499f9b3934f88beffe53f021c6834350c2f1/certbot-apache/src/certbot_apache/_internal/configurator.py#L1795-L1829),
enabling HSTS or OCSP stapling on an HTTP vhost seems wrong/dangerous,
and since we don't have cert and key information in these [enhance
calls](https://github.com/certbot/certbot/blob/867b499f9b3934f88beffe53f021c6834350c2f1/certbot/src/certbot/interfaces.py#L255),
any SSL vhost we create will be left with snakeoil certs which also
seems very wrong
of course, this simple change to certbot-nginx's prod code required many
changes to its tests. the config file for headers.com was introduced
[here](https://github.com/certbot/certbot/pull/6068) specifically for
testing HSTS so i added the SSL configuration it needs to make work with
the choose_vhost changes. that then broke the IP tests for headers.com
that were added in https://github.com/certbot/certbot/pull/10145/ so i
created a new no-listens.com vhost for testing that
if this is merged, my plan in the next PR or two is to:
1. since choose_or_make_vhosts is now always called with
create_if_no_match=True in prod code, i plan to remove that variable,
make that the default behavior of the function, and fix up tests
2. then, since choose_or_make_vhosts is only called in deploy_cert, i
plan to pass the cert and key to it so it can be used in
_make_server_ssl instead of the dummy certs currently being used there
i could do more of this in this PR if you want ohemorange, but i figured
it rarely hurts to break things up especially when the code is kind of
tricky like it is (to me) here
this is the quickest somewhat sane fix for the current CI problem that i
could think of
the domain being used in the self-signed cert really seems irrelevant.
on my main test VPS, socket.gethostname returns "brad-certbot-dev". on
my laptop, it's "MacBookPro". i manually tested this branch a bit on my
VPS and nginx seems content
using a simple static string like this seems unlikely to break anything
to me and i think helps clearly identify where the self-signed cert is
coming from if ever causes a problem for anyone in the future
Several of these have been fixed, so let's update the requirement if
necessary and remove the warning catching.
`python-dateutil 2.9.0` was released Feb 29, 2024, so it's not widely
packaged in non-EOL major distros yet.
`pytest-cov 4.1.0` was released May 24, 2023.
Our pinned versions were already higher than these requirements.
Alternatively, we could just remove the warnings and not update the
minimum requirement, but I think it's nicer to note it in requirements
for anyone running our tests, like packagers.
We already require `poetry-plugin-export>=1.9.0`. `1.7.0` updated its
`requests-toolbelt` requirement to `>=1.0.0`, which is greater than the
minimum version needed to remove the warning.
It's a [drop-in
replacement](https://docs.astral.sh/uv/pip/compatibility/) that speeds
things up. I don't see any reason why not.
`--use-pep517` is [set by default](
https://docs.astral.sh/uv/pip/compatibility/#pep-517-build-isolation),
so we don't need it.
`--disable-pip-version-check` also does nothing on uv.
`uv` [uses
separate](https://docs.astral.sh/uv/pip/compatibility/#build-constraints)
`UV_BUILD_CONSTRAINT` and `UV_CONSTRAINT`. I just added it to both to do
the simplest thing here. We could split them.
We probably don't actually need to pipstrap pip anymore, I could take
that out.
What's happening with `parsedatetime` and `python-digitalocean` is that
they were always secretly wrong. Since `pip` compiles bytecode by
default, it was suppressing the errors. If you add the
`--compile-bytecode` flag to `uv`, it passes, but I don't think we
should do that. You can see the failure happen on main by passing
`--no-compile` to the pip args and running `certbot -r -e oldest`.
Now what I don't understand is that some places seem to say the `'\/'`
error from `parsedatetime` only started in python 3.12, whereas others
see it on earlier python. Perhaps pytest is vendorizing python or
something. Not too worried about that, needed to get updated anyway, and
it's an accurate oldest version based on our oldest OSes.
`python-digitalocean` is techincally newer than debian 11, but we've
made that decision before so it seems fine to me.
This uses the new `storage.atomic_rewrite` function to write only the
ARI-related value to the config file, leaving other values the same.
Updates `storage.atomic_rewrite` to handle the case where the
`"renewalparams"` section might be empty (which occurs in the new
unittests), and adds some comments.
Note: `atomic_rewrite` is mocked out as a no-op in
`test_resilient_ari_directory_fetches` and `test_renew_before_expiry`
because those tests have a simplistic config object on their
mock_renewable_cert that doesn't include a filename. If we try to write
the config in those tests we'd get an error (trying to write to `None`).
Since those tests aren't intended to test the "store and obey
Retry-After" behavior, I figured it's reasonable to skip it in the name
of testing; but of course, open to idea about the best way to navigate
this.
Part of #10355
Part of https://github.com/certbot/certbot/issues/10403
We were never actually updating the versions in certbot-ci and letstest.
Not that it really matters, but let's do that there as well.
Final part of https://github.com/certbot/certbot/issues/10403
I tested running `tools/snap/generate_dnsplugins_snapcraft.sh
certbot-dns-dnsimple` and it put the correct description in to the
`snapcraft.yaml` file.
Part of https://github.com/certbot/certbot/issues/10403.
As far as I can tell, "stick it in setup.py" is the official way of
handling complex dependencies. But since the version is static, we have
a little more choice here than we had with `certbot/pyproject.toml`.
We could put the version in the respective `pyproject.toml`s and read it
directly from the toml file with something like
[this](https://stackoverflow.com/a/78082561). Or otherwise load and
parse that file. The benefit of doing it that way is that all
non-certbot versions would be canonically in the `pyproject.toml`, and
also if we wanted we could use that same toml parsing to change the
version at release time instead of `sed`. I actually suspect `acme`,
`certbot-ci`, and `certbot-compatibility-test` will be the only ones
where we can completely delete `setup.py`, as the others all have
lockstep dependencies. (side note - we just never update `certbot-ci`
version. it's still set at `0.32.0.dev0`. there's no way this matters
but just noting.) I chose to do it this way instead because it seems
cleaner since we have to keep `setup.py` around anyway, but I don't have
a strong preference.
Based on what I've read, there's not actually a clean way to grab and
insert the version number within the toml file. This is due to [design
decisions](https://github.com/toml-lang/toml/issues/77) by the toml
authors. The clean `all` extras specification that we used in
`certbot/pyproject.toml` [seems to be an
outlier](https://github.com/pypa/setuptools/discussions/3627#discussioncomment-6476654)
because it's pip handling the self-reference, not toml.
This was causing oldest tests to fail on my mac, which has an open file
limit of 256. Locks were being released at exit, but there were more
than 256 tests being run at once. Holding onto the file descriptor for
temporary files was making us keep the files open.
I also removed unnecessary setUps and tearDowns in subclasses so that
this could be fixed in only one spot.
If you wanted to do any testing locally, I was throwing this in places:
```
import errno, os, resource
open_file_handles = []
for fd in range(resource.getrlimit(resource.RLIMIT_NOFILE)[0]):
try: os.fstat(fd)
except OSError as e:
if e.errno == errno.EBADF: continue
open_file_handles.append(fd)
print(f'location description: {len(open_file_handles)}')
```
Alternative to https://github.com/certbot/certbot/pull/10408/ and
https://github.com/certbot/certbot/pull/10415/ that fixes production
code for account meta and puts autouse fixtures in certbot and acme
tests. Overrides all `time.sleep` calls while we're at it.
Fixes the production code where it's simple/clean, and fixes the tests
for HTTPServer-based code because we just don't have that many mac users
using standalone.
The responsibility for atomic updates to a config file was previously
spread across different functions. This moves the atomic update to the
lowest level of the call graph.
Also, factor out the code that creates a ConfigObj based on various
inputs (archive dir, cert locations, and renewal params). This allows
cleanly reusing it across the "update" and "create new" paths.
Now the "create new config" code path doesn't have to do any renaming,
and the "update config" code path can assume the input file exists (and
error if it doesn't).
This will make it easier and cleaner to reuse the config-writing code in
#10377.
Part of #10355
this is the final part of
https://github.com/certbot/certbot/issues/10195. this fixes
https://github.com/certbot/certbot/issues/10195
the changes in the first commit were done automatically with the
command:
```
ruff check --fix --extend-select UP006 --unsafe-fixes
```
the second commit configures ruff to check for this to avoid regressions
thanks for bearing with me thru these somewhat large automatically
generated PRs ohemorange 🙏
this is another part of https://github.com/certbot/certbot/issues/10195
these changes were done automatically with the command:
```
ruff check --fix --extend-select UP006 --unsafe-fixes certbot-apache
```
This PR is modeled on https://github.com/certbot/certbot/pull/10373/,
and is part of https://github.com/certbot/certbot/issues/10183.
relevant requirements:
use_tls13 >= 1.13.0
session_tix_off implemented: nginx >= 1.5.9 and openssl_version >=
1.0.2l
session tix off by default: >= 1.23.2
oldest non-deprecated major distros nginx versions:
debian 11 1.18.0
epel 8 1.23.1
ubuntu 22.04 1.18.0
Therefore, we can stop testing for use tls 13 and session tix off
allowed, with the same caveat as [in this
comment](https://github.com/certbot/certbot/pull/10373#issuecomment-3134101604).
While we could add a new split for configs that don't require session
tickets off to be set explicitly since it's the default, I don't think
it's worth doing now. I added a note in the comments about this.
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
this is another part of https://github.com/certbot/certbot/issues/10195
the first commit was done automatically with the command:
```
ruff check --fix --extend-select UP006 --unsafe-fixes certbot/src/certbot/_internal
```
this was unfortunately insufficient as it left a line in webroot.py with
over 100 characters. i fixed this manually in my second commit
this is another part of https://github.com/certbot/certbot/issues/10195
these changes were done automatically with the commands:
```
ruff check --fix --extend-select UP006 --unsafe-fixes certbot
git checkout certbot/src/certbot/_internal
```
fixing up the files under certbot/src/certbot/_internal will be done in
another PR
This sets up a `pyproject.toml` file for certbot, initially generated
[using](https://hatch.pypa.io/latest/intro/#existing-project) `hatch new
--init` and modifying from there. Since we dynamically require acme of a
matching version, I kept that around in `setup.py` to do the simplest
thing in this PR.
Other possible (future) implementations:
- setuptools has a beta implementation to read from a
`requirements.txt`. we could generate one of those.
- we could just hardcode it and update at release time. I like not
having to keep the version up to date in various places but maybe it's
actually fine
- something something integration with poetry pinning?
I think the syntax for setting version dynamically in `pyproject.toml`
is much nicer than what we do in `setup.py`. It's a little silly to do
it there after we've bothered to calculate it, but I put it there in the
hopes of being able to remove it from `setup.py` someday/somehow.
It would be nice to access the version dynamically set in
`pyproject.toml` in `setup.py`, but I do not think it is likely to be
possible.
Here are some useful links regarding this migrations:
- [How to modernize a `setup.py` based
project?](https://packaging.python.org/en/latest/guides/modernize-setup-py-project/)
- [Writing your
`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/)
- [Configuring setuptools using `pyproject.toml` files](
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)
- [`pyproject.toml`
specification](https://packaging.python.org/en/latest/specifications/pyproject-toml/)
- [Platform compatibility
tags](https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/)
this is in response to
https://github.com/certbot/certbot/pull/10399#issuecomment-3166305086
this PR does two things:
1. it clarifies what is meant by "build dependencies" in DESIGN.md
2. fixes our workaround for
https://github.com/python-poetry/poetry/issues/4103 which broke when we
moved most of our code under `src` directories. i kept the previous `rm
-rf ${REPO_ROOT}/*/*.egg-info` line around for `letstest` and to
hopefully add some robustness for us if we ever move our code around
again
Alternative implementation for #7908.
In this PR:
- set up ruff in CI (add to `tox.ini`, mark dep in `certbot/setup.py`)
- add a `ruff.toml` that ignores particularly annoying errors. I think
line length isn't actually necessary to set with this workflow since
we're not checking it but putting it there for future usage.
- either fix or ignore the rest of the errors that come with the default
linting configuration. fixed errors are mostly unused variables. ignored
are usually where we're doing weird import things for a specific reason.
Part of #10355.
This allows combining the NamespaceConfig object with a cache of ACME
clients, so we don't have to make the whole large NamespaceConfig object
available all the way down the renewal call stack.
The new AriClientPool is responsible for instantiating ACME clients for
the purpose of ARI fetching. It provides only a simple user agent,
listing the Certbot version. The only CLI flag it observes is
--no-verify-ssl.
