The manual plugin offers environment variables for its hook called
CERTBOT_DOMAIN and CERTBOT_ALL_DOMAINS. I added CERTBOT_IDENTIFIER and
CERTBOT_ALL_IDENTIFIERS, while keeping the old variables for backwards
compatibility. Certbot will pass IP addresses in the CERTBOT_DOMAIN
environment variable rather than erroring out.
Part of #10346
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
IPv6 addresses in URLs should be enclosed in square brackets.
Note: I chose to fix this by parsing the identifier rather than changing
the method signature. The obvious choice to change the method signature
would be to take `messages.Identifier`. We could even do this backwards
compatibly by taking `str | messages.Identifier`. However, `messages`
imports `challenges`, so referencing `messages.Identifier` here would
require an import loop.
I also considered implementing a new method on, e.g.
messages.Authorization that would take a challenge as a parameter. But
this would be suboptimal because the `uri` method really is specific to
the http-01 challenge type, so it's nice to have it implemented only on
the relevant class.
Fixes#10506.
When --webroot-path was specified multiple times, Certbot was erroring
with `DNSName SAN compared to non-SAN`. That's because, in the
_WebrootPathAction that builds `namespace.webroot_path`, we were passing
`domain` (type `san.DNSName`) as the keys. The other code that modifies
or accesses `namespace.webroot_path` expects the keys to be of type
`str`. In particular `webroot.Authenticator._set_webroots` does:
```python
for achall in achalls:
self.conf("map").setdefault(achall.domain, webroot_path)
```
Where `achall.domain` is a `str`.
Two existing unittests would have caught this: `test_multiwebroot` and
`test_webroot_map_partial_without_perform`. However, they faked out the
parsing of the `--domains` flag, and that faked out code was not updated
in #10468. Since this bug is caused by an interaction between the types
produced by the `--domains` flag and those produced by the
`--webroot-path` flag, the tests failed to catch the problem. I've
updated the tests and confirmed that they fail before the fix is
applied.
This field is optional to maintain backwards compatibility. Note that
`AnnotatedChallenge` inherits from `jose.ImmutableMap`, which has a
[check in
__init__](https://github.com/certbot/josepy/blob/4b747476703fe4fff1aaccd76ebe570698bbf4f0/src/josepy/util.py#L125-L131)
that all slots are provided. That check would not allow us to do a
backwards-compatible addition, so I implemented an `__init__` for each
of these subclasses that fills the fields without calling the parent
`__init__`, and so doesn't hit an error when `identifier` is absent.
I chose to use `acme.messages.Identifier` rather than
`certbot._internal.san.SAN` here because these are wrapped ACME types,
so they should use the ACME representation. Also, `AnnotatedChallenge`
is passed to plugins, so we need to pass a type that the plugins can
understand.
Additionally, `domain` is marked as deprecated.
Part of #10346
/cc @bmw, who noticed the issue with `AnnotatedChallenge`
[here](https://github.com/certbot/certbot/pull/10468#issuecomment-3403294394)
and provided additional feedback
[here](https://github.com/jsha/certbot/pull/2#issuecomment-3534895793).
Note that there's still some work to do to finish excising `domain`
assumptions from this portion of the code.
---------
Co-authored-by: ohemorange <ebportnoy@gmail.com>
In #10478 we added a `san.SAN` class, with two subclasses `san.DNSName`
and `san.IPAddress`, so we can carry type information about identifiers
through the Certbot code. This PR plumbs through those types in most
Certbot-internal code. Note that this does not change the `acme` module,
which uses `messages.Identifier`. It also tries to leave alone the code
paths into plugins.
This does not add a CLI flag to request an IP address certificate. That
will be in a followup PR.
Part of #10346
Contains san.DNSName, san.IPAddress, and a parent class san.SAN.
Split out from #10468 as a standalone PR. To see examples of how it's
intended to be used, please see that PR.
The constructor for DNSName incorporates the same validation done in
`enforce_domain_sanity`, and the tests from `enforce_domain_sanity` are
copied here as well. The goal is to delete `enforce_domain_sanity`
entirely as part of #10468.
In support of #10346.
This uses the new `storage.atomic_rewrite` function to write only the
ARI-related value to the config file, leaving other values the same.
Updates `storage.atomic_rewrite` to handle the case where the
`"renewalparams"` section might be empty (which occurs in the new
unittests), and adds some comments.
Note: `atomic_rewrite` is mocked out as a no-op in
`test_resilient_ari_directory_fetches` and `test_renew_before_expiry`
because those tests have a simplistic config object on their
mock_renewable_cert that doesn't include a filename. If we try to write
the config in those tests we'd get an error (trying to write to `None`).
Since those tests aren't intended to test the "store and obey
Retry-After" behavior, I figured it's reasonable to skip it in the name
of testing; but of course, open to idea about the best way to navigate
this.
Part of #10355
The responsibility for atomic updates to a config file was previously
spread across different functions. This moves the atomic update to the
lowest level of the call graph.
Also, factor out the code that creates a ConfigObj based on various
inputs (archive dir, cert locations, and renewal params). This allows
cleanly reusing it across the "update" and "create new" paths.
Now the "create new config" code path doesn't have to do any renaming,
and the "update config" code path can assume the input file exists (and
error if it doesn't).
This will make it easier and cleaner to reuse the config-writing code in
#10377.
Part of #10355
Part of #10355.
This allows combining the NamespaceConfig object with a cache of ACME
clients, so we don't have to make the whole large NamespaceConfig object
available all the way down the renewal call stack.
The new AriClientPool is responsible for instantiating ACME clients for
the purpose of ARI fetching. It provides only a simple user agent,
listing the Certbot version. The only CLI flag it observes is
--no-verify-ssl.
Previously, we were constructing an ACME client for ARI checking that
used the global value for `server`, not the one recorded in a lineage's
renewal file.
This resulted in errors in the logs and failure to observe ARI for
lineages that used a non-default `--server` (e.g. staging or non-Let's
Encrypt CAs).
---------
Co-authored-by: ohemorange <ebportnoy@gmail.com>
This depends on a pending Pebble pull request and so will fail
integration tests until/unless that lands:
https://github.com/letsencrypt/pebble/pull/501
However, I'd appreciate some eyes on this PR in this regard: is the
interface we're using in Pebble useful and appropriate? If not, we can
adjust the Pebble PR.
Inspired based on conversation on
https://github.com/certbot/certbot/pull/10307, but note that this just
tests the general case; it does not test the "default server differs
from lineage server" case yet; when I try adding that I get some bugs
that may reflect a problem in #10307 I need to fix (or may reflect that
I need to inhibit the `--server` flag rather than trying to override it
late in the command line).
Follow-up to #10241. The acme module code is mostly the same, except the
switch to return a tuple containing Retry-After.
This includes the CLI-side work to call out to the new `renewal_time`
method when checking for renewal.
I moved `should_autorenew` from `storage.py` into `renewal.py`, where it
fits better (and also this solves an import cycle problem). To make the
edits more visible I split this into one commit for the move and [one
commit for the subsequent
edits](https://github.com/certbot/certbot/pull/10272/commits/4e137d9b00fb0a299723978d7a289b881716d36b#diff-fad906e31304c767d620bfd243f4c7adf1e63a3420fd634ee57a0f6651c182cf).
This does not yet attempt to store the Retry-After info, or failure
retries, in renewal configs. I figured since that's a pretty big chunk
of work and design on its own, I wanted to get interim feedback as is. I
think this PR would be okay to land with the current default crons /
systemd timers that run twice a day. I think we should implement storage
of retry information before increasing the frequency of runs. And if the
team would like to hold off on landing any ARI until that storage is
done, I'm good with that too. 👍🏻
Previously we defaulted to renewing at 30 days before expiry, and
allowed users to customize the config file to set a different value.
Instead, we should renew when 1/3 of the lifetime is left, or for
shorter certificates (<10 days), when 1/2 of the lifetime is left.
This still allows explicitly configured values to take precedence.
---------
Co-authored-by: Will Greenberg <ifnspifn@gmail.com>
Give better instructions on running all unittests, and on running
specific test cases.
Replace `python` with `python3` in venv setup invocations because some
systems don't have a plain `python` command.
---------
Co-authored-by: ohemorange <ebportnoy@gmail.com>
We strongly encouraged providing an email address because we wanted
people to get expiration notices to ensure that even if their Certbot
install broke, they could fix it before their site goes down.
Now that Let's Encrypt is getting rid of expiration notices
(https://letsencrypt.org/2025/01/22/ending-expiration-emails/), we can
remove some of the encouragement, providing a smoother user experience.
Recognizes the profiles map in the "meta" section of directory.
Allows sending a "profile" field in order objects.
Adds an optional "profile" parameter to new_order in client.py.
Related to #10194.
* Edit certs -> certificates in user-facing text.
To reduce confusion, we should consistently use the full term.
* Edit certs->certificates in more user-facing text.
* fix failing lint (line too long)
* fix typo
Co-authored-by: Jacob Hoffman-Andrews <github@hoffman-andrews.com>
Co-authored-by: Alex Zorin <alex@zorin.id.au>
* Remove unnecessary account ID match check.
Right now the Account object calculates an ID using md5. This is
unnecessary and causes problems on FIPS systems that forbid md5. It's
just as good to pick a random series of bytes for the ID, since the ID
gets read out of renewal/foo.conf.
However, if we switched the algorithm right now, we could wind up
breaking forward compatibility / downgradeability, since older versions
would run into this check.
Removing this check now lays the ground to change the ID-calculation
algorithm in the future.
Related to #1948 and
https://github.com/certbot/certbot/pull/1013#issuecomment-149983479.
* Remove test.
* Remove unused import.
The value for FAKE_DNS is now always the same because Boulder's
docker-compose hardcodes it, so skip some sed.
Set a time limit on how long we'll wait for boulder to come up.
This adds two new classes in messages: Order and OrderResource. It also adds methods to ClientV2 to create orders, and poll orders then request issuance.
The CSR is stored on the OrderResource so it can be carried along and submitted when it's time to finalize the order.
* Change "Attempting to parse" warning to info.
This message shows up on every renewal run when the config was updated
by a newer version of Certbot than the one being run. For instance, if a
user has the certbot packages installed from PPA (currently 0.18.2), but
runs certbot-auto once to try out the latest version (0.21.1), they will
start getting this message via email every 12 hours.
* Implement ACMEv2 signing of POST bodies.
* Add account, and make acme_version explicit.
* Remove separate NewAccount.
* Rename to add v2.
* Add terms_of_service_agreed.
* Split out wrap_in_jws_v2 test.
* Re-add too-many-public-methods.
* Split Client into ClientBase / Client / ClientV2
* Use camelCase for newAccount.
* Make acme_version optional parameter on .post().
This allows us to instantiate a ClientNetwork before knowing the version.
* Add kid unconditionally.
This produces a super-long log line that wraps to 30-60 lines, depending on
screen width. Even though it's just at debug level, it clutters up the integration
test output without providing proportional debugging value.
* Remove Default Detector log line.
This produces about 30 lines of log output. Even though it's just at debug
level, it clutters up the integration test output without providing proportional
debugging value.
* Add more useful logs.
pip install generates a lot of lines of output that make it harder to see what
tox is running in general. This adds the -q flag to pip install.
At the same time, add `set -x` in install_and_test.sh and pip_install.sh so they
echo the commands they are running. This makes it a little clearer what's going
on in tests.
I didn't put `set -x` at the top or in the shebang, because moving it lower lets
us avoid echoing some of the messy if/then setup statements in these scripts,
which focussed attention on the pip install command.
These tests are retained in the test-everything branch, which has a Travis cron
job to run nightly.
Removing these speeds up the Certbot Travis builds dramatically for two reasons:
- The Boulder integration tests are slow (10-12 minutes), and it's exceedingly
rare for them to fail on one Python environment but not another.
- The macOS tests take a very long time to run, because they need to wait for
build slots on the limited number of macOS instances, which are often in high
demand.
Fixes#4535
Extracts the relevant fields using a regex. We considered catching
specific exception types, and referencing their fields, but the types
raised by `requests` are not well documented and may not be long
term stable. If the regex fails to match, for instance due to a change
in the exception message, the new exception message will just be
passed through.
* Add an account deactivate utility script.
This is handy if you created an account with a tool other than Certbot, and want
to deactivate the account.
* Move deactivate.py to tools.
* Add test for ConflictError.
* Fix lint error.
* Document how to set server.
* Move 'jwk' and 'alg' fields to protected header.
Previously, these were in the unprotected JWS header, which Boulder currently
allows. However, the next version of the spec doesn't allow anything in the
unprotected header. Moving these fields now allows server implementers who are
implementing the Certbot/Boulder version of ACME
(https://github.com/letsencrypt/boulder/blob/master/docs/acme-divergences.md) to
use JOSE libraries that don't support unprotected headers.
Fixes#4417.
* Only protect existing headers.
* Add url and kid to jws.
This will be required in order to implement the latest ACME spec, which uses
these protected header fields.
* Add comments and fix lint.
* Enforce mutual exclusivity of jwk and kid.
It's relatively finicky to make a CSR appropriate to pass to poll_and_request_issuance. I think most users want to be able to give a list of domains and a private key, and get back a CSR. This branch adds that functionality to crypto_util.
Note that the two new functions take arguments, and return values, as PEM-encoded buffers. This is a departure from some existing ACME interfaces that take PyOpenSSL types. I've discussed with the Certbot team, and we agree that this is broadly the direction the ACME API should take, so that users of the module don't need to import PyOpenSSL themselves, or use its primitives.
* Add make_csr.
* accept privkey
* Tweak API.
* Remove make_csr from certbot package.
* Skip test in older Pythons.
* Move get_Extensions call under protection.
* Remove assertIn because not backwards-compatible.
* Fix encoding, and use PEM.
* Fix test
* Fix tests on py35.
* Fix error in test.
* Make import_csr_file always return PEM.
Also delete get_sans_from_csr (unused) and get_names_from_csr (newly unused).
* Fix function doc.
* Fix indent
* Fix call of obtain_certificate_from_Csr
* lint
* Handle review feedback.
* Fix test.
Might help debug #4363.
Also:
make "bash" vs "sh" explicit
move the paranoia flags (-ex) from the shebang into the body
add -u (fail on unset variables)
change _common to work with -u
remove some env vars that were no longer used
remove shebang from _common.sh because it's meant to be sourced, not run
Move "Writing your own plugin" under Code components and layout, with the other
plugin docs.
Include instructions on how to install a plugin into a virtualenv and how to check
for its presence.
Document that users can install third-party plugins systemwide, but not with
certbot-auto.
Remove obsolete information from Authenticators section and make the section
more informative.
Remove IDisplay sub-section since it repeats information in the main "Plugin
architecture" section.
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
* Revert "Revert "Remove Link rel=next for authzs and new-certs." (#4277)"
This reverts commit 11ec1eb911.
* Save new_authzr_uri with account for older clients.
* Add test that new_authzr_uri exists in regr.
* Restore backwards compatibility for new_authzr_uri.
* Fix account_test.
* Add test for deprecated URI argument to request_challenges.
* Review feedback.
* Fix test
* Add omitempty to new_cert_uri.
These are from certbot/certbot#4174
Add more documentation, and help for NoCredentialsError.
Allow multiple DNS records to be provisioned at once and waited for together.
Fix doc strings to use "Certbot" instead of "Let's Encrypt."
Set TTL to 0.
Create a single boto3 session rather than one per API call.
Use pagination in Route53 API in case there are many domains.
Add a maximum wait time for update to propagate (10 minutes).
Correct tagging format.
Add request for random offsets for renewal.
Make all bulleted lists consistent.
Remove obsolete `letsencrypt` package for Fedora.
Remove discouraged letshelp-certbot package.
* Remove UnexpectedUpdate exceptions.
These exceptions trigger when the server sends the client back an object with a
field that doesn't exactly match what the client previously sent.
This causes unnecessary breakage in various cases, doesn't prevent any problems,
and isn't required by spec.
* Back out all UnexpectedUpdate removals except registration update.
Recommend faster-running commands with clearer output first, building up to
slow, noisy, `tox` last.
Remove outdated Boulder testing advice, and outdated Boulder testing scripts
(Vagrantfile, mac-bootstrap.sh, boulder-start.sh). Update Boulder testing
section.
Fill out "directory structure" section of contributing guide a little better.
The #letsencrypt channel on Freenode is much more active, and is the defacto place for questions about Certbot. Users posting questions on #certbot on OFTC are not getting prompt answers.
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
Per @patf at https://community.letsencrypt.org/t/-/21318/2:
> [The message about account recovery] is not really true anymore - account recovery via email was planned to be added at some point, but has been removed from the spec since.
* Fix non-ASCII domain check.
Previously, the code would convert to utf-8, check for non-ASCII, and then try
to use .format() to interpolate the result into an error message. This would
generate a second error that would cause the whole message to get dropped, and
the program to silently exit. The problem can be succinctly observed like so:
$ python
>>> "{0}".format("ウェブ.crud.net")
'\xe3\x82\xa6\xe3\x82\xa7\xe3\x83\x96.crud.net'
>>> "{0}".format(u"ウェブ.crud.net")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)
Note for the curious: This problem only seems to happen with .format():
>>> "%s" % ("ウェブ.crud.net")
'\xe3\x82\xa6\xe3\x82\xa7\xe3\x83\x96.crud.net'
>>> "%s" % (u"ウェブ.crud.net")
u'\u30a6\u30a7\u30d6.crud.net'
Print request and response bodies with newlines, rather than all on one line.
Remove "Omitted empty field" log, which gets logged meaninglessly for every JSON
serialization.
Remove duplicated logging of responses.
Log the base64 version of the nonce, rather than turning it into bytes and
logging the backslash-escaped version of those bytes.
Only pass -vv in tests.
* Update Where Are My Certs section.
This combines the `cert.pem` and `chain.pem` sections into a single paragraph,
making it clearer that they are closely connected. It also adds text indicating
that they are less common and moves them below the section for `fullchain.pem`.
* Update "Getting Help" section.
* Add link to document missing intermediate.
* Remove incorrect line about ordering.
Also remove "(as the filename suggests)," and clarify file ordering in the
fullchain.pem section.
* Update Where Are My Certs section.
This combines the `cert.pem` and `chain.pem` sections into a single paragraph,
making it clearer that they are closely connected. It also adds text indicating
that they are less common and moves them below the section for `fullchain.pem`.
* Update "Getting Help" section.
* Add link to document missing intermediate.
* Remove incorrect line about ordering.
Also remove "(as the filename suggests)," and clarify file ordering in the
fullchain.pem section.
This is a quick fix to https://github.com/certbot/certbot/issues/3382, so that
Python log rotation is less likely to delete files before logrotate can get to
them. Specifically, if logrotate's config says to preserve fewer than 1000 logs,
it will do the right thing.
Right now the ACME client checks that the returned registration matches the
registation posted, but there's no guarantee this will always be the case, and
this only introduces unnecessary fragility.
Don't run pep8 for directories that we don't actually enforce pep8 on.
Install dependencies with -q.
Don't print reports, they make it hard to find the actual errors.
Remove deprecated fields from acme .pylintrc, they cause unnecessary messages
about deprecation.
