* add amazon linux to auto targets
* disable updates outside of debian and rhel
* test certbot-auto with disabled upgrades
* try new approach to testing
* remove bad space
* tweak error text
* add changelog entry
* fix bad certbot-auto commit
* test new error text
* update changelog
* update error text
* Remove deprecated options as early as possible using an explicit list
* add deprecated options to cli init import list
* use correct dict comprehension syntax for py3
* lint
* add test for renewal reconstitution code
* add test to ensure we're not saving deprecated values
* comment code
Fixes#8389#8584.
This PR makes the necessary modifications to officially drop Python 2 support in the Certbot project.
I did not remove the specific Python 2 compatibility branches that has been added in various places in the codebase, to reduce the size of this PR and this will be done in a future one
* Update classifiers and python_requires in setup.py
* Remove warnings about Python 2 deprecation
* Remove Azure jobs on Python 2.7
* Remove references to python 2 in documentation
* Pin dnspython to 2.1.0
* Update changelog
* Remove warning ignore
Fixes https://github.com/certbot/certbot/issues/8580.
With this PR, it should now be possible to run the oldest tests natively on Linux, at least when using an older version of Python 3, which hasn't been possible in a long time. Unfortunately, this isn't possible on macOS which I opened https://github.com/certbot/certbot/issues/8589 to track.
You can see the full test suite running with these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=3283&view=results.
I took the version numbers for the packages I updated by searching for the oldest version of the dependency I think we should try and support based on the updated comments at the top of `oldest_constraints.txt`. While kind of annoying, I think it'd be a good idea for the reviewer to double check that I didn't make a mistake with the versions I used here.
To find these versions, I used https://packages.ubuntu.com, https://packages.debian.org, and a CentOS 7 Docker image with EPEL 7 installed. For the latter, not all packages are available in Python 3 yet (which is something Certbot's EPEL package maintainers are working on) and in that case I didn't worry about the system because I think they can/will package the newest version available. If they end up hitting any issues here when trying to package Certbot on Python 3, we can always work with them to fix it.
* remove py27 from oldest name
* update min cryptography version
* remove run_oldest_tests.sh
* upgrade setuptools and pyopenssl
* update cffi, pyparsing, and idna
* expand oldest_constraints comments
* clarify oldest comment
* update min configobj version
* update min parsedatetime version
* quote tox env name
* use Python 3.6 in the oldest tests
* use Python 3.6 for oldest integration tests
* properly pin asn1crypto
* update min six version
* set basepython for a nicer error message
* remove outdated python 2 oldest constraints
* --preferred-chain: only match root name
Currently, when certbot is given the `--preferred-chain='Some Name'`
flag, it iterates through all alternate chains offered by the ACME
server until it finds any certificate which has `'Some Name'` as its
Issuer Common Name. Unfortunately, this means that if the desired
alternate chain is a strict subset of any earlier chain (e.g. the
default chain is 'EE <-- Int <-- Root1 <-- Root2', but the desired
chain is 'EE <-- Int <-- Root1'), there is no name which can be
provided by the user which will allow the client to select the desired
chain.
This change makes it so that the `find_chain_with_issuer` logic only
cares about the Issuer Common Name found in the last certificate in
each chain. In the example above, the user would then be able to get
their desired chain by specifying `--preferred-chain='Root1'`: although
that name appears in the default chain, it does not appear in the
highest certificate of that chain.
This change is technically backwards-incompatible. However, the only
advice that has been given to users of certbot (and the only usecase
that we believe has existed so far) involved setting the flag to a
value that is the name of a root, not an intermediate, so we don't
expect any real-world configurations or use-cases to be broken.
Fixes#8577
* Update interfaces.py
* test: certbot-ci crash due to no p521 on boulder
The bugfix in #8598 added an integration test to request a certificate
for an EC P-521 key, which is unsupported when ACME_SERVER=boulder,
failing our nightly integration tests.
* add an integration test for all EC curves
* Fix EC curve name typo in crypto_util
Fix typo of secp521r1 in crypto util module.
- secp521r1 is to be supported by certbot, but a typo of "SECP521R1" in the input validation section of the make_key function results in an error being thrown
* Add myself to authors.md
Add myself to authors.md ^^
* Add test for secp521r1 key generation
Add test for secp521r1 key generation to cli-tests
For some time, SUSE distributions have had both an apachectl
executable and an apache2ctl compat symlink so both could be used
but apachectl is preferred since that's the official upstream name.
This is currently the case in SLE 15 SP2 and openSUSE Leap 15.2
(and every release since SLE 12 SP1)
OTOH, openSUSE Tumbleweed removed the apache2ctl compat symlink
some weeks ago and both SLE/Leap will follow in one of the next
releases so it's better to change certbot to use the official name,
apachectl.
* clean up some Sphinx warnings
* first attempt at a doc-test pipeline job
* fix formatting
* fix test name
* set env for bash
* try bash vs script
* maybe it didn't like me setting 'PATH'...derp
* drop use of venv
* sphinx-build isn't a py script
* try activating venv
* docs: remove unused html_static tags
* clean up final sphinx build errors for certbot
* clean up final sphinx build errors for acme
* better names for docs pipeline
* fix spelling
* add docs_extras to setup.py
* remove temp doc-testing pipeline; add template to main.yml
* rearrange pipeline execution; run sphinx builds in one job
* add documentation note to compat.os
* add uninstall.rst as a sub-toctree to avoid build error
Now that we have a new pipstrap script with recent version of pip, dependencies for Windows can be resolved correctly on Python 3.8.
This PR enables tests on Python 3.8, and package Certbot for Windows on Python 3.8 also. I do not move up to Python 3.9 since some dependencies (`cryptography`, `pynacl`) do not provide wheels for Python 3.9 yet on Windows, which would require a complete C++ build system to compile them.
* Enable windows tests on Python 3.8 and package it on Python 3.8 also.
* Upgrade pynsist, nsis and pywin32, remove old workarounds
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Fix plugin param type in updater
The command used to do this was:
sed -i 's/\(:type .*plugins:\) `list` of `str`/\1 certbot._internal.plugins.disco.PluginsRegistry/g' certbot/certbot/_internal/updater.py
* fix plugin param type in main.py
The command used to do this was:
sed -i 's/\(:type .*plugins:\) `list` of `str`/\1 plugins_disco.PluginsRegistry/g' certbot/certbot/_internal/main.py
The method `os.readlink()` has a significant behavior change with Python 3.8+ on Windows.
Starting with this version, it will return the resolved path in its "extended-style" form unconditionally, a form which allows to use more than 259 characters in a Windows path, and its string representation is prepended with "\\\\?\\".
See https://docs.microsoft.com/fr-fr/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#maximum-path-length-limitation
Problem is that `os.readlink()` does it for any path, including paths that could be represented with the normal form. As a consequence, any string comparison with a path provided in the normal form will fail even if it represents the same path. This makes Certbot partially break on Windows with Python 3.8.
My proposition in this PR is to forbid `os.readlink()`, and provide `certbot.compat.filesystem.readlink()` which serves the same purpose at resolving the pointed path of a link, and has a consistent behavior over supported Python versions.
* Forbid os.readlink()
* Use readlink
* Raise error with long paths on Windows
* Add unit tests
* Update certbot/certbot/compat/filesystem.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Edit certs -> certificates in user-facing text.
To reduce confusion, we should consistently use the full term.
* Edit certs->certificates in more user-facing text.
* fix failing lint (line too long)
* fix typo
Co-authored-by: Jacob Hoffman-Andrews <github@hoffman-andrews.com>
Co-authored-by: Alex Zorin <alex@zorin.id.au>
* Fix TTL mismatch leading to HTTP 412
This PR is a follow up from #8521 where we address the
issue of potentially having a mismatch of TTL when executing
a DNS change (transaction = deletion + additions). Let's say
we have a record `foo.org 30 IN TXT foo-content` with TTL 30s,
when creating challenge or cleaning we might need to perform
a deletion operation in the transaction. Currently certbot
would ask Google API to delete the foo record like this:
`foo.org 60 in TXT foo-content` ignoring the record's original
TTL and using 60s instead. This leads to HTTP 412 as Google would
expect a perfect match of what we want to delete with what it is
on the DNS. See also #8523
* remove ttl from default data to avoid confusions
* Refactor tests and add a missing case
This commit adds a test that covers the case when we are
deleting a TXT record which contains a single rrdatas. Also,
refactoring a couple of tests.
* Make get_existing_txt_rrset documentation more precise about return value
* Add missing assertions in tests.
* fix linting issues
* Mention fix on changelog
* Explain fix around user impact
* Explain what happens when no records are returned
* Update certbot/CHANGELOG.md
* Update certbot/CHANGELOG.md
* update_account: print correct message for -m ""
When -m "" was passed on the CLI, Certbot would print that it updated
the email to '' (an empty string) rather than printing that it removed
the contact details.
This commit also refactors the update_account tests to be a bit more
modern.
* use addCleanup instead of tearDown in tests
* Fix fetch of existing records from Google DNS
There has been many complaints regarding `certbot_dns_google` plugin
failing with:
* HTTP 412 - Precondition not met
* HTTP 409 - Conflict
See #6036. This PR fixes that situation. The bug lies on how we
fetch the TXT records from google. For large amount of records
the Google API paginates the result but we ignore the subsequent
pages and assume that if the record is not in the first response then
it doesn't exist. This leads to either HTTP 409, or HTTP 412 or both.
In this PR we leverage the use of filters on the API to get exactly
the records we are looking for. Apart from fixing the problem stated
above, it has the extra benefit of making the process faster by
reducing the amount of API calls and it doesn't require us to handle
any pagination logic
* Explain changes on CHANGELOG
* Edit AUTHORS.md
* make execute static
* Update certbot/CHANGELOG.md
Being more specific for which plugin this fix bug is meant for.
Co-authored-by: alexzorin <alex@zor.io>
* Fix if expression to be more python-idiomatic
Co-authored-by: alexzorin <alex@zor.io>
* Sort AUTHORS.md
* Simplify tests
Make rrs_mock modeling simpler and refactor
* Revert "Simplify tests"
This reverts commit 9de9623ba7.
* Reimplement conditional mock
We still want to use a conditional mock by make it more
simple to understand by using MagicMock.
* Revert "Sort AUTHORS.md"
This reverts commit b3aa35bcf1.
* Add name in AUTHORS.md
Co-authored-by: alexzorin <alex@zor.io>
Fixes https://github.com/certbot/certbot/issues/8519.
I left the `certbot-auto` docs in `install.rst` to avoid breaking links and to help propagate information about our changes there. I moved it closer to the bottom of the doc though since I think our documentation about OS packages and Docker is more helpful to most people.
* clean up certbot-auto docs
* add more info to changelog
* remove more certbot-auto references
Although Certbot is a classic snap, it shouldn't load Python code from
the host system. This change prevents packages being loaded from the
"user site-packages directory" (PEP-370). i.e. Certbot will no longer
load DNS plugins installed via `pip install --user certbot-dns-*`.
* cli: clean up `certbot renew` summary
- Unduplicate output which was being sent to both stdout and stderr
- Don't use IDisplay.notification to buffer output
- Remove big "DRY RUN" guards above and below, instead change language
to "renewal" or "simulated renewal"
- Reword "Attempting to renew cert ... produced an unexpected error"
to be more concise.
* add newline to docstring
Co-authored-by: ohemorange <ebportnoy@gmail.com>
Co-authored-by: ohemorange <ebportnoy@gmail.com>
Fixes https://github.com/certbot/certbot/issues/8495.
To further explain the problem here, `modify_kwargs_for_default_detection` as called in `add` is simplistic and doesn't always work. See https://github.com/certbot/certbot/issues/6164 for one other example.
In this case, were bitten by the code https://github.com/certbot/certbot/blob/d1e7404358c05734aaf436ef3c9d709029d62b09/certbot/certbot/_internal/cli/helpful.py#L393-L395
The action used for deprecated arguments isn't in `ZERO_ARG_ACTIONS` so it assumes that all deprecated flags take one parameter.
Rather than trying to fix this function (which I think can only realistically be fixed by https://github.com/certbot/certbot/issues/4493), I took the approach that was previously used in `HelpfulArgumentParser.add_deprecated_argument` of bypassing this extra logic entirely. I adapted that function to now call `HelpfulArgumentParser.add` as well for consistency and to make testing easier.
* Rename deprecated arg action class
* Skip extra parsing for deprecated arguments
* Add back test of --manual-public-ip-logging-ok
* Add changelog entry
(cherry picked from commit 5f73274390)
* nginx: fix py2 unicode sandwich
The nginx parser would crash when saving configuraitons containing
Unicode, because py2's `str` type does not support Unicode.
This change fixes that crash by ensuring that a string type supporting
Unicode is used in both Python 2 and Python 3.
* nginx: add unicode to the integration test config
* update CHANGELOG
Erica Portnoy