Fixes https://github.com/certbot/certbot/issues/8495.
To further explain the problem here, `modify_kwargs_for_default_detection` as called in `add` is simplistic and doesn't always work. See https://github.com/certbot/certbot/issues/6164 for one other example.
In this case, were bitten by the code https://github.com/certbot/certbot/blob/d1e7404358c05734aaf436ef3c9d709029d62b09/certbot/certbot/_internal/cli/helpful.py#L393-L395
The action used for deprecated arguments isn't in `ZERO_ARG_ACTIONS` so it assumes that all deprecated flags take one parameter.
Rather than trying to fix this function (which I think can only realistically be fixed by https://github.com/certbot/certbot/issues/4493), I took the approach that was previously used in `HelpfulArgumentParser.add_deprecated_argument` of bypassing this extra logic entirely. I adapted that function to now call `HelpfulArgumentParser.add` as well for consistency and to make testing easier.
* Rename deprecated arg action class
* Skip extra parsing for deprecated arguments
* Add back test of --manual-public-ip-logging-ok
* Add changelog entry
(cherry picked from commit 5f73274390)
* nginx: fix py2 unicode sandwich
The nginx parser would crash when saving configuraitons containing
Unicode, because py2's `str` type does not support Unicode.
This change fixes that crash by ensuring that a string type supporting
Unicode is used in both Python 2 and Python 3.
* nginx: add unicode to the integration test config
* update CHANGELOG
Fixes https://github.com/certbot/certbot/issues/8134.
* Test on Python 3.9.
* Mention Python 3.9 support in changelog.
* s/\( *'Pro.*3\.\)8\(',\)/\18\2\n\19\2/
* undo changes to tox.ini
* Move more tests to Python 3.9
* Update PyYAML and packages which pinned it back
* Upgrade typed-ast
* Use <= to "pin" dnspython
* Fix lint by telling pylint it cannot be trusted
* Disable mypy on RFC plugin
* add comment about <= support
* Add timeout to DNS query function calls
* Modify tests to account for new timeout variable
* Add change to CHANGELOG
* Add `dns.exception.Timeout` to exception handler
* Move changelog to 1.10.0
Fixes https://github.com/certbot/certbot/issues/8292.
This uses the same approach that worked well for us in https://github.com/certbot/certbot/pull/7926. I'm sure we could delete more code or refactor things here, but I think we should make the most conservative changes we can to certbot-auto until we can just delete the entire thing.
I ran the full test suite on these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=2773&view=results and manually tested things on OpenSUSE and it worked as expected. certbot-auto refused to create new installations and refused to update old ones while continuing to allow the old version of Certbot to run.
* Deprecate cb-auto outside of Debian and RHEL.
* Don't deprecate Amazon Linux yet.
The ErrorHandler context manager could produce very verbose CLI output
when handling long exception chains (PIP 3134 enhanced reporting).
Rather than logging every exception with its traceback to the CLI, this
commit changes ErrorHandler so that only the final exception in the
chain, without traceback, is logged to the CLI.
This is consistent with a previous change made in the global except
hook (#8000).
* nginx: reduced CLI logging when reloading nginx
Hides the output of `nginx -s reload` from the CLI, moving it to
debug-level logging.
Additionally, fixes an issue where Certbot did not properly capture the
output of the nginx reload and restart commands.
Fixes#8231
* remove leftover debugging
* reorder CHANGELOG
* don't use bare asserts
Fixes#7585
This PR removes the specific configuration to configure the test runner included in `setuptools` to use pytest, the deprecated parameters related to setuptools testing in `setup.py`, and update the packaging guide to use `python -m pytest` instead of `python setup.py test`.
The farm test `test_sdist.sh` is also updated to use directly pytest. This test is designed to reproduce the steps used by OS integrators when they package `certbot`, and ensure that we are not breaking something that will impact their work. We discussed with integrators from RHEL/CentOS and Debian, and they are fine with us testing sdist directly with pytest.
One execution of the `test_sdist.sh` farm test with the modifications made by this PR can be seen here: https://dev.azure.com/certbot/certbot/_build/results?buildId=2606&view=results
* Remove setuptools deprecated features about testing
* Updating packaging guide
* Add changelog entry
This commit fixes an issue with the nginx parser where it would perform
case-sensitive matching against server_name.
This would cause the authenticator and installer to ignore existing
virtualhosts containing uppercase characters, resulting in duplicate
virtualhosts and broken configurations.
"Exact" and "wildcard" matching is now case-insensitive. Regex-based
matching will continue to respect the case mode of the pattern.
Fixes#6776.
* snap: Fix "stack smashing" error in wrapper
certbot.wrapper had implicit dependencies on sed, awk and coreutils,
which were being accidentally provided through the host system. Because
certbot.wrapper modifies LD_LIBRARY_PATH, this was causing some systems
to load an incompatible combination of shared libraries, resulting sed
crashing.
This commit reduces the dependencies of this script to just gawk, and
explicitly stages it as part of the Certbot snap.
It additionally moves invocations of all host system programs to a
moment prior to the modification of LD_LIBRARY_PATH, and the invocation
of snapped programs to after the modification.
Fixes#8245
* snap: Don't modify LD_LIBRARY_PATH
* leftover tracing
* snap: revert curl/jq in wrapper, use gawk for now
Fixing a mistake in pull request #8212 where I recorded my changes in an already released version 😳.
- Moving new changes out of a previous changelog and into the next
releases' changelog
* Allow user to remove email using update command
Fixes#3162. Slight change to control flow to replace current email
addresses with an empty list. Also add appropriate result message when
an email is removed.
* Update ACME to allow update to remove fields
- New field type "UnFalseyField" that treats all non-None fields as
non-empty
- Contact changed to new field type to allow sending of empty contact
field
- Certbot update adjusted to use tuple instead of None when empty
- Test updated to check more logic
- Unrelated type hint added to keep pycharm gods happy
* Moved some mocks into decorators
* Restore default to `contact` but do not serialize
- Add `to_partial_json` and `fields_to_partial_json` to Registration
- Store private variable noting if the value of the `contact` field was
provided by the user.
- Change message when updating without email to reflect removal of
all contact info.
- Add note in changelog that `update_account` with the
`--register-unsafely-without-email` flag will remove contact
from an account.
* Reverse logic for field handling on serialization
Now forcably add contact when serilizing, but go back to base `jose`
field type.
* Responding to Review
- change out of date name
- update several comments
- update `from_data` function of `Registration`
- Update test to remove superfluous mock
* Responding to review
- Change comments to make from_data more clear
- Remove code worried about None (omitempty has got my back)
- Update test to be more reliable
- Add typing import with comment to avoid pylint bug
* nginx: add --nginx-sleep-seconds
As described in #7422, reloading nginx is an asynchronous process and
Certbot does not know when it is complete. In an environment where this
reload takes a long time, the nginx plugin suffers from an issue where
it responds to and fails the ACME challenge before the nginx server is
ready to serve it.
Following the discussion in a previous PR #7740, this commit introduces
a new flag, --nginx-sleep-seconds, which may be used to increase the
duration that Certbot will wait for nginx to reload, from its previously
hard-coded value of 1s.
Fixes#7422
* update CHANGELOG
* nginx: update docstring for nginx_restart
Fixes#8149
This PR adds warnings to warn about the incoming deprecation of Python 3.5 in Certbot.
* Add warnings about Python 3.5 deprecation in Certbot
* Update certbot/certbot/__init__.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
«When you add or change DNS zones or records, your changes will now be
reflected at our authoritative nameservers in under 60 seconds. This is
down from the previous “every quarter hour” approach that we had for so
long.» - https://www.linode.com/blog/linode/linode-turns-17/
Fixes#4351
This PR proposes a solution to use the third party plugins with the prefix `pip_package_name:` in the plugin name, plugin specific flags and keys in dns plugin credential files.
A first solution has been proposed in #6372, and a more advanced one in #7026. In #7026 was also added a deprecation warning when the old plugin name `pip_package_name:plugin_name` was used.
However there were some limitations with #7026, in particular the fact that existing flags of type `pip_package_name:dns_plugin_option` or keys like `pip_package_name:key` in dns plugin credential files were not read anymore. This would have led to silent failures during renewals if the configuration was not explicitly updated by the user.
I tried to fix that based on #7026, but the changes needed are complex, and create new problems on their own, like unexpected erasure of values in the renewal configurations.
Instead I try in this PR a new approach: the `PluginsRegistry` in `certbot._internal.plugins.disco` module register two plugins for a given entrypoint refering to a third party plugin when `find_all()` is called:
* one plugin with the name `plugin_name`
* one plugin with the name `pip_package_name:plugin_name` (like before)
This way, every existing configuration continues to work without any change (credentials, renewal configuration, CLI flags). And new configurations can refer to the new plugin name without prefix, and use the approriate CLI flags, credentials without this prefix.
On top of it I added the deprecation path given in #7026 (thanks @coldfix!):
* the plugin named `pip_package_name:plugin_name` is hidden from `certbot plugins` output
* the help for this plugin is still displayed, and a deprecation warning is displayed in the description
* when invoked, the same deprecation warning is displayed in the terminal
* Support both prefixed and not prefix third party plugins
* Adapt tests
* Add deprecation path
* Named parameters
* Add deprecation warning in CLI
* Add a changelog
* acme: add support for alternative cert. chains
* certbot: add --preferred-chain
* remove support for issuer SKI matching
* show --preferred-chain in "run" help
* warn if no chain matched and it's not a dry-run
* fix existing failing tests
* add unit, integration tests
* bump acme dependency to dev version
* simplify test to avoid py2.7 recursion bug
* add preferred_chain to STR_CONFIG_ITEMS
* reduce preferred_chain warning to info level
* acme: fix some docstrings in .messages
* certbot: fix docstring in crypto_util
* try to fix certbot-nginx acme dep problem
Fixes#8093.
This PR modifies and audits all uses of `subprocess` and `Popen` outside of tests, `certbot-ci/`, `certbot-compatibility-test/`, `letsencrypt-auto-source/`, `tools/`, and `windows-installer/`. Calls to outside programs have their `env` modified to remove the `SNAP` components of paths, if they exist. This includes any calls made from hooks, calls to `apachectl` and `nginx`, and to `openssl` from `ocsp.py`.
For testing manually, rsync flags will look something like:
```
rsync -avzhe ssh root@focal.domain:/home/certbot/certbot/certbot_*_amd64.snap .
rsync -avzhe ssh certbot_*_amd64.snap root@centos7.domain:/root/certbot/
```
With these modifications, `certbot plugins --prepare` now passes on Centos 7.
If I'm wrong and we package the `openssl` binary, the modifications should be removed from `ocsp.py`, and `env` should be passed into `run_script` rather than set internally in its calls from nginx and apache.
One caveat with this approach is the disconnect between why it's a problem (packaging) and where it's solved (internal to Certbot). I considered a wrapping approach, but we'd still have to audit specific calls. I think the best way to address this is robust testing; specifically, running the snap on other systems.
For hooks, all calls will remove the snap paths if they exist. This is probably fine, because even if the hook intends to call back into certbot, it can do that, it'll just create a new snap.
I'm not sure if we need these modifications for the Mac OS X/ Darwin calls, but they can't hurt.
* Add method to plugins util to get env without snap paths
* Use modified environment in Nginx plugin
* Pass through env to certbot.util.run_script
* Use modified environment in Apache plugin
* move env_no_snap_for_external_calls to certbot.util
* Set env internally to run_script, since we use that only to call out
* Add env to mac subprocess calls in certbot.util
* Add env to openssl call in ocsp.py
* Add env for hooks calls in certbot.compat.misc.
* Pass env into execute_command to avoid circular dependency
* Update hook test to assert called with env
* Fix mypy type hint to account for new param
* Change signature to include Optional
* go back to using CERTBOT_PLUGIN_PATH
* no need to modify PYTHONPATH in env
* robustly detect when we're in a snap
* Improve env util fxn docstring
* Update changelog
* Add unit tests for env_no_snap_for_external_calls
* Import compat.os
According to `distutils/version.py`, StrictVersion is pretty strict in
what version numbers to accept:
> A version number consists of two or three dot-separated numeric
> components, with an optional "pre-release" tag on the end. The
> pre-release tag consists of the letter 'a' or 'b' followed by a number.
This assumption already fails for some pretty basic python libraries
itself, like setuptools, also available in `46.1.3.post20200610`, a
completely valid version number according to
https://www.python.org/dev/peps/pep-0440/#post-releases.
There doesn't seem to be a particular reason on why StrictVersion has
been used here, so let's use LooseVersion, to be compatible with these
versions.
Co-authored-by: Adrien Ferrand <adferrand@users.noreply.github.com>
* Tweaks for improved Cloudflare API
* Update docs for dns-cloudflare
* Update tests and changelog
* Fix bad merge
* Fix error code for record add
* Improve error message
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Add snap plugin support
Switch to a PoC branch of certbot that supports the new
CERTBOT_PLUGIN_PATH and wrap the snap to set this variable correctly
based on the content interfaces connected.
(cherry picked from commit 7076a55fd82116d068e2aca7239209b7203917d2)
* Modify certbot.wrapper to append to PYTHONPATH instead of separate CERTBOT_PLUGIN_PATH variable
* Update certbot-wrapper to python3.6 version
* add source field
* Update changelog
* Use bash instead of sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Exit if something goes wrong
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* No leading : when modifying empty PYTHONPATH
* Improve bash handling of PYTHONPATH manipulation
Co-authored-by: Robie Basak <robie.basak@canonical.com>
Co-authored-by: Brad Warren <bmw@eff.org>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This PR gets its root from an observation I did on current version of Certbot (1.3.0): the `renewal-hooks` directory in Certbot configuration directory is created on Windows with write permissions to everybody.
I thought it was a critical bug since this directory contains hooks that are executed by Certbot, and you certainly do not want this folder to be open to any malicious hook that could be inserted by everyone, then executed with administrator privileges by Certbot.
Turns out for this specific problem that the bug is not critical for the hooks, because the scripts are expected to be in subdirectories of `renewal-hooks` (namely `pre`, `post` and `deploy`), and these subdirectories have proper permissions because we set them explicitly when Certbot is starting.
Still, there is a divergence here between Linux and Windows: on Linux all Certbot directories without explicit permissions have at maximum `0o755` permissions by default, while on Windows it is a `0o777` equivalent. It is not an immediate security risk, but it is definitly error-prone, not expected, and so a potential breach in the future if we forget about it.
Root cause is that umask is not existing in Windows. Indeed under Linux the umask defines the default permissions when you create a file or a directory. Python takes that into account, with an API for `os.open` and `os.mkdir` that expose a `mode` parameter with default value of `0o777`. In practice it is never `0o777` (either you the the `mode` explictly or left the default one) because the effective mode is masked by the current umask value in the system: on Linux it is `0o022`, so files/directories have a maximum mode of `0o755` if you did not set the umask explicitly, and it is what it is observed for Certbot.
However on Windows, the `mode` value passed (and got from default) to the `open` and `mkdir` of `certbot.compat.filesystem` module is taken verbatim, since umask does not exit, and then is used to calculate the DACL of the newly created file/directory. So if the mode is not set explicitly, we end up with files and directories with `0o777` permissions.
This PR fixes this problem by implementing a umask behavior in the `certbot.compat.filesystem` module, that will be applied to any file or directory created by Certbot since we forbid to use the `os` module directly.
The implementation is quite straight-forward. For Linux the behavior is not changed. On Windows a `mask` parameter is added to the function that calculates the DACL, to be invoked appropriately when file or directory are created. The actual value of the mask is taken from an internal class of the `filesystem` module: its default value is `0o755` to match default umasks on Linux, and can be changed with the new method `umask` that have the same behavior than the original `os.umask`. Of course `os.umask` becomes a forbidden function and `filesystem.umask` must be used instead.
Existing code that is impacted have been updated, and new unit tests are created for this new function.
* Implement umask for Windows
* Set umask at the beginning of tests
* Fix lint, update local oldest requirements
* Update certbot-apache/setup.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Improve tests
* Adapt filesystem.makedirs for Windows
* Fix
* Update certbot-apache/setup.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Changelog entries
* Fix lint
* Update certbot/CHANGELOG.md
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Brad Warren