This PR supersedes #7353.
It fixes the execution of nginx oldest tests when these tests are executed on top of the modifications made in #7337. This execution failure revealed the fact that in some cases, the wrong version of certbot logic was used during integration tests (namely the logic lying in the codebase of the branch built, instead of the logic from the version of certbot declared by certbot-nginx for instance).
I let you appreciate my inline comment for the explanation and the workaround.
Thanks a lot to @bmw who found this python/pytest madness.
You can see the oldest tests succeeding with the logic of #7337 + this PR here: https://travis-ci.com/certbot/certbot/builds/124816254
* Remove certbot root from PYTHONPATH during integration tests
* Add a biiiiig comment.
On Windows you can have several drives (`C:`, `D:`, ...), that is the roughly (really roughly) equivalent of mount points, since each drive is usually associated to a specific physical partition.
So you can have paths like `C:\one\path`, `D:\another\path`.
In parallel, `os.path.relpath(path, start='.')` calculates the relative path between the given `path` and a `start` path (current directory if not provided). In recent versions of Python, `os.path.relpath` will fail if `path` and `start` are not on the same drive, because a relative path between two paths like `C:\one\path`, `D:\another\path` is not possible.
In saw unit tests failing because of this in two locations. This occurs when the certbot codebase that is tested is on a given drive (like `D:`) while the default temporary directory used by `tempfile` is on another drive (most of the time located in `C:` drive).
This PR fixes that.
* Use travis_retry in travis builds to retry the farm tests
* travis_retry is a bash function, so it can be called only from current bash
* Update .travis.yml
* Update .travis.yml
* Fix account_tests
* Fix hook executable test
* Remove the temporary decorator @broken_on_windows
* Fix util_test
* No broken unit test on Windows anymore
* More elegant mock
* Fix context manager
* Fix lint
* Fix mypy
* Adapt coverage
* Corrections
* Fix lint
* Adapt coverage
* Update certbot/tests/compat/filesystem_test.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update util_test.py
* Fix pylint
* Forbid os.access
* Update os_test.py
* Update os.py
* Fix lint
* Update filesystem.py
* Update filesystem.py
* Update filesystem.py
* Update os.py
* Start fixing tests
* Platform independent hooks
* Fix probe fd close
* Add broken_on_windows for integration tests
* Fix a lot of tests
* Use a python hook script, to prepare cross-platform
* New approach to be compliant with Linux and Windows on hook scripts
* New tests fixed
* Test for permissions on Windows
* Permissions comparison for Windows
* No broken tests in certbot core anymore
* Change mode
* Specific config for appveyor
* Use forked pebble for now
* Various fixes
* Assert file permissions for world on private keys
* Clean code
* Fix several things
* Add integration target
* Optimize integration env
* Re-enable all AppVeyor envs
* Use again official pebble
* Update pebble_artifacts.py
* Set PYTEST_ADDOPTS silently
* Update appveyor.yml
* Pin pywin32 for tests, give a minimal requirement for certbot.
* Remove injection of nginx in PATH
* Clean debug code
* Various cleanup, ensure to remove workspace after tests
* Update tox target
* Improve assertions. Control the keyword echoed in hooks
* Fix for virtualenv on Python 3.7.4 for Windows
* Update certbot-ci/certbot_integration_tests/certbot_tests/assertions.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add conditionally pywin in certbot-ci like in certbot
Since #7191, TLS configuration files for Apache have been moved to a dedicated folder tls_configs. Then the entries in MANIFEST.in removed by this PR do not correspond to an existing path, and so are not useful anymore.
Following discussions in #7298.
This PR moves the three Nginx TLS configuration files into a specific folder, tls_configs, update the MANIFEST to include this folder and its content into the certbot-nginx package, and update tests accordingly.
* Move tls configuration files in a specific folder
* Move new file
* Implement a logic, miss the private key of pebble
* Complete process
* Fix nginx cert path
* Check conditionnally docker
* Update gitignore, fix apacheconftest
* Full object
* Carriage return
* Work in progress
* Move to official v2.1.0 of pebble
* Fix name
* Update acme_server.py
* Link things together with new version of pebble
* Plug the logic to tests
* Update config
* Reinitiate config
* Add OCSP config to pebble
* Working.
* Simplify logic
* Clean code
* Use forked pebble for now
# Conflicts:
# certbot-ci/certbot_integration_tests/utils/pebble_artifacts.py
* Move full logic of mock at the acme server config
* Continue work
* Finish fixing the date parsing
* Update module name
* Use again official pebble
* Activate mock OCSP server
* Clean code
* Update pebble_artifacts.py
* Remove OCSP stale test
* Add executable permissions
* Clean code
* Update setup.py
* Simplify code
* On-demand import of pebble_ocsp_server
* Revert "Remove OCSP stale test"
This reverts commit 2e4c985b42.
# Conflicts:
# certbot-ci/certbot_integration_tests/utils/misc.py
* Fix for virtualenv on Python 3.7.4 for Windows
* Update acme_server.py
AppVeyor recently upgrade the Python 3.7.x installed in their VM to 3.7.4. However, virtualenv 16.6.1 is broken on that specific version of Python for Windows.
This PR upgrade virtualenv installed for a dev/test environment from 16.6.1 to 16.6.2 in order to fix this issue, and repair the CI jobs execute by AppVeyor on PRs.
Fixes#6850
This PR makes the last corrections needed to run all unit tests on Windows:
add a function to check if a hook is executable in a cross-platform compatible way
handle correctly the PATH surgery for Windows during hook execution
handle correctly an account compatibility over both ACMEv1 and ACMEv2
remove (finally!) the @broken_on_windows decorator.
* Fix account_tests
* Fix hook executable test
* Remove the temporary decorator @broken_on_windows
* Fix util_test
* No broken unit test on Windows anymore
* More elegant mock
* Fix context manager
* Adapt coverage
* Corrections
* Adapt coverage
* Forbid os.access
* Implement the logic
* Update tests
* Fix lint and changelog
* Update configurator.py
* Move the TLS configs in a dedicated folder. Fix the formalism of their naming and location.
* Improve existing test to check all TLS config have their hash registered in Certbot
* Corrections after review
* Improve a test
* Remove commented useless lines in TLS configs
* Add a nice warning. Because I am nice.
* Fix lint
* Add a test
Fixes#7115
This PR creates a `realpath` method in `filesystem`, whose goal is to replace any call to `os.path.realpath` in Certbot. The reason is that `os.path.realpath` is broken on some versions of Python for Windows. See https://bugs.python.org/issue9949. The function created here works consistently across Linux and Windows.
As for the other forbidden functions in `os` module, our `certbot.compat.os` will raise an exception if its `path.realpath` function is invoked, and using the `os` module from Python is forbidden from the pylint check implemented in our CI.
Every call to `os.path.realpath` is corrected in `certbot` and `certbot-apache` modules.
* Forbid os.path.realpath
* Finish implementation
* Use filesystem.realpath
* Control symlink loops also for Linux
* Add a test for forbidden method
* Import a new object from os.path module
* Use same approach of wrapping than certbot.compat.os
* Correct errors
* Fix dependencies
* Make path module internal
There is a unit test to check that the default directories for Certbot are not diverging, in certbot.tests.cli_test:FlagDefaultTests:test_linux_directories.
But this test is not done on Windows.
This PR fixes that.
This PR implements the filesystem.copy_ownership_and_apply_mode method from #6497.
This method is used in two places in Certbot, replacing os.chown, to copy the owner and group owner from a file to another one, and apply to the latter the given POSIX mode.
* Implement copy_ownership_and_apply_mode
* Update certbot/compat/os.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Remove default values
* Rewrite a comment.
* Relaunch CI
* Pass as keyword arguments
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Make the private key permissions transfer platform specific
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Rename variable
* Fix comment0
* Add unit test for copy_ownership_and_apply_mode
* Adapt coverage
* Execute unconditionally chmod with copy_ownership_and_apply_mode. Improve doc.
This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform.
Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore.
As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows.
Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables.
Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability.
Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble.
* Complete process
* Fix nginx cert path
* Check conditionnally docker
* Update gitignore, fix apacheconftest
* Full object
* Carriage return
* Move to official v2.1.0 of pebble
* Fix name
* Update acme_server.py
* Relaunch CI
* Update certbot-ci/certbot_integration_tests/utils/acme_server.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/acme_server.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docstring
* Update documentation
* Configure a stdout to ACMEServer
* Map all process through defined stdout
* Remove unused variable
* Handle using signals
* Use failsafe entering context
* Remove failsafe rmtree, that is not needed anymore
Since Pebble v2.1.0, new controls have been added on ACME specs compliance on Pebble with strict mode enabled. These controls are described here: letsencrypt/pebble@3a2ce1c
Currently Certbot is not compliant enough to pass these new controls. One part of the work to do is described here: #7171
As a consequence, our CI is currently broken, both on PR builds and nightly builds.
This PR disables the strict mode during integration tests, fixing temporarily our CI. This will give us some time to fix theses deviations, and add back the strict mode in a future PR once it is merged.
* Remove -strict mode on Pebble for now.
* Refer to relevant Certbot PR
* Clean code
Because some users were complaining about staled workflow when flags (https://docs.codecov.io/docs/flags) are enabled, Codecov decided to remove them when calculating the coverage on branches until they improved this functionality.
See: https://docs.codecov.io/docs/flags#section-flags-in-the-codecov-ui
The flags are still taken into account on PR builds, but not on based branch.
This is a problem for us, because we use the flags to compare specifically the coverage of a PR against its base branch for Windows on one side, and Linux on the other side. Without flags taken into account on the base branch, the CI fails because the coverage on Windows is too low.
As a temporary fix until the situation is clarified on Codecov side, this PR replaces the validation condition, that was a comparison against the base branch, to a fixed coverage registered in the local .codecov.yml file in Certbot repository.
This way, the coverage on PR builds, that takes into account the flags, is validated against an appropriate value.
This is a temporary solution, that will require an explicit update of .codecov.yml in the mean time if the coverage significantly increases, or decreases on some developments. But until the situation is fixed, this will allow to have a functional quality gate.
This PR implements the filesystem.chmod method from #6497.
* Implement filesystem.chmod
* Conditionally add pywin32 on setuptools versions that support environment markers.
* Update apache plugin requirements
* Use a try/except import approach similar to lock
* Add comments about well-known SIDs
* Add main command
* Call filesystem.chmod in tests, remove one test
* Add test for os module
* Update environment marker
* Ensure we are not building wheels using an old version of setuptools
* Added a link to list of NTFS rights
* Simplify sid comparison
* Enable coverage
* Sometimes, double-quote is the solution
* Add entrypoint
* Add unit tests to filesystem
* Resolve recursively the link, add doc
* Move imports to the top of the file
* Remove string conversion of the ACL, fix setup
* Ensure admins have all permissions
* Simplify dacl comparison
* Conditionally raise for windows temporary workaround
* Add a test to check filesystem.chown is protected against symlink loops
This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform.
I initially used the fully-fledged HTTP proxy [Traefik](https://docs.traefik.io/) to distribute HTTP challenges among several pytest nodes, and so parallelize the integration tests. Traefik for this purpose is overkill. We just want to redirect the ACME server to a pytest node depending on the `Host` header, and we use here a production-grade HTTP proxy for that.
However it was not a problem on Linux, as soon as you can have Docker, because this instance is deployed through it.
But this becomes a problem for Windows, where Docker is not available everywhere, very compelling on its setup, and limited by the implemented network drivers. See my comments here https://github.com/letsencrypt/pebble/pull/240 for more details.
Hopefully Python ships with everything needed to implement a simple HTTP proxy, with strictly what we need for the parallelization of integration tests.
This PR implements this kind of HTTP proxy, and remove the coupling to Traefik.
This PR has been tested successfully with integration tests on Pebble under Linux for Python 2.x and Python 3.x, and the proxy alone has been also tested successfully on Windows (no integration tests can be run for now on this platform).
* Create a python proxy
* Refactor proxy config
* Working logic
* Resolve from the path
* Give proxy process to the ACMEServer context manager
Since #7073 for Certbot and letsencrypt/boulder@3918714 for Boulder have landed, the bash scripts that remained after certbot-ci are not useful anymore outside of Certbot.
Only remaining place is the apacheconftest-with-pebble tox target, which leverages pebble-fetch.py script to expose a running ACME server to the apache-conf-test script.
This PR refactor apacheconftest-with-pebble to use certbot-ci instead. Finally, this PR remove the remaining integration tests bash scripts, that are _common.sh, boulder-fetch.py and pebble-fetch.py.
* Disconnect common and boulder-fetch
* Prepare reconnection of apacheconftest to new pebble deployment logic
* Finish the configuration for apacheconftest
* Add executable flag to python script
* Fix shebang
* Delete pebble-fetch.sh
Currently integration tests against Boulder fail during nightly tests. See https://travis-ci.com/certbot/certbot/builds/115373954.
This is due to a failure to cleanup the workspace associated to the Boulder docker started during the integration tests. Indeed this docker compile several artifacts whose owner is root, and permissions are 0744. These files are persisted in the workspace folder attached to the Docker.
Since tox is run as a non-root user (but this user still have access to the Docker daemon), everything works fine until the end of the test suite, when all resources are cleaned up. At this point, pytest fires a PermissionError when failing to delete these artifacts, return with a non-zero exit code, and so fail the build.
Since this situation could happen outside of the CI, I made appropriate corrections to allow the integration tests to be run as a non-root user, instead of changing Travis to execute tests as root user.
The correction is to add a step to the cleanup process: the deletion of these artifacts through an ad-hoc docker instance.
During review of #6989, we saw that some of our test bash scripts were still used in the Boulder project in particular. It is about `tests/integration/_common.sh` in particular, to expose the `certbot_test` bash function, that is an appropriate way to execute a local version of certbot in test mode: define a custom server, remove several checks, full log and so on.
This PR is an attempt to assert this goal: exposing a new `certbot_test` executable for test purpose. More generally, this PR is about giving well suited scripts to quickly make manual tests against certbot without launching the full automated pytest suite.
The idea here is to leverage the existing logic in certbot-ci, and expose it as executable scripts. This is done thanks to the `console_scripts` entry of setuptools entrypoint feature, that install scripts in the `PATH`, when `pip install` is invoked, that delegate to specific functions in the installed packages.
Two scripts are defined this way:
* `certbot_test`: it executes certbot in test mode in a very similar way than the original `certbot_test` in `_common.sh`, by delegating to `certbot_integration_tests.utils.certbot_call:main`. By default this execution will target a pebble directory url started locally. The url, and also http-01/tls-alpn-01 challenge ports can be configured using ad-hoc environment variables. All arguments passed to `certbot_test` are transferred to the underlying certbot command.
* `acme_server`: it set up a fully running instance of an ACME server, ready for tests (in particular, all FQDN resolves to localhost in order to target a locally running `certbot_test` command) by delegating to `certbot_integration_tests.utils.acme_server:main`. The choice of the ACME server is given by the first parameter passed to `acme_server`, it can be `pebble`, `boulder-v1` or `boulder-v2`. The command keeps running on foreground, displaying the logs of the ACME server on stdout/stderr. The server is shut down and resources cleaned upon entering CTRL+C.
This two commands can be run also through the underlying python modules, that are executable.
Finally, a typical workflow on certbot side to run manual tests would be:
```
cd certbot
tools/venv.py
source venv/bin/activate
acme_server pebble &
certbot_test certonly --standalone -d test.example.com
```
On boulder side it could be:
```
# Follow certbot dev environment setup instructions, then ...
cd boulder
docker-compose run --use-aliases -e FAKE_DNS=172.17.0.1 --service-ports boulder ./start.py
SERVER=http://localhost:4001/directory certbot_test certonly --standalone -d test.example.com
```
* Configure certbot-ci to expose a certbot_test console script calling certbot in test mode against a local pebble instance
* Add a command to start pebble/boulder
* Use explicit start
* Add execution permission to acme_server
* Add a docstring to certbot_test function
* Change executable name
* Increase sleep to 3600s
* Implement a context manager to handle the acme server
* Add certbot_test workspace in .gitignore
* Add documentation
* Remove one function in context, split logic of certbot_test towards capturing non capturing
* Use an explicit an properly configured ACMEServer as handler.
* Add doc. Put constants.
* Modifications for misc
* Add some types
* Use os_rename
* Move rename into filesystem
* Use our os package
* Rename filesystem.rename to filesystem.replace
* Disable globally function redefined lint in os module
With the various optimizations already done and upcoming (certbot-ci), the time execution of integration tests have significantly decreased, allowing potentially a complete execution of a Travis PR job to be done within 5min30s.
However, one job is significantly longer that the other ones after this migration: this is nginx_compat, that takes more that 11min to finish. I tried to split the nginx_compat in terms of tested configuration and of tests to execute (auth, install, enhance). Both are not satisfactory:
splitting by configuration may work, but add a significant complexity in the tests
splitting by tests type is supported almost out-of-the-box, but fails to make two fast tests (see https://travis-ci.org/adferrand/certbot/builds/525892885?utm_source=github_status&utm_medium=notification for instance)
Since these tests are designed to check corner cases on the nginx parser, this is mostly useless to execute them on each PR, as the nginx parser is rarely updated.
After some discussion with @bmw, I think that we can just move the nginx_compat from the PR tests to the nightly tests. This PR does that.
* Validate OCSP response for responders that are not the certificate's issuer.
* Improve OCSP tests using a issuer/responder pair for OCSP responses
* Clean code
* Update ocsp_test.py
* Add various comments
* Add several cases of ocsp responder. More factories for the resilience tests.
* Update ocsp_test.py
Following discussion in #6947 (comment), I have second thoughts about relying on acme in certbot-ci.
Indeed, I think it is a good design to not rely in tests on the code you are testing. Obviously in unit tests it is very difficult, since most of the time the unit that is tested needs input generated by other part of the code. However it is not really a problem in a unit test, as its purpose is to make assertions about a specific portion of the code, not the others parts.
In the scope of integration tests, the software tested is treated as a black box. In this case, having some parts of the test logic that use in fact part of the code in the black box, increase the risk that some assertions compared two results coming from the same flawed logic from the tested software.
Since using acme in certbot-ci is only saving few lines of code, I think it does not worth the risk and the added complexity to declare acme as a dependency. I prefer to duplicate these lines and keep certbot-ci free of any dependency coming from the certbot project.
* Connect certbot-ci to travis. Remove old bash files.
* Configure test-everything
* Protect against import error
* Remove unused ignore
* Better handling of urllib3
* Correct path
* Remove a warning
* Correct call
* Protect atexit register execution
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add again some bash scripts to avoid breaking to much retro-compatiblity on third party scripts
* Move boulder-v1 and boulder-v2 in nightly tests
* Separate oldest unit tests and oldest integration tests
* Remove try/except
* Test integration included in toxenv
* Add a wait to avoid a transient issue on OCSP status in oldest tests
* Clean travis.yml, split other tests
* Remove useless config
* Update .travis.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update tox.ini
* Update tox.ini
* Remove pytest-sugar
* Remove empty pytest.ini, tests are working without it
Revert #6702
After some discussions, we realized that changing the path for FreeBSD users, if it corresponds to the path used when Certbot is installed using ports, will break for users that installed it through certbot-auto.
Indeed in this case, the path used was the one for Linux. After #6702, Certbot would not find anymore the existing config path by default.
It would require, to be integrated, a proper documentation and a migration path. For now, it is preferable to revert it.
This reverts commit 7fe82cf1ac.
Fixes#7031
I use the same approach than in `CreateVenv()` and `CompareVersions()`: a new bash function `CheckPathPermissions()` is declared an execute a python script passed to the interpreter through stdin.
This allows:
* to not require the temp_dir that holds a temporary script to be executed
* to reduce at the bare minimum the change to make on the order of bash command to execute (including when the temp_dir is created)
* Fix check permissions logic in certbot-auto by making a temp dir useless
* Update CHANGELOG.md
In #7019, a solution has been integrated to fix oldest tests execution in the corner cases described in #7014.
However this solution was not very satisfactory, as it consists in making a --force-reinstall for all requirements on each oldest tests (apache, certbot, acme, each dns plugin ...). As a consequence, the overall execution time of these tests increased from 5 min to 10 min.
In this PR I propose a more elegant solution: instead of reinstalling all dependencies, we force reinstall only the requirements themselves describe in the relevant oldest-requirements.txt files. This way only the packages that are potentially ignored by pip because they exists locally (acme, certbot, ...) are reinstalled.
The result is the same than in #7019 (we are sure that all packages are really installed by pip), but the very limited number of force reinstalled package here make the impact on execution time negligible.
As a consequence, I revert back also the tox environments to execute all oldest tests together.
A successful execution of oldest tests using this PR material in the context of a point release can be seen here: https://travis-ci.org/adferrand/certbot/builds/527513101
Fixes#7014.
Using a --force-reinstall (only for oldest tests), dependencies are properly reinstalled. Since this action significantly increases the execution time of oldest tests, I split them into two parts to allow their parallel execution by Travis.
We will need to find a better way to solve this in the future.
An example of successful execution of oldest tests in the situation of a point release can be found here: https://travis-ci.org/adferrand/certbot/builds/527475532
* Fix for oldest requirements
* Split oldest tests
* Update a comment
(cherry picked from commit b19d4801c9)
Fixes#7014.
Using a --force-reinstall (only for oldest tests), dependencies are properly reinstalled. Since this action significantly increases the execution time of oldest tests, I split them into two parts to allow their parallel execution by Travis.
We will need to find a better way to solve this in the future.
An example of successful execution of oldest tests in the situation of a point release can be found here: https://travis-ci.org/adferrand/certbot/builds/527475532
* Fix for oldest requirements
* Split oldest tests
* Update a comment
Fixes#6974.
This PR removes the fallback that consists in retrying to send the keyAuthorization field during a challenge request in case of malformed request.
* Remove keyAuthorization fallback dump in challenges response
* Correct import
* Add changelog entry
This PR is the part 4 to implement #6541. It adds the integration tests for the nginx certbot plugin, and corresponds to the certbot-ci translation of certbot-nginx/tests/boulder-integration.sh that is executed for each PR.
As with certbot core tests, tests are written in Python, and executed by pytest, against a dynamic Boulder/Pebble instance setup. Tests are parallelized, of course, and a specific IntegrationTestsContext class, extended the one from certbot core tests, is crafter for these specific tests: its main goal is to setup a specific nginx instance for the current test.
On top of that, I use the test parametrization feature of Pytest, to drastically reduce the size of the actual code: indeed, the 6 tests from the original bash script share the same logic. So using a parametrization, one unique test is written, that is then executed 6 times against 6 different sets of parameters.
Note that the module integration_tests.nginx_tests.nginx_config do the same, but in Python, than certbot-nginx/tests/boulder-integration.conf.sh. The latter will be removed in a future PR, with all other bash scripts.
* Add nginx tests
* Distribute the other_port
* Load a pre-generated key/cert for nginx config
* Correct preload, remove a test, simplify a variable
* Integrate assertion directly in the test function
* Check process is not terminated
* Add spaces in the nginx config
* Add comments
* Use indirection
* Allow external cert
* Add coverage threshold for certbot-nginx
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering checks on L430-447 in tests/certbot-boulder-integration.sh. Previous lines are covered with existing tests, or by #6946, #6947, #6948, #6949.
* Add tests
* Change param
* Increase coverage min to 64%
* Disable OCSP Must-Staple test for Pebble
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering checks on L531 to the end on tests/certbot-boulder-integration.sh. Previous lines are covered with existing tests, or by #6946, #6947, #6948, #6949, #6951, #6952.
* Add tests
* Add load resource
* Separate OCSP in two tests
* Copy new asset
* Load the asset
* Add coverage limit
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering checks on L448-530 in tests/certbot-boulder-integration.sh. Previous lines are covered with existing tests, or by #6946, #6947, #6948, #6949, #6951.
* Add tests
* Normalize paths
* Fix merge error in git
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering checks on L397-429 in tests/certbot-boulder-integration.sh. Previous lines are covered with existing tests, or by #6946, #6947 and #6948.
* Add tests
* Change a variable name
* Fix merge errors from git
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering about renew, on L283-396 in tests/certbot-boulder-integration.sh (by including existing test_renew_files_permissions and test_renew_with_hook_scripts). Previous lines are covered with existing tests, or by #6946 and #6947.
* Add tests
* Correct assertion about world permission
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering on L268-282 in tests/certbot-boulder-integration.sh. Previous lines are covered with existing tests, or by #6946.
* Add tests
* Fix CSR generation
* Add dependency
Following #6821, this PR continues to convert certbot integration tests into certbot-ci.
This PR add tests covering on L185-222 in tests/certbot-boulder-integration.sh.
* Add tests
* Correct some assertions
This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
Following #6934, this PR finalize two things, as explained in #6934:
disable the default aggregated report
validate linux and windows reports against the PR base branch
So, we observed lately several inconsistencies in how Codecov behave toward the CI pipeline for PRs in Certbot. One example is #6888. The most annoying thing is that the build of PR is **temporary** marked as failed, until all coverage are run.
The correction on the latter is done in two PRs. This is the first part.
TL;DR
This PR separates the Codecov report in two: one for coverage executed on Windows, one for Linux. This is the correct way to do regarding our current CI pipeline. Actions are required by a GitHub administrator of Certbot once this PR is merged.
Complete explanation
So the failure stated in the introduction is essentially due to several things interacting together:
* AppVeyor generates a coverage report for Windows, that have a coverage value a little lower than on Linux (96%)
* Travis generates a coverage report for Linux. Its coverage is higher, and slowly decrease as more specific Windows code is added to Certbot, that cannot be tested on Travis
* Since AppVeyor saw its capacity increasing, it finishes its coverage job before the one from Travis
* Certbot GitHub repo is configured to require the coverage pipeline to succeed (in whatever that means) to success the overall PR build
So here the suite of events:
1) PR is issued. GitHub expect three pipeline to succeed: AppVeyor CI, Travis CI and Codecov (displayed in the PR page)
2) Codecov receive first the report of AppVeyor coverage. It is 96%. It is a failure for now, because coverage in master (AppVeyor+Travis) is 98.6%.
3) GitHub is reported of the failure on Codecov, so fail the PR build
4) Codecov receive then the report of Travis coverage. It is 98%. It merges it with the report from AppVeyor, leading to the 98.6%. The failure becomes a success.
5) GitHub is reported of the success on Codecov, so, nevermind, the PR build is a success finally!
So we have a CI flow that change its mind. Great. This is because of 2) and 4), and we could expect that Codecov should handle that. This is not the case: it is somewhat misleading, because Codecov adverts a lot about its capability to merge reports, including from different CI. But it is about the final state, not about the transient state, while reports are progressively received.
Two things to things that a transient state is existing, with a result that can change:
* first, from Codecov doc itself, explaining that reports should not be trusted during the CI pipeline execution: https://docs.codecov.io/docs/ci-service-relationship#section-checking-ci-status
* second, is an example of transient state of `cryptography` project, this is advert by Codecov to be a reference of the implementation:
As you can see above, build state of `cryptography` is failing after the first report is received, and until all coverage reports from Travis are received.
So, what can we do about it? Thing is, we are aggregating coverage from very two unrelated sources (two different OS systems), and Codecov has something for that. This is flags: https://docs.codecov.io/docs/flags
Flags allow to flag coverage material depending on any logic you apply to the command that uploaded the coverage report (eg. `codecov -F a_flag`). Then, several logics can be applied on it, for instance having in Codecov UI the capability to filter the coverage other a flag, having status of build for each flag and ... having a report for a specific flag.
So:
1) I modified Travis and AppVeyor to send their report under a specific flag: `linux` or `windows`
2) I created a project specific `.codecov.yml` configuration in Certbot repository, to instruct Codecov to push two separate reports on GitHub build: one for Linux, one for Windows. Each report can be validated against its specific coverage from the `master` branch (more on this just after)
With all of this, now the GitHub is succeeding, because each coverage is validated independently.
I think it is the good approach, because it solves the specific issue here, and because it reflects the logic behind: merging coverage from different OS architectures does not make much sense. It would be a long-term problem, because as I said at the beginning, coverages will slowly decrease as more platform specific code is added in Certbot.
Now, it is not finished. Two things need to be done: an administrator action, and a second PR
Administrator action
Certbot GitHub as a a branch protection rule (Settings > Branches > Branch protection rules). It needs to be changed.
Indeed this rule is expecting the full coverage report (named `codecov/project`) to be valid on a PR. It needs to be changed to expect two coverage reports: `codecov/project/linux` and `codecov/project/windows`. The `codecov/project` needs to be removed.
This can be done once this PR is merged, and the specific coverage reports have been generated on master.
Second PR
Once this PR is merged and administrative actions have been done. I will make a new PR modifying `.codecov.yml` with two things:
* disable the faulty full coverage report, that is not required anymore by GitHub branch protection rules
* modify the `linux` and `windows` reports to validate against the relevant coverage calculated from `master` (indeed, in this PR it is a fixed ratio rule, since the coverage to compare on master is the full coverage one, significantly higher)
* Tag reports
* Set per-project codecov configuration
Fixes#6912
Bash evaluate all condition in a predicate statement, eg. `"$SOMEVAR" = "test" -a "$ANOTHERVAR" = "test2"`, even if it is not necessary, for instance if the first condition is false in the example here.
As a consequence, on non-Fedora distributions, an evaluation of the distribution version could be done on non numeric value, eg. `"6.7" -eq "29"`, making certbot-auto failing in this case.
This PR fixes that, by evaluating the version on RPM distributions only if we are on Fedora. Otherwise, version will be "0".
Dependencies generated by the script introduced with #6839 were not including anymore the fix about enum34 for CentOS 6.
This PR reinserts this fix, and updates the script overrides to ensure that this fix will stay in next dependencies generation.
* Add the environment marker back. Ensure that it will stay by adding an override to dependencies generator.
* Add comments, for future fix
* Update letsencrypt-auto-source/rebuild_dependencies.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update comment
This PR is a part of the effort to remove the last broken unit tests in certbot codebase for Windows, as described in #6850.
It solves the problems associated to ErrorHandler in Windows (enlighted by tests errors) by ... wipping out the problem: no signal is handled by ErrorHandler on Windows. See the relevant inline comment in certbot.error_handler for explanation and sources.
This PR is the first part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This step 1 refactor existing certbot.compat module into certbot.compat.misc, without any logic changed. Package certbot.compat will host the new modules that constitute the security model for Windows.
* Create the certbot.compat package. Move logic in certbot.compat.misc
* Add doc
* Fix lint
* Correct mypy
* Update client.py
This PR is a part of the tls-sni-01 removal plan described in #6849.
As `acme` is a library, we need to put some efforts to make a decent deprecation path before totally removing tls-sni in it. While initialization of `acme.challenges.TLSSNI01` was already creating deprecation warning, not all cases were covered.
For instance, and innocent call like this ...
```python
if not isinstance(challenge, acme.challenges.TLSSNI01):
print('I am not using this TLS-SNI deprecated stuff, what could possibly go wrong?')
```
... would break if we suddenly remove all objects related to this challenge.
So, I use the _Deprecator Warning Machine, Let's Pacify this Technical Debt_ (Guido ®), to make `acme.challenges` and `acme.standalone` patch themselves, and display a deprecation warning on stderr for any access to the tls-sni challenge objects.
No dev should be able to avoid the deprecation warning. I set the deprecation warning in the idea to remove the code on `0.34.0`, but the exact deprecation window is open to discussion of course.
* Modules challenges and standalone patch themselves to generated deprecation warning when tls-sni related objects are accessed.
* Correct unit tests
* Correct lint
* Update challenges_test.py
* Correct lint
* Fix an error during tests
* Update coverage
* Use multiprocessing for coverage
* Add coverage
* Update test_util.py
* Factor the logic about global deprecation warning when accessing TLS-SNI-01 attributes
* Fix coverage
* Add comment for cryptography example.
* Use warnings.
* Add a changelog
* Fix deprecation during tests
* Reload
* Update acme/acme/__init__.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update CHANGELOG.md
* Pick a random free port.
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes --tls-sni-01-port, --tls-sni-01-address and tls-sni-01/tls-sni options from --preferred-challenges. They are replace by deprecation warning, indicating that these options will be removed soon.
This deprecation, instead of complete removal, is done to avoid certbot instances to hard fail if some automated scripts still use these flags for some users.
Once this PR lands, we can remove completely theses flags in one or two release.
* Remove tls-sni related flags in cli. Add a deprecation warning instead.
* Adapt tests to cli and renewal towards tls-sni flags deprecation
* Add https_port option. Make tls_sni_01_port show a deprecation warning, but silently modify https_port if set
* Migrate last items
* Fix lint
* Update certbot/cli.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Ensure to remove all occurences of tls-sni-01
* Remove unused parameter
* Revert modifications on cli-help.txt
* Use logger.warning instead of sys.stderr
* Update the logger warning message
* Remove standalone_supported_challenges option.
* Fix order of preferred-challenges
* Remove supported_challenges property
* Fix some tests
* Fix lint
* Fix tests
* Add a changelog
* Clean code, fix test
* Update CI
* Reload
* No hard date for tls-sni removal
* Remove useless cast to list
* Update certbot/tests/renewal_test.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add entry to the changelog
* Add entry to the changelog
Currently, `tox -e le_auto_jessie` job fails. It breaks in particular the cron pipeline that test everything each night.
The failure occurs while setting up the Jessie Docker container to run the tests for certbot-auto, when `apt-get update` is invoked, with this error:
```
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages 404 Not Found
```
Indeed, if there are `stretch-updates`, `buster-updates` and so on in the repository, there is no `jessie-updates`. I do not know exactly the logic of Debian here, but as `*-updates` folders store stable updates, a distribution moving to LTS support like Jessie has no stable updates anymore. I suppose `jessie-updates have been decommissioned recently, and the official Docker has not been updated yet to use the LTS configuration for repositories.
This PR does that live in the Dockerfile, using official instructions from https://wiki.debian.org/LTS/Using, and fixes this specific job.
An example of a successful job with this modification can be found here: https://travis-ci.com/certbot/certbot/jobs/187864341
Explicit is better than implicit
When calling raise without an argument, Python will raise the last error occured from the caller except block. This makes my PyCharm very sad however. So this PR makes the function handling the error raising explicitly the error received as an argument.
To fix one of the two uncovered lines in certbot-apache, given in #6880. Instead of adding a test to just increase the coverage, this fixes the uncovered line by removing the unused code.
This PR is a part of the effort to remove the last broken unit tests in certbot codebase for Windows, as described in #6850.
This PR fixes various unit tests on Windows, whose resolution was only to modify some logic in the tests, or minor changes in certbot codecase impacting Windows only (like handling correctly paths with DOS-style).
* Correct several tests
* Skip test definitively
* Test to be reactivated with #6497
* Mock log system to avoid errors due to multiple calls to main in main_test
* Simplify mock
* Update cli_test.py
* One test to be repaired when windows file permissions PR is merged
* Remove tls-sni from nginx config
* Add a dedicated configuration to define what is the HTTPS port for this certbot instance.
* Correct some tests
* Reestablish default vhost creation
* Clean tls references for nginx integration tests
* Associate https_port only to tests and nginx
* Add a dedicated configuration to define what is the HTTPS port for this certbot instance.
* Remove tls-sni in apache plugin
* Update constants.py
* Update interfaces.py
* Remove option
* Simplify a test
* Reconfigure compatibility tests to use http challenge
* Correct simple test
* Add a fake DNS resolution for HTTP simple_verify
* Debug
* More subtle approach: we monkey patch urllib3 to fake a dns resolution to the target IP, allowing every host header to be preserved.
* Private package
* Relaxed permissions on certbot temp working dir
* Move the fake DNS logic in compatibility test, to avoid degrading the acme coverage
* Fix lint
* Update certbot-compatibility-test/certbot_compatibility_test/configurators/common.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes the tls-sni-01 challenge tests during the integration tests. The approach I used here is not to remove completely the existing test code, but simply editing it to use a http-01 challenge. Indeed:
* the current integration tests are strongly coupled, and would require more modifications that it is worth, because ...
* the certbot-ci project, that has already no tls-sni tests, will soon replace completely the current integration tests code.
Currently coverage invocation during integration tests on certbot core is misplaced, just before the OCSP statuses tests.
This PR move back the coverage invocation at the end of the script.
Fixes#6836.
OCSP responses contains a thisUpdate and nextUpdate that allow to calculate its validity. Certbot currently uses datetime.now() to get the current time when OCSP check is done through cryptography. But datetime.now() expresses the date in the machine local time, and comparison operators on datetime do not take into account the offset between two datetime objects expressed in difference timezones.
As a consequence, a given thisUpdate may be seen as a future date depending on the local time, failing the OCSP check process.
The error is not critical for certbot, as it will just make some valid OCSP responses giving an EXPIRED status been ignored.
This PR fixes this comparison by taking the current time in UTC using datetime.utctime().
Fixes#6698
Fedora maintainers engaged a deprecation path for Python 2.x with Fedora 29. As a first step, python2-virtualenv does not install the virtualenv binary anymore, in favor of python3-virtualenv, and so the installation of Python 3 virtual environments by default.
However, certbot-auto installs python2-virtualenv for all recent RPM distributions, and relies of the execution of virtualenv, and this is failing the process.
Since the plan in the future is to remove Python 2.x from Fedora, this PR follows this logic to fix certbot-auto: started to Fedora 29, certbot-auto will install and execute certbot on Python 3. This implies to detect that we are on Fedora 29+, install python3-virtualenv that will install also Python 3 dependencies and virtualenv binary, then instruct the process to use Python 3. This is in fact similar to EOL distributions shipping with Python 2.6, and for which Python 3.4 from EPEL is installed and used.
Older versions of Fedora continue to use Python 2.x, and their process is untouched. Four scenarios are covered here:
fresh Fedora 28: old process is used, nothing changes
fresh Fedora 29: new process is used, Python 3 is installed, certbot runs on it
update Fedora 29 from 28, already installed certbot-auto without rebootstrapping required: existing venv continue to be used, certbot runs on it
update Fedora 29 from 28, already installed certbot-auto with rebootstrapping required: new process is used, installing python3-virtualenv, python3-devel and python3-rpm-macros, Python 3 is installed, certbot runs on it
* Add a step to handle python3 on fedora29
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto-source/letsencrypt-auto.template
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update rpm_python3.sh
* Rebuild certbot-auto
* Empty commit to relaunch CI pipeline
* Add changelog
* Update CHANGELOG.md
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update CHANGELOG.md
Fixes#6228.
Since OpenSUSE Leap 15, python-virtualenv became a source package, breaking certbot-auto bootstrap on this version. Then python2-virtualenv must be used to create Python 2.x virtual environments.
This PR makes certbot-auto compatible to prior and after Leap 15, by testing the existence of python-virtualenv on current OpenSUSE system, and then use appropriate packages.
* Cover case of OpenSUSE Leap 15+ in certbot-auto
* Revert increment on bootstrap for OpenSUSE
* Fix configuration for Leap15+
* Add comment about explicit installation of python2-setuptools
* Update letsencrypt-auto-source/pieces/bootstrappers/suse_common.sh
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update letsencrypt-auto
By removing all the builds on push to master, done in #6804, we also removing the coveralls reports that are necessary to calculate the effect of a PR on code coverage, that is part of the quality gate process.
This PR is reverting #6804, and another implementation preserving the coveralls reports will be done soon.
This reverts commit a468a3b255.
Fixes#5789
This PR is about allowing Certbot to respect the Retry-After HTTP header that an ACME CA server can return to a client POSTing to a challenge, to instruct him and retry the request later.
However, this feature was not easily implementable in the current code of certbot.auth_handler, because the code became really hard to read. In fact, @bmw was thinking that the code was really deceiving, and a lot of supposed functionalities declared in the comments were in fact not implemented or not functional.
So I took the time to understand what was going on, and effectively, most of the code is in fact not usable or not used. Then I did a refactoring against the bare ACME spec about what to do to prepare challenges, instruct the ACME CA server to perform them, then polling regularly the authorization resources until they are decided (valid or invalid).
And of course this implementation takes care of Retry-After ^^
I added a lot of comments in the new implementation, to explain what is going on for a future developer. The workflow I used is relying on the relationships between authorizations and challenges states as described in section 7.1.6 of the ACME spec draft: https://datatracker.ietf.org/doc/draft-ietf-acme-acme/
* Clean auth_handler a bit, and implement retry-after.
* Remove a debug logger
* Correct tests
* Fix mypy and lint. Setup max retries and default retry after accordingly.
* Ease a comparison in tests
* Update documentation
* Add tests
* Adapt windows coverage threshold to the global LOC reduction
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Corrections under review
* Correction under review
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Corrections under review
* Update auth_handler_test.py
* Reimplementing user readable report for failed authorizations
* Fixes two tests
* Fix another test + lint + mypy
* Update auth_handler.py
* Update auth_handler_test.py
* Fix tests
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Raise directly the exception on polling timout
* Improve interface documentation
* Move the wait on top of the loop, to be used initially or after a new loop iteration. Do not wait for negative values.
* Always display the report about failed authorizations.
* Clarify an exception.
* Return, instead of break
* Use setdefault
* Remove useless assertion
* Adapt tests
* Improve a test about retry after value.
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add a complete test on best_effort
* Add entry to the changelog
* Gather all failed authzrs to be reported in one unique report in case of best_effort
* Build complete warn/report/raise process about failed authzrs
Fixes#6746.
Every commit on master is always the result of a merged PR, that has been tested by Travis. So retesting the merge commit on master is superfluous. This PR uses build conditions to avoid to launch a build for a commit push on master.
I also added the equivalent logic for AppVeyor. Builds cannot received conditions, so it needs to be done on init using Exit-AppVeyorBuild. This command does not fail the build, it finishes it prematurely with success.
* Disable build for commit pushed on master (PR are still tested of course)
* Equivalent exclusion code for AppVeyor
If you execute `tests/lock_test.py` or `tox -e integration` on a fairly recent machine, you will get the following error during tests executing against a live Nginx instance:
```
no "ssl_certificate" is defined in server listening on SSL port while SSL handshaking, client: x.x.x.x, server: y:y:y:y:z
```
Indeed, having no defined ssl certificate for a ssl port would inevitably lead to an error during the handshake SSL process between a client and this mis-configured nginx instance.
However it was not a problem one year before, because the handshake was not occurring in practice: the test just need to have a nginx started, and then immediately proceed to modify the configuration with a correct SSL setup. And nginx was able to start with a mis-configuration on SSL.
But then this fix has been done: https://trac.nginx.org/nginx/ticket/178
Basically with this, validation of the configuration is done during nginx startup, that will refuse to start with invalid configuration on SSL. Consequently, all related tests are failing with a sufficiently up-to-date nginx. For now, it is not seen on Travis because Ubuntu Trusty is used, with an old Nginx.
The PR fixes that, by generating on the fly self-signed certificates in the two impacted tests, and pushing the right parameters in the Nginx configuration.
* Fix nginx configuration with self-signed certificates generated on the fly
* Fix lint/mypy
* Fix old cryptography
* Unattended openssl
* Update lock_test.py
* First part
* Several optimizations about the docker env setup
* Documentation
* Various corrections and documentation. Add acme and certbot explicitly as dependencies of certbot-ci.
* Correct a variable misinterpreted as a pytest hook
* Correct strict parsing option on pebble
* Refactor acme setup to be executed from pytest hooks.
* Pass TRAVIS env variable to trigger specific xdist logic
* Retrigger build.
* Work in progress
* Config operational
* Propagate to xdist
* Corrections on acme and misc
* Correct subnet for pebble
* Remove gobetween, as tls-sni challenges are not tested anymore.
* Improve pebble setup. Reduce LOC.
* Update acme.py
* Optimize acme ca setup, with less temporary assets
* Silent setup
* Clean code
* Remove unused workspace
* Use default network driver
* Remove bridge
* Update package documentation
* Remove rerun capability for integration tests, not needed.
* Add documentation
* Variable for all ports and subnets used by the stack
* Update certbot-ci/certbot_integration_tests/conftest.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/acme.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/misc.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update tox.ini
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/misc.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/acme.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/acme.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/conftest.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Rename to acme_server
* Add comment
* Refactor in a unique context fixture
* Remove the need of CERTBOT_ACME_XDIST environment variable
* Remove nonstrict/strict options in pebble
* Clean dependencies
* Clean tox
* Change function name
* Add comment about coveragerc specificities
* Change a comment.
* Update setup.py
* Update conftest.py
* Use the production-ready docker-compose.yml file for Pebble
* New style class
* Tune pebble to have a stable test environment
* Pin a dependency
The rerun capability in a test campaign can be a nice feature. It can also a bad design.
It is right that with high level tests, like performance or end-to-end tests, a given test runtime can depend on an external component, outside the scope of the developers, that would spuriously fail. In theses situations, having the capacity to rerun several time a test can be a great benefit. Indeed, as these tests are inherently flaky, the rerun greatly reduces their failure because of external reasons, reducing also the Pierre et le Loup effect (from a well-known french book for children): because tests constantly fail for no reason, you stop to listen to them and to not see when they fail for a real reason.
However this not apply to unit tests, or operations about code quality. For theses executions, the flakiness should not exist: a unit test is supposed to have no external dependency. A rerun approach would just hide a situation that is not desirable for this kind of tests
Also effectively I see that our tests are usually not flaky: so the only effect of the rerun is to give the failure state about a test three times slower. If a test becomes flaky, this should be fixed, and so be visible immediately in our CI.
For these reasons, I remove the travis_retry in the script section of .travis-ci.yml, to call directly tox and let the pipeline fails on the first error.
* Disable rerun feature of Travis
* Update .travis.yml
* Remove completely the retry logic
* Fix pipstrap on windows
* Pipstrap pin setuptools version, so explicit install it is not needed anymore.
* Rebuild letsencrypt-auto source
* Use sys.executable in pipstrap to allow straightforward execution in a venv by choosing the python interpreter from this venv.
* Update letsencrypt-auto
* Simulate test-everything
* Revert "Simulate test-everything"
This reverts commit b62c4d719a.
* Clean pipstrap code
* Reenabling OCSP cryptography support
* Refactor the validation logic of OCSP response to match the OpenSSL one
* Prepare runtime for OCSP response test
* Move unrelated test to another relevant place
* Reimplement OCSP status checks in integration tests
* Clean script
* Protect OCSP check against connection errors
* Update tests/certbot-boulder-integration.sh
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Cleaning
* Add a specific script for letsencrypt-auto install+help
* Remove inconsistent assertion
* Add executable permissions
* Remove unused variable
* Move testdata
* Corrected cleanup code
* Empty commit
Fixes#6755.
POSTing the `keyAuthorization` in a JWS token when answering an ACME challenge, has been deprecated for some time now. Indeed, this is superfluous as the request is already authentified by the JWS signature.
Boulder still accepts to see this field in the JWS token, and ignore it. Pebble in non strict mode also. But Pebble in strict mode refuses the request, to prepare complete removal of this field in ACME v2.
Certbot still sends the `keyAuthorization` field. This PR removes it, and makes Certbot compliant with current ACME v2 protocol, and so Pebble in strict mode.
See also [letsencrypt/pebble#192](https://github.com/letsencrypt/pebble/issues/192) for implementation details server side.
* New implementation, with a fallback.
* Add deprecation on changelog
* Update acme/acme/client.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Fix an instance parameter
* Update changelog, extend coverage
* Update comment
* Add unit tests on keyAuthorization dump
* Update acme/acme/client.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Restrict the magic of setting a variable in immutable object in one place. Make a soon to be removed method private.
So merging the study from @bmw and me, here is what happened.
Each invocation of `certbot.logger.post_arg_parse_setup` create a file handler on `letsencrypt.log`. This function also set an atexit handler invoking `logger.shutdown()`, that have the effect to close all logger file handler not already closed at this point. This method is supposed to be called when a python process is close to exit, because it makes all logger unable to write new logs on any handler.
Before #6667 and this PR, for tests, the atexit handle would be triggered only at the end of the pytest process. It means that each test that launches `certbot.logger.post_arg_parse_setup` add a new file handler. These tests were typically connecting the file handler on a `letsencrypt.log` located in a temporary directory, and this directory and content was wipped out at each test tearDown. As a consequence, the file handles, not cleared from the logger, were accumulating in the logger, with all of them connected to a deleted file log, except the last one that was just created by the current test. Considering the number of tests concerned, there were ~300 file handler at the end of pytest execution.
One can see that, on prior #6667, by calling `print(logger.getLogger().handlers` on the `tearDown` of these tests, and see the array growing at each test execution.
Even if this represent a memory leak, this situation was not really a problem on Linux: because a file can be deleted before it is closed, it was only meaning that a given invocation of `logger.debug` for instance, during the tests, was written in 300 log files. The overhead is negligeable. On Windows however, the file handlers were failing because you cannot delete a file before it is closed.
It was one of the reason for #6667, that added a call to `logging.shutdown()` at each test tearDown, with the consequence to close all file handlers. At this point, Linux is not happy anymore. Any call to `logger.warn` will generate an error for each closed file handler. As a file handler is added for each test, the number of errors grows on each test, following an arithmetical suite divergence.
On `test_sdists.py`, that is using the bare setuptools test suite without output capturing, we can see the damages. The total output takes 216000 lines, and 23000 errors are generated. A decent machine can support this load, but a not a small AWS instance, that is crashing during the execution. Even with pytest, the captured output and the memory leak become so large that segfaults are generated.
On the current PR, the problem is solved, by resetting the file handlers array on the logging system on each test tearDown. So each fileHandler is properly closed, and removed from the stack. They do not participate anymore in the logging system, and can be garbage collected. Then we stay on always one file handler opened at any time, and tests can succeed on AWS instances.
For the record, here is all the places where the logging system is called and fail if there is still file handlers closed but not cleaned (extracted from the original huge output before correction):
```
Logged from file account.py, line 116
Logged from file account.py, line 178
Logged from file client.py, line 166
Logged from file client.py, line 295
Logged from file client.py, line 415
Logged from file client.py, line 422
Logged from file client.py, line 480
Logged from file client.py, line 503
Logged from file client.py, line 540
Logged from file client.py, line 601
Logged from file client.py, line 622
Logged from file client.py, line 750
Logged from file cli.py, line 220
Logged from file cli.py, line 226
Logged from file crypto_util.py, line 101
Logged from file crypto_util.py, line 127
Logged from file crypto_util.py, line 147
Logged from file crypto_util.py, line 261
Logged from file crypto_util.py, line 283
Logged from file crypto_util.py, line 307
Logged from file crypto_util.py, line 336
Logged from file disco.py, line 116
Logged from file disco.py, line 124
Logged from file disco.py, line 134
Logged from file disco.py, line 138
Logged from file disco.py, line 141
Logged from file dns_common_lexicon.py, line 45
Logged from file dns_common_lexicon.py, line 61
Logged from file dns_common_lexicon.py, line 67
Logged from file dns_common.py, line 316
Logged from file dns_common.py, line 64
Logged from file eff.py, line 60
Logged from file eff.py, line 73
Logged from file error_handler.py, line 105
Logged from file error_handler.py, line 110
Logged from file error_handler.py, line 87
Logged from file hooks.py, line 248
Logged from file main.py, line 1071
Logged from file main.py, line 1075
Logged from file main.py, line 1189
Logged from file ops.py, line 122
Logged from file ops.py, line 325
Logged from file ops.py, line 338
Logged from file reporter.py, line 55
Logged from file selection.py, line 110
Logged from file selection.py, line 118
Logged from file selection.py, line 123
Logged from file selection.py, line 176
Logged from file selection.py, line 231
Logged from file selection.py, line 310
Logged from file selection.py, line 66
Logged from file standalone.py, line 101
Logged from file standalone.py, line 88
Logged from file standalone.py, line 97
Logged from file standalone.py, line 98
Logged from file storage.py, line 52
Logged from file storage.py, line 59
Logged from file storage.py, line 75
Logged from file util.py, line 56
Logged from file webroot.py, line 165
Logged from file webroot.py, line 186
Logged from file webroot.py, line 187
Logged from file webroot.py, line 204
Logged from file webroot.py, line 223
Logged from file webroot.py, line 234
Logged from file webroot.py, line 235
Logged from file webroot.py, line 237
Logged from file webroot.py, line 91
```
* Reapply #6667
* Make setuptools delegates tests execution to pytest, like in acme module.
* Clean handlers at each tearDown to avoid memory leaks.
* Update changelog
This PR fixes certbot-nginx and relevant tests to make them succeed on Windows.
Next step will be to enable integration tests through certbot-ci in a future PR.
* Fix tests and incompabilities in certbot-nginx for Windows
* Fix lint, fix oldest local dependencies
The method `lock_and_call`, in `certbot.tests.util` is designed to acquire a lock on a foreign process, then execute a callable in the current process. This is done to closely reproduce the lock mechanism involved between two certbot instances that are running in parallel.
This method uses the `multiprocessing` module. But its implementation in `lock_and_call` is broken for Windows: the two processes fail to communicate, leading to a deadlock.
In fact, `multiprocessing` module is using the fork mechanism on Linux, and the spawn mechanism on Windows, leading to behavior inconsistencies between the two platforms.
As this method is for tests, and not for production code, I did not try to make two implementations "by the book", one suitable for Windows, the other for Linux, like for the `certbot.lock` module.
Instead, I use a `subprocess` approach with a trigger file allowing to coordinate the current process and the subprocess. With this, `lock_and_call` is running from the same code both on Linux and Windows.
Relevant tests in the `certbot.tests.lock_test` test module are now enabled for Windows.
* Implement new lock_and_call method
* Reactivate tests for Windows
First PR about this issue, #6440, involved to much refactoring to ensure a correct behavior on Linux and installer plugins.
This PR proposes a new implementation of a lock mechanism for Linux and Windows, without implying a refactoring. It takes strictly the existing behavior for Linux, and add the appropriate logic for Windows. The `lock` module formalizes two independant mechanism dedicated to each platform, to improve maintainability.
Tests related to locking are re-activated for Windows, or definitively skipped because of irrelevancy for this platform. 6 more tests are enabled overall.
* Reimplement lock file on basic level
* Remove unused code
* Re-activate some tests
* Update doc
* Reactivate tests relevant to locks in Windows. Correct a test that was not testing what is supposed to test.
* Clean compat.
* Move close sooner in Windows lock implementation
* Add strong mypy types
* Use os.name
* Refactor lock mechanism logic
* Enable more tests
* Update lock.py
* Update lock_test.py
This PR updates and fixes `pebble-fetch.sh` considering latest improvements done on Pebble, to start a working instance.
* Fix the pebble fetch script
* Update pebble-fetch.sh
* Update tox.ini
In response to #6594. [Fixes #6594.]
To execute OCSP requests, certbot relies currently on a openssl binary execution. If openssl is not present in the PATH, the OCSP check will be silently ignored. Since version 2.4, cryptography has support for OCSP requests, without the need to have openssl binary available locally.
This PR takes advantage of it, and will use the built-in support of OCSP in cryptography for versions >= 2.4. Otherwise, fallback is done do a direct call to openssl binary, allowing oldest requirements to still work with legacy cryptography versions.
Update: requirement is now cryptography >= 2.5, to avoid to rely on a private method from cryptography.
* Implement logic using cryptography
* Working OSCP using pure cryptography
* Fix openssl usage in unit tests
* Reduce verbosity
* Add tests
* Improve naive skipIf
* Test resiliency
* Update ocsp.py
* Validate OCSP response. Unify OCSP URL get
* Improve resiliency checks, correct lint/mypy
* Improve hash selection
* Fix warnings when calling openssl bin
* Load OCSP tests assets as vectors.
* Update ocsp.py
* Protect against invalid ocsp response.
* Add checks to OCSP response
* Add more control on ocsp response
* Be lenient about assertion that next_update must be in the future, similarly to openssl.
* Construct a more advanced OCSP response mock to trigger more logic in ocsp module.
* Add test
* Refactor signature process to use crypto_util
* Fallback for cryptography 2.4
* Avoid a collision with a meteor.
* Correct method signature documentation
* Relax OCSP update interval
* Trigger built-in ocsp logic from cryptography with 2.5+
* Update pinned version of cryptography
* Update certbot/ocsp.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update ocsp.py
* Update ocsp_test.py
* Update CHANGELOG.md
* Update CHANGELOG.md
