This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
This PR is the first part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This step 1 refactor existing certbot.compat module into certbot.compat.misc, without any logic changed. Package certbot.compat will host the new modules that constitute the security model for Windows.
* Create the certbot.compat package. Move logic in certbot.compat.misc
* Add doc
* Fix lint
* Correct mypy
* Update client.py
* Extraction from #6541 to add flag to disable shuffle sleep on renew action
* Move the logic of random sleep to execute it only if there is effectively a certificate to renew.
* Add comments
* Correct lint
* Suspend lint rule
* Revert code cleaning
* Hide the flag
* Ignore lint
* Update cli.py
* WIP on adding a random sleep for noninteractive renewal
* Update changelog
* Log the fact that we're randomly sleeping
* stdin may better define interactivity than stdout
* Try mocking time.sleep for all tests
* Move mocked sleep elsewhere
* mock the right object
* Somewhat ugly synthetic PTY trick
* Move set -u down below self-exec
* Revert "Move set -u down below self-exec"
This reverts commit 6bde65a738.
* Revert "Somewhat ugly synthetic PTY trick"
This reverts commit 89c704a4be.
* Log specific duration of random sleep
* Test coverage for random sleep() logic in main.py
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Add and use a compatibility layer to allow certbot to be run on windows.
* Fix path comparison
* Corrections on compat and util for tests
* Less intrusive way to parse prefix in webroot plugin working for both linux and windows.
* Disable pylint import-error for some optional imports in compat.py
* Ensure path is normalized before prefixes are generated in webroot plugin
* Same prefixes in linux and windows, in fact root path is not needed in webroot plugin
* Check that user has administrative rights before continuing on windows (necessary for symlink creation)
* More straightforward way to test administrative rights on windows
* Try to resolve import error in travis ci
* OK. We go for full introspection to trick the ci.
* Move the administrative rights control to the certbot entrypoint
* Add comment for a really non trivial code.
* Allow some commands to be run on a shell without admin rights
* Avoid races conditions on windows for lock files
* Add sphinx doc to the compat functions.
* Remove irrelevant Windows error in the lock mechanism.
* Some corrections on compat
This stops us from printing messages like:
"Could not choose appropriate plugin for updaters: Could not select or initialize the requested installer None."
when certbot renew --force-renewal is run with a lineage that doesn't have an installer.
* unquote None
* Test None values aren't saved in config file.
* find the correct url when deactivating an acmev1 account on the acmev2 endpoint
* set regr in ClientNetwork.account after deactivating on the server
* update self.net.account
* move logic into update_registration
* return methods to their original order to please git
* factor out common code
* update test_fowarding to use a method that still gets forwarded
* add acme module test coverage
* pragma no cover on correct line
* use previous regr uri
* strip unnecessary items from regr before saving
* add explanation to main.py
* add extra check to client_test.py
* use empty dict instead of empty string to indicate lack of body that we save to disk
interfaces.GenericUpdater and new enhancement interface updater functions get run on every invocation of Certbot with "renew" verb for every lineage. This causes performance problems for users with large configurations, because of plugin plumbing and preparsing happening in prepare() method of installer plugins. This PR moves the responsibility to call prepare() to the plugin (possibly) implementing a new style enhancement interface.
Fixes: #6153
* Do not call IPlugin.prepare() for updaters when running renew
* Check prepare called in tests
* Refine pydoc and make the function name more informative
* Verify the plugin type
If either --cert-name or both --key-path and --cert-path (in which case the user requests installation for a certificate not managed by Certbot) are not provided, prompt the user with managed certificates and let them choose.
Fixes: #5824
This PR adds the functionality to enhance Apache configuration to include HTTP Strict Transport Security header with a low initial max-age value.
The max-age value will get increased on every (scheduled) run of certbot renew regardless of the certificate actually getting renewed, if the last increase took place longer than ten hours ago. The increase steps are visible in constants.AUTOHSTS_STEPS.
Upon the first actual renewal after reaching the maximum increase step, the max-age value will be made "permanent" and will get value of one year.
To achieve accurate VirtualHost discovery on subsequent runs, a comment with unique id string will be added to each enhanced VirtualHost.
* AutoHSTS code rebased on master
* Fixes to match the changes in master
* Make linter happy with metaclass registration
* Address small review comments
* Use new enhancement interfaces
* New style enhancement changes
* Do not allow --hsts and --auto-hsts simultaneuously
* MyPy annotation fixes and added test
* Change oldest requrements to point to local certbot core version
* Enable new style enhancements for run and install verbs
* Test refactor
* New test class for main.install tests
* Move a test to a correct test class
When Certbot is run with --dry-run, skip running GenericUpdater and RenewDeployer interface methods.
This PR also makes the parameter order of updater.run_generic_updaters and updater.run_renewal_deployer consistent.
Fixes#5927
* Do not call updaters and deployers when run with --dry-run
* Use ConfigTestCase instead of mocking config objects manually
This change will allow registering/updating account with multi emails.
Detail is enclosed in #4242
* support multi emails register
* add more test cases
* update test to unregister before register
* update create path to support multi emaill
* refactor payload updating
* fix typo
* move command line doc to another place
* revert the change for updating account registration info, added unit test
* rearrange text for consistency
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
Include a line break before "(None)" to maintain consistency with output
for lists that are not empty.
Previous result as expected for non-empty lists:
>>> _format_list('+', ['one', 'two', 'three'])
'\n+ one\n+ two\n+ three'
Previous unexpected result for empty lists:
>>> _format_list('+', [])
'\n+ '
New result as expected (unchanged) for non-empty lists:
>>> _format_list('+', ['one', 'two', 'three'])
'\n+ one\n+ two\n+ three'
New behavior more explicit for empty lists:
>>> _format_list('+', [])
'\n(None)'
Resolves#5886
* ServerTLSUpdater and InstallerSpecificUpdater implementation
* Fixed tests and added disables for linter :/
* Added error logging for misconfigurationerror from plugin check
* Remove redundant parameter from interfaces
* Renaming the interfaces
* Finalize interface renaming and move tests to own file
* Refactored the runners
* Refactor the cli params
* Fix the interface args
* Fixed documentation
* Documentation and naming fixes
* Remove ServerTLSConfigurationUpdater
* Remove unnecessary linter disable
* Rename run_renewal_updaters to run_generic_updaters
* Do not raise exception, but make log message more informative and visible for the user
* Run renewal deployer before installer restart
* Allow revoke to pass in a url
* Add revocation support to ACMEv2.
* Provide regr for account based revocation.
* Add revoke wrapper to BackwardsCompat client
* update order object with returned authorizations
* major structure of finalize_order shim refactor
* util methods and imports for finalize_order shim refactor
* update certbot.tests.client_test.py
* extraneous client_test imports
* remove correct import
* update renewal call
* add test for acme.dump_pyopenssl_chain
* Add test for certbot.crypto_util.cert_and_chain_from_fullchain
* add tests for acme.client and change to fetch chain failure to TimeoutError
* s/rytpe/rtype
* remove ClientV1 passthrough
* dump the wrapped cert
* remove dead code
* remove the correct dead code
* support earlier mock
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
Detects acme version by checking for newNonce field in the directory, since it's mandatory. Also updates ClientNetwork.account on register and update_registration.
* add mechanism to detect acme version
* update ClientNetwork.account comment
* switch to MultiVersionClient object in acme
* add shim methods
* add returns
* use backwards-compatible format and implement register
* update to actual representation of tos v2
* add tos fields and pass through to v1 for partial updates
* update tests
* pass more tests
* allow instance variable pass-through and lint
* update certbot and tests to use new_account_and_tos method
* remove --agree-tos test from main_test for now because we moved the callback into acme
* add docstrings
* use hasattr
* all most review comments
* use terms_of_service for both v1 and v2
* add tests for acme/client.py
* tests for acme/messages.py
* Fix install verb
* Fix error message, tests and remove global pylint change
* Fix boulder integration test keypath
* Also use chain_path from lineage if not defined on CLI
* Add --delete-after-revoke flags
* Use delete_after_revoke value
* Add delete_after_revoke unit tests
* Add integration tests for delete-after-revoke.
