This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes --tls-sni-01-port, --tls-sni-01-address and tls-sni-01/tls-sni options from --preferred-challenges. They are replace by deprecation warning, indicating that these options will be removed soon.
This deprecation, instead of complete removal, is done to avoid certbot instances to hard fail if some automated scripts still use these flags for some users.
Once this PR lands, we can remove completely theses flags in one or two release.
* Remove tls-sni related flags in cli. Add a deprecation warning instead.
* Adapt tests to cli and renewal towards tls-sni flags deprecation
* Add https_port option. Make tls_sni_01_port show a deprecation warning, but silently modify https_port if set
* Migrate last items
* Fix lint
* Update certbot/cli.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Ensure to remove all occurences of tls-sni-01
* Remove unused parameter
* Revert modifications on cli-help.txt
* Use logger.warning instead of sys.stderr
* Update the logger warning message
* Remove standalone_supported_challenges option.
* Fix order of preferred-challenges
* Remove supported_challenges property
* Fix some tests
* Fix lint
* Fix tests
* Add a changelog
* Clean code, fix test
* Update CI
* Reload
* No hard date for tls-sni removal
* Remove useless cast to list
* Update certbot/tests/renewal_test.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add entry to the changelog
* Add entry to the changelog
* Extraction from #6541 to add flag to disable shuffle sleep on renew action
* Move the logic of random sleep to execute it only if there is effectively a certificate to renew.
* Add comments
* Correct lint
* Suspend lint rule
* Revert code cleaning
* Hide the flag
* Ignore lint
* Update cli.py
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Initial work on new version of --reuse-key
* Test for reuse_key
* Make lint happier
* Also test a non-dry-run reuse_key renewal
* Test --reuse-key in boulder integration test
* Better reuse-key integration testing
* Log fact that key was reused
* Test that the certificates themselves are different
* Change "oldkeypath" to "old_keypath"
* Simply appearance of new-key generation logic
* Reorganize new-key logic
* Move awk logic into TotalAndDistinctLines function
* After refactor, there's now explicit None rather than missing param
* Indicate for MyPy that key can be None
* Actually import the Optional type
* magic_typing is too magical for pylint
* Remove --no-reuse-key option
* Correct pylint test disable
When Certbot is run with --dry-run, skip running GenericUpdater and RenewDeployer interface methods.
This PR also makes the parameter order of updater.run_generic_updaters and updater.run_renewal_deployer consistent.
Fixes#5927
* Do not call updaters and deployers when run with --dry-run
* Use ConfigTestCase instead of mocking config objects manually
* ServerTLSUpdater and InstallerSpecificUpdater implementation
* Fixed tests and added disables for linter :/
* Added error logging for misconfigurationerror from plugin check
* Remove redundant parameter from interfaces
* Renaming the interfaces
* Finalize interface renaming and move tests to own file
* Refactored the runners
* Refactor the cli params
* Fix the interface args
* Fixed documentation
* Documentation and naming fixes
* Remove ServerTLSConfigurationUpdater
* Remove unnecessary linter disable
* Rename run_renewal_updaters to run_generic_updaters
* Do not raise exception, but make log message more informative and visible for the user
* Run renewal deployer before installer restart
* update order object with returned authorizations
* major structure of finalize_order shim refactor
* util methods and imports for finalize_order shim refactor
* update certbot.tests.client_test.py
* extraneous client_test imports
* remove correct import
* update renewal call
* add test for acme.dump_pyopenssl_chain
* Add test for certbot.crypto_util.cert_and_chain_from_fullchain
* add tests for acme.client and change to fetch chain failure to TimeoutError
* s/rytpe/rtype
* remove ClientV1 passthrough
* dump the wrapped cert
* remove dead code
* remove the correct dead code
* support earlier mock
* Refactor main to simplify logic
* Update tests and comments
* Correct main test
* increase timeout limit
* reset timeout limit
* call renew_cert in appropriate main test
* Update docstrings and revert signatures of _report_new_cert and _suggest_donation_of_appropriate
* replace renew_cert logic
* update tests
* rename _csr_obtain_cert and add a check to _report_new_cert
* use challenge type strings, not objectS
* Factor out parse_preferred_challenges
* restore pref_challs
* save pref_challs
* Make CheckCertCount more flexible
* improve integration tests
* Make pref_challs more flexible
* Remove incorrect hook message
* Add generalized msg about skipping hooks
* Properly report when hooks were skipped
* Also print message about renew hook
* Change quotes to help OCD
* organize cert_manager.py
* add delete files to cert manager and storage
* add tests
* add to main and cli
* Clean up all related files we can find, even if some are missing.
* error messages, debug logs, and remove RenewerConfiguration
* add logs for failure to remove
* remove renewer_config_file
* Rename and simplify main functions
* pass certname to auth method
* find cert by certname flag
* Implement --cert-name command
* don't ask to confirm new cert when we have domains and no existing certs with the lineage name
* Refactor and add --new-cert-name flag
* add interactivity to rename verb
* allow noninteractive and more descriptive function names
* Switch to using absolute path in symlink
* save archive_dir to config and read it back
* cli_config.archive_dir --> cli_config.default_archive_dir
* Use archive_dir specified in renewal config file
* add helpful broken symlink info
* add docstring to method
* Add tests
* remove extraneous test imports
* fix tests
* py2.6 syntax fix
* git problems
* Add list-certs command
* no dict comprehension in python2.6
* add test coverage
* More py26 wrangling
* update tests for py3 and lint
* remove extra dep from test
* test coverage
* test shouldn't be based on dict representation order
* Redo report UX and add tests to cover
* remove storage str test
* lint and use mock properly
* mock properly
* address code review comments
* lineage --> certificate name and print fullchain and privkey paths
* make py26 happy
* actually make py26 happy
* don't wrap text
Re-do the fix for #3497 to ensure it works in all cases.
* If lineages are in an inconsistent (non-deployed) state, deploy them
* Test new _handle_identical_cert case
* Move lineage.has_pending_deployment() check up to _auth_from_domains
Less conceptually nice, but in the "renew" verb case it wasn't being called :(
* Swap _auth_from_domains return type
* It now matches _treat_as_renewal & _handle_identical_cert_request etc
* Revert "Move lineage.has_pending_deployment() check up to _auth_from_domains"
This reverts commit a7fe734d73.
* Move test back to handle_identical_cert_request
* We need to check for non-deployment on two separate code paths
- Once high up in "renew" (because failure to be deployed stops us from
divind down the stack)
- Once way down in _handle_identical_cert_request (because that's where it
makes the most sense for run / certonly)
- So refactor that work into storage.py
* We don't necessarily reinstall
Log and show warning with real exception message to make it more clear what exactly happened.
Currently we show `config is broken` when in fact we have broken symlinks in live folder.
