* cli: improve Obtaining/Renewing wording
* dont use logger, and use new phrasing
* .display_util.notify: dont wrap
As this function is supposed to be an analogue for print, we do not want
it to wrap by default.
* tests: fix leaking patch in eff_test.py
* tests: PrintTest->NotifyTest in .display.util
The function was renamed during #8432. This change renames the test as
well.
* IDisplay.notification: add `decorate` param.
The flag allows the caller to control whether the message will be
printed in a decorated way (wrapped by hlines) or in an undecorated
way (similar to print).
It is set to true by default, to reflect the existing behavior of the
function.
* IDisplay.notification: write message to debug log
In the same vein as IReporter, this ensures that all notifications which
are shown to the user also make an appearance in the debug log, which
will aid in troubleshooting.
* restore accidentally deleted newline in decoration
* add helper function for printing status messages
* register: use notify rather than logger
Undoes the change in #8393 in favor of the new helper
* comment .display and ._internal.log
Describing when it is suitable to use each
* add more comments to log.py
* make IDisplay.notification decorate arg private
* rename notify->print and move to .display.util
* rename .display.print back to .display.notify
because linters complain about print being a redefined builtin
This PR is exactly the same as #7895, but know we know a little bit more about what was going on with `mypy`.
Part of #7886.
This PR conditionally installs mock in `certbot/setup.py` based on setuptools version and python version, when possible. It then updates `certbot` tests to use `unittest.mock` when `mock` isn't available.
* Conditionally install mock in certbot
* use unittest.mock when third-party mock isn't available in certbot
* Add type:ignores because of https://github.com/python/mypy/issues/1153
* error out on newer python and older setuptools
* error when trying to build wheels with old setuptools
* Revert "Do not require mock in Python 3 in certbot module (#7895)"
This reverts commit 77871ba71c.
* Revert "Do not require mock in Python 3 in acme module (#7894)"
This reverts commit cd0acf5dcc.
Part of #7886.
This PR conditionally installs mock in `certbot/setup.py` based on setuptools version and python version, when possible. It then updates `certbot` tests to use `unittest.mock` when `mock` isn't available.
* Conditionally install mock in certbot
* use unittest.mock when third-party mock isn't available in certbot
* Add type:ignores because of https://github.com/python/mypy/issues/1153
* error when trying to build wheels with old setuptools
* Fix dangerous default argument
* Remove unused imports
* Remove unnecessary comprehension
* Use literal syntax to create data structure
* Use literal syntax instead of function calls to create data structure
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Fixes#7594.
Removes the code asking interactively if the user would like to add a redirect.
* Remove interactive redirect ask
* display.enhancements is no longer used, so remove it.
* update changelog
* remove references to removed display.enhancements
* add redirect_default flag to enhance_config to conditionally set default for redirect value
* Update default in help text.
This PR builds on #7657 and cleans up additional unnecessary pylint comments and some stray comments referring to pylint: disable comments that have been deleted that I didn't notice in my review of that PR.
* Remove stray pylint link.
* Cleanup more pylint comments
* Cleanup magic_typing imports
* Remove unneeded pylint: enable comments
Part of #7550
This PR makes appropriate corrections to run pylint on Python 3.
Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.
Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.
Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.
Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.
That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.
I also made PyLint and MyPy checks run correctly on Windows.
* Start configuration
* Reconfigure travis
* Suspend a check specific to python 3. Start fixing code.
* Repair call_args
* Fix return + elif lints
* Reconfigure development to run mainly on python3
* Remove incompatible Python 3.4 jobs
* Suspend pylint in some assertions
* Remove pylint in dev
* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path
* Various return + else lint errors
* Find a set of deps that is working with current mypy version
* Update local oldest requirements
* Remove all current pylint errors
* Rebuild letsencrypt-auto
* Update mypy to fix pylint with new astroid version, and fix mypy issues
* Explain type: ignore
* Reconfigure tox, fix none path
* Simplify pinning
* Remove useless directive
* Remove debugging code
* Remove continue
* Update requirements
* Disable unsubscriptable-object check
* Disable one check, enabling two more
* Plug certbot dev version for oldest requirements
* Remove useless disable directives
* Remove useless no-member disable
* Remove no-else-* checks. Use elif in symetric branches.
* Add back assertion
* Add new line
* Remove unused pylint disable
* Remove other pylint disable
* Move display/completer.py to _internal/
* Move display/dummy_readline.py to _internal/
* Move display/enhancements.py to _internal/
* Create __init__.py in _internal/display
* Create _internal package for Certbot's non-public modules
* Move account.py to _internal
* Move auth_handler.py to _internal
* Move cert_manager.py to _internal
* Move client.py to _internal
* Move error_handler.py to _internal
* Move lock.py to _internal
* Move main.py to _internal
* Move notify.py to _internal
* Move ocsp.py to _internal
* Move renewal.py to _internal
* Move reporter.py to _internal
* Move storage.py to _internal
* Move updater.py to _internal
* update apache and nginx oldest requirements
* Keep the lock file as certbot.lock
* nginx oldest tests still need to rely on newer certbot
* python doesn't have good dependency resolution, so specify the transitive dependency
* update required minimum versions in nginx setup.py
This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
This PR is the first part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This step 1 refactor existing certbot.compat module into certbot.compat.misc, without any logic changed. Package certbot.compat will host the new modules that constitute the security model for Windows.
* Create the certbot.compat package. Move logic in certbot.compat.misc
* Add doc
* Fix lint
* Correct mypy
* Update client.py
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Add and use a compatibility layer to allow certbot to be run on windows.
* Fix path comparison
* Corrections on compat and util for tests
* Less intrusive way to parse prefix in webroot plugin working for both linux and windows.
* Disable pylint import-error for some optional imports in compat.py
* Ensure path is normalized before prefixes are generated in webroot plugin
* Same prefixes in linux and windows, in fact root path is not needed in webroot plugin
* Check that user has administrative rights before continuing on windows (necessary for symlink creation)
* More straightforward way to test administrative rights on windows
* Try to resolve import error in travis ci
* OK. We go for full introspection to trick the ci.
* Move the administrative rights control to the certbot entrypoint
* Add comment for a really non trivial code.
* Allow some commands to be run on a shell without admin rights
* Avoid races conditions on windows for lock files
* Add sphinx doc to the compat functions.
* Remove irrelevant Windows error in the lock mechanism.
* Some corrections on compat
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
* Addressing #4071 Wrote an ImmutableReturnMixin to prevent developers overriding return_value in certain Mock objects
* Language
* Loosening the assumption that underlying _mock objects need to be Immutable-like simplifies implementation
* Addressing #4071
* Ensure side_effects and return_values are pushed down to the underlying _mock in FreezableMocks. And IDisplay mocks are no longer frozen in _create_get_utility_mock()
* Edit a handful of tests to not override the mock_get_utility return_value
* Brief explainer of FreezableMock.__setattr__
* Incorporating PR feedback and some compatibility
* FreezableMock __getattr__ needs a shortcut in case of return_value or side_effect
* Changing return_value only forbidden if set before freezing
* Remove unnecessary else block
* Expanded doc strings
* Bring a couple new tests in line with patch_get_utility() norms
* display: support validation of user input
To avoid each caller of `display.input` and `display.directory_select`
needing to implement validation logic, this allows for a validator to be
supplied as a part of the call.
Following the existing pattern from `webroot`, this validator is expected
to throw a `Error` when it encounters invalid input. The user then
receives a `notification` is re-prompted.
Testing Done:
* tox -e py27
* tox -e lint
* plugins: update webroot to use display's validation functionality
This change updates the webroot plugin to use the now-built-in validation
functionality in display, reducing duplicated code.
Testing Done:
* tox -e py27
* tox -e lint
* display: move validation logic to ops
To avoid adding complexity to `IDisplay` methods, move validation logic
to helper methods in `display.ops`.
Testing Done:
* tox -e py27
* tox -e lint
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
* Revert "Revert "Remove Link rel=next for authzs and new-certs." (#4277)"
This reverts commit 11ec1eb911.
* Save new_authzr_uri with account for older clients.
* Add test that new_authzr_uri exists in regr.
* Restore backwards compatibility for new_authzr_uri.
* Fix account_test.
* Add test for deprecated URI argument to request_challenges.
* Review feedback.
* Fix test
* Add omitempty to new_cert_uri.
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
Fixes#3996.
I'm pretty confident this PR solves the problem. I've audited all calls to IDisplay methods and the assertions done in certbot.display.util are now done in all our unit tests.
With that said, it wouldn't hurt to have someone else double check I didn't miss anything. The easiest way to do this is to grep for IDisplay in our code and ensure all calls to IDisplay methods are valid. This means every method call other than notification (because a notification call is always OK) either provides a value for default or force_interactive. This is defined in interfaces.py.
I've also been considering removing the assertion that's been causing us trouble here from our release. The only argument I have for not doing so is it may hinder 3rd party plugin development. When they use IDisplay, they have the same problem as we do with prompting users without a TTY. Not keeping this assertion in makes it more likely they won't notice the issue and Certbot will crash on an unsuspecting user.
With that said, none of our known 3rd party plugins use IDisplay at all.
* Provide force_interactive in _get_certname
* Use force_interactive when asking for webroot
* Factor IDisplay assertion into it's own function
* Add util.patch_get_utility()
* Allow custom path to patch_get_utiity
* Change GetEmailTest to use patch_get_utility
* Use new_callable to create new objects
* Modify tests to use patch_get_utility
* Improve FreezableMock documentation
* Add user facing error to TTY magic
* Comment out assert_valid_call
* Add test_input_assertion_fail2()
