So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Add and use a compatibility layer to allow certbot to be run on windows.
* Fix path comparison
* Corrections on compat and util for tests
* Less intrusive way to parse prefix in webroot plugin working for both linux and windows.
* Disable pylint import-error for some optional imports in compat.py
* Ensure path is normalized before prefixes are generated in webroot plugin
* Same prefixes in linux and windows, in fact root path is not needed in webroot plugin
* Check that user has administrative rights before continuing on windows (necessary for symlink creation)
* More straightforward way to test administrative rights on windows
* Try to resolve import error in travis ci
* OK. We go for full introspection to trick the ci.
* Move the administrative rights control to the certbot entrypoint
* Add comment for a really non trivial code.
* Allow some commands to be run on a shell without admin rights
* Avoid races conditions on windows for lock files
* Add sphinx doc to the compat functions.
* Remove irrelevant Windows error in the lock mechanism.
* Some corrections on compat
* Addressing #4071 Wrote an ImmutableReturnMixin to prevent developers overriding return_value in certain Mock objects
* Language
* Loosening the assumption that underlying _mock objects need to be Immutable-like simplifies implementation
* Addressing #4071
* Ensure side_effects and return_values are pushed down to the underlying _mock in FreezableMocks. And IDisplay mocks are no longer frozen in _create_get_utility_mock()
* Edit a handful of tests to not override the mock_get_utility return_value
* Brief explainer of FreezableMock.__setattr__
* Incorporating PR feedback and some compatibility
* FreezableMock __getattr__ needs a shortcut in case of return_value or side_effect
* Changing return_value only forbidden if set before freezing
* Remove unnecessary else block
* Expanded doc strings
* Bring a couple new tests in line with patch_get_utility() norms
* display: support validation of user input
To avoid each caller of `display.input` and `display.directory_select`
needing to implement validation logic, this allows for a validator to be
supplied as a part of the call.
Following the existing pattern from `webroot`, this validator is expected
to throw a `Error` when it encounters invalid input. The user then
receives a `notification` is re-prompted.
Testing Done:
* tox -e py27
* tox -e lint
* plugins: update webroot to use display's validation functionality
This change updates the webroot plugin to use the now-built-in validation
functionality in display, reducing duplicated code.
Testing Done:
* tox -e py27
* tox -e lint
* display: move validation logic to ops
To avoid adding complexity to `IDisplay` methods, move validation logic
to helper methods in `display.ops`.
Testing Done:
* tox -e py27
* tox -e lint
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
* Revert "Revert "Remove Link rel=next for authzs and new-certs." (#4277)"
This reverts commit 11ec1eb911.
* Save new_authzr_uri with account for older clients.
* Add test that new_authzr_uri exists in regr.
* Restore backwards compatibility for new_authzr_uri.
* Fix account_test.
* Add test for deprecated URI argument to request_challenges.
* Review feedback.
* Fix test
* Add omitempty to new_cert_uri.
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
Fixes#3996.
I'm pretty confident this PR solves the problem. I've audited all calls to IDisplay methods and the assertions done in certbot.display.util are now done in all our unit tests.
With that said, it wouldn't hurt to have someone else double check I didn't miss anything. The easiest way to do this is to grep for IDisplay in our code and ensure all calls to IDisplay methods are valid. This means every method call other than notification (because a notification call is always OK) either provides a value for default or force_interactive. This is defined in interfaces.py.
I've also been considering removing the assertion that's been causing us trouble here from our release. The only argument I have for not doing so is it may hinder 3rd party plugin development. When they use IDisplay, they have the same problem as we do with prompting users without a TTY. Not keeping this assertion in makes it more likely they won't notice the issue and Certbot will crash on an unsuspecting user.
With that said, none of our known 3rd party plugins use IDisplay at all.
* Provide force_interactive in _get_certname
* Use force_interactive when asking for webroot
* Factor IDisplay assertion into it's own function
* Add util.patch_get_utility()
* Allow custom path to patch_get_utiity
* Change GetEmailTest to use patch_get_utility
* Use new_callable to create new objects
* Modify tests to use patch_get_utility
* Improve FreezableMock documentation
* Add user facing error to TTY magic
* Comment out assert_valid_call
* Add test_input_assertion_fail2()
* CLI flag for forcing interactivity
* add --force-interactive
* Add force_interactive error checking and tests
* Add force_interactive parameter to FileDisplay
* add _can_interact
* Add _return_default
* Add **unused_kwargs to NoninteractiveDisplay
* improve _return_default assertion
* Change IDisplay calls and write tests
* Document force_interactive in interfaces.py
* Don't force_interactive with a new prompt
* Warn when skipping an interaction for the first time
* add specific logger.debug message
* Begin breaking out cli_test.py
* simplify main
* refactor porse tests
* move determine account tests to main_test.py
* move duplicate cert test to main_test.py
* move cli stuff out of the way
* add test_renewal.py
* move error test into error_handler_test.py
* move test_read_file
* move test_no_gui out of MainTest
* move test_install_abspath to parsetest
* Move main tests into main_test.py
* move cli tests back into cli_test.py
* clean up cli_test.py
* move punycode test to util_test.py
* Fix NameError from missing plugins_disco
* Fix linting errors
* test_renewal.py -> renewal_test.py
* rm not_cli_test.py
* Move main._handle_exception test to main_test.py
* Move renewal import in renewal_test.py
from @ohemorange comments
* certbot.tests.test_util -> certbot.tests.util
* Fix issues from rebasing.
* Fix testing issue with option_was_set
* fix linting issue
* Output status for `revoke` operation. Fixes#2819.
- Added method to `certbot.display.ops` to output confirmation of `revoke`.
- Wrapped call to `acme.client.Client.revoke` in a try to statement to
handle possible error.
- Added test for `main.revoke`.
* Added test for failure of certificate revocation.
Moved creation of mocks into RevokeTest setup function.
Stopped mocks in RevokeTest teardown function.
* Fixed lint errors.
* Do not call `unittest.TestCase.assertRaises` as a context manager (to work with py26).
* Fixed spelling error in successful revocation notification.
Added test for the notification.
* Changed informational messages because of confusing message on reinstallation.
Certbot prompts the user when it detects that an appropriately fresh certificate
is already available:
You have an existing certificate that contains exactly the same domains you requested and isn't close to expiry.
(ref: <path>)
What would you like to do?
-------------------------------------------------------------------------------
1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (limit ~5 per 7 days)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
On selecting '1' (reinstall), the resulting message is:
-------------------------------------------------------------------------------
Your existing certificate has been successfully reinstalled, and the new
certificate has been installed.
The new certificate covers the following domains: https://<whatever>
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=<whatever>
-------------------------------------------------------------------------------
"Your existing certificate has been successfully reinstalled" <-- Okay
"and the new certificate has been installed." <-- Wait, what?
The issue appears to come from assumptions in certbot/certbot/main.py
It uses `len(lineage.available_versions("cert"))` to determine if this was a
fresh install or renewal, and then calls either `display_ops.success_renewal()`
(which produces the "existing certificate ... and the new certificate" language)
or `display_ops.success_installation()` (which has no messaging about existing
vs. new certificates).
The len(lineage) test isn't the right way to make this choice. The certificate's
lineage length doesn't imply anything about whether we've just obtained a new
certificate, because there is no new certificate in the case of a "reinstall"
action.
The new logic calls `display_ops.success_installation()` on all "reinstall"
actions, and otherwise employs the existing `len(lineage)` test.
Additionally the `display_ops.success_installation()` has been enhanced to
accept an action parameter, and has the message reworded slightly to make
sense regardless of the action passed. The messaging is mostly unchanged if it's
called without the action parameter:
Original message:
-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://<whatever>
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=<whatever>
-------------------------------------------------------------------------------
New message on initial install:
-------------------------------------------------------------------------------
Congratulations! You have successfully installed a certificate for
https://<whatever>
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=<whatever>
-------------------------------------------------------------------------------
New message on re-install:
-------------------------------------------------------------------------------
Congratulations! You have successfully reinstalled a certificate for
https://<whatever>
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=<whatever>
-------------------------------------------------------------------------------
* Typo in display message.
* Typo, characters transposed.
* undo changes to certbot/display/ops.py
* remove invalid todos
* Test success_installation() called for reinstall
* Simplify display_ops.success* functions
* refactor and expand run() tests
* Punycode is about to be permitted; stop rejecting it
* Remove spurious bracket
* More brackets rather than fewer!
* Change ops_test's notion of valid domains
* Remove spurious "certonly" from new test
* Make test more localized
* Remove commented-out punycode prohibition
* remove unhelpful question about servernames and default vhosts
* add prefix about names found in config files
* test we include configuration files prefix
* Tell the user what kind of conf files were missing domains
* Revert "Tell the user what kind of conf files were missing domains"
This reverts commit 1066a88dae.
Erica Portnoy