On Linux, you can invoke os.rename(src, dst) even if dst already exists. In this case, destination file will be atomically replaced by the source file.
On Windows, this will lead to an OSError because changes are not atomic. This cause certbot renew to fail in particular, because the old certificate configuration needs to be replace by the new when a certificate is effectively renewed.
One could use the cross-platform function os.replace, but it is available only on Python >= 3.3.
This PR add a function in compat to handle correctly this case on Windows, and delegating everything else to os.rename.
* Cross platform compatible os.rename (we can use os.replace if its python 3)
* Use os.replace instead of custom non-atomic code.
* Avoid errors for lint and mypy. Add a test.
* flip challenge preference in Nginx
* Fix Nginx tests
* Flip challenge preference in Apache
* Flip challenge preference in standalone
* update changelog
* continue to run with tls-sni in integration tests for coverage
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
We want to discourage people from moving things around in `/etc/letsencrypt/live`! So we dropped an extra README in the `/etc/` directory when it's first created.
* Warn users not to remove/undo previous challenges
* Even more specific DNS challenge message
* Fix spacing and variable names
* Create a second test DNS challenge for UI testing
* Changelog for subsequent manual challenge behavior
* Add and use a compatibility layer to allow certbot to be run on windows.
* Fix path comparison
* Corrections on compat and util for tests
* Less intrusive way to parse prefix in webroot plugin working for both linux and windows.
* Disable pylint import-error for some optional imports in compat.py
* Ensure path is normalized before prefixes are generated in webroot plugin
* Same prefixes in linux and windows, in fact root path is not needed in webroot plugin
* Check that user has administrative rights before continuing on windows (necessary for symlink creation)
* More straightforward way to test administrative rights on windows
* Try to resolve import error in travis ci
* OK. We go for full introspection to trick the ci.
* Move the administrative rights control to the certbot entrypoint
* Add comment for a really non trivial code.
* Allow some commands to be run on a shell without admin rights
* Avoid races conditions on windows for lock files
* Add sphinx doc to the compat functions.
* Remove irrelevant Windows error in the lock mechanism.
* Some corrections on compat
Fixes#6207.
As noted by Erica:
- we no longer need to check if it exists before linking to it, because we delete properly.
- the previously excisting check on if server is in `LE_REUSE_SERVERS` before unlinking is nice, but probably not necessary, especially since we don't officially support people doing weird things with symlinks in our directories, and because we rmdir which will fail if it's not empty anyway.
* Create single account symlink.
* refactor _delete_accounts_dir_for_server_path
* add symlinked account dir deletion
* add tests
Implement an Authenticator which can fulfill a dns-01 challenge using the OVH DNS API. Applicable only for domains using OVH DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-ovh -d`, specifying a credentials file as a command line argument. Verified that a certificate was successfully obtained without user interaction.
* Used `certbot certonly --dns-ovh -d`, without specifying a credentials file as a command line argument. Verified that the user was prompted and that a certificate was successfully obtained.
* Used `certbot certonly -d`. Verified that the user was prompted for a credentials file after selecting dnsimple interactively and that a certificate was successfully obtained.
* Used `certbot renew --force-renewal`. Verified that certificates
were renewed without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Path to credentials file with an invalid application key.
* Path to credentials file with an invalid application secret.
* Path to credentials file with an invalid consumer key.
* Path to credentials file with missing properties.
* Domain name not registered to OVH account.
Implement an Authenticator which can fulfill a dns-01 challenge using
the Gehirn DNS (Gehirn Infrastructure Service) API.
Applicable only for domains using Gehirn DNS for DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-gehirn -d`, specifying a
credentials file as a command line argument. Verified that a
certificate was successfully obtained without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Domain name not registered to Gehirn DNS account.
This stops us from printing messages like:
"Could not choose appropriate plugin for updaters: Could not select or initialize the requested installer None."
when certbot renew --force-renewal is run with a lineage that doesn't have an installer.
* unquote None
* Test None values aren't saved in config file.
Implement an Authenticator which can fulfill a dns-01 challenge using
the Sakura Cloud DNS API.
Applicable only for domains using Sakura Cloud for DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-sakuracloud -d`, specifying a
credentials file as a command line argument. Verified that a
certificate was successfully obtained without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Domain name not registered to Sakura Cloud account.
* Added DNS based authenticator plugin for Linode
* Added linode plugin to docs
* Added Dockerfile
* Added .gitignore and readthedocs.org.requirements.txt
* Updated default_propagation_seconds
* Updated according to changes requested
* Bump version to 0.26.0
* Advertise our packages work on Python 3.7.
* find the correct url when deactivating an acmev1 account on the acmev2 endpoint
* set regr in ClientNetwork.account after deactivating on the server
* update self.net.account
* move logic into update_registration
* return methods to their original order to please git
* factor out common code
* update test_fowarding to use a method that still gets forwarded
* add acme module test coverage
* pragma no cover on correct line
* use previous regr uri
* strip unnecessary items from regr before saving
* add explanation to main.py
* add extra check to client_test.py
* use empty dict instead of empty string to indicate lack of body that we save to disk
Brad Warren