After getting a +1 from everyone on the team, this PR removes the use of `codecov` from the Certbot repo because we keep having problems with it.
Two noteworthy things about this PR are:
1. I left the text at https://github.com/certbot/certbot/blob/4ea98d830bcc3d1b980a4055243c6a6a25d8dc54/.azure-pipelines/INSTALL.md#add-a-secret-variable-to-a-pipeline-like-codecov_token because I think it's useful to document how to set up a secret variable in general.
2. I'm not sure what the text "Option -e makes sure we fail fast and don't submit to codecov." in `tox.cover.py` refers to but it seems incorrect since `-e` isn't accepted or used by the script so I just deleted the line.
As part of this, I said I'd open an issue to track setting up coveralls (which seems to be the only real alternative to codecov) which is at https://github.com/certbot/certbot/issues/7810.
With my change, failure output looks something like:
```
$ tox -e py27-cover
...
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------------------
certbot/certbot/__init__.py 1 0 100%
certbot/certbot/_internal/__init__.py 0 0 100%
certbot/certbot/_internal/account.py 191 4 98% 62-63, 206, 337
...
certbot/tests/storage_test.py 530 0 100%
certbot/tests/util_test.py 374 29 92% 211-213, 480-484, 489-499, 504-511, 545-547, 552-554
------------------------------------------------------------------------------------------
TOTAL 14451 647 96%
Command '['/path/to/certbot/dir/.tox/py27-cover/bin/python', '-m', 'coverage', 'report', '--fail-under', '100', '--include', 'certbot/*', '--show-missing']' returned non-zero exit status 2
Test coverage on certbot did not meet threshold of 100%.
ERROR: InvocationError for command /Users/bmw/Development/certbot/certbot/.tox/py27-cover/bin/python tox.cover.py (exited with code 1)
_________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________
ERROR: py27-cover: commands failed
```
I printed the exception just so we're not throwing away information.
I think it's also possible we fail for a reason other than the threshold not meeting the percentage, but I've personally never seen this, `coverage report` output is not being captured so hopefully that would inform devs if something else is going on, and saying something like "Test coverage probably did not..." seems like overkill to me personally.
* remove codecov
* remove unused variable group
* remove codecov.yml
* Improve tox.cover.py failure output.
I want to do what I did in https://github.com/certbot/certbot/pull/7733 to our Azure Pipelines setup, but unfortunately this isn't currently possible. The only filters available for service hooks for the "build completed" trigger are the pipeline and build status. See
To accomplish this, I propose splitting the "advanced" pipeline into two cases. One is for builds on protected branches where we want to be notified if they fail while the other is just used to manually run tests on certain branches.
As discussed in #7539, we need proper tests of the Windows installer itself in order to variety that all the logic contained in a production-grade runtime of Certbot on Windows is correctly setup by each version of the installer, and so for a variety of Windows OSes.
This PR handles this requirement. The new `windows_installer_integration_tests` module in `certbot-ci` will:
* run the given Windows installer
* check that Certbot is properly installed and working
* check that the scheduled renew task is set up
* check that the scheduled task actually launch the Certbot renew logic
The Windows nightly tests are updated accordingly, in order to have the tests run on Windows Server 2012R2, 2016 and 2019.
These tests will evolve as we add more logic on the installer.
* Configure an integration test testing the windows installer
* Write the test module
* Configurable installer path, prepare azure pipelines
* Fix option
* Update test_main.py
* Add confirmation for this destructive test
* Use regex to validate certbot --version output
* Explicit dependency on a log output
* Use an exception to ask confirmation
* Use --allow-persistent-changes
This PRs extends the installer tests on Azure Pipeline, in order to run the integration tests on a certbot instance installed with the Windows installer for several Windows versions, corresponding to the scope of supported versions on Certbot:
* Windows Server 2012 R2
* Windows Server 2016
* Windows Server 2019
One can see the result on: https://dev.azure.com/adferrand/certbot/_build/results?buildId=311
* Try specific installer-build step
* Install Python manually
* Add tests on windows 2019
Clean up some places missed by #7544.
Found this when running test farm tests. They were working as of 5d90544, and I will truly shocked if subsequent changes (all to the windows installer) made them stop working.
* Release script needs to target new CHANGELOG location
* Clean up various other CHANGELOG path references
* Update windows paths for new certbot location
* Add certbot to packages list for windows installer
In Travis, the full test suite doesn't run on PRs for point release branches, just on commits for them. I think this behavior makes sense because what we actually want to test before a point release is the exact commit we want to release after any squashing/merging has been done. This PR modifies Azure to match this behavior.
After this PR lands, I need to update the tests required to pass on GitHub.
This PR creates a pipeline triggered on tag push matching v0.* (eg. v0.40.0).
Once triggered, this pipeline will build the windows installer, and run integration tests on it, like for the pipeline run nightly.
I also add a simple script to extract from CHANGELOG.md file to extract the relevant part to put it in the body of the GitHub release. I believe it makes things nicer.
* Create release pipeline
* Relax condition on tags
* Put beta keyword
* Update job name
* Fix release pipeline
When setting up Azure Pipelines, I didn't like having to define codecov_token for each pipeline. This works around it by using a shared variable group.
You can see this working successfully at https://dev.azure.com/certbot/certbot/_build/results?buildId=3.
* Use certbot-common.
* update instructions
This PR defines pipelines that can be run on Azure Pipelines. Currently there are two:
* `.azure-pipelines/main.yml` is the main one, executed on PRs for master, and pushes to master,
* `.azure-pipelines/advanced.yml` add installer testing on top of the main pipeline, and is executed for `test-*` branches, release branches, and nightly run for master.
These two pipelines covers all existing stuff done by AppVeyor currently, and so AppVeyor can be decommissioned once Azure Pipelines is operational.
You can see working pipeline in my fork:
* a PR for `master` (so using main pipeline): https://github.com/adferrand/certbot/pull/65
* a PR for `test-something` (so using advanced pipeline): https://github.com/adferrand/certbot/pull/66
* uploaded coverage from Azure Pipelines: https://codecov.io/gh/adferrand/certbot/commit/499aa2cbf25e1e0ab4c93ab64057db92dfec0fba/build
Once this PR is merged, we need to enable Azure Pipelines for Certbot. Instructions are written in `azure-pipelines/INSTALL.md`. This document also references all access rights required to Azure Pipelines onto GitHub to make the CI process work.
Future work for future PRs:
* create a CD pipeline for the releases that will push the installer to GitHub releases
* implement a solution to generate notification on IRC or Mattermost when a nightly build fails
* Define pipelines
* Update locations
* Update nightly
* Use x86
* Update nightly.yml for Azure Pipelines
* Run script
* Use script
* Update install
* Use local installation
* Register warnings
* Fix pywin32 loading
* Clean context
* Enable coverage publication
* Consume codecov token
* Document installation
* Update tool to upload coverage
* Prepare pipeline artifacts
* Update artifact ignore
* Protect against codecov failures
* Add a comment about codecov
* Add a comment on RW access asked by Azure
* Add instructions
* Rename pipeline file
* Update instructions
* Update .azure-pipelines/templates/tests-suite.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update .azure-pipelines/INSTALL.md
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Modified scheduled pipeline
* Add comment
* Remove dynamic version-based installer name
