starred/certbot-certbot
1
0
Fork 0
mirror of https://github.com/certbot/certbot.git synced 2026-07-27 16:30:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

Disable rerun feature of Travis (#6800)

Browse Source 
The rerun capability in a test campaign can be a nice feature. It can also a bad design.

It is right that with high level tests, like performance or end-to-end tests, a given test runtime can depend on an external component, outside the scope of the developers, that would spuriously fail. In theses situations, having the capacity to rerun several time a test can be a great benefit. Indeed, as these tests are inherently flaky, the rerun greatly reduces their failure because of external reasons, reducing also the Pierre et le Loup effect (from a well-known french book for children): because tests constantly fail for no reason, you stop to listen to them and to not see when they fail for a real reason.

However this not apply to unit tests, or operations about code quality. For theses executions, the flakiness should not exist: a unit test is supposed to have no external dependency. A rerun approach would just hide a situation that is not desirable for this kind of tests

Also effectively I see that our tests are usually not flaky: so the only effect of the rerun is to give the failure state about a test three times slower. If a test becomes flaky, this should be fixed, and so be visible immediately in our CI.

For these reasons, I remove the travis_retry in the script section of .travis-ci.yml, to call directly tox and let the pipeline fails on the first error.

* Disable rerun feature of Travis

* Update .travis.yml

* Remove completely the retry logic
This commit is contained in:
Adrien Ferrand
2019-02-28 12:49:36 -08:00
committed by Brad Warren
parent 5e849e03f6
commit efc8d49806
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.travis.yml
+3 -3
View File
@@ -197,10 +197,10 @@ addons:
- nginx-light
- openssl
install: "travis_retry $(command -v pip || command -v pip3) install codecov tox"
install: "$(command -v pip || command -v pip3) install codecov tox"
script:
- travis_retry tox
- '[ -z "${BOULDER_INTEGRATION+x}" ] || (travis_retry tests/boulder-fetch.sh && tests/tox-boulder-integration.sh)'
- tox
- '[ -z "${BOULDER_INTEGRATION+x}" ] || (tests/boulder-fetch.sh && tests/tox-boulder-integration.sh)'
after_success: '[ "$TOXENV" == "py27-cover" ] && codecov'