From 8b684e9b9543c015669844222b8960e1b9a71e97 Mon Sep 17 00:00:00 2001 From: Brad Warren Date: Fri, 17 May 2019 16:30:20 -0700 Subject: [PATCH] Enable test farm tests in Travis (#7072) You can see the full test suite running at https://travis-ci.com/certbot/certbot/builds/112291892. A few noteworthy things: --fast is included because without, the tests would sometimes reach Travis' 50 minute timeout even with 1 test script per Travis build. The only script that is run at release time which is not being run here is https://github.com/certbot/certbot/blob/master/tests/letstest/scripts/test_tests.sh because that script runs tests on the packages installed by certbot-auto which won't be updated until midway through a release. We check TRAVIS_PULL_REQUEST and error out if it is not false for simplicity which should be fine because these tests are never run on PRs. The reason it's more complex to run test farm tests on PRs is the test farm tests need a named branch to pull from and Travis effectively merges the PR into the target branch before running tests complicating this. I don't think this should block this PRs, but the one final change we may want to make to the current setup is #7071. * Add encrypted private key. * Add test farm tests to tox and travis. * Change magic profile name. * Further split test farm tests. * Build local branch. * more depth --- .travis.yml | 22 +++++++++++ tests/letstest/travis-setup.sh | 10 +++++ tests/letstest/travis-test-farm.pem.enc | Bin 0 -> 1680 bytes tox.ini | 47 ++++++++++++++++++++++++ 4 files changed, 79 insertions(+) create mode 100755 tests/letstest/travis-setup.sh create mode 100644 tests/letstest/travis-test-farm.pem.enc diff --git a/.travis.yml b/.travis.yml index 3a054b419..52595c111 100644 --- a/.travis.yml +++ b/.travis.yml @@ -94,6 +94,28 @@ matrix: <<: *not-on-master # Extended test suite on cron jobs and pushes to tested branches other than master + - python: "2.7" + env: + - TOXENV=travis-test-farm-apache2 + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-leauto-upgrades + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + git: + depth: false # This is needed to have the history to checkout old versions of certbot-auto. + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-certonly-standalone + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-sdists + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + <<: *extended-test-suite - python: "3.7" dist: xenial env: TOXENV=py37 CERTBOT_NO_PIN=1 diff --git a/tests/letstest/travis-setup.sh b/tests/letstest/travis-setup.sh new file mode 100755 index 000000000..261a1504f --- /dev/null +++ b/tests/letstest/travis-setup.sh @@ -0,0 +1,10 @@ +#!/bin/bash -ex +# +# Preps the test farm tests to be run in Travis. + +if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then + echo This script must be run in Travis on a non-pull request build + exit 1 +fi + +openssl aes-256-cbc -K "${encrypted_9a387195a62e_key}" -iv "${encrypted_9a387195a62e_iv}" -in travis-test-farm.pem.enc -out travis-test-farm.pem -d diff --git a/tests/letstest/travis-test-farm.pem.enc b/tests/letstest/travis-test-farm.pem.enc new file mode 100644 index 0000000000000000000000000000000000000000..f8b1d576c606b37bdbe3a83eec29cc800ee8a91d GIT binary patch literal 1680 zcmV;B25PgWUK3ZX;ZSeD~a@MsC>R2Vhw9&ZJ^HRZAa#;e4JlfV_K`n4=9$ zR9AN;O;*0zS7P{g0u1bb5O0yIZvZgsi7QbZ+9SARB+rAlNJu8$aWZJ_9aJBxDyCL- zE(CIXCA-OpUGWW}JhL(zk(YBsF&1yzCcbQv!vi0T7dVjB@RR>%OSp~5y!X`3)faVV zi(`inrw}Eu7Ikvi5uYaKq#{Tj@s}>_VNPX8bLzxA{JnijfPFFwj4YSB@{J#%9^&;u z*-ao(ZC1{@Q-{|Fwr!fC>zD-*VNkMd&eQ6w|@=(dSz$aGHzI+^juRd;of zRlVmwybUV+jU_=Zg@JuUj9+j%4nVdgy4VBwVPcVhu!a zVdFx$9k|t_O#rDTV|#-VAWi>Dv=mkTN=;)!XXxX~iZ-g{R~JtrXR(2wRpFQ{3<;%A z2ZO~*^CGF+ZU2!H2;i4Z(Mlf!$h%Liw|?xTio~SZH`22pLLfB#>?buX&U!kqs5zizErXcy38IMCL$F++8h335amCg z7&EpoUU>Z8DJFzW%V-e`j@?Yd&o^$NVw%XGdC9pkTF z4gG7Fu5F$ohKkvapk$#In}c58>l|D<7o9UGcy<5QcWq^-ePwGG_nGz-p%hE&a za$+8HkNzQkgbYO|_1Djr1R|X-UIyRZU<70kt?Qy-{)CWT5Yot``$)4HsJ_ZXW&w-t z940nXpXnQXFS!0VM4uzG5e~K#<=LSbr}fveh?lEPf{qoJ!ao!I6n7nFVozh+a%Isz zhr1_!+~`NIfzy1)?7A1j$Q3zvcy+Un@yfkM~F;ro779ngvmwL9a zlpd11`!5t(BgUAbXm{IPhUOS7`wUv($Mo{^~u{idTzQb@v}w z@ug%2Jey!zKo6fz6f&IYXXBk86L^jQfVP*MFHup`4?Nar7?$I<_Dyw07I+ z4ca729~;?=njh<1rkZ3&-_xMF)@zc|^RhT{|5f5p9jTQKbtT8FJDc_Pb^*|o7QIq2 zU-PjrYB6i^J<+GTi1P>9<#!v5D)BbcZ6MS@PkJ64P?oPvhz9=R&5xOcbB^;NX zU{^db*B>m?i&?4v*-vbwT}$8VU_xHZaBOQ@{)V@Dz><5W_$}|pkY4^PEEQ0`a^2gm z$Oq*vPL;2Gyr8IG(>qyomTja+kOCt)m$tuF)3R7E2i$a-+BwK7K_h&qj=MuF6jT?M zwR5atV~lCuDzvTN&jQ$2?mPl0pN)~DT-Mb*`c<`wy8W1JF?_JbM)w3&`cL9IO2LmZYz zp!$`d=Z(w;xkm-KYiK+Wn$ev_q1Hv`jIe|gLa!VQJXv(TORhUf*063UUa`}|3-3}% zkoMbq9~g za8vX4dCi9-(59t%l0@!!4*^&|%Z*nCtvSejOVA0NIS<%=g--al-&g|zv