* check_untyped_defs in mypy with clean output for acme
* test entire acme module
* Add typing as a dependency because it's only in the stdlib for 3.5+
* Add str_utils, modified for python2.7 compatibility
* make mypy happy in acme
* typing is needed in prod
* we actually only need typing in acme so far
* add tests and more docs for str_utils
* pragma no cover
* add magic_typing
* s/from typing/from magic_typing/g
* move typing to dev_extras
* correctly set up imports
* remove str_utils
* only type: ignore for OpenSSL.SSL, not crypto
* Since we only run mypy with python3 anyway and we're fine importing it when it's not actually there, there's no actual need for typing to be present as a dependency
* comment magic_typing.py
* disable wildcard-import im magic_typing
* disable pylint errors
* add magic_typing_test
* make magic_typing tests work alongside other tests
* make sure temp_typing is set
* add typing as a dev dependency for python3.4
* run mypy with python3.4 on travis to get a little more testing with different environments
* don't stick typing into sys.modules
* reorder imports
Fixes#5490.
There's a lot of possibilities discussed in #5490, but I'll try and explain what I actually did here as succinctly as I can. Unfortunately, there's a fair bit to explain. My goal was to break lockstep and give us tests to ensure the minimum specified versions are correct without taking the time now to refactor our whole test setup.
To handle specifying each package's minimum acme/certbot version, I added a requirements file to each package. This won't actually be included in the shipped package (because it's not in the MANIFEST).
After creating these files and modifying tools/pip_install.sh to use them, I created a separate tox env for most packages (I kept the DNS plugins together for convenience). The reason this is necessary is because we currently use a single environment for each plugin, but if we used this approach for these tests we'd hit issues due to different installed plugins requiring different versions of acme/certbot. There's a lot more discussion about this in #5490 if you're interested in this piece. I unfortunately wasted a lot of time trying to remove the boilerplate this approach causes in tox.ini, but to do this I think we need negations described at complex factor conditions which hasn't made it into a tox release yet.
The biggest missing piece here is how to make sure the oldest versions that are currently pinned to master get updated. Currently, they'll stay pinned that way without manual intervention and won't be properly testing the oldest version. I think we should solve this during the larger test/repo refactoring after the release because the tests are using the correct values now and I don't see a simple way around the problem.
Once this lands, I'm planning on updating the test-everything tests to do integration tests with the "oldest" versions here.
* break lockstep between packages
* Use per package requirements files
* add local oldest requirements files
* update tox.ini
* work with dev0 versions
* Install requirements in separate step.
* don't error when we don't have requirements
* install latest packages in editable mode
* Update .travis.yml
* Add reminder comments
* move dev to requirements
* request acme[dev]
* Update pip_install documentation
* Use newer boulder config
* Use ACMEv2 endpoint if requested
* Add v2 integration tests
* Work with unset variables
* Add wildcard issuance test
* quote domains
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* Add OSX Python2 tests
* Make sure python2 is originating from homebrew on macOS
* Upgrade the already installed python2 instead of trying to reinstall
* Remove extra le-auto tests from master
* Remove dockerfile-dev test from master
* Remove intermediate Python 3.x tests from master
* Reorder travis jobs for speed
These tests are retained in the test-everything branch, which has a Travis cron
job to run nightly.
Removing these speeds up the Certbot Travis builds dramatically for two reasons:
- The Boulder integration tests are slow (10-12 minutes), and it's exceedingly
rare for them to fail on one Python environment but not another.
- The macOS tests take a very long time to run, because they need to wait for
build slots on the limited number of macOS instances, which are often in high
demand.
Occasionally a network error prevents Docker from starting boulder causing
Travis tests to fail like it did at
https://travis-ci.org/certbot/certbot/jobs/282923098. This works around the
problem by using travis_retry to try to start boulder again if it fails.
This also moves the logic of waiting for boulder to start into
tests/boulder-fetch.sh so people running integration tests locally can benefit.
* Remove py26 oldest tests.
The only systems where we support Python 2.6 use certbot-auto so the oldest
supported versions of our dependencies are never used when using supported
installation methods. Let's remove this unnecessary and slow test.
* Make tox.ini happy
* Remove py26-oldest from Travis
* Add OS X test
* Fix apache tests
* Use absolute path first so that certbot works with non-standard $PATH
Some tests use a fake $PATH, which prevents invoking `sw_vers`
* Also test Python 3 on Mac
* Set ulimit to fix "Too many open files"
* add py36 to tox
* Add Python 3.6 tests to Travis
* Provide real path to python stdlib during tests
* set logs_dir in config_test
* set *_dirs in DetermineAccountTest
* Fix TLSSNI01Test
* Fix RenewalTest
* fix test_ancient_webroot_renewal_conf
* Adding Debian 7 (Wheezy) to LE tests
* Adding Debian 8 (Jessie) to LE tests
* Fixing Debian Wheezy certificate addition error
* Adding packages to LEA Debian Jessie test and refining the code commenting
* Adding installing OpenSSL to the Debian Wheezy LEA test script
* Removing LEA tests for Debian Jessie
* Fixing nits
One more tiny tweak, placing the slower sudo environments back up the list. I expect this should save us another 10-20 seconds.
- there's about a 50 second delay in starting apacheconftest, so move
it back up the priority queue of jobs to start
* Tweak for Travis performance
- merge cover and py27 BOULDER_INTEGRATION into one matrix entry
- re-order to put the fastest environments last, improving average
case parallelism
* Also put the things most likely to fail at the top
This removes a lot of setup code we used to need in order to get Boulder to run,
and should reduce brittleness of tests based on Boulder changes.
This also unblocks Boulder from upgrading to MariaDB 10.1 in integration tests,
since changing to 10.1 syntax for user creation would break the current certbot
integration tests (which run 10.0).
- hardcoded
`LETSENCRYPT=/home/travis/build/letsencrypt/letsencrypt/.tox/apacheconftest/bin/letsencrypt`
causes Travis tests to fail if running under any other Travis user
(from e.g. a fork)
- `sudo env "PATH=$PATH" letsencrypt` should make sure that sudo can
find letsencrypt binary from virtualenv; realpath is not necessary
- sudo is called already from within the test script, no need to sudo
the entire script
