Also, switch timeout to 30 so it has every opportunity to actually work, even in bad network weather. (I posit that people are used to 30-second timeouts.)
Stop catching URLError explicitly, since it's a subclass of the already-caught IOError.
bmw ran into a problem on his own machine in which cryptography was built with an old version of openssl, then openssl was upgraded to fix the Drown attack, and the API change (in a bugfix release, mind you) broke the cached wheel.
This should solve our unicode error (https://github.com/erikrose/peep/issues/125) as well as many other errors caused by pip, setuptools, or wheel being really, really old. It also means I don't have to maintain peep anymore for LE's sake.
Revert the patch that added the InstallRequirements function, since we're back to needing only 1 requirements file.
Turn off all the Travis addons for the GCE le_auto job, since the MariaDB one broke there and the rest weren't necessary. See https://github.com/travis-ci/travis-ci/issues/5759. TheNavigat collaborated on this.
When pip-installing any of these packages, pip hit our permissive, any-version "setuptools" dependency first and then ignored all subsequent, more constrained ones, like cryptography's "setuptools>=1.0". See https://github.com/pypa/pip/issues/988. It thus, on a box with setuptools 0.9.8, stuck with that version. Then, at runtime, letsencrypt crashed because pkg_resources couldn't satisfy cryptography's setuptools>=1.0 requirement.
This change lets us pip-install our packages and have it work. We'll need to make sure our direct requirements (all of them) satisfy the more constrained requirements of our dependencies. Yes, it is disgusting.
There's no particular reason this *should* fix#2499, but it changes how pycparser gets installed (to a more modern way: pip vs. setuptools), so it may.
I prefer to err toward simplicity here. Yes, there's an assumption necessary for this to work--that the shell doesn't do multiple open() calls to the script path throughout the life of the interpreter--but I think it's reasonable. The alternative of exec-ing out to a dedicated update script which then execs back to le-auto has more moving parts (like extra files that we have to clean up) and is longer.
It will always be a copy of the latest release version, 0.4 in this case. (Modify the release script to make that so.) This way, people using the old method of running le-auto from a git checkout will not end up using a bleeding-edge version, letting us work on the tip-of-tree version more freely.
Ported from #1751.
* It's more lines but fewer tokens, less room for quote errors, and more idiomatic (see any init.d script).
* Also, fix a bug in which any option containing "-v", e.g. --eat-vertical-pizza, would be construed as --verbose.
Ported from #1751.
* Make sure any Python passed in as $LE_PYTHON actually exists.
* Dodge a word-splitting bug: `a='a b'; export a=${a:-c}; echo $a` gives `a` instead of `a b` under shells that respect POSIX.1, like dash.
Otherwise, we sometimes end up using the system Python, for which we'd need to use sudo to install virtualenv. Brew complicates this by yelling at you if you do use sudo. So let's simplify things by always using the homebrew python, which is more up to date anyway.
...except for $SUDO, which is always either "sudo", "su_sudo", or "", never having a quote-needing char in it.
It's unlikely that $PYVER would have a space in it, but it doesn't hurt.
Python 2.4 doesn't support --version, and we want to be able to at least complain that it's too old without crashing.
Also, bring built le-auto up to date.
--no-self-upgrade metamorphosed from a private flag to a public one, so add a new private flag, --le-auto-phase2 to take its original role of marking the division between phases. This flag must come first and, consequently, can be stripped off the arg list before calling through to letsencrypt, which means the client doesn't need to know about it.
The downside is that anyone still (deprecatedly) running le-auto out of the root of a (recently updated) master checkout will get a "Hey, the current release version le-auto I just self-upgraded to doesn't understand the --le-auto-phase2 flag" error from when we merge this until the next release is made, but that's better than a documented option not working right.
Also, remove a needless folder creation from the Dockerfile.
Leaving broken venvs around can, if it got as far as installing the venv/bin/letsencrypt script, wreck future le-auto runs, since the presence of that script means "a working LE is installed" to it. Waiting until a new version of le-auto comes out and running it would recover, but this lets re-running the same version recover as well.
Remove the old bootstrap scripts, which have been subsumed into letsencrypt-auto-source/pieces/bootstrappers. They no longer need to be dispatched among manually: everyone can just run letsencrypt-auto --os-packages-only, regardless of OS.
Make the root-level le-auto a symlink to the canonical version. It should thus still work for people running le-auto from a git checkout.
ConfigArgParse has a conditional dependency for Pythons < 2.7. On my local machine, I had a cached ConfigArgParse wheel built under 2.7, so it didn't carry those dependencies, and the pip freeze I used to determine the le-auto requirements thus missed it. From now on, we'll do those passes with --no-cache-dir.
* We choose a different Travis infra for one of the jobs, as in https://github.com/numpy/numpy/blob/master/.travis.yml#L49.
* We keep the language as "python" so the installation of packages (like tox, which we need) doesn't fail.
* Override the before_install to disable the dpkg stuff the other jobs need.
* adduser is redundant with `--groups sudo` above, so we delete it.
This will avoid crashing when used with pip 8.x, which was released today and is already the 3rd most used client against PyPI. (7.1.2 and 1.5.4 take spots 1 and 2, respectively.)
This had more of a purpose when we were moving mock to test_requires. I'll reintroduce this in the separate PR for that.
Also bring back the testing extra in tox for now.
Bring everything to the latest versions.
Make dependencies unconditional: argparse, ndg-httpsclient, and pyasn1 get in all the time, to match the state of master as of 0.2.0.
None of this is ideal, since we're making the dependencies tighter than they theoretically need to be, but the behavior of the old le-auto makes this necessary to make it succeed in practice (when using LE wheels).
Once we move to the new le-auto (which pins everything and makes setup.py dependencies irrelevant for auto installs), we should redo this using env markers as in https://github.com/letsencrypt/letsencrypt/pull/2177. We're too afraid to do it now.
Similarly, we're too afraid to change how we handle argparse right now, despite that it should be required directly by us under 2.6. In practice, ConfigArgParse pulls it in for us, so we're okay as long as it continues to do that.
We're stepping off that infra briefly, to keep it the same as boulder's. When we retire the old le-auto, we'll step back on and change boulder to use it as well.
If the new le-auto works well in the minutes or hours after release, we'll make another commit to master that removes the old le-auto and bootstrap scripts.
Close https://github.com/erikrose/letsencrypt/pull/2.
I didn't backport their imports, so they had NameErrors in the failure case anyway. And, because of the docker image, these tests currently are run under only 2.7 at the moment.
This fixes an "OSError: [Errno 2] No such file or directory" on Fedora 23. Note that openssl-devel was not sufficient to install the openssl commandline tool.
The current manual-testing build of le-auto now crashes with #1548, but that should have been resolved when we upgraded the cryptography lib and so should go away when we build a new version.
This should make the linter pass and allow us to merge the letsencrypt-auto-release branch when it's ready. IHNI why it passes on master without this disabled.
Switch to a MySQL 5.6 setup based on https://github.com/mozilla/treeherder/pull/1080/files and Travis's beta trusty infra, which runs on Google Compute Engine.
Remove MariaDB addon, which conflicts with the socket used by the treeherder approach's mysql package. Remove maria service (which has no effect).
The motivation is to free us of a reliance on a rather modern version of setuptools, which caused le-auto failures for people on Wheezy and other older distros. (The alternative would have been to forcibly upgrade setuptools as the old le-auto did, but less is more.)
Mock is used only in tests, so we move it to tests_require. It will still be installed automatically when setup.py test is run. Give all packages a test_suite so this works.
The "testing" extra remains for optional packages not required for the nose tests but used in tox. However, the extra is much less useful now and is a candidate for deletion. We could roll the list of packages therein into the tox config so as not to favor any particular package.
Remove tests_require=install_requires, which I don't think does anything useful, since install requirements are implicitly installed when running setup.py test.
Fix tests to pass with mock removed. We had to stop them pulling down LE from PyPI, since the current version there (0.1.1) requires mock and explodes when `letsencrypt` is run.
Originally, I had it in mind to move letsencrypt-auto inside this dir. However, now we'd like to copy it or link it to the root level, where people are used to finding it (at least for awhile). Since it would be confusing to have a letsencrypt-auto and a letsencrypt_auto right next to each other, we rename this folder.
We're going to keep the built artifact in the tree as per https://github.com/letsencrypt/letsencrypt/issues/1572#issuecomment-161379131 so that...
1. People's current behavior of cloning from git and running the le-auto script still works.
2. We don't have a deprecation timeline and process to babysit.
We'll enforce its up-to-dateness with a test.
To run it, cd letsencrypt_auto && ./build.py && docker build -t lea . && docker run --rm -t -i lea
So as not to depend on the state of the host machine, the test runs within an Ubuntu Docker image. This lets us sidestep interaction challenges by setting up passwordless sudo. It also lets us insert our own local CA for the mock HTTPS server. (openssl's SSL_CERT_FILE env var replaces rather than adds to the accepted CAs, meaning later connections to PyPI within the same process chain fail. SSL_CERT_DIR seems not to work at all on OS X.) This also demonstrates a way to test across various Linux distros, even within Travis if we like,
Also...
* Switch to an official release of ConfigArgParse.
* Don't redundantly re-bootstrap on --no-self-upgrade (that is, phase 2).
This should let us create a harness that won't force us to mess with GitHub or PyPI just to test.
(I haven't tried this commit yet, but you can if you want to get a head start on testing.)
Bourne does the dumb thing when substituting vars; this helps that. Bash does the smart thing but is unhurt by this.
We don't do it to $SUDO because Bourne takes "" as a command rather than a no-op, and we don't want the SUDO= case to generate command-not-found errors.
* Temp dir creation is now always done in shell.
* Split download_upgrade.py into 2 phases itself so we can have it report back the latest LE version and make a decision based on it before doing and major downloading. Rename it for clarity.
* Teach the build script how to do special vars. Factor up file reading.
* Use a static string for the PyPI JSON location, as it will soon be overrideable via an env var for testing.
This replaces --_skip-to-install and is suitable for people who have audited letsencrypt-auto and wish to run it as is, without upgrading to the latest version.
Also...
* rm temp dirs when done. No longer reuse a single temp dir across phases so the user doesn't have to pass a temp dir with --no-self-upgrade as phase 1 itself used to.
* Swap stanzas in the big "if" so we aren't testing negatives all the time.
* Fix a bug in which we ran peep with $LE_PYTHON rather than the python in the venv.
* Bootstrap only if it looks like we never got to the point of making a venv before.
* Move venv creation into Phase 2. Besides the practical benefit of ensuring there's a venv if a user passes --no-self-upgrade, this has the philosophical advantage of making Phase 1 more minimal, giving us more latitude to change behavior in updates.
Change template language to reference files, saving me some boilerplate over the dict-based .format() thing I originally had in mind.
Put newlines at the ends of bootstrap scripts. It makes the built le-auto script prettier.
They're now used only by the le-auto build process. The new public interface for OS-level bootstrapping is le-auto --os-packages-only, which dispatches by OS automatically. That obsoletes install-deps.sh as well, saving some repetition.
Also, switch to mustache-style templating to avoid colliding with shell variable references.
To optimize for the docker cache, we could later add a shim script that sources just deb_common.sh and calls its bootstrap function.
Erik Rose