ConfigArgParse has a conditional dependency for Pythons < 2.7. On my local machine, I had a cached ConfigArgParse wheel built under 2.7, so it didn't carry those dependencies, and the pip freeze I used to determine the le-auto requirements thus missed it. From now on, we'll do those passes with --no-cache-dir.
* We choose a different Travis infra for one of the jobs, as in https://github.com/numpy/numpy/blob/master/.travis.yml#L49.
* We keep the language as "python" so the installation of packages (like tox, which we need) doesn't fail.
* Override the before_install to disable the dpkg stuff the other jobs need.
* adduser is redundant with `--groups sudo` above, so we delete it.
This will avoid crashing when used with pip 8.x, which was released today and is already the 3rd most used client against PyPI. (7.1.2 and 1.5.4 take spots 1 and 2, respectively.)
This had more of a purpose when we were moving mock to test_requires. I'll reintroduce this in the separate PR for that.
Also bring back the testing extra in tox for now.
Bring everything to the latest versions.
Make dependencies unconditional: argparse, ndg-httpsclient, and pyasn1 get in all the time, to match the state of master as of 0.2.0.
None of this is ideal, since we're making the dependencies tighter than they theoretically need to be, but the behavior of the old le-auto makes this necessary to make it succeed in practice (when using LE wheels).
Once we move to the new le-auto (which pins everything and makes setup.py dependencies irrelevant for auto installs), we should redo this using env markers as in https://github.com/letsencrypt/letsencrypt/pull/2177. We're too afraid to do it now.
Similarly, we're too afraid to change how we handle argparse right now, despite that it should be required directly by us under 2.6. In practice, ConfigArgParse pulls it in for us, so we're okay as long as it continues to do that.
We're stepping off that infra briefly, to keep it the same as boulder's. When we retire the old le-auto, we'll step back on and change boulder to use it as well.
If the new le-auto works well in the minutes or hours after release, we'll make another commit to master that removes the old le-auto and bootstrap scripts.
Close https://github.com/erikrose/letsencrypt/pull/2.
I didn't backport their imports, so they had NameErrors in the failure case anyway. And, because of the docker image, these tests currently are run under only 2.7 at the moment.
This fixes an "OSError: [Errno 2] No such file or directory" on Fedora 23. Note that openssl-devel was not sufficient to install the openssl commandline tool.
The current manual-testing build of le-auto now crashes with #1548, but that should have been resolved when we upgraded the cryptography lib and so should go away when we build a new version.
This should make the linter pass and allow us to merge the letsencrypt-auto-release branch when it's ready. IHNI why it passes on master without this disabled.
Switch to a MySQL 5.6 setup based on https://github.com/mozilla/treeherder/pull/1080/files and Travis's beta trusty infra, which runs on Google Compute Engine.
Remove MariaDB addon, which conflicts with the socket used by the treeherder approach's mysql package. Remove maria service (which has no effect).
The motivation is to free us of a reliance on a rather modern version of setuptools, which caused le-auto failures for people on Wheezy and other older distros. (The alternative would have been to forcibly upgrade setuptools as the old le-auto did, but less is more.)
Mock is used only in tests, so we move it to tests_require. It will still be installed automatically when setup.py test is run. Give all packages a test_suite so this works.
The "testing" extra remains for optional packages not required for the nose tests but used in tox. However, the extra is much less useful now and is a candidate for deletion. We could roll the list of packages therein into the tox config so as not to favor any particular package.
Remove tests_require=install_requires, which I don't think does anything useful, since install requirements are implicitly installed when running setup.py test.
Fix tests to pass with mock removed. We had to stop them pulling down LE from PyPI, since the current version there (0.1.1) requires mock and explodes when `letsencrypt` is run.
Originally, I had it in mind to move letsencrypt-auto inside this dir. However, now we'd like to copy it or link it to the root level, where people are used to finding it (at least for awhile). Since it would be confusing to have a letsencrypt-auto and a letsencrypt_auto right next to each other, we rename this folder.
We're going to keep the built artifact in the tree as per https://github.com/letsencrypt/letsencrypt/issues/1572#issuecomment-161379131 so that...
1. People's current behavior of cloning from git and running the le-auto script still works.
2. We don't have a deprecation timeline and process to babysit.
We'll enforce its up-to-dateness with a test.
To run it, cd letsencrypt_auto && ./build.py && docker build -t lea . && docker run --rm -t -i lea
So as not to depend on the state of the host machine, the test runs within an Ubuntu Docker image. This lets us sidestep interaction challenges by setting up passwordless sudo. It also lets us insert our own local CA for the mock HTTPS server. (openssl's SSL_CERT_FILE env var replaces rather than adds to the accepted CAs, meaning later connections to PyPI within the same process chain fail. SSL_CERT_DIR seems not to work at all on OS X.) This also demonstrates a way to test across various Linux distros, even within Travis if we like,
Also...
* Switch to an official release of ConfigArgParse.
* Don't redundantly re-bootstrap on --no-self-upgrade (that is, phase 2).
This should let us create a harness that won't force us to mess with GitHub or PyPI just to test.
(I haven't tried this commit yet, but you can if you want to get a head start on testing.)
Bourne does the dumb thing when substituting vars; this helps that. Bash does the smart thing but is unhurt by this.
We don't do it to $SUDO because Bourne takes "" as a command rather than a no-op, and we don't want the SUDO= case to generate command-not-found errors.
* Temp dir creation is now always done in shell.
* Split download_upgrade.py into 2 phases itself so we can have it report back the latest LE version and make a decision based on it before doing and major downloading. Rename it for clarity.
* Teach the build script how to do special vars. Factor up file reading.
* Use a static string for the PyPI JSON location, as it will soon be overrideable via an env var for testing.
This replaces --_skip-to-install and is suitable for people who have audited letsencrypt-auto and wish to run it as is, without upgrading to the latest version.
Also...
* rm temp dirs when done. No longer reuse a single temp dir across phases so the user doesn't have to pass a temp dir with --no-self-upgrade as phase 1 itself used to.
* Swap stanzas in the big "if" so we aren't testing negatives all the time.
* Fix a bug in which we ran peep with $LE_PYTHON rather than the python in the venv.
* Bootstrap only if it looks like we never got to the point of making a venv before.
* Move venv creation into Phase 2. Besides the practical benefit of ensuring there's a venv if a user passes --no-self-upgrade, this has the philosophical advantage of making Phase 1 more minimal, giving us more latitude to change behavior in updates.
Change template language to reference files, saving me some boilerplate over the dict-based .format() thing I originally had in mind.
Put newlines at the ends of bootstrap scripts. It makes the built le-auto script prettier.
They're now used only by the le-auto build process. The new public interface for OS-level bootstrapping is le-auto --os-packages-only, which dispatches by OS automatically. That obsoletes install-deps.sh as well, saving some repetition.
Also, switch to mustache-style templating to avoid colliding with shell variable references.
To optimize for the docker cache, we could later add a shim script that sources just deb_common.sh and calls its bootstrap function.
This makes the script more readable and easier to work on. We'll stitch it together with a build process.
Also, stop passing the sudo command as an arg to the experimental bootstrappers. They will be inlined into the main script and can just reference $SUDO. As a result, stop recommending devs run the scripts manually, instead running le-auto --os-packages-only. This has the nice side effect of making dev documentation simpler.
Name the folder "letsencrypt_auto" rather than "letsencrypt-auto" because git yield endless pain when replacing a file with a dir. Perhaps we can change it with impunity in a latter commit.
These are the exceptions that are likely to happen, so we give them extra, human-readable descriptions.
Also, name them more in line with stdlib exceptions.
PyPI does not appear to give it to us for free through its JSON interface. distutils gives us a sufficient (though not foolproof) comparator without having to go outside the stdlib.
Erik Rose