When working on an update to our packages in Ubuntu Xenial, @NCommander noticed that importing code through acme.jose no longer works since josepy became a separate package and remembers having to fix up some code that was using acme.jose himself.
This PR should fix that problem by making all of josepy accessible through acme.jose. This is primarily beneficial to our OS package maintainers who want to avoid subtle API changes when updating packages in stable repositories. They will likely backport this change, but I figure we might as well add it ourselves to minimize divergences in our OS packages in the future and avoid problems on the off chance someone hasn't upgraded acme and was relying on this feature.
GitHub notified us about a security vulnerability in our pinned version of `urllib3` earlier this week. It doesn't affect us, but we might as well upgrade anyway. I checked:
* There are no backwards incompatible features we care about listed at https://github.com/urllib3/urllib3/blob/master/CHANGES.rst.
* urllib3's dependencies don't also need to be updated according to https://github.com/urllib3/urllib3/blob/1.24.1/setup.py.
* The hashes match when obtained from different network vantage points.
While reducing noise in test output is valuable, this flag has made a couple aspects of Certbot's development difficult:
1. We test with different sets of dependencies and running pip in quiet mode removes all output about the packages being installed which has made reviewing changes to these tests more difficult.
2. When pip fails, it provides significantly less output about the failure in quiet mode than it does normally. The output is reduced so much that in the two times I've hit this issue in the last month, I was only able to see that installing package X failed rather than what the cause of that failure was which could be seen with `--quiet` removed.
Also, since running pip without `--quiet` is the tox default, I expect Python developers to be familiar with what they see here.
Currently the release script in master fails for a few reasons. First, it's trying to use --numprocesses which comes from a pytest plugin that we're not installing in the release script. Second, many new warnings are raised when we're not using pinned versions of our dependencies.
I'm not sure I agree, but one could argue that we should fix these issues and use the file during the release. I'm particularly hesitant for us to do this when it comes to warnings. We currently do not pin our dependencies in the release script. Do we really want to stop the release because a new package was released and is warning about something? One could argue we do because these warnings may be visible to the user, but they very rarely are and I think this makes the release process much too painful.
I especially do not think we should block the release on this now as we are not up to date on the warnings raised by the latest versions of our packages so there is a lot to work through.
* Don't use pytest.ini during the release.
* State that pytest.ini isn't used in release script.
This PR has the value of VENV_NAME override any value set in the tools/venv* scripts.
I also removed the use of VENV_ARGS. This was used in _venv_common.sh as a means of passing arguments for virtualenv between the scripts, however, there is no other use of the variable in this repository and passing the arguments through a function call is much more natural in Python.
* Respect VENV_NAME in tools/venv*.
* Stop using VENV_ARGS
* Remove VENV_NAME_ENV_VAR and add docstrings.
Main piece of #5810.
* Rename Certbot integration tests
* Remove nginx from certbot tests
* allow for running individual integration tests
* fail under 65
* Add set -e
* Track Nginx coverage and omit it from report later.
* Use INTEGRATION_TEST in script
* add INTEGRATION_TEST=all
* update min certbot percentage
Fixes#4686.
In Sphinx 1.6, they changed how they handle images in latex and PDF files. You can learn more about this by reading the linked issue (or I can answer any questions), but the shortish version is we now need to use the extension sphinx.ext.imgconverter. This is only available in Sphinx 1.6+.
I also updated our pinned versions to use the latest Sphinx and a new dependency it pulled in called sphinxcontrib-websupport. To build the latex and PDF docs, you must first run:
apt-get install imagemagick latexmk texlive texlive-latex-extra
Afterwards, if you create the normal Certbot dev environment using this branch, activate the virtual environment, and from the root of the repo run make -C docs clean latex latexpdf, you'll successfully build the PDF docs.
* fix#4686
* bump minimum Sphinx req
Fixes#6207.
As noted by Erica:
- we no longer need to check if it exists before linking to it, because we delete properly.
- the previously excisting check on if server is in `LE_REUSE_SERVERS` before unlinking is nice, but probably not necessary, especially since we don't officially support people doing weird things with symlinks in our directories, and because we rmdir which will fail if it's not empty anyway.
* Create single account symlink.
* refactor _delete_accounts_dir_for_server_path
* add symlinked account dir deletion
* add tests
(cherry picked from commit 9b0d2714c1)
Fixes#6207.
As noted by Erica:
- we no longer need to check if it exists before linking to it, because we delete properly.
- the previously excisting check on if server is in `LE_REUSE_SERVERS` before unlinking is nice, but probably not necessary, especially since we don't officially support people doing weird things with symlinks in our directories, and because we rmdir which will fail if it's not empty anyway.
* Create single account symlink.
* refactor _delete_accounts_dir_for_server_path
* add symlinked account dir deletion
* add tests
This stops us from printing messages like:
"Could not choose appropriate plugin for updaters: Could not select or initialize the requested installer None."
when certbot renew --force-renewal is run with a lineage that doesn't have an installer.
* unquote None
* Test None values aren't saved in config file.
Now that yaml/pyyaml#126 is resolved, #6170 can be reverted by bumping the pinned version of PyYAML.
You can see this code passing with full macOS and integration tests at https://travis-ci.org/certbot/certbot/builds/400957729.
* Revert "Allow py37 testing (#6170)"
This reverts commit cad95466b0.
* Bump pyyaml pinning to work on Python 3.7.
We released josepy 1.1.0 a while ago to work around newer versions of cryptography deprecating some of the functionality we were using. We haven't yet upgraded our pinned josepy version though and since #6169 has landed, we're now seeing these deprecation warnings in our tests. This would be shown to certbot-auto users as well.
This PR removes these warnings by upgrading our pinned version of josepy.
* update pinned josepy version
* build leauto
* update pinned dev version of josepy
* Remove apacheconftest packages.
The apacheconftests handle installing Apache dependencies, so let's remove it from the general case.
* We don't need to run dpkg -s in before_install.
* Remove augeas sources.
We only needed it for Ubuntu Precise which is dead and it doesn't work in Ubuntu Xenial.
* Upgrade Python 3.6 tests to 3.7.
Let's continue the approach of testing on the oldest and newest versions of Python 3. We will continue testing on Python 3.6 in the nightly tests.
* Revert "We don't need to run dpkg -s in before_install."
This reverts commit e5d35099a7.
* let apacheconftest handle deps
This allows us to depend on packages like acme>=0.26.0.dev0 during development
and automatically change it to acme>=0.26.0 during the release. We use `git add
-p` to be safe, but if .dev0 is used at all in our released setup.py files,
we're probably doing something wrong.
* Reorganize packages in tox to allow for py37 tests
certbot-dns-cloudflare doesn't currently work in Python 3.7 because it transitively depends on pyYAML which doesn't yet support Python 3.7. See https://github.com/yaml/pyyaml/issues/126 for more info.
* add py37 tox environment
* Use greater than or equal to in requirements.
This changes the existing requirements using strictly greater than to greater
than or equal to so that they're more conventional.
* Use >= for certbot-postfix.
Despite it previously saying 'certbot>0.23.0', certbot-postfix/local-oldest-requirements.txt was pinned to 0.23.0 so let's just use certbot>=0.23.0.
This partially reverts commit 15f1405fff.
A basic tls-alpn-01 implementation is left so we can successfully parse the
challenge so it can be used in boulder's tests.
* Revert "Revert "switch signature verification to use pure cryptography (#6000)" (#6074)"
This reverts commit 3cffe1449c.
* Fixes#6073.
This silences the deprecation warnings from cryptography. I looked into only
silencing the cryptography warning specifically in the function, however,
CryptographyDeprecationWarning doesn't seem to be publicly documented, so we
probably shouldn't depend on it.
For the past couple of releases, twine has errored while trying to upload
packages and this is fixed by upgrading to a newer version of twine. This
commit updates our pinned version installed when using tools/venv.sh to the
latest available version. pkginfo had to be upgraded as well to support the
latest version of twine.
Festival isn't available via Homebrew and is only needed to read the hash
aloud, so let's not make it a strict requirement that it's installed. You can
simply read the hash from the terminal instead.
Debian Wheezy is no longer supported (see https://wiki.debian.org/LTS) and
Amazon shut down their Debian 7 mirrors so let's stop trying to use Debian 7
during testing.
* Remove unneeded sys import.
Once upon a time we needed this in some of these setup.py files because we were
using sys in the file, but we aren't anymore so let's remove the import.
* use setuptools instead of distutils
When importing a module, Python first searches the current directory. See
https://docs.python.org/3/tutorial/modules.html#the-module-search-path. This
means that running something like `import certbot` from the root of the Certbot
repo will use the local Certbot files regardless of the version installed on
the system or virtual environment.
Normally this behavior is fine because the local files are what we want to
test, however, during our "oldest" tests, we test against older versions of our
packages to make sure we're keeping compatibility. To make sure our tests use
the correct versions, this commit has our tests cd to an empty temporary
directory before running tests.
We also had to change the package names given to pytest to be the names used in
Python to import the package rather than the name of the files locally to
accommodate this.
* Fixes#5570.
The issue is calls to atexit aren't mocked out. During the tests there are many
repeated calls registering functions to be called when the process exits so
when the tests finishes, it prints a ton of output from running those
registered functions. This suppresses that by mocking out atexit.
* Mock at a lower level.
This ensures we don't mess with any other mocks in this test class by mocking
at the lowest level we can. Other tests shouldn't be mocking out specific
internals of functions in other modules, so this should work just fine.
* Clarify UNIX only
* Have people develop natively.
Some systems like Arch Linux and macOS require --debug in order to install
dependencies.
Our bootstrapping script for macOS works, so let's let people who want to
develop natively.
* briefly mention docker as dev option
* remove bad _common.sh info
* update OS dep section
* Remove sudo from certbot-auto usage
When sudo isn't available, Certbot is able to fall back to su. Removing it from
the instructions here allows the command to work when its run in minimal
systems like Docker where sudo may not be installed.
* copy advice about missing interpreters
* Improve integration tests docs
Explain what a boulder is and tell people they probably should just let the
tests run in Travis.
* Don't tell people to run integration tests.
I don't think any paid Certbot devs run integration tests locally and instead
rely on Travis. Let's not make others do it.
* fix spacing
* you wouldn't download a CA
* address review comments
See https://wiki.ubuntu.com/Releases.
Ubuntu 15.* repositories have been shut down for months now causing our tests
to always fail on these systems. While the tests on Ubuntu 12.04 still work, it
has been unsupported by Canonical for almost a year and I don't think we should
hamstring ourselves trying to continue to support it ourselves.
