From ebf5170d1270659d272b4a745b6e0314ad071785 Mon Sep 17 00:00:00 2001 From: Jacob Hoffman-Andrews Date: Thu, 16 Feb 2017 12:39:17 -0800 Subject: [PATCH] Update testing docs. (#4211) Recommend faster-running commands with clearer output first, building up to slow, noisy, `tox` last. Remove outdated Boulder testing advice, and outdated Boulder testing scripts (Vagrantfile, mac-bootstrap.sh, boulder-start.sh). Update Boulder testing section. Fill out "directory structure" section of contributing guide a little better. --- Vagrantfile | 41 ------------- docs/contributing.rst | 116 +++++++++++++---------------------- tests/boulder-integration.sh | 2 +- tests/boulder-start.sh | 9 --- tests/mac-bootstrap.sh | 26 -------- 5 files changed, 43 insertions(+), 151 deletions(-) delete mode 100644 Vagrantfile delete mode 100755 tests/boulder-start.sh delete mode 100755 tests/mac-bootstrap.sh diff --git a/Vagrantfile b/Vagrantfile deleted file mode 100644 index 23d3ddf13..000000000 --- a/Vagrantfile +++ /dev/null @@ -1,41 +0,0 @@ -# -*- mode: ruby -*- -# vi: set ft=ruby : - -# Vagrantfile API/syntax version. Don't touch unless you know what you're doing! -VAGRANTFILE_API_VERSION = "2" - -# Setup instructions from docs/contributing.rst -# Script installs dependencies for tox and boulder integration -$ubuntu_setup_script = <> /home/vagrant/.profile; fi -if ! grep -Fxq "export PATH=\\$GOROOT/bin:\\$PATH" /home/vagrant/.profile ; then echo "export PATH=\\$GOROOT/bin:\\$PATH" >> /home/vagrant/.profile; fi -if ! grep -Fxq "export GOPATH=\\$HOME/go" /home/vagrant/.profile ; then echo "export GOPATH=\\$HOME/go" >> /home/vagrant/.profile; fi -if ! grep -Fxq "cd /vagrant/; ./tests/boulder-start.sh &" /etc/rc.local ; then sed -i -e '$i \cd /vagrant/; ./tests/boulder-start.sh &\n' /etc/rc.local; fi -export DEBIAN_FRONTEND=noninteractive -sudo -E apt-get -q -y install git make libltdl-dev mariadb-server rabbitmq-server nginx-light -SETUP_SCRIPT - -Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| - - config.vm.define "ubuntu-trusty", primary: true do |ubuntu_trusty| - ubuntu_trusty.vm.box = "ubuntu/trusty64" - ubuntu_trusty.vm.provision "shell", inline: $ubuntu_setup_script - ubuntu_trusty.vm.provider "virtualbox" do |v| - # VM needs more memory to run test suite, got "OSError: [Errno 12] - # Cannot allocate memory" when running - # letsencrypt.client.tests.display.util_test.NcursesDisplayTest - # We may no longer need this. - v.memory = 1024 - - # Handle cases when the host is behind a private network by making the - # NAT engine use the host's resolver mechanisms to handle DNS requests. - v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"] - end - end - -end diff --git a/docs/contributing.rst b/docs/contributing.rst index 040c22864..5f939e947 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -54,104 +54,68 @@ where appropriate. Once you've got a working branch, you can open a pull request. All changes in your pull request must have thorough unit test coverage, pass our -`integration`_ tests, and be compliant with the :ref:`coding style -`. +tests, and be compliant with the :ref:`coding style `. .. _github issue tracker: https://github.com/certbot/certbot/issues .. _Good Volunteer Task: https://github.com/certbot/certbot/issues?q=is%3Aopen+is%3Aissue+label%3A%22Good+Volunteer+Task%22 +.. _testing: + Testing ------- -The following tools are there to help you: +When you are working in a file ``foo.py``, there should also be a file ``foo_test.py`` +either in the same directory as ``foo.py`` or in the ``tests`` subdirectory +(if there isn't, make one). While you are working on your code and tests, run +``python foo_test.py`` to run the relevant tests. -- ``tox`` starts a full set of tests. Please note that it includes - apacheconftest, which uses the system's Apache install to test config file - parsing, so it should only be run on systems that have an - experimental, non-production Apache2 install on them. ``tox -e - apacheconftest`` can be used to run those specific Apache conf tests. +For debugging, we recommend running ``pip install ipdb`` and putting +``import ipdb; ipdb.set_trace()`` statements inside the source +code. Alternatively, you can use Python's standard library `pdb`, +but you won't get TAB completion. -- ``tox --skip-missing-interpreters`` runs tox while ignoring missing versions - of Python needed for running the tests. +Once you are done with your code changes, and the tests in ``foo_test.py`` pass, +run all of the unittests for Certbot with ``tox -e py27`` (this uses Python +2.7). -- ``tox -e py27``, ``tox -e py26`` etc, run unit tests for specific Python - versions. +Once all the unittests pass, check for sufficient test coverage using +``tox -e cover``, and then check for code style with ``tox -e lint`` (all files) +or ``pylint --rcfile=.pylintrc path/to/file.py`` (single file at a time). -- ``tox -e cover`` checks the test coverage only. Calling the - ``./tox.cover.sh`` script directly (or even ``./tox.cover.sh $pkg1 - $pkg2 ...`` for any subpackages) might be a bit quicker, though. - -- ``tox -e lint`` checks the style of the whole project, while - ``pylint --rcfile=.pylintrc path`` will check a single file or - specific directory only. - -- For debugging, we recommend ``pip install ipdb`` and putting - ``import ipdb; ipdb.set_trace()`` statement inside the source - code. Alternatively, you can use Python's standard library `pdb`, - but you won't get TAB completion... +Once all of the above is successful, you may run the full test suite, +including integration tests, using ``tox``. We recommend running the +commands above first, because running all tests with ``tox`` is very +slow, and the large amount of ``tox`` output can make it hard to find +specific failures when they happen. Also note that the full test suite +will attempt to modify your system's Apache config if your user has sudo +permissions, so it should not be run on a production Apache server. +If you have trouble getting the full ``tox`` suite to run locally, it is +generally sufficient to open a pull request and let Github and Travis run +integration tests for you. .. _integration: -Integration testing with the boulder CA +Integration testing with the Boulder CA ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Generally it is sufficient to open a pull request and let Github and Travis run -integration tests for you. - -However, if you prefer to run tests, you can use Vagrant, using the Vagrantfile -in Certbot's repository. To execute the tests on a Vagrant box, the only -command you are required to run is:: - - ./tests/boulder-integration.sh - -Otherwise, please follow the following instructions. - -macOS users: Run ``./tests/mac-bootstrap.sh`` instead of -``boulder-start.sh`` to install dependencies, configure the -environment, and start boulder. - -Otherwise, install `Go`_ 1.5, ``libtool-ltdl``, ``mariadb-server`` and -``rabbitmq-server`` and then start Boulder_, an ACME CA server. - -If you can't get packages of Go 1.5 for your Linux system, -you can execute the following commands to install it: +To run integration tests locally, you need Docker and docker-compose installed +and working. Fetch and start Boulder using: .. code-block:: shell - wget https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz -P /tmp/ - sudo tar -C /usr/local -xzf /tmp/go1.5.3.linux-amd64.tar.gz - if ! grep -Fxq "export GOROOT=/usr/local/go" ~/.profile ; then echo "export GOROOT=/usr/local/go" >> ~/.profile; fi - if ! grep -Fxq "export PATH=\\$GOROOT/bin:\\$PATH" ~/.profile ; then echo "export PATH=\\$GOROOT/bin:\\$PATH" >> ~/.profile; fi + ./tests/boulder-fetch.sh -These commands download `Go`_ 1.5.3 to ``/tmp/``, extracts to ``/usr/local``, -and then adds the export lines required to execute ``boulder-start.sh`` to -``~/.profile`` if they were not previously added +If you have problems with Docker, you may want to try `removing all containers and +volumes`_ and making sure you have at least 1GB of memory. -Make sure you execute the following command after `Go`_ finishes installing:: +Run the integration tests using: - if ! grep -Fxq "export GOPATH=\\$HOME/go" ~/.profile ; then echo "export GOPATH=\\$HOME/go" >> ~/.profile; fi +.. code-block:: shell -Afterwards, you'd be able to start Boulder_ using the following command:: - - ./tests/boulder-start.sh - -The script will download, compile and run the executable; please be -patient - it will take some time... Once its ready, you will see -``Server running, listening on 127.0.0.1:4000...``. Add ``/etc/hosts`` -entries pointing ``le.wtf``, ``le1.wtf``, ``le2.wtf``, ``le3.wtf`` -and ``nginx.wtf`` to 127.0.0.1. You may now run (in a separate terminal):: - - ./tests/boulder-integration.sh && echo OK || echo FAIL - -If you would like to test `certbot_nginx` plugin (highly -encouraged) make sure to install prerequisites as listed in -``certbot-nginx/tests/boulder-integration.sh`` and rerun -the integration tests suite. - -.. _Boulder: https://github.com/letsencrypt/boulder -.. _Go: https://golang.org + ./tests/boulder-integration.sh +.. _removing all containers and volumes: https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes Code components and layout ========================== @@ -159,7 +123,11 @@ Code components and layout acme contains all protocol specific code certbot - all client code + main client code +certbot-apache and certbot-nginx + client code to configure specific web servers +certbot.egg-info + configuration for packaging Certbot Plugin-architecture diff --git a/tests/boulder-integration.sh b/tests/boulder-integration.sh index 4a2131006..2ddd3c04b 100755 --- a/tests/boulder-integration.sh +++ b/tests/boulder-integration.sh @@ -1,7 +1,7 @@ #!/bin/sh -xe # Simple integration test. Make sure to activate virtualenv beforehand # (source venv/bin/activate) and that you are running Boulder test -# instance (see ./boulder-start.sh). +# instance (see ./boulder-fetch.sh). # # Environment variables: # SERVER: Passed as "certbot --server" argument. diff --git a/tests/boulder-start.sh b/tests/boulder-start.sh deleted file mode 100755 index acf8f0bbf..000000000 --- a/tests/boulder-start.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash - -export GOPATH="${GOPATH:-/tmp/go}" -export PATH="$GOPATH/bin:$PATH" - -./tests/boulder-fetch.sh - -cd $GOPATH/src/github.com/letsencrypt/boulder -./start.py diff --git a/tests/mac-bootstrap.sh b/tests/mac-bootstrap.sh deleted file mode 100755 index 66036ce56..000000000 --- a/tests/mac-bootstrap.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/sh - -#Check Homebrew -if ! hash brew 2>/dev/null; then - echo "Homebrew Not Installed\nDownloading..." - ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" -fi - -brew install libtool mariadb rabbitmq coreutils go - -mysql.server start - -rabbit_pid=`ps | grep rabbitmq | grep -v grep | awk '{ print $1}'` -if [ -n "$rabbit_pid" ]; then - echo "RabbitMQ already running" -else - rabbitmq-server & -fi - -hosts_entry=`cat /etc/hosts | grep "127.0.0.1 le.wtf"` -if [ -z "$hosts_entry" ]; then - echo "Adding hosts entry for le.wtf..." - sudo sh -c "echo 127.0.0.1 le.wtf >> /etc/hosts" -fi - -./tests/boulder-start.sh