From 027a7d755d593e7c00be00df83c6909feffa53d5 Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 14:58:55 -0700 Subject: [PATCH 1/8] Update developer docs - Discuss letsencrypt-dev - Be clearer in general --- docs/contributing.rst | 188 +++++++++++++++++++++++++----------------- 1 file changed, 112 insertions(+), 76 deletions(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index 60e7f35c2..d7feb9edc 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -7,40 +7,33 @@ Contributing Hacking ======= -All changes in your pull request **must** have 100% unit test coverage, pass -our `integration`_ tests, **and** be compliant with the -:ref:`coding style `. +Running a local copy of the client +---------------------------------- - -Bootstrap ---------- - -Start by :ref:`installing Let's Encrypt prerequisites -`. Then run: +Running the client in developer mode from your local tree is very similar to +running the released versions of the client; just replace ``letsencrypt-auto`` +with ``letsencrypt-dev``: .. code-block:: shell - ./bootstrap/dev/venv.sh + git clone https://github.com/letsencrypt/letsencrypt + cd letsencrypt + ./letsencrypt-dev -Activate the virtualenv: -.. code-block:: shell +Find issues to work on +---------------------- - source ./$VENV_NAME/bin/activate +You can find the open issues in the `github issue tracker`_. If you're +starting work on something, post a comment to let others know and seek +feedback on your plan where appropriate. -This step should prepend you prompt with ``($VENV_NAME)`` and save you -from typing ``./$VENV_NAME/bin/...``. It is also required to run some -of the `testing`_ tools. Virtualenv can be disabled at any time by -typing ``deactivate``. More information can be found in `virtualenv -documentation`_. - -Note that packages are installed in so called *editable mode*, in -which any source code changes in the current working directory are -"live" and no further ``./bootstrap/dev/venv.sh`` or ``pip install -...`` invocations are necessary while developing. - -.. _`virtualenv documentation`: https://virtualenv.pypa.io +Once you've got a working branch, you can open a pull request. All changes in +your pull request must have thorough unit test coverage, pass our +`integration`_ tests, and be compliant with the :ref:`coding style +`. +.. _github issue tracker: https://github.com/letsencrypt/letsencrypt/issues Testing ------- @@ -64,8 +57,12 @@ The following tools are there to help you: but you won't get TAB completion... -Integration -~~~~~~~~~~~ +Integration testing with the boulder CA +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Generally it is sufficient to open a pull request and let Github and Travis run +integration tests for you. + Mac OS X users: Run `./tests/mac-bootstrap.sh` instead of `boulder-start.sh` to install dependencies, configure the environment, and start boulder. @@ -91,55 +88,6 @@ the integration tests suite. .. _Go: https://golang.org -Vagrant -------- - -If you are a Vagrant user, Let's Encrypt comes with a Vagrantfile that -automates setting up a development environment in an Ubuntu 14.04 -LTS VM. To set it up, simply run ``vagrant up``. The repository is -synced to ``/vagrant``, so you can get started with: - -.. code-block:: shell - - vagrant ssh - cd /vagrant - ./venv/bin/pip install -r requirements.txt .[dev,docs,testing] - sudo ./venv/bin/letsencrypt - -Support for other Linux distributions coming soon. - -.. note:: - Unfortunately, Python distutils and, by extension, setup.py and - tox, use hard linking quite extensively. Hard linking is not - supported by the default sync filesystem in Vagrant. As a result, - all actions with these commands are *significantly slower* in - Vagrant. One potential fix is to `use NFS`_ (`related issue`_). - -.. _use NFS: http://docs.vagrantup.com/v2/synced-folders/nfs.html -.. _related issue: https://github.com/ClusterHQ/flocker/issues/516 - - -Docker ------- - -OSX users will probably find it easiest to set up a Docker container for -development. Let's Encrypt comes with a Dockerfile (``Dockerfile-dev``) -for doing so. To use Docker on OSX, install and setup docker-machine using the -instructions at https://docs.docker.com/installation/mac/. - -To build the development Docker image:: - - docker build -t letsencrypt -f Dockerfile-dev . - -Now run tests inside the Docker image: - -.. code-block:: shell - - docker run -it letsencrypt bash - cd src - tox -e py27 - - Code components and layout ========================== @@ -293,6 +241,94 @@ directory. .. _prerequisites: + +Other methods for running the client +==================================== + +Lower level venv scripts +------------------------ + +You can get slightly lower level exposure to virtualenv by using these +scripts as an alternative to ``letsencrypt-dev``. + +these +by :ref:`installing Let's Encrypt prerequisites `. Then run: + +.. code-block:: shell + + ./bootstrap/dev/venv.sh + +Activate the virtualenv: + +.. code-block:: shell + + source ./$VENV_NAME/bin/activate + +This step should prepend you prompt with ``($VENV_NAME)`` and save you +from typing ``./$VENV_NAME/bin/...``. It is also required to run some +of the `testing`_ tools. Virtualenv can be disabled at any time by +typing ``deactivate``. More information can be found in `virtualenv +documentation`_. + +Note that packages are installed in so called *editable mode*, in +which any source code changes in the current working directory are +"live" and no further ``./bootstrap/dev/venv.sh`` or ``pip install +...`` invocations are necessary while developing. + +.. _`virtualenv documentation`: https://virtualenv.pypa.io + + + +Vagrant +------- + +If you are a Vagrant user, Let's Encrypt comes with a Vagrantfile that +automates setting up a development environment in an Ubuntu 14.04 +LTS VM. To set it up, simply run ``vagrant up``. The repository is +synced to ``/vagrant``, so you can get started with: + +.. code-block:: shell + + vagrant ssh + cd /vagrant + ./venv/bin/pip install -r requirements.txt .[dev,docs,testing] + sudo ./venv/bin/letsencrypt + +Support for other Linux distributions coming soon. + +.. note:: + Unfortunately, Python distutils and, by extension, setup.py and + tox, use hard linking quite extensively. Hard linking is not + supported by the default sync filesystem in Vagrant. As a result, + all actions with these commands are *significantly slower* in + Vagrant. One potential fix is to `use NFS`_ (`related issue`_). + +.. _use NFS: http://docs.vagrantup.com/v2/synced-folders/nfs.html +.. _related issue: https://github.com/ClusterHQ/flocker/issues/516 + + +Docker +------ + +OSX users will probably find it easiest to set up a Docker container for +development. Let's Encrypt comes with a Dockerfile (``Dockerfile-dev``) +for doing so. To use Docker on OSX, install and setup docker-machine using the +instructions at https://docs.docker.com/installation/mac/. + +To build the development Docker image:: + + docker build -t letsencrypt -f Dockerfile-dev . + +Now run tests inside the Docker image: + +.. code-block:: shell + + docker run -it letsencrypt bash + cd src + tox -e py27 + + + Notes on OS depedencies ======================= From ad56bbbceb486cf1b189a17f46ce3b6f98d557e8 Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Thu, 22 Oct 2015 17:30:37 -0700 Subject: [PATCH 2/8] Cleanup plugin docs --- docs/contributing.rst | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index d7feb9edc..61621424a 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -119,7 +119,8 @@ which implement bindings to alternative UI libraries. Authenticators -------------- -Authenticators are plugins designed to solve challenges received from +Authenticators are plugins designed prove that this client deserves a +certificate for some domain name by solving challenges received from the ACME server. From the protocol, there are essentially two different types of challenges. Challenges that must be solved by individual plugins in order to satisfy domain validation (subclasses @@ -143,17 +144,23 @@ in a separate branch). Installer --------- -Installers classes exist to actually setup the certificate and be able -to enhance the configuration. (Turn on HSTS, redirect to HTTPS, -etc). You can indicate your abilities through the -:meth:`~.IInstaller.supported_enhancements` call. We currently only +Installers classes exist to actually setup the certificate in a server, +possibly tweak the security configuration to make it more correct and secure +(Fix some mixed content problems, turn on HSTS, redirect to HTTPS, etc). +Installer plugins tell the main client about their abilities to do the latter +via the :meth:`~.IInstaller.supported_enhancements` call. We currently only have one Installer written (still developing), `~.ApacheConfigurator`. -Installers and Authenticators will oftentimes be the same -class/object. Installers and Authenticators are kept separate because +Installers and Authenticators will oftentimes be the same class/object +(because for instance both tasks can be performed by a webserver like nginx) +though this is not always the case (the standalone plugin is an authenticator +that listens on port 443, but it cannot install certs; a postfix plugin would +be an installer but not an authenticator). + +Installers and Authenticators are kept separate because it should be possible to use the `~.StandaloneAuthenticator` (it sets up its own Python server to perform challenges) with a program that -cannot solve challenges itself. (Imagine MTA installers). +cannot solve challenges itself (Such as MTA installers). Installer Development From 096b5ec32ade4ba33cf976f289c3b853d94a22c2 Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 16:12:33 -0700 Subject: [PATCH 3/8] Back out ./letsencrypt-dev; document "activate" more clearly --- bootstrap/install-deps.sh | 40 ++++++++++++++++++++++++++++++++++++ docs/contributing.rst | 43 ++++++++++++++++++++++++++++++--------- 2 files changed, 73 insertions(+), 10 deletions(-) create mode 100644 bootstrap/install-deps.sh diff --git a/bootstrap/install-deps.sh b/bootstrap/install-deps.sh new file mode 100644 index 000000000..159027c84 --- /dev/null +++ b/bootstrap/install-deps.sh @@ -0,0 +1,40 @@ +#!/bin/sh -e +# +# Install OS dependencies. In the glorious future, letsencrypt-auto will +# source this... + +if test "`id -u`" -ne "0" ; then + SUDO=sudo +else + SUDO= +fi + +BOOTSTRAP=`dirname $0`/bootstrap +if [ ! -f $BOOTSTRAP/debian.sh ] ; then + echo "Cannot find the letsencrypt bootstrap scripts in $BOOTSTRAP" + exit 1 +fi +if [ -f /etc/debian_version ] ; then + echo "Bootstrapping dependencies for Debian-based OSes..." + $SUDO $BOOTSTRAP/_deb_common.sh +elif [ -f /etc/arch-release ] ; then + echo "Bootstrapping dependencies for Archlinux..." + $SUDO $BOOTSTRAP/archlinux.sh +elif [ -f /etc/redhat-release ] ; then + echo "Bootstrapping dependencies for RedHat-based OSes..." + $SUDO $BOOTSTRAP/_rpm_common.sh +elif uname | grep -iq FreeBSD ; then + echo "Bootstrapping dependencies for FreeBSD..." + $SUDO $BOOTSTRAP/freebsd.sh +elif uname | grep -iq Darwin ; then + echo "Bootstrapping dependencies for Mac OS X..." + echo "WARNING: Mac support is very experimental at present..." + $BOOTSTRAP/mac.sh +else + echo "Sorry, I don't know how to bootstrap Let's Encrypt on your operating system!" + echo + echo "You will need to bootstrap, configure virtualenv, and run a pip install manually" + echo "Please see https://letsencrypt.readthedocs.org/en/latest/contributing.html#prerequisites" + echo "for more info" + exit 1 +fi diff --git a/docs/contributing.rst b/docs/contributing.rst index 61621424a..a6c93de4a 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -10,23 +10,41 @@ Hacking Running a local copy of the client ---------------------------------- -Running the client in developer mode from your local tree is very similar to -running the released versions of the client; just replace ``letsencrypt-auto`` -with ``letsencrypt-dev``: +Running the client in developer mode from your local tree is a little +different than running ``letsencrypt-auto``. To get set up, do these things +once: .. code-block:: shell git clone https://github.com/letsencrypt/letsencrypt cd letsencrypt - ./letsencrypt-dev + ./bootstrap/install-deps.sh + ./bootstrap/dev/venv.sh + +Then in each shell where you're working on the client, do: + +.. code-block:: shell + + ./venv/bin/activate + +After that, your shell will be using the virtual environment, and you run the +client by typing: + +.. code-block:: shell + + letsencrypt + +Activating a shell in this way makes it easier to run unit tests +with ``tox`` and integration tests, as described below. Find issues to work on ---------------------- -You can find the open issues in the `github issue tracker`_. If you're -starting work on something, post a comment to let others know and seek -feedback on your plan where appropriate. +You can find the open issues in the `github issue tracker`_. Comparatively +easy ones are marked `Good Volunteer Task`_. If you're starting work on +something, post a comment to let others know and seek feedback on your plan +where appropriate. Once you've got a working branch, you can open a pull request. All changes in your pull request must have thorough unit test coverage, pass our @@ -34,6 +52,7 @@ your pull request must have thorough unit test coverage, pass our `. .. _github issue tracker: https://github.com/letsencrypt/letsencrypt/issues +.. _Good Volunteer Task: https://github.com/letsencrypt/letsencrypt/issues?q=is%3Aopen+is%3Aissue+label%3A%22Good+Volunteer+Task%22 Testing ------- @@ -60,6 +79,8 @@ The following tools are there to help you: Integration testing with the boulder CA ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _integration: + Generally it is sufficient to open a pull request and let Github and Travis run integration tests for you. @@ -144,12 +165,14 @@ in a separate branch). Installer --------- -Installers classes exist to actually setup the certificate in a server, +Installers plugins exist to actually setup the certificate in a server, possibly tweak the security configuration to make it more correct and secure (Fix some mixed content problems, turn on HSTS, redirect to HTTPS, etc). Installer plugins tell the main client about their abilities to do the latter -via the :meth:`~.IInstaller.supported_enhancements` call. We currently only -have one Installer written (still developing), `~.ApacheConfigurator`. +via the :meth:`~.IInstaller.supported_enhancements` call. We currently +have two Installers in the tree, the `~.ApacheConfigurator`. and the +`~.NginxConfigurator`. External projects have made some progress toward +support for IIS, Icecast and Plesk. Installers and Authenticators will oftentimes be the same class/object (because for instance both tasks can be performed by a webserver like nginx) From e19e0dc2e9b6289ad3d7a8649344d6b9f9e83d6b Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 16:18:08 -0700 Subject: [PATCH 4/8] This section is now redundant --- docs/contributing.rst | 34 ---------------------------------- 1 file changed, 34 deletions(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index a6c93de4a..a32bc0e63 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -275,40 +275,6 @@ directory. Other methods for running the client ==================================== -Lower level venv scripts ------------------------- - -You can get slightly lower level exposure to virtualenv by using these -scripts as an alternative to ``letsencrypt-dev``. - -these -by :ref:`installing Let's Encrypt prerequisites `. Then run: - -.. code-block:: shell - - ./bootstrap/dev/venv.sh - -Activate the virtualenv: - -.. code-block:: shell - - source ./$VENV_NAME/bin/activate - -This step should prepend you prompt with ``($VENV_NAME)`` and save you -from typing ``./$VENV_NAME/bin/...``. It is also required to run some -of the `testing`_ tools. Virtualenv can be disabled at any time by -typing ``deactivate``. More information can be found in `virtualenv -documentation`_. - -Note that packages are installed in so called *editable mode*, in -which any source code changes in the current working directory are -"live" and no further ``./bootstrap/dev/venv.sh`` or ``pip install -...`` invocations are necessary while developing. - -.. _`virtualenv documentation`: https://virtualenv.pypa.io - - - Vagrant ------- From f66334a7827bcf08854bcc3f70ac19250600999c Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 17:22:26 -0700 Subject: [PATCH 5/8] chmod a+x --- bootstrap/install-deps.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 bootstrap/install-deps.sh diff --git a/bootstrap/install-deps.sh b/bootstrap/install-deps.sh old mode 100644 new mode 100755 From aa3f4933419232767952175e00343ee1f71d9774 Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 17:23:26 -0700 Subject: [PATCH 6/8] Fix fragment link nit --- docs/contributing.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index 9dfc35882..9faf82582 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -76,11 +76,11 @@ The following tools are there to help you: but you won't get TAB completion... +.. _integration: + Integration testing with the boulder CA ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. _integration: - Generally it is sufficient to open a pull request and let Github and Travis run integration tests for you. From 6b6a20b5e50db568dc0e59fa205124c524ebd89a Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 17:24:17 -0700 Subject: [PATCH 7/8] typo --- docs/contributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index 9faf82582..118906c0c 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -140,7 +140,7 @@ which implement bindings to alternative UI libraries. Authenticators -------------- -Authenticators are plugins designed prove that this client deserves a +Authenticators are plugins designed to prove that this client deserves a certificate for some domain name by solving challenges received from the ACME server. From the protocol, there are essentially two different types of challenges. Challenges that must be solved by From efa1d6d38e4c56c03c5d5c6da7d8449e3a28367b Mon Sep 17 00:00:00 2001 From: Peter Eckersley Date: Fri, 23 Oct 2015 17:27:55 -0700 Subject: [PATCH 8/8] Reintroduce a bit more verbosity about virtualenv? --- docs/contributing.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index 118906c0c..7859e14bc 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -35,8 +35,10 @@ client by typing: letsencrypt Activating a shell in this way makes it easier to run unit tests -with ``tox`` and integration tests, as described below. +with ``tox`` and integration tests, as described below. To reverse this, you +can type ``deactivate``. More information can be found in the `virtualenv docs`_. +.. _`virtualenv docs`: https://virtualenv.pypa.io Find issues to work on ----------------------