[Unix] Create a framework for certbot integration tests: PART 4 (#6958)

This PR is the part 4 to implement #6541. It adds the integration tests for the nginx certbot plugin, and corresponds to the certbot-ci translation of certbot-nginx/tests/boulder-integration.sh that is executed for each PR.

As with certbot core tests, tests are written in Python, and executed by pytest, against a dynamic Boulder/Pebble instance setup. Tests are parallelized, of course, and a specific IntegrationTestsContext class, extended the one from certbot core tests, is crafter for these specific tests: its main goal is to setup a specific nginx instance for the current test.

On top of that, I use the test parametrization feature of Pytest, to drastically reduce the size of the actual code: indeed, the 6 tests from the original bash script share the same logic. So using a parametrization, one unique test is written, that is then executed 6 times against 6 different sets of parameters.

Note that the module integration_tests.nginx_tests.nginx_config do the same, but in Python, than certbot-nginx/tests/boulder-integration.conf.sh. The latter will be removed in a future PR, with all other bash scripts.

* Add nginx tests

* Distribute the other_port

* Load a pre-generated key/cert for nginx config

* Correct preload, remove a test, simplify a variable

* Integrate assertion directly in the test function

* Check process is not terminated

* Add spaces in the nginx config

* Add comments

* Use indirection

* Allow external cert

* Add coverage threshold for certbot-nginx

certbot-ci/certbot_integration_tests/assets/nginx_cert.pem

@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFlTCCA32gAwIBAgIUR3wbM8qFE68f8NxfciHhUjR1GeUwDQYJKoZIhvcNAQEL
+BQAwWTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
+GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDESMBAGA1UEAwwJbmdpbngud3RmMCAX
+DTE5MDQxODIwMDUwM1oYDzIyOTMwMTMwMjAwNTAzWjBZMQswCQYDVQQGEwJBVTET
+MBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQ
+dHkgTHRkMRIwEAYDVQQDDAluZ2lueC53dGYwggIiMA0GCSqGSIb3DQEBAQUAA4IC
+DwAwggIKAoICAQC/W+yxYE0PWJOS4df71Yx596fDjW03I9JZuu9kfP7mneMgy+OC
+HyRm0TEhl6FPUp9tD9YeEHloUZNjHEOg/qrnbEOspv3Ha3RFinzrzkMwbzEPR3Xf
+0go+aVsWelDhapFl8fccw4tWwijVZQquhBsWOUnPenS3Txe96kEv2NNJlJ0qFUa+
+rOTruzRzOzlbgKv5WRb4+BxxWonHLkAQ5IT87GBlsCerVIyPD+BnZveZGl6e9oMH
+ZlZvUT6aWRnzFWjAnQGiJpVIw7l9r4EW0jq1z7wqb37FrqrFbtWrOfUZVE7AlqXH
+aKIR82/xwkcZfFk3sCAM0IcZc8B2SDLi4zNZtDivW6qQgTC/3z5yf1hnJ+j00dtE
+X5qYlgXRaM2raOn31lxcerk5pjgagQ7Zj+v3YZS0QnenrgyXJcdnXLDj+cIARzx4
+QHtoO0nyP0RJqxvwX/H98513JTkeqFBc/Bx11UWYsUv20Qoo9IAuz0VDARu6rquu
+k9anv56yvxo77qZ8r80l3z8eMyDA+UjuSD2p1Za09RAHfva7o8rMUqULHNQ4pfFH
+JIUozHoinAg/9lBC/W80fcbILks+Sdi6E9WQ0n8PLl7oFLx9prEDCycKuC0z76J/
+Shb6R6sWr1YtzUFUc5EH2g9pMriaqT8uGO4CMOeRemXahrdT/H+Xg5m4TQIDAQAB
+o1MwUTAdBgNVHQ4EFgQU46gJeu9ZOfTQ6c4vfbWbSLUpEMowHwYDVR0jBBgwFoAU
+46gJeu9ZOfTQ6c4vfbWbSLUpEMowDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
+AQsFAAOCAgEAcnfkXDUTsEGs0MleegkGbRCVy72a3U7tv1KVTLB8qLPc3tpPJJoT
+D4PbOuw9+yIE+HetZTZooOpaZoorLQdiwAEjlQ44RVuXSHSARQ8KW9ZZeiWN/Qvl
+Ip4xJ/cHxcKTFKSc/99o8M+kmPKEXF9SUMfKPc5jXarNxCsnA3VriYqJ+CnYEox2
+duNUEe3A9Y2d8ZxjmscBqlcXpk1kFwsCRT5UYVoUYwyjYznLkO5A+GJ0ZnMyRMQp
+obUiB34hUrNgyOaBvizk+pNh9EV4rEBPRQwhy4vDMco4AjQcwLWQAQ9G4GSt/E+Q
+62XdVDa6CAuOvBCudDPki7kEqNLbj1tMY1K/gsbgb6TYA/xTOVulAnqm4OEZ2svJ
+0Jqw3BzMfRTaxsNU6jxm8WehVL15GjoJUzfs7Te+l7Vm/QNc1Dv2pmEhVfBibwMa
+YxUZ8ClQtQ1lsOpne97Og0p/Cm93kKELNBLTjzXtpXGGPPYisAyNwe0Hadq8SiOd
+pXeNwXa5vHOXHv8xBENzBvFJ3TRN2GmMlHBp/eOfVUx/huNSpcnh2gO3fn5EbMj7
+43IaR133JW5yWbneYAMJOEAMdEB5EthRmEDtLVA7kLqLc/ywFTQ4VbS2b+PsOr5O
+501nzt0OTMMEz+UafvGXj5OPJBhe26RtnYXzVwwLfto/F5udM5zglWo=
+-----END CERTIFICATE-----

certbot-ci/certbot_integration_tests/assets/nginx_key.pem

@@ -0,0 +1,52 @@
+-----BEGIN PRIVATE KEY-----
+MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQC/W+yxYE0PWJOS
+4df71Yx596fDjW03I9JZuu9kfP7mneMgy+OCHyRm0TEhl6FPUp9tD9YeEHloUZNj
+HEOg/qrnbEOspv3Ha3RFinzrzkMwbzEPR3Xf0go+aVsWelDhapFl8fccw4tWwijV
+ZQquhBsWOUnPenS3Txe96kEv2NNJlJ0qFUa+rOTruzRzOzlbgKv5WRb4+BxxWonH
+LkAQ5IT87GBlsCerVIyPD+BnZveZGl6e9oMHZlZvUT6aWRnzFWjAnQGiJpVIw7l9
+r4EW0jq1z7wqb37FrqrFbtWrOfUZVE7AlqXHaKIR82/xwkcZfFk3sCAM0IcZc8B2
+SDLi4zNZtDivW6qQgTC/3z5yf1hnJ+j00dtEX5qYlgXRaM2raOn31lxcerk5pjga
+gQ7Zj+v3YZS0QnenrgyXJcdnXLDj+cIARzx4QHtoO0nyP0RJqxvwX/H98513JTke
+qFBc/Bx11UWYsUv20Qoo9IAuz0VDARu6rquuk9anv56yvxo77qZ8r80l3z8eMyDA
++UjuSD2p1Za09RAHfva7o8rMUqULHNQ4pfFHJIUozHoinAg/9lBC/W80fcbILks+
+Sdi6E9WQ0n8PLl7oFLx9prEDCycKuC0z76J/Shb6R6sWr1YtzUFUc5EH2g9pMria
+qT8uGO4CMOeRemXahrdT/H+Xg5m4TQIDAQABAoICAAGGL+pxw+tdXz+KQPgmiUnn
+aRSrqbUIugIw9Pst67HWjBqUxSkiKl4PSH7mAEjrdY2e1KvEodLs42mkrf04ShAx
+0pArfFX8Sx7KrZgLOonGOPPQM+YmfCJnIGybaM2C1cmkFb3K6O81+LFKbr1ZHAYf
+SrE2XnufS6cdmItTBMvPPTk6lieqpOAjy5UnYZuS+Muxo/czsrZMbFCD08rOpyiE
+kXf94TMCJ2R0UetA7LPxe9N0TzLd485bLU55azV+dCkklwC9oe7EcFPJ9BNEdWdB
+UlRcMvxMGdwct+L3QTaEb2QlTwi5kqDl+XxJeduAHA3Pf1Haz1iqjVvj01PvT1di
+Cs0+ZeFBsa+BfiGDe9ONwuSQljda1CuI+vDv5bGUExulOSG1dHJ7RK9PBaXFaR/b
+/9tRBwAw1Erm7s1JIkjda5Oc46jFb3HzDaZYB1n5hUmEIrYM8HhUOGITyVT3hxDO
+AWlaV3aveQ0MmMXLptVXDgbjPGbWDGMLD9d5vUE9R7IyOLeXOmjthYlCH2rj378M
+r2PkgX2tD0A/yoEZ8XCFdtBWSVajLdL0/gkm7sKosMABBy3yrSCxbHeq5TFuTAXA
+hOdypX4NOZkA6WJU+hn3GkQyIScLqSrvGRA9kzHGoEWVZDKkB9DXg+dmTARZDWXD
+mCnHkJo6+FcbhUpXniuZAoIBAQDmE94vvdstB+HEtXxN1uNDY7H8gPc/BUonU6a9
+G5YOIbjByCfEDcXF8AUWekc6lc8DNG3ydx0dnb2ZAkxmdlsaD8GLqHGILzlSsOwR
+sez8nR4+4n9vYMfx9Qal8Ren5xEP9Z9sJcNqbKVGta1WFtQzrgYbpVXXf/Luv0xS
+YoVK8KaEACciD6XX4wmajrAXPPQgThvqQtXuTn/AxWsUDg1DK0tw1VRUuOJuJwpw
+f6qocM9AyqUNvdeVyjFx8Slag34ZI7fmxPtHX/e6opTg3zVXab1Ow8AMICOHMRL6
+m5/+wnWa9xMoKI4kYfk/QFqeTccnLDlwi6kQM8WRfbwr9AyPAoIBAQDU60wrX6Lm
+0vIfngv1/4j/w+AGAwjvxiuJ7Q7LwQ2fGsZGOIfMK/lpBxCn543kGbQT+KQKNOjO
++EywObftnJ6Y2+om2NoLkCnCiptsfr5WlN8pxtIPQu2iu5xXA67WpQv4Nc4769PM
+wJGVW3pmPKi6H0QjjqYAZd1NAXdN9Au14zZVh3KBWoz82kTHWKSL6Ld1UClG728W
+k/moyCFFMMGTXX/LVliQzDVLM6L5jbAOaG317qAuxZIqFJ9NLwHFW9uH/i1S6Qfp
++lOmOfVYKu1O/qh1DUBQfuJkR1XIn2ifZEjxOsxeTmWu1LXpyoZy526JRu49pk8Z
+DdEu+w7hsdNjAoIBAD1YWsub8Y6GJXpPcX9HpnzXXiOXN1VEUcs+kJyneFD4SMzS
+U1gA3BS0tIaTv94tB28xUYdunwLAhkb/x+Mh95RxUwert+m5va0Ao1DsgeWw9tmJ
+hrTptyYaUNV5/Pa1s2Tv9rvdLcd4hHDgDAGCQL4uzk4cvVCiOuHRe8YTorqig6N6
+bvSz+2IelPbyyJzJkcXzTZoei+/oWkPJ340PWhXou0qwdrXIPgdkvXHVeGlE+t2p
+qmyJi6vSp3Bb/sy1dq+5SFVtfBpBykmnA88ZdJ2EAge4RcJ150MqoIbVa8l/i9/v
+tNnmRlAJF233+LFwx4L4VbBebIt3YlwyjDOj9J0CggEAIknKOGnsV/O8ni7bikAe
+leG7X/x5IfPt6wZMDbAHO4oaSBCufcjPH4TNv9xgU014XIb8E9C1dS8zWmXRIujH
++aHgsWTWqGoM75FWukAm8taCob2s8lw63KwN301uiI6HwO8ZSTkPILgaOc1DhtdZ
+7K9AT+GXBhVhcBc+WUVl5WKzy05GuGIWtlmIHfo+dXGCqdfA7fV9FEu8NtwTz4qs
+gcja3aoIFTltk7C7HCkfIxLaMnK9RQr4IOK1TL63MEs8rUfXkLSKW7m+YtSOmCZB
+lSkZg9AgfVYRq0h5nhddx91kicSISN+jLGaA7Sd6Q2LVwDG2CCOSNVyuRTyVBu+W
+NQKCAQAWN6vB6oToNIoBLdOThm0HD07cNHcrnBjtaKsYsQDgqbr2m8LRCRzNRML4
+jG0IAOWpuCiEGsgUPxywiI1Ufvyq7ZSNT1QQNzCR47NM3Ve6S2abrQkMIk9VJ+za
+CB9c1BH92GokoRxqswb/BiMttG2EIP8L8/pSRYEcVnaaxAkf9QOhEwj4LJPGX0mS
+t7kWIUVHPdFJ67F25dYr3mUHgyV+QJupQICkkkgY3nBOU1fS42vAugaxqH0wAP3T
+53FlpY3NuE7+kYC3FjfcBer99F1pOac3X9jxhk26w9dr2/QNA33xhDXHKYvoLUCG
+RPQylahJByU7IrtQzSCf/RE7q4v0
+-----END PRIVATE KEY-----

certbot-ci/certbot_integration_tests/certbot_tests/context.py

@@ -25,6 +25,7 @@ class IntegrationTestsContext(object):
         self.directory_url = acme_xdist['directory_url']
         self.tls_alpn_01_port = acme_xdist['https_port'][self.worker_id]
         self.http_01_port = acme_xdist['http_port'][self.worker_id]
+        self.other_port = acme_xdist['other_port'][self.worker_id]
         # Challtestsrv REST API, that exposes entrypoints to register new DNS entries,
         # is listening on challtestsrv_port.
         self.challtestsrv_port = acme_xdist['challtestsrv_port']

certbot-ci/certbot_integration_tests/nginx_tests/context.py

@@ -1,5 +1,58 @@
+import os
+import subprocess
+
 from certbot_integration_tests.certbot_tests import context as certbot_context
+from certbot_integration_tests.utils import misc
+from certbot_integration_tests.nginx_tests import nginx_config as config
 
 
 class IntegrationTestsContext(certbot_context.IntegrationTestsContext):
     """General fixture describing a certbot-nginx integration tests context"""
+    def __init__(self, request):
+        super(IntegrationTestsContext, self).__init__(request)
+
+        self.nginx_root = os.path.join(self.workspace, 'nginx')
+        os.mkdir(self.nginx_root)
+
+        self.webroot = os.path.join(self.nginx_root, 'webroot')
+        os.mkdir(self.webroot)
+        with open(os.path.join(self.webroot, 'index.html'), 'w') as file_handler:
+            file_handler.write('Hello World!')
+
+        self.nginx_config_path = os.path.join(self.nginx_root, 'nginx.conf')
+        self.nginx_config = None
+
+        default_server = request.param['default_server']
+        self.process = self._start_nginx(default_server)
+
+    def cleanup(self):
+        self._stop_nginx()
+        super(IntegrationTestsContext, self).cleanup()
+
+    def certbot_test_nginx(self, args):
+        """
+        Main command to execute certbot using the nginx plugin.
+        :param list args: list of arguments to pass to nginx
+        """
+        command = ['--authenticator', 'nginx', '--installer', 'nginx',
+                   '--nginx-server-root', self.nginx_root]
+        command.extend(args)
+        return self._common_test(command)
+
+    def _start_nginx(self, default_server):
+        self.nginx_config = config.construct_nginx_config(
+            self.nginx_root, self.webroot, self.http_01_port, self.tls_alpn_01_port,
+            self.other_port, default_server, wtf_prefix=self.worker_id)
+        with open(self.nginx_config_path, 'w') as file:
+            file.write(self.nginx_config)
+
+        process = subprocess.Popen(['nginx', '-c', self.nginx_config_path, '-g', 'daemon off;'])
+
+        assert process.poll() is None
+        misc.check_until_timeout('http://localhost:{0}'.format(self.http_01_port))
+        return process
+
+    def _stop_nginx(self):
+        assert self.process.poll() is None
+        self.process.terminate()
+        self.process.wait()

certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py

@@ -0,0 +1,126 @@
+"""General purpose nginx test configuration generator."""
+import getpass
+
+import pkg_resources
+
+
+def construct_nginx_config(nginx_root, nginx_webroot, http_port, https_port, other_port,
+                           default_server, key_path=None, cert_path=None, wtf_prefix='le'):
+    """
+    This method returns a full nginx configuration suitable for integration tests.
+    :param str nginx_root: nginx root configuration path
+    :param str nginx_webroot: nginx webroot path
+    :param int http_port: HTTP port to listen on
+    :param int https_port: HTTPS port to listen on
+    :param int other_port: other HTTP port to listen on
+    :param bool default_server: True to set a default server in nginx config, False otherwise
+    :param str key_path: the path to a SSL key
+    :param str cert_path: the path to a SSL certificate
+    :param str wtf_prefix: the prefix to use in all domains handled by this nginx config
+    :return: a string containing the full nginx configuration
+    :rtype: str
+    """
+    key_path = key_path if key_path \
+        else pkg_resources.resource_filename('certbot_integration_tests', 'assets/nginx_key.pem')
+    cert_path = cert_path if cert_path \
+        else pkg_resources.resource_filename('certbot_integration_tests', 'assets/nginx_cert.pem')
+    return '''\
+# This error log will be written regardless of server scope error_log
+# definitions, so we have to set this here in the main scope.
+#
+# Even doing this, Nginx will still try to create the default error file, and
+# log a non-fatal error when it fails. After that things will work, however.
+error_log {nginx_root}/error.log;
+
+# The pidfile will be written to /var/run unless this is set.
+pid {nginx_root}/nginx.pid;
+
+user {user};
+worker_processes 1;
+
+events {{
+  worker_connections 1024;
+}}
+
+http {{
+  # Set an array of temp, cache and log file options that will otherwise default to
+  # restricted locations accessible only to root.
+  client_body_temp_path {nginx_root}/client_body;
+  fastcgi_temp_path {nginx_root}/fastcgi_temp;
+  proxy_temp_path {nginx_root}/proxy_temp;
+  #scgi_temp_path {nginx_root}/scgi_temp;
+  #uwsgi_temp_path {nginx_root}/uwsgi_temp;
+  access_log {nginx_root}/error.log;
+  
+  # This should be turned off in a Virtualbox VM, as it can cause some
+  # interesting issues with data corruption in delivered files.
+  sendfile off;
+  
+  tcp_nopush on;
+  tcp_nodelay on;
+  keepalive_timeout 65;
+  types_hash_max_size 2048;
+  
+  #include /etc/nginx/mime.types;
+  index index.html index.htm index.php;
+  
+  log_format   main '$remote_addr - $remote_user [$time_local] $status '
+    '"$request" $body_bytes_sent "$http_referer" '
+    '"$http_user_agent" "$http_x_forwarded_for"';
+    
+  default_type application/octet-stream;
+  
+  server {{
+    # IPv4.
+    listen {http_port} {default_server};
+    # IPv6.
+    listen [::]:{http_port} {default_server};
+    server_name nginx.{wtf_prefix}.wtf nginx2.{wtf_prefix}.wtf;
+    
+    root {nginx_webroot};
+    
+    location / {{
+      # First attempt to serve request as file, then as directory, then fall
+      # back to index.html.
+      try_files $uri $uri/ /index.html;
+    }}
+  }}
+  
+  server {{
+    listen {http_port};
+    listen [::]:{http_port};
+    server_name nginx3.{wtf_prefix}.wtf;
+    
+    root {nginx_webroot};
+    
+    location /.well-known/ {{
+      return 404;
+    }}
+    
+    return 301 https://$host$request_uri;
+  }}
+  
+  server {{
+    listen {other_port};
+    listen [::]:{other_port};
+    server_name nginx4.{wtf_prefix}.wtf nginx5.{wtf_prefix}.wtf;
+  }}
+  
+  server {{
+    listen {http_port};
+    listen [::]:{http_port};
+    listen {https_port} ssl;
+    listen [::]:{https_port} ssl;
+    if ($scheme != "https") {{
+      return 301 https://$host$request_uri;
+    }}
+    server_name nginx6.{wtf_prefix}.wtf nginx7.{wtf_prefix}.wtf;
+
+    ssl_certificate {cert_path};
+    ssl_certificate_key {key_path};
+  }}
+}}
+'''.format(nginx_root=nginx_root, nginx_webroot=nginx_webroot, user=getpass.getuser(),
+           http_port=http_port, https_port=https_port, other_port=other_port,
+           default_server='default_server' if default_server else '', wtf_prefix=wtf_prefix,
+           key_path=key_path, cert_path=cert_path)

certbot-ci/certbot_integration_tests/nginx_tests/test_main.py

@@ -1,3 +1,7 @@
+"""Module executing integration tests against certbot with nginx plugin."""
+import os
+import ssl
+
 import pytest
 
 from certbot_integration_tests.nginx_tests import context as nginx_context
@@ -13,5 +17,40 @@ def context(request):
         integration_test_context.cleanup()
 
 
-def test_hello(context):
+@pytest.mark.parametrize('certname_pattern, params, context', [
-    print(context.directory_url)
+    ('nginx.{0}.wtf', ['run'], {'default_server': True}),
+    ('nginx2.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': True}),
+    # Overlapping location block and server-block-level return 301
+    ('nginx3.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': True}),
+    # No matching server block; default_server exists
+    ('nginx4.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': True}),
+    # No matching server block; default_server does not exist
+    ('nginx5.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': False}),
+    # Multiple domains, mix of matching and not
+    ('nginx6.{0}.wtf,nginx7.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': False}),
+], indirect=['context'])
+def test_certificate_deployment(certname_pattern, params, context):
+    """
+    Test various scenarios to deploy a certificate to nginx using certbot.
+    """
+    domains = certname_pattern.format(context.worker_id)
+    command = ['--domains', domains]
+    command.extend(params)
+    context.certbot_test_nginx(command)
+
+    lineage = domains.split(',')[0]
+    server_cert = ssl.get_server_certificate(('localhost', context.tls_alpn_01_port))
+    with open(os.path.join(context.workspace, 'conf/live/{0}/cert.pem'.format(lineage)), 'r') as file:
+        certbot_cert = file.read()
+
+    assert server_cert == certbot_cert
+
+    command = ['--authenticator', 'nginx', '--installer', 'nginx',
+               '--nginx-server-root', context.nginx_root,
+               'rollback', '--checkpoints', '1']
+    context._common_test_no_force_renew(command)
+
+    with open(context.nginx_config_path, 'r') as file_h:
+        current_nginx_config = file_h.read()
+
+    assert context.nginx_config == current_nginx_config

certbot-ci/certbot_integration_tests/utils/acme_server.py

@@ -57,6 +57,8 @@ def _construct_acme_xdist(acme_server, nodes):
                                in zip(nodes, range(5200, 5200 + len(nodes)))}
     acme_xdist['https_port'] = {node: port for (node, port)
                                 in zip(nodes, range(5100, 5100 + len(nodes)))}
+    acme_xdist['other_port'] = {node: port for (node, port)
+                                in zip(nodes, range(5300, 5300 + len(nodes)))}
 
     return acme_xdist
 

tox.ini

@@ -260,5 +260,6 @@ commands =
         --cov-config={toxinidir}/certbot-ci/certbot_integration_tests/.coveragerc \
         -W 'ignore:Unverified HTTPS request'
     coverage report --include 'certbot/*' --show-missing --fail-under=64
+    coverage report --include 'certbot-nginx/*' --show-missing --fail-under=74
 passenv =
     DOCKER_*