Move compatibility tests off of certbot-auto and Python 2 (#8248)

Fixes https://github.com/certbot/certbot/issues/8162.

I had to update the base of the Dockerfile to get a new enough version of Python 3. I also simplified things a lot and removed a lot of the comments that were essentially just describing how Dockerfiles work.

The most complicated changes here are in `testdata`. You can find a diff of the changes to `nginx.tar.gz` at https://gist.github.com/c7727db0cecf3f15f02439f085c73848.

The first problem was that there were some complaints from the new Apache/nginx/OpenSSL version about the 1024 bit RSA key so I updated `empty_cert.pem` both inside and outside of the tarball as well as the corresponding private key in the tarball to use a 2048 bit key.

The 2nd problem is trickier to understand. If you look at the output from nginx after loading the config from `lots/` you'll see it complaining about conflicting `server_name` directives for the directives I deleted. See https://dev.azure.com/certbot/certbot/_build/results?buildId=2578&view=logs&j=250aa146-b243-5f8f-bf86-17a529c9fb7e&t=9baa2014-9673-5e78-8f4f-7a463caf2bfa&l=1516.

After switching the tests to Python 3, tests on that domain started failing. What I believe to be happening is we were just lucky these tests were passing to begin with. In both the Apache and Nginx plugin, if there are conflicting virtual hosts like this, we just arbitrarily pick one. The relevant code here for nginx is https://github.com/certbot/certbot/blob/575092d6030330ed8379babaa4cbbfe43e7bf721/certbot-nginx/certbot_nginx/_internal/configurator.py#L455

I played around with a debugger and confirmed that before I removed the conflicting server names, there were two exact matches for the domain we were searching for here.

I think all that's going on is with the switch to Python 3, the vhost we happen to choose changes and "breaks" the test. I suspect this to be due to something like getting values out of a dict somewhere where the order of items in a dict while iterating over it is different between Python 2 and 3. I didn't track where this difference happens down, but I personally don't think it's a good use of time since I think the real problem here is that the nginx config being tested was invalid with conflicting `server` blocks.

I removed all references to the `server_name` causing conflicts in that nginx configuration because both server blocks had other domains that are being tested, but I could add either back if you prefer. You can see the `nginx_compat` test passing with these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=2587&view=logs&j=250aa146-b243-5f8f-bf86-17a529c9fb7e.

* update Dockerfile

* Fix apache_compat on py3.

* Update empty_cert.pem.

The command used here was `openssl req -key
certbot/certbot/tests/testdata/rsa2048_key.pem -new -subj '/CN=example.com'
-x509 >
certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem`.

* update nginx.tar.gz

* Remove conflicting server_names

certbot-compatibility-test/Dockerfile

@@ -1,47 +1,18 @@
-FROM debian:stretch
+FROM debian:buster
 MAINTAINER Brad Warren <bmw@eff.org>
 
-# no need to mkdir anything:
+RUN apt-get update && \
-# https://docs.docker.com/reference/builder/#copy
+    apt install python3-dev python3-venv gcc libaugeas0 libssl-dev \
-# If <dest> doesn't exist, it is created along with all missing
+                 libffi-dev ca-certificates openssl -y
-# directories in its path.
 
-# TODO: Install non-default Python versions for tox.
+WORKDIR /opt/certbot/src
-# TODO: Install Apache/Nginx for plugin development.
-COPY letsencrypt-auto-source /opt/certbot/src/letsencrypt-auto-source
-RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only
 
-# the above is not likely to change, so by putting it further up the
+# We copy all contents of the build directory to allow us to easily use
-# Dockerfile we make sure we cache as much as possible
+# things like tools/venv3.py which expects all of our packages to be available.
+COPY . .
 
-COPY certbot/setup.py certbot/README.rst certbot/CHANGELOG.md certbot/MANIFEST.in linter_plugin.py tox.cover.py tox.ini .pylintrc /opt/certbot/src/
+RUN tools/venv3.py
-
+ENV PATH /opt/certbot/src/venv3/bin:$PATH
-# all above files are necessary for setup.py, however, package source
-# code directory has to be copied separately to a subdirectory...
-# https://docs.docker.com/reference/builder/#copy: "If <src> is a
-# directory, the entire contents of the directory are copied,
-# including filesystem metadata. Note: The directory itself is not
-# copied, just its contents." Order again matters, three files are far
-# more likely to be cached than the whole project directory
-
-COPY certbot /opt/certbot/src/certbot/
-COPY acme /opt/certbot/src/acme/
-COPY certbot-apache /opt/certbot/src/certbot-apache/
-COPY certbot-nginx /opt/certbot/src/certbot-nginx/
-COPY certbot-compatibility-test /opt/certbot/src/certbot-compatibility-test/
-COPY tools /opt/certbot/src/tools
-
-RUN VIRTUALENV_NO_DOWNLOAD=1 virtualenv -p python2 /opt/certbot/venv && \
-    /opt/certbot/venv/bin/pip install -U setuptools && \
-    /opt/certbot/venv/bin/pip install -U pip
-ENV PATH /opt/certbot/venv/bin:$PATH
-RUN /opt/certbot/venv/bin/python \
-    /opt/certbot/src/tools/pip_install_editable.py \
-    /opt/certbot/src/acme \
-    /opt/certbot/src/certbot \
-    /opt/certbot/src/certbot-apache \
-    /opt/certbot/src/certbot-nginx \
-    /opt/certbot/src/certbot-compatibility-test
 
 # install in editable mode (-e) to save space: it's not possible to
 # "rm -rf /opt/certbot/src" (it's stays in the underlaying image);

certbot-compatibility-test/certbot_compatibility_test/test_driver.py

@@ -102,8 +102,10 @@ def _create_achalls(plugin):
         prefs = plugin.get_chall_pref(domain)
         for chall_type in prefs:
             if chall_type == challenges.HTTP01:
+                # challenges.HTTP01.TOKEN_SIZE is a float but os.urandom
+                # expects an integer.
                 chall = challenges.HTTP01(
-                    token=os.urandom(challenges.HTTP01.TOKEN_SIZE))
+                    token=os.urandom(int(challenges.HTTP01.TOKEN_SIZE)))
                 challb = acme_util.chall_to_challb(
                     chall, messages.STATUS_PENDING)
                 achall = achallenges.KeyAuthorizationAnnotatedChallenge(
@@ -137,7 +139,7 @@ def test_deploy_cert(plugin, temp_dir, domains):
     """Tests deploy_cert returning True if the tests are successful"""
     cert = crypto_util.gen_ss_cert(util.KEY, domains)
     cert_path = os.path.join(temp_dir, "cert.pem")
-    with open(cert_path, "w") as f:
+    with open(cert_path, "wb") as f:
         f.write(OpenSSL.crypto.dump_certificate(
             OpenSSL.crypto.FILETYPE_PEM, cert))
 
@@ -273,7 +275,7 @@ def _dirs_are_unequal(dir1, dir2):
             logger.error(str(dircmp.diff_files))
             return True
 
-        for subdir in dircmp.subdirs.itervalues():
+        for subdir in dircmp.subdirs.values():
             dircmps.append(subdir)
 
     return False

certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem

@@ -1,13 +1,17 @@
 -----BEGIN CERTIFICATE-----
-MIICATCCAWoCCQCvMbKu4FHZ6zANBgkqhkiG9w0BAQsFADBFMQswCQYDVQQGEwJB
+MIICqDCCAZACCQCRC1UKg2WfRTANBgkqhkiG9w0BAQsFADAWMRQwEgYDVQQDDAtl
-VTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0
+eGFtcGxlLmNvbTAeFw0yMDA4MTkyMzM5MjdaFw0yMDA5MTgyMzM5MjdaMBYxFDAS
-cyBQdHkgTHRkMB4XDTE1MDcyMzIzMjc1MFoXDTE2MDcyMjIzMjc1MFowRTELMAkG
+BgNVBAMMC2V4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
-A1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0
+AQEA5tViHnJx4y+BbCb8Qz9uxsnqp1ynONR7ET/XL+M/jQ4xPeJg4L2uZ3YnogPc
-IFdpZGdpdHMgUHR5IEx0ZDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAws3o
+WdEoey17WXBg3KRqKfg+7PqIdGqVeonSCfXhD1HoGJRsThSUJ2fK3uoQ+zGgJTWR
-y46PMLM9Gr68pbex0MhdPr7Cq4rRe9BBpnOuHFdF35Ak0aPrzFwVzLlGOir94U11
+FYWa8Cb6xsuq0xaYtw2jaJBp+697Np60PWs4pY5FkadT50wZ0TYDnYt3NSAdn+Pt
-e5JYJDWJi+4FwLBRkOAfanjJ5GJ9BnEHSOdbtO+sv9uhbt+7iYOOUOngKSiJyUrM
+j3cpI4ocZZ2FLiOFn+UFOaRcetGtpnU1QwvmygD9tiL7kJ55B4CWGEv6DMRQk/UE
-i1THAE+B1CenxZ1KHRQCke708zkK8jVuxLeIAOMCAwEAATANBgkqhkiG9w0BAQsF
+eMUETzse1NkVlaxQ1TCd5iAfBTluiV30EpmmWa+OsXJWxCK+EEOkXD1r3CdXAldY
-AAOBgQCC3LUP3MHk+IBmwHHZAZCX+6p4lop9SP6y6rDpWgnqEEeb9oFleHi2Rvzq
+nRYxJrn4udrFe69QX95wiRZNXwIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCJvtDC
-7gxl6nS5AsaSzfAygJ3zWKTwVAZyU4GOQ8QTK+nHk3+LO1X4cDbUlQfm5+YuwKDa
+875CK7SKNf006gSciXsNPNSVORGPjc/5OQ23baK4iPhxftI4LGZN8773N14jWp3E
-4LFKeovmrK6BiMLIc1J+MxUjLfCeVHYSdkZULTVXue0zif0BUA==
+QnQLL1gZ9/G+98SlI5lm97a4m4XZyNaULbmQwRKgI22H0F1AWbvsG0SppjnhVlJ+
+93ZUqSQBXgbXelFHSsNfk1AB6Kvo6+UvS8s0vkz7SfkPOZGx0b+3RJSJZnZHvYih
+ggudN/jJggSgRrb+F6lpaelJE9pZsznJFb9R7mFI33AGBpQWV4r3p1ZbM1vGMqGc
+4PGBzDzi28BhLBplSOPZZxqRiINQzGiQ5T2SfN06usr7EafFr6+7YKNhgrCdlVjU
+thzJ5MgHZgALNXsh
 -----END CERTIFICATE-----