starred/certbot
1
0
Fork 0
mirror of https://github.com/certbot/certbot.git synced 2026-07-28 16:54:56 +02:00
Code Issues Packages Projects Releases Wiki Activity

Enable again build isolation with proper pinning of build dependencies (#8443)

Browse Source 
Fixes #8256

First let's sum up the problem to solve. We disabled the build isolation available in pip>=19 because it could potential break certbot build without a control on our side. Basically builds are not reproductible. Indeed the build isolation triggers build of PEP-517 enabled transitive dependencies (like `cryptography`) with the build dependencies defined in their `pyproject.toml`. For `cryptography` in particular these requirements include `setuptools>=40.6.0`, and quite logically pip will install the latest version of `setuptools` for the build. And when `setuptools` broke with the version 50, our build did the same.

But disabling the build isolation is not a long term solution, as more and more project will migrate on this approach and it basically provides a lot of benefit in how dependencies are built.

The ideal solution would be to be able to apply version constraints on our side on the build dependencies, in order to pin `setuptools` for instance, and decide precisely when we upgrade to a newer version. However for now pip does not provide a mechanism for that (like a `--build-constraint` flag or propagation of existing `--constraint` flag).

Until I saw https://github.com/pypa/pip/issues/9081 and https://github.com/pypa/pip/issues/8439.

Apart the fact that https://github.com/pypa/pip/issues/9081 shows that pip maintainers are working on this issue, it explains how pip works regarding PEP-517 and infers which workaround can be used to still pin the build dependencies. It turns out that pip invokes itself in each build isolation to install the build dependencies. It means that even if some flags (like `--constraint`) are not explicitly passed to the pip sub call, the global environment remains, in particular the environment variables.

Thus it is known that every pip flag can alternatively be set by environment variable using the following pattern for the variable name: `PIP_[FLAG_NAME_UPPERCASE]`. So for `--constraint`, it is `PIP_CONSTRAINT`. And so you can pass a constraint file to the pip sub call through that mechanism.

I made some tests with a constraint file containing pinning for `setuptools`: indeed under isolation zone, the constraint file has been honored and the provided pinned version has been used to build the dependencies (I tested it with `cryptography`).

Finally this PR takes advantage of this mechanism, by setting `PIP_CONSTRAINT` to `pip_install`, the snap building process, the Dockerfiles and the windows installer building process.

I also extracted out the requirements of the new `pipstrap.py` to be reusable in these various build processes.

* Use workaround to fix build requirements in build isolation, and renable build isolation

* Clean imports in pipstrap

* Externalize pipstrap reqs to be reusable

* Inject pipstrap constraints during pip_install

* Update docker build

* Update snapcraft build

* Prepare installer build

* Fix pipstrap constraints in snap build

* Add back --no-build-cache option in Docker images build

* Update snap/snapcraft.yaml

* Use proper flags with pip

Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This commit is contained in:
Adrien Ferrand
2020-12-16 10:49:31 -08:00
committed by GitHub
co-authored by Brad Warren
parent c5a0b1ae5d
commit d38766e05c
7 changed files with 72 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
snap/snapcraft.yaml
+10 -4
View File
@@ -40,7 +40,6 @@ parts:
certbot:
plugin: python
source: .
constraints: [$SNAPCRAFT_PART_SRC/snap-constraints.txt]
python-packages:
- git+https://github.com/certbot/python-augeas.git@certbot-patched
- ./acme
@@ -74,14 +73,21 @@ parts:
build-packages: [gcc, libffi-dev, libssl-dev, git, libaugeas-dev, python3-dev]
build-environment:
- SNAPCRAFT_PYTHON_VENV_ARGS: --upgrade
- PIP_NO_BUILD_ISOLATION: "no"
# Constraints are passed through the environment variable PIP_CONSTRAINTS instead of using the
# parts.[part_name].constraints option available in snapcraft.yaml when the Python plugin is
# used. This is done to let these constraints be applied not only on the certbot package
# build, but also on any isolated build that pip could trigger when building wheels for
# dependencies. See https://github.com/certbot/certbot/pull/8443 for more info.
- PIP_CONSTRAINT: $SNAPCRAFT_PART_SRC/snap-constraints.txt
override-build: |
python3 -m venv "${SNAPCRAFT_PART_INSTALL}"
"${SNAPCRAFT_PART_INSTALL}"/bin/python3 "${SNAPCRAFT_PART_SRC}/tools/pipstrap.py"
"${SNAPCRAFT_PART_INSTALL}/bin/python3" "${SNAPCRAFT_PART_SRC}/tools/pipstrap.py"
snapcraftctl build
override-pull: |
snapcraftctl pull
python3 "${SNAPCRAFT_PART_SRC}/tools/strip_hashes.py" "${SNAPCRAFT_PART_SRC}/letsencrypt-auto-source/pieces/dependency-requirements.txt" | grep -v python-augeas > "${SNAPCRAFT_PART_SRC}/snap-constraints.txt"
python3 "${SNAPCRAFT_PART_SRC}/tools/strip_hashes.py" "${SNAPCRAFT_PART_SRC}/letsencrypt-auto-source/pieces/dependency-requirements.txt" | grep -v python-augeas >> "${SNAPCRAFT_PART_SRC}/snap-constraints.txt"
python3 "${SNAPCRAFT_PART_SRC}/tools/strip_hashes.py" "${SNAPCRAFT_PART_SRC}/tools/pipstrap_constraints.txt" >> "${SNAPCRAFT_PART_SRC}/snap-constraints.txt"
echo "$(python3 "${SNAPCRAFT_PART_SRC}/tools/merge_requirements.py" "${SNAPCRAFT_PART_SRC}/snap-constraints.txt")" > "${SNAPCRAFT_PART_SRC}/snap-constraints.txt"
snapcraftctl set-version `grep -oP "__version__ = '\K.*(?=')" "${SNAPCRAFT_PART_SRC}/certbot/certbot/__init__.py"`
shared-metadata:
plugin: dump
tools/docker/core/Dockerfile
+4 -6
View File
@@ -42,9 +42,7 @@ RUN apk add --no-cache --virtual .build-deps \
musl-dev \
libffi-dev \
&& python tools/pipstrap.py \
&& pip install --no-build-isolation \
-r letsencrypt-auto-source/pieces/dependency-requirements.txt \
&& pip install --no-build-isolation --no-cache-dir --no-deps \
--editable src/acme \
--editable src/certbot \
&& apk del .build-deps
&& python tools/pip_install.py --no-cache-dir \
--editable src/acme \
--editable src/certbot \
&& apk del .build-deps
tools/docker/plugin/Dockerfile
+1 -1
View File
@@ -11,4 +11,4 @@ COPY qemu-${QEMU_ARCH}-static /usr/bin/
COPY . /opt/certbot/src/plugin
# Install the DNS plugin
RUN tools/pip_install.py --no-cache-dir --editable /opt/certbot/src/plugin
RUN python tools/pip_install.py --no-cache-dir --editable /opt/certbot/src/plugin
tools/pip_install.py
+23 -15
View File
@@ -59,9 +59,13 @@ def certbot_normal_processing(tools_path, test_constraints):
certbot_requirements = os.path.normpath(os.path.join(
repo_path, 'letsencrypt-auto-source/pieces/dependency-requirements.txt'))
with open(certbot_requirements, 'r') as fd:
data = fd.readlines()
certbot_reqs = fd.readlines()
with open(os.path.join(tools_path, 'pipstrap_constraints.txt'), 'r') as fd:
pipstrap_reqs = fd.readlines()
with open(test_constraints, 'w') as fd:
data = "\n".join(strip_hashes.process_entries(data))
data_certbot = "\n".join(strip_hashes.process_entries(certbot_reqs))
data_pipstrap = "\n".join(strip_hashes.process_entries(pipstrap_reqs))
data = "\n".join([data_certbot, data_pipstrap])
fd.write(data)
@@ -72,7 +76,8 @@ def merge_requirements(tools_path, requirements, test_constraints, all_constrain
# Here is the order by increasing priority:
# 1) The general development constraints (tools/dev_constraints.txt)
# 2) The general tests constraints (oldest_requirements.txt or
# certbot-auto's dependency-requirements.txt for the normal processing)
# certbot-auto's dependency-requirements.txt + pipstrap's constraints
# for the normal processing)
# 3) The local requirement file, typically local-oldest-requirement in oldest tests
files = [os.path.join(tools_path, 'dev_constraints.txt'), test_constraints]
if requirements:
@@ -82,17 +87,18 @@ def merge_requirements(tools_path, requirements, test_constraints, all_constrain
fd.write(merged_requirements)
def call_with_print(command):
def call_with_print(command, env=None):
if not env:
env = os.environ
print(command)
subprocess.check_call(command, shell=True)
subprocess.check_call(command, shell=True, env=env)
def pip_install_with_print(args_str, disable_build_isolation=True):
command = ['"', sys.executable, '" -m pip install --disable-pip-version-check ']
if disable_build_isolation:
command.append('--no-build-isolation ')
command.append(args_str)
call_with_print(''.join(command))
def pip_install_with_print(args_str, env=None):
if not env:
env = os.environ
command = ['"', sys.executable, '" -m pip install --disable-pip-version-check ', args_str]
call_with_print(''.join(command), env=env)
def main(args):
@@ -113,20 +119,22 @@ def main(args):
else:
certbot_normal_processing(tools_path, test_constraints)
env = os.environ.copy()
env["PIP_CONSTRAINT"] = all_constraints
merge_requirements(tools_path, requirements, test_constraints, all_constraints)
if requirements: # This branch is executed during the oldest tests
# First step, install the transitive dependencies of oldest requirements
# in respect with oldest constraints.
pip_install_with_print('--constraint "{0}" --requirement "{1}"'
.format(all_constraints, requirements))
pip_install_with_print('--requirement "{0}"'.format(requirements),
env=env)
# Second step, ensure that oldest requirements themselves are effectively
# installed using --force-reinstall, and avoid corner cases like the one described
# in https://github.com/certbot/certbot/issues/7014.
pip_install_with_print('--force-reinstall --no-deps --requirement "{0}"'
.format(requirements))
pip_install_with_print('--constraint "{0}" {1}'.format(
all_constraints, ' '.join(args)))
pip_install_with_print(' '.join(args), env=env)
if __name__ == '__main__':
tools/pipstrap.py
+3 -32
View File
@@ -1,46 +1,17 @@
#!/usr/bin/env python
"""Uses pip to upgrade Python packaging tools to pinned versions."""
from __future__ import absolute_import
import os
import shutil
import tempfile
import pip_install
# We include the hashes of the packages here for extra verification of
# the packages downloaded from PyPI. This is especially valuable in our
# builds of Certbot that we ship to our users such as our Docker images.
#
# An older version of setuptools is currently used here in order to keep
# compatibility with Python 2 since newer versions of setuptools have dropped
# support for it.
REQUIREMENTS = r"""
pip==20.2.4 \
--hash=sha256:51f1c7514530bd5c145d8f13ed936ad6b8bfcb8cf74e10403d0890bc986f0033 \
--hash=sha256:85c99a857ea0fb0aedf23833d9be5c40cf253fe24443f0829c7b472e23c364a1
setuptools==44.1.1 \
--hash=sha256:27a714c09253134e60a6fa68130f78c7037e5562c4f21f8f318f2ae900d152d5 \
--hash=sha256:c67aa55db532a0dadc4d2e20ba9961cbd3ccc84d544e9029699822542b5a476b
wheel==0.35.1 \
--hash=sha256:497add53525d16c173c2c1c733b8f655510e909ea78cc0e29d374243544b77a2 \
--hash=sha256:99a22d87add3f634ff917310a3d87e499f19e663413a52eb9232c447aa646c9f
"""
_REQUIREMENTS_PATH = os.path.join(os.path.dirname(__file__), "pipstrap_constraints.txt")
def main():
with pip_install.temporary_directory() as tempdir:
requirements_filepath = os.path.join(tempdir, 'reqs.txt')
with open(requirements_filepath, 'w') as f:
f.write(REQUIREMENTS)
pip_install_args = '--requirement ' + requirements_filepath
# We don't disable build isolation because we may have an older
# version of pip that doesn't support the flag disabling it. We
# expect these packages to already have usable wheels available
# anyway so no building should be required.
pip_install.pip_install_with_print(pip_install_args,
disable_build_isolation=False)
pip_install_args = '--requirement "{0}"'.format(_REQUIREMENTS_PATH)
pip_install.pip_install_with_print(pip_install_args)
if __name__ == '__main__':
tools/pipstrap_constraints.txt
+18
View File
@@ -0,0 +1,18 @@
# Constraints for pipstrap.py
#
# We include the hashes of the packages here for extra verification of
# the packages downloaded from PyPI. This is especially valuable in our
# builds of Certbot that we ship to our users such as our Docker images.
#
# An older version of setuptools is currently used here in order to keep
# compatibility with Python 2 since newer versions of setuptools have dropped
# support for it.
pip==20.2.4 \
--hash=sha256:51f1c7514530bd5c145d8f13ed936ad6b8bfcb8cf74e10403d0890bc986f0033 \
--hash=sha256:85c99a857ea0fb0aedf23833d9be5c40cf253fe24443f0829c7b472e23c364a1
setuptools==44.1.1 \
--hash=sha256:27a714c09253134e60a6fa68130f78c7037e5562c4f21f8f318f2ae900d152d5 \
--hash=sha256:c67aa55db532a0dadc4d2e20ba9961cbd3ccc84d544e9029699822542b5a476b
wheel==0.35.1 \
--hash=sha256:497add53525d16c173c2c1c733b8f655510e909ea78cc0e29d374243544b77a2 \
--hash=sha256:99a22d87add3f634ff917310a3d87e499f19e663413a52eb9232c447aa646c9f
windows-installer/construct.py
+13 -6
View File
@@ -46,9 +46,11 @@ def _compile_wheels(repo_path, build_path, venv_python):
wheels_project = [os.path.join(repo_path, package) for package in certbot_packages]
with _prepare_constraints(repo_path) as constraints_file_path:
command = [venv_python, '-m', 'pip', 'wheel', '-w', wheels_path, '--constraint', constraints_file_path]
env = os.environ.copy()
env['PIP_CONSTRAINT'] = constraints_file_path
command = [venv_python, '-m', 'pip', 'wheel', '-w', wheels_path]
command.extend(wheels_project)
subprocess.check_call(command)
subprocess.check_call(command, env=env)
def _prepare_build_tools(venv_path, venv_python, repo_path):
@@ -61,15 +63,20 @@ def _prepare_build_tools(venv_path, venv_python, repo_path):
@contextlib.contextmanager
def _prepare_constraints(repo_path):
requirements = os.path.join(repo_path, 'letsencrypt-auto-source', 'pieces', 'dependency-requirements.txt')
constraints = subprocess.check_output(
[sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), requirements],
reqs_certbot = os.path.join(repo_path, 'letsencrypt-auto-source', 'pieces', 'dependency-requirements.txt')
reqs_pipstrap = os.path.join(repo_path, 'tools', 'pipstrap_constraints.txt')
constraints_certbot = subprocess.check_output(
[sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), reqs_certbot],
universal_newlines=True)
constraints_pipstrap = subprocess.check_output(
[sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), reqs_pipstrap],
universal_newlines=True)
workdir = tempfile.mkdtemp()
try:
constraints_file_path = os.path.join(workdir, 'constraints.txt')
with open(constraints_file_path, 'a') as file_h:
file_h.write(constraints)
file_h.write(constraints_pipstrap)
file_h.write(constraints_certbot)
file_h.write('pywin32=={0}'.format(PYWIN32_VERSION))
yield constraints_file_path
finally: