From 0f0000298bbaac04452cb01dd3a37bf5cf1ebd79 Mon Sep 17 00:00:00 2001 From: Brad Warren Date: Thu, 12 Dec 2024 12:00:11 -0800 Subject: [PATCH] improve repinning (#10082) this PR hopefully improves two things that i hit while working on #10035 1) i found that repinning our dependencies took ~6 minutes! digging into it a bit, the biggest culprit i found was the inclusion of `--no-cache-dir` here which seemed to cause poetry to redownload the same packages over and over in a single run. this comes from https://github.com/certbot/certbot/pull/9453 which fixed a problem i (but not alex) was having with a major performance penalty. i removed the flag here and instead included instructions on clearing poetry's caches in case anyone ever hits this in the future. with this change, the script now takes about 40 seconds on my laptop 2) every run of this script ended with the output: ``` Warning: poetry-plugin-export will not be installed by default in a future version of Poetry. In order to avoid a breaking change and make your automation forward-compatible, please install poetry-plugin-export. explicitly. See https://python-poetry.org/docs/plugins/#using-plugins for details on how to install a plugin. To disable this warning run 'poetry config warnings.export false'. ``` setting `POETRY_WARNINGS_EXPORT=false` fixes this which i believe is safe to do because of https://github.com/certbot/certbot/blob/2c8609464c3e371053f78e515fc7beb26446f5a3/certbot/setup.py#L53-L56 --- .../common/export-pinned-dependencies.sh | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/tools/pinning/common/export-pinned-dependencies.sh b/tools/pinning/common/export-pinned-dependencies.sh index 06f1da8a1..a136472b8 100755 --- a/tools/pinning/common/export-pinned-dependencies.sh +++ b/tools/pinning/common/export-pinned-dependencies.sh @@ -37,15 +37,22 @@ if [ -f poetry.lock ]; then rm poetry.lock fi -echo "If this takes more than a few minutes, you can try running this script again" >&2 -echo "with arguments for poetry like -vvv on the command line to help see where" >&2 -echo "poetry is getting stuck." >&2 +echo "If this takes more than a few minutes, you may want to try clearing poetry's" >&2 +echo "caches with a command like the following and running this script again:" >&2 +echo >&2 +echo " poetry cache list | xargs -L1 poetry cache clear --all" >&2 +echo >&2 +echo "If that doesn't help, you can also try running this script with extra arguments" >&2 +echo 'for "poetry lock" on the command line such as -vvv to help see where poetry is' >&2 +echo "getting stuck." >&2 extra_args="${*:2}" -# If you're running this with different Python versions (say to update both our -# "current" and "oldest" pinnings), poetry's cache can become corrupted causing -# poetry to hang indefinitely. --no-cache avoids this. -poetry lock --no-cache ${extra_args:+"$extra_args"} >&2 +poetry lock ${extra_args:+"$extra_args"} >&2 trap 'rm poetry.lock' EXIT -# We need to remove local packages from the output. -poetry export --format constraints.txt --without-hashes | sed '/^acme @/d; /certbot/d;' +# POETRY_WARNINGS_EXPORT is set to remove warning output about +# poetry-plugin-export no longer being installed with poetry by default in the +# future which we can ignore because we explicitly depend on the plugin +# package. +# +# sed is then used to remove local packages from the output. +POETRY_WARNINGS_EXPORT=false poetry export --format constraints.txt --without-hashes | sed '/^acme @/d; /certbot/d;'