Command-line UX overhaul (#8852)

Streamline and reorganize Certbot's CLI output.

This change is a substantial command-line UX overhaul,
based on previous user research. The main goal was to streamline
and clarify output. To see more verbose output, use the -v or -vv flags.

---

* nginx,apache: CLI logging changes

- Add "Successfully deployed ..." message using display_util
- Remove IReporter usage and replace with display_util
- Standardize "... could not find a VirtualHost ..." error

This changes also bumps the version of certbot required by certbot-nginx
and certbot-apache to take use of the new display_util function.

* fix certbot_compatibility_test

since the http plugins now require IDisplay, we need to inject it

* fix dependency version on certbot

* use better asserts

* try fix oldest deps

because certbot 1.10.0 depends on acme>=1.8.0, we need to use
acme==1.8.0 in the -oldest tests

* cli: redesign output of new certificate reporting

Changes the output of run, certonly and certonly --csr. No longer uses
IReporter.

* cli: redesign output of failed authz reporting

* fix problem sorting to be stable between py2 & 3

* add some catch-all error text

* cli: dont use IReporter for EFF donation prompt

* add per-authenticator hints

* pass achalls to auth_hint, write some tests

* exclude static auth hints from coverage

* dont call auth_hint unless derived from .Plugin

* dns fallback hint: dont assume --dns-blah works

--dns-blah won't work for third-party plugins, they need to be specified
using --authenticator dns-blah.

* add code comments about the auth_hint interface

* renew: don't restart the installer for dry-runs

Prevents Certbot from superfluously invoking the installer restart
during dry-run renewals. (This does not affect authenticator restarts).

Additionally removes some CLI output that was reporting the fullchain
path of the renewed certificate.

* update CHANGELOG.md

* cli: redesign output when cert installation failed

- Display a message when certificate installation begins.
- Don't use IReporter, just log errors immediately if restart/rollback
  fails.
- Prompt the user with a command to retry the installation process once
  they have fixed any underlying problems.

* vary by preconfigured_renewal

and move expiry date to be above the renewal advice

* update code comment

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* update code comment

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* fix lint

* derve cert name from cert_path, if possible

* fix type annotation

* text change in nginx hint

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* print message when restarting server after renewal

* log: print "advice" when exiting with an error

When running in non-quiet mode.

* try fix -oldest lock_test.py

* fix docstring

* s/Restarting/Reloading/ when notifying the user

* fix test name

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* type annotations

* s/using the {} plugin/installer: {}/

* copy: avoid "plugin" where possible

* link to user guide#automated-renewals

when not running with --preconfigured-renewal

* cli: reduce default logging verbosity

* fix lock_test: -vv is needed to see logger.debug

* Change comment in log.py to match the change to default verbosity

* Audit and adjust logging levels in apache module

* Audit and adjust logging levels in nginx module

* Audit, adjust logging levels, and improve logging calls in certbot module

* Fix tests to mock correct methods and classes

* typo in non-preconfigured-renewal message

Co-authored-by: ohemorange <ebportnoy@gmail.com>

* fix test

* revert acme version bump

* catch up to python3 changes

* Revert "revert acme version bump"

This reverts commit fa83d6a51c.

* Change ocsp check error to warning since it's non-fatal

* Update storage_test in parallel with last change

* get rid of leading newline on "Deploying [...]"

* shrink renewal and installation success messages

* print logfile rather than logdir in exit handler

* Decrease logging level to info for idempotent operation where enhancement is already set

* Display cert not yet due for renewal message when renewing and no other action will be taken, and change cert to certificate

* also write to logger so it goes in the log file

* Don't double write to log file; fix main test

* cli: remove trailing newline on new cert reporting

* ignore type error

* revert accidental changes to dependencies

* Pass tests in any timezone by using utcfromtimestamp

* Add changelog entry

* fix nits

* Improve wording of try again message

* minor wording change to changelog

* hooks: send hook stdout to CLI stdout

includes both --manual and --{pre,post,renew} hooks

* update docstrings and remove TODO

* add a pending deprecation on execute_command

* add test coverage for both

* update deprecation text

Co-authored-by: ohemorange <ebportnoy@gmail.com>

Co-authored-by: Alex Zorin <alex@zorin.id.au>
Co-authored-by: alexzorin <alex@zor.io>

tests/lock_test.py

@@ -133,7 +133,7 @@ def set_up_command(config_dir, logs_dir, work_dir, nginx_dir):
     return (
         'certbot --cert-path {0} --key-path {1} --config-dir {2} '
         '--logs-dir {3} --work-dir {4} --nginx-server-root {5} --debug '
-        '--force-renewal --nginx --verbose '.format(
+        '--force-renewal --nginx -vv '.format(
             test_util.vector_path('cert.pem'),
             test_util.vector_path('rsa512_key.pem'),
             config_dir, logs_dir, work_dir, nginx_dir).split())
@@ -213,7 +213,8 @@ def check_error(command, dir_path):
     if ret == 0:
         report_failure("Certbot didn't exit with a nonzero status!", out, err)
 
-    match = re.search("Please see the logfile '(.*)' for more details", err)
+    # this regex looks weird in order to deal with a text change between HEAD and -oldest
+    match = re.search("ee the logfile '?(.*?)'? ", err)
     if match is not None:
         # Get error output from more verbose logfile
         with open(match.group(1)) as f: