interfaces.GenericUpdater and new enhancement interface updater functions get run on every invocation of Certbot with "renew" verb for every lineage. This causes performance problems for users with large configurations, because of plugin plumbing and preparsing happening in prepare() method of installer plugins. This PR moves the responsibility to call prepare() to the plugin (possibly) implementing a new style enhancement interface.
Fixes: #6153
* Do not call IPlugin.prepare() for updaters when running renew
* Check prepare called in tests
* Refine pydoc and make the function name more informative
* Verify the plugin type
If either --cert-name or both --key-path and --cert-path (in which case the user requests installation for a certificate not managed by Certbot) are not provided, prompt the user with managed certificates and let them choose.
Fixes: #5824
This PR adds the functionality to enhance Apache configuration to include HTTP Strict Transport Security header with a low initial max-age value.
The max-age value will get increased on every (scheduled) run of certbot renew regardless of the certificate actually getting renewed, if the last increase took place longer than ten hours ago. The increase steps are visible in constants.AUTOHSTS_STEPS.
Upon the first actual renewal after reaching the maximum increase step, the max-age value will be made "permanent" and will get value of one year.
To achieve accurate VirtualHost discovery on subsequent runs, a comment with unique id string will be added to each enhanced VirtualHost.
* AutoHSTS code rebased on master
* Fixes to match the changes in master
* Make linter happy with metaclass registration
* Address small review comments
* Use new enhancement interfaces
* New style enhancement changes
* Do not allow --hsts and --auto-hsts simultaneuously
* MyPy annotation fixes and added test
* Change oldest requrements to point to local certbot core version
* Enable new style enhancements for run and install verbs
* Test refactor
* New test class for main.install tests
* Move a test to a correct test class
When Certbot is run with --dry-run, skip running GenericUpdater and RenewDeployer interface methods.
This PR also makes the parameter order of updater.run_generic_updaters and updater.run_renewal_deployer consistent.
Fixes#5927
* Do not call updaters and deployers when run with --dry-run
* Use ConfigTestCase instead of mocking config objects manually
In order to give more flexibility for plugins using interfaces.GenericUpdater interface, lineage needs to be passed to the updater method instead of individual domains. All of the (present and potential) installers do not work on per domain basis, while the lineage does contain a list of them for installers which do.
This also means that we don't unnecessarily run the updater method multiple times, potentially invoking expensive tooling up to $max_san_amount times.
* Make GenericUpdater use lineage as parameter and get invoked only once per lineage
* ServerTLSUpdater and InstallerSpecificUpdater implementation
* Fixed tests and added disables for linter :/
* Added error logging for misconfigurationerror from plugin check
* Remove redundant parameter from interfaces
* Renaming the interfaces
* Finalize interface renaming and move tests to own file
* Refactored the runners
* Refactor the cli params
* Fix the interface args
* Fixed documentation
* Documentation and naming fixes
* Remove ServerTLSConfigurationUpdater
* Remove unnecessary linter disable
* Rename run_renewal_updaters to run_generic_updaters
* Do not raise exception, but make log message more informative and visible for the user
* Run renewal deployer before installer restart
The base class for Installer plugins `certbot.plugins.common.Installer` now provides functionality of `PluginStorage` to all installer plugins. This allows a plugin to save and retrieve variables in between of invocations.
The on disk storage is basically a JSON file at `config_dir`/`.pluginstorage.json`, usually `/etc/letsencrypt/.pluginstorage.json`. The JSON structure is automatically namespaced using the internal plugin name as a namespace key. Because the actual storage is JSON, the supported data types are: dict, list, tuple, str, unicode, int, long, float, boolean and nonetype.
To add a variable from inside the plugin class:
`self.storage.put("my_variable_name", my_var)`
To fetch a variable from inside the plugin class:
`my_var = self.storage.fetch("my_variable_key")`
The storage state isn't written on disk automatically, but needs to be called:
`self.storage.save()`
* Plugin storage implementation
* Added config_dir to existing test mocks
* PluginStorage test cases
* Saner handling of bad config_dir paths
* Storage moved to Installer and not initialized on plugin __init__
* Finetuning and renaming
In `deploy_cert()` and `enhance()`, the user will be presented with a dialog to choose from the VirtualHosts that can be covered by the wildcard domain name. The (multiple) selection result will then be handled in a similar way that we previously handled a single VirtualHost that was returned by the `_find_best_vhost()`.
Additionally the selected VirtualHosts are added to a dictionary that maps selections to a wildcard domain to be reused in the later `enhance()` call and not forcing the user to select the same VirtualHosts again.
* Apache plugin wildcard support
* Present dialog only once per domain, added tests
* Raise exception if no VHosts selected for wildcard domain
* Fix install verb
* Fix error message, tests and remove global pylint change
* Fix boulder integration test keypath
* Also use chain_path from lineage if not defined on CLI
Unfortunately, the way that Apache merges the configuration directives is different for mod_rewrite and <Location> / <Directory> directives.
To work around basic auth in VirtualHosts, the challenge override Include had to be split in two. The first part handles overrides for RewriteRule and the other part will handle overrides for <Directory> and <Location> directives.
* Add OSX Python2 tests
* Make sure python2 is originating from homebrew on macOS
* Upgrade the already installed python2 instead of trying to reinstall
Make gitignore take pytest cache directories in to account, even if
they reside in subdirectories.
If pytest is run for a certain module, ie. `pytest certbot-apache` the
cache directory is created under `certbot-apache` directory.
Class inheritance based approach to distro specific overrides.
How it works:
The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available.
Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file.
Generic changes:
Parser initialization has been moved to separate class method, allowing easy override where needed.
Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py
Split Debian specific code from configurator.py to debian_override.py
Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands.
Moved add_parser_mod() from configurator to parser add_mod()
Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules().
Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out).
Moved OS based constants to their respective override classes.
Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class.
tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class.
This PR includes two major generic additions that should vastly improve our parsing accuracy and quality:
Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file.
Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such).
Distribution specific changes
Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are:
CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off.
Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values.
Debian
Moved the Debian specific parts from configurator.py to Debian specific override.
CentOS
Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump.
Added CentOS default Apache configuration tree for realistic test cases.
Gentoo
Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps.
Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases.
* Distribution specific override functionality based on class inheritance
* Need to patch get_systemd_os_like to as travis has proper os-release
* Added pydoc
* Move parser initialization to a method and fix Python 3 __new__ errors
* Parser changes to parse HTTPD config
* Try to get modules and includes from httpd process for better visibility over the configuration
* Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214)
* CentOS tests and linter fixes
* Gentoo override, tests and linter fixes
* Mock the process call in all the tests that require it
* Fix CentOS test mock
* Restore reseting modules list functionality for cleanup
* Move OS fingerprinting and constant mocks to parent class
* Fixes requested in review
* New entrypoint structure and started moving OS constants to override classes
* OS constants move continued, test and linter fixes
* Removed dead code
* Apache compatibility test changest to reflect OS constant restructure
* Test fix
* Requested changes
* Moved Debian specific tests to own test file
* Removed decorator based override class registration in favor of entrypoint dict
* Fix for update_includes for some versions of Augeas
* Take fedora fix into account in tests
* Review fixes
* Nginx IPv6 support
* Test and lint fixes
* IPv6 tests to Nginx plugin
* Make ipv6_info() port aware
* Named tuple values for readability
* Lint fix
* Requested changes
This changes the apache plugin behaviour to only parse enabled configuration files and respecting the --apache-vhost-root CLI parameter for new SSL vhost creation. If --apache-vhost-root isn't defined, or doesn't exist, the SSL vhost will be created to originating non-SSL vhost directory.
This PR also implements actual check for vhost enabled state, and makes sure parser.parse_file() does not discard changes in Augeas DOM, by doing an autosave.
Also handles enabling the new SSL vhost, if it's on a path that's not parsed by Apache.
Fixes: #1328Fixes: #3545Fixes: #3791Fixes: #4523Fixes: #4837Fixes: #4905
* First changes
* Handle rest of the errors
* Test fixes
* Final fixes
* Make parse_files accessible and fix linter problems
* Activate vhost at later time
* Cleanup
* Add a new test case, and fix old
* Enable site later in deploy_cert
* Make apache-conf-test default dummy configuration enabled
* Remove is_sites_available as obsolete
* Cleanup
* Brought back conditional vhost_path parsing
* Parenthesis
* Fix merge leftovers
* Fix to work with the recent changes to new file creation
* Added fix and tests for non-symlink vhost in sites-enabled
* Made vhostroot parameter for ApacheParser optional, and removed extra_path
* Respect vhost-root, and add Include statements to root configuration if needed
* Fixed site enabling order to prevent apache restart error while enabling mod_ssl
* Don't exclude Ubuntu / Debian vhost-root cli argument
* Changed the SSL vhost directory selection priority
* Requested fixes for paths and vhost discovery
* Make sure the Augeas DOM is written to disk before loading new files
* Actual checking for if the file is parsed within existing Apache configuration
* Fix the order of dummy SSL directives addition and enabling modules
* Restructured site_enabled checks
* Enabling vhost correctly for non-debian systems
These changes allow developers to run tests directly from the host machine using Docker, and to enable ipdb inside the container.
docker-compose.yml is upgraded to version 2 format. This means that you need docker-engine version >= 1.10.0 instead of previous requirement of version >= 1.9.1. The reason for this is to be able to use custom Dockerfile (Dockerfile-dev in this case) in build context.
ipdb has been added to dev dependencies to be able to be able to debug the code without installing it on every docker run. This is also what we recommend for debugging in the developer documentation, so there really is no reason not to install it with the dev dependencies.
setuptools is being upgraded to a newer version to be able to run coverage tests. This was using the older version of setuptools for some reason, and without the upgrade, coverage tests would fail horribly. Upgrading remedies the situation.
Few examples:
Run unit tests for certbot-apache
`docker-compose run --rm --service-ports development bash -c 'cd src;nosetests -v certbot-apache'`
Run coverage tests
`docker-compose run --rm --service-ports development bash -c 'cd src;./tox.cover.sh'`
Run linter
`docker-compose run --rm --service-ports development bash -c 'cd src;tox -e lint'`
* Script plugin initial commit
* Fix auth script path
* Return correct responses
* Added DNS-01 support
* Report the challenge pref correctly
* Use config root from certbot constants rather than hardcoded
* Remove prehook and rename posthook to cleanup for clarity
* Refactoring
* Docs
* Refactoring
* Refactoring continued, working now
* Use global preferred-challenges argument in favor of local
* Added http-01 as fallback challenge if not defined
* Do not continue if auth script not defined
* Skip unnecessary steps when running
* Read config values from correct places
* Tests and minor fixes
* Make Python 2.6 happy again
* Added CERTBOT_AUTH_OUTPUT and better tests
* Lint & Py3 fixes
* Make Python 2.6 happy again
* Doc changes
* Refactor hook execute and reuse in script plugin
* Refactored hook validation
* Added long_description for plugin help text
* Refactored env var writing
