* Fix galaxy publish sha256 value format.
The multipart/form content used for the body
of the POST to /api/automation-hub/v3/collections
was missing a newline before the line with the value
of the sha256.
automation-hub/galaxy/django skips the field entirely in
that case and automation-hub code will use None for default
to indicate that no sha256 is provided (an available option).
Fixesansible/galaxy-dev#246
* Add changelog fragment
Co-authored-by: Matt Martz <matt@sivel.net>
This causes multiple unneeded http redirects.
Since each http redirect takes a few hundred
milliseconds, handling the redirects starts to
take up a significant amount of wall time.
Fixes: #63281
In addition to trying the configured url (for ex, a migrated
'https://galaxy.ansible.com/') there is an attempt at that
URL with '/api' postpended.
If the extended URL works, update GalaxyAPI.api_server to
use the extended URL. Previously it only used it for finding
the API root info ('available_versions', etc)
* Stop appending '/api' to configured galaxy server url
Since not all galaxy REST api server URLs live
at '/api', stop always appending it to the
'url' value loaded from config.
* Add note about manually migrated galaxy configs and /api
* Add '/api/' to galaxy url and guessing if galaxy API
* Fix most unit tests (update to expect /api/)
* Fix test_initialise_unknown unit test
Since we retry now with an added /api/, mock it as well.
* Update fallback default avail_ver to new format
* Add galaxy collections API v3 support
Issue: ansible/galaxy-dev#60
- Determine if server supports v3
Use 'available_versions' from `GET /api`
to determine if 'v3' api is available on
the server.
- Support v3 pagination style
ie, 'limit/offset style', with the paginated
responses based on https://jsonapi.org/format/#fetching-pagination
v2 galaxy uses pagination that is more or less
'django rest framework style' or 'page/page_size style',
based on the default drf pagination described
at https://www.django-rest-framework.org/api-guide/pagination/#pagenumberpagination
- Support galaxy v3 style error response
The error objects returned by the galaxy v3 api are based
on the JSONAPI response/errors format
(https://jsonapi.org/format/#errors).
This handles that style response. At least for publish_collection
for now. Needs extracting/generalizing.
Handle HTTPError in CollectionRequirement.from_name()
with _handle_http_error(). It will raise AnsibleError
based on the json in an error response.
- Update unit tests
update test/unit/galaxy/test_collection*
to paramaterize calls to test against
mocked v2 and v3 servers apis.
Update artifacts_versions_json() to tale an
api version paramater.
Add error_json() for generating v3/v3 style error
responses.
So now, the urls generated and the pagination schema
of the response will use the v3 version if
the passed in GalaxyAPI 'galaxy_api' instance
has 'v3' in it's available_api_versions
* Move checking of server avail versions to collections.py
collections.py needs to know the server api versions
supported before it makes collection related calls,
so the 'lazy' server version check in api.GalaxyAPI
is never called and isn't set, so 'v3' servers weren't
found.
Update unit tests to mock the return value of the
request instead of GalaxyAPI itself.
* Port bundled distro to use optparse instead of argparse (py2.6)
* Use an absolute import to satisfy the current import testing harness
* Port from subprocess.check_output to subprocess.Popen.communicate() (py2.6)
* Add license location
The changes have been proposed upstream here:
https://github.com/nir0s/distro/pull/232
Upstream is contemplating a branch where everyone wanting python-2.6
support can collaborate without it becoming part of the regularly
supported releases.
Since the 'platform.dist()' and 'platform.linux_distribution()'
methods will be removed from future versions of python, this
provides an alternative to replace ansibles use of those
methods.
lib/ansible/module_utils/distro.py is a copy of
https://github.com/nir0s/distro/blob/master/distro.py
This module is originally from https://github.com/nir0s/distro
and is license under the Apache License, Version 2.0.
* Fix errors decrypted non-ascii vault vars
AnsibleVaultEncryptedUnicode was just using b"".decode()
instead of to_text() on the bytestrings returned from
vault.decrypt() and could cause errors on python2
if non-ascii since decode() defaults to ascii.
Use to_text() to default to decoding utf-8.
add intg and unit tests for value of vaulted vars
being non-ascii utf8
based on https://github.com/ansible/ansible/issues/37258Fixes#37258
* yamllint fixups
* Fix 'New Vault password' on vault 'edit'
ffe0ddea96 introduce a
change on 'ansible-vault edit' that tried to check
for --encrypt-vault-id in that mode. But '--encrypt-vault-id'
is not intended for 'edit' since the 'edit' should always
reuse the vault secret that was used to decrypt the text.
Change cli to not check for --encrypt-vault-id on 'edit'.
VaultLib.decrypt_and_get_vault_id() was change to return
the vault secret used to decrypt (in addition to vault_id
and the plaintext).
VaultEditor.edit_file() will now use 'vault_secret_used'
as returned from decrypt_and_get_vault_id() so that
an edited file always gets reencrypted with the same
secret, regardless of any vault id configuration or
cli options.
Fixes#35834
* Fix redundant yaml error blurbs on ModArgs parse errors
Some of the AnsibleParserErrors from parsing.mod_args
are created with the obj=some_yaml_ds options but
some are not.
If they were, we don't want to add another yaml_ds to
it, because that will result in double yaml error blurbs.
And since we dont need to add info, we can just re raise it.
But if there is no ._obj, add it here so we get the extra
detail in the error message (see issue #14790) and raise
a new AnsibleParserError instance.
Fixes#36848
* cleanup existing test_tasks pep8/sanity issues
ansible-console was calling setup_vault_secrets
twice. Once directly and once via _play_prereqs()
The direct invocation was not setting auto_prompt=False.
However, the direct invocation isn't need at all so
this removes it so only _play_reqs() is used.
That fixes the unrequested vault password
prompting.
Fixes#33027
This fixes an indention bug introduced in
45a9f96774
that was causing the virtualization_role to be set to 'host'
for many cases incorrectly.
Fixes https://github.com/ansible/ansible/pull/21621
(the issue mentioned in the pr)
Add deps/requires for fact collectors
Fact collectors can now set a required_facts
class attribute that will be a set of the names
of fact collectors they require to be run first.
ie, if a collector needs to know the ansible_distribution,
it should set it's required_facts to include 'distribution'
required_facts = set(['distribution'])
If a collector requires another collector, it gets added
to the selected collector names.
We then topological sort the ordering of the collectors
so that deps work out (ie, 'distribution' will run before
'service_mgr')
required_facts were added to the collectors for:
- network (requires 'distribution', 'platform')
- hardware (requires 'platform')
- service_mgr (requires 'distribution', 'platform')
Fix name references for facts (need 'ansible_' prefix)
is service_mgr
Fixes#30753
Enforce that there can be only one --new-vault-id or
--new-vault-password-file and use this instead of
--encrypt-vault-id
* Add a config option for default vault encrypt id
The accumulated collected_facts was being update
with new facts _after_ filtering them. So only
facts that pass the filter would ever be passed
to other fact collectors.
For 'filter=ansible_service_mgr', even though it requires
the platform and distribution facts and even collects them,
they would get filtered out and never passed to the other
collectors that need them (service_mgr for ex).
Fix is just to add the unfiltered facts to collected_facts.
Adds unit tests for fact filter and collected_facts.
Fixes#32286
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.
Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'
Also add a meta fact indicating which search string
was matched.
Test case info from:
https://github.com/ansible/ansible/issues/31501#issuecomment-340861535Fixes#31501
Extract vault related bits of DataLoader._get_file_contents to DataLoader._decrypt_if_vault_data
When loading vault password files, detect if they are vault encrypted, and if so, try to decrypt with any already known vault secrets.
This implements the 'Allow vault password files to be vault encrypted' (#31002) feature card from
the 2.5.0 project at https://github.com/ansible/ansible/projects/9Fixes#31002
* Fix _clean_result for debug callback with 1 char var names
The check in _clean_results was removing any keys that happened
to be one of the chars in ('invocation') instead of the string
'invocation'. This was meant to be a tuple but there was no comma
so the for iterated the string instead of the tuple.
Introduced in 9dba580204
Update unit test to catch this.
Fixes#33723
* Use .pop() to remove invocation from results dict
In base callback _clean_results, simplify the way the
'invocation' item is removed.
Add some more unit tests.
* Fix vault --ask-vault-pass with no tty
2.4.0 added a check for isatty() that would skip setting up interactive
vault password prompts if not running on a tty.
But... getpass.getpass() will fallback to reading from stdin if
it gets that far without a tty. Since 2.4.0 skipped the interactive
prompts / getpass.getpass() in that case, it would never get a chance
to fall back to stdin.
So if 'echo $VAULT_PASSWORD| ansible-playbook --ask-vault-pass site.yml'
was ran without a tty (ie, from a jenkins job or via the vagrant
ansible provisioner) the 2.4 behavior was different than 2.3. 2.4
would never read the password from stdin, resulting in a vault password
error like:
ERROR! Attempting to decrypt but no vault secrets found
Fix is just to always call the interactive password prompts based
on getpass.getpass() on --ask-vault-pass or --vault-id @prompt and
let getpass sort it out.
* up test_prompt_no_tty to expect prompt with no tty
We do call the PromptSecret class if there is no tty, but
we are back to expecting it to read from stdin in that case.
* Fix logic for when to auto-prompt vault pass
If --ask-vault-pass is used, then pretty much always
prompt.
If it is not used, then prompt if there are no other
vault ids provided and 'auto_prompt==True'.
Fixes vagrant bug https://github.com/hashicorp/vagrant/issues/9033Fixes#30993
* Better handling of malformed vault data envelope
If an embedded vaulted variable ('!vault' in yaml)
had an invalid format, it would eventually cause
an error for seemingly unrelated reasons.
"Invalid" meaning not valid hexlify (extra chars,
non-hex chars, etc).
For ex, if a host_vars file had invalid vault format
variables, on py2, it would cause an error like:
'ansible.vars.hostvars.HostVars object' has no
attribute u'broken.example.com'
Depending on where the invalid vault is, it could
also cause "VARIABLE IS NOT DEFINED!". The behavior
can also change if ansible-playbook is py2 or py3.
Root cause is errors from binascii.unhexlify() not
being handled consistently.
Fix is to add a AnsibleVaultFormatError exception and
raise it on any unhexlify() errors and to handle it
properly elsewhere.
Add a _unhexlify() that try/excepts around a binascii.unhexlify()
and raises an AnsibleVaultFormatError on invalid vault data.
This is so the same exception type is always raised for this
case. Previous it was different between py2 and py3.
binascii.unhexlify() raises a binascii.Error if the hexlified
blobs in a vault data blob are invalid.
On py2, binascii.Error is a subclass of Exception.
On py3, binascii.Error is a subclass of TypeError
When decrypting content of vault encrypted variables,
if a binascii.Error is raised it propagates up to
playbook.base.Base.post_validate(). post_validate()
handles exceptions for TypeErrors but not for
base Exception subclasses (like py2 binascii.Error).
* Add a display.warning on vault format errors
* Unit tests for _unhexlify, parse_vaulttext*
* Add intg test cases for invalid vault formats
Fixes#28038
The platform/distro/etc facts were being passed in
correctly, but service_mgr.py was looking up the
wrong names ('system' vs 'ansible_system') resulting
in service_mgr falling back to default 'service' result.
Fixes#30753, #31095
This adds a new type of vault-password script (a 'client') that takes advantage of and enhances the
multiple vault password support.
If a vault password script basename ends with the name '-client', consider it a vault password script client.
A vault password script 'client' just means that the script will take a '--vault-id' command line arg.
The previous vault password script (as invoked by --vault-password-file pointing to an executable) takes
no args and returns the password on stdout. But it doesnt know anything about --vault-id or multiple vault
passwords.
The new 'protocol' of the vault password script takes a cli arg ('--vault-id') so that it can lookup that specific
vault-id and return it's password.
Since existing vault password scripts don't know the new 'protocol', a way to distinguish password scripts
that do understand the protocol was needed. The convention now is to consider password scripts that are
named like 'something-client.py' (and executable) to be vault password client scripts.
The new client scripts get invoked with the '--vault-id' they were requested for. An example:
ansible-playbook --vault-id my_vault_id@contrib/vault/vault-keyring-client.py some_playbook.yml
That will cause the 'contrib/vault/vault-keyring-client.py' script to be invoked as:
contrib/vault/vault-keyring-client.py --vault-id my_vault_id
The previous vault-keyring.py password script was extended to become vault-keyring-client.py. It uses
the python 'keyring' module to request secrets from various backends. The plain 'vault-keyring.py' script
would determine which key id and keyring name to use based on values that had to be set in ansible.cfg.
So it was also limited to one keyring name.
The new vault-keyring-client.py will request the secret for the vault id provided via the '--vault-id' option.
The script can be used without config and can be used for multiple keyring ids (and keyrings).
On success, a vault password client script will print the password to stdout and exit with a return code of 0.
If the 'client' script can't find a secret for the --vault-id, the script will exit with return code of 2 and print an error to stderr.
The /etc/os-release based distro detection doesn't
seem to work for Ubuntu 10.04 (no /etc/os-release?).
So it was testing the next case which was /etc/lsb-release to
see if it is 'Mandriva'. Since the check for existence of
(/etc/lsb-release, Mandrive) was the first non-empty dist
file match, 'ansible_distribution' was being set to 'Mandriva'
expecting to be corrected by the data from the dist file content.
But since the dist file parsing for Mandriva didn't match for
Ubuntu 10.04 /etc/lsb-release _and_ there is no Debian specific
lsb-release check, 'ansible_distribution' stayed at 'Mandriva'
and the dist file checking loop keeps going and eventually off
the end of the list before finding a better match.
Adding a debian/ubuntu specific check for /etc/lsb-release after
the debian os-release sets the info correctly and stops further
checking of dist files.
Fixes#30693
'distribution' facts were being set after checking
the existence of the dist file, and then being set
again with more detail after they were succesfully parsed.
But if the dist file was not succesfully parsed and
matched the required names, the loop continues
without resetting the earlier set facts. This is
how 'Mandriva' would end up being the 'distribution'
file for unrelated cases (it would find /etc/lsb-release,
set distro to 'Mandriva', then fail to parse/match and
continue the loop. If no other checks worked, 'Mandriva'
would stick).
* parse_dist_file_NA should check 'name' not distro for NA
parse_distribution_file_NA was checking the incoming
'distribution' fact to be 'NA', but the fact itself can
be specific at that point ('KDE Neon', for ex) but the
check is really if the 'name' it was passed is NA.
* for matches on OS_RELEASE_ALIAS (ie, 'Archlinux') do
not continue if the dist file content doesn't match. Previously
it had to because of the 'Mandriva' bug mentioned above.
This is a more general fix for #30693 than #30723
Fixes #30693
Related to #30600
In cli.CLI.unfrack_path callback, special case if the
value of '--output' is '-', and avoid expanding
it to a full path.
vault cli already has special cases for '-', so it
just needs to get the original value to work.
Fixes#30550
* Fix fact failures cause by ordering of collectors
Some fact collectors need info collected by other facts.
(for ex, service_mgr needs to know 'ansible_system').
This info is passed to the Collector.collect method via
the 'collected_facts' info.
But, the order the fact collectors were running in is
not a set order, so collectors like service_mgr could
run before the PlatformFactCollect ('ansible_system', etc),
so the 'ansible_system' fact would not exist yet.
Depending on the collector and the deps, this can result
in incorrect behavior and wrong or missing facts.
To make the ordering of the collectors more consistent
and predictable, the code that builds that list is now
driven by the order of collectors in default_collectors.py,
and the rest of the code tries to preserve it.
* Flip the loops when building collector names
iterate over the ordered default_collectors list
selecting them for the final list in order instead
of driving it from the unordered collector_names set.
This lets the list returned by select_collector_classes
to stay in the same order as default_collectors.collectors
For collectors that have implicit deps on other fact collectors,
the default collectors can be ordered to include those early.
* default_collectors.py now uses a handful of sub lists of
collectors that can be ordered in default_collectors.collectors.
fixes#30753fixes#30623
* Use vault_id when encrypted via vault-edit
On the encryption stage of
'ansible-vault edit --vault-id=someid@passfile somefile',
the vault id was not being passed to encrypt() so the files were
always saved with the default vault id in the 1.1 version format.
When trying to edit that file a second time, also with a --vault-id,
the file would be decrypted with the secret associated with the
provided vault-id, but since the encrypted file had no vault id
in the envelope there would be no match for 'default' secrets.
(Only the --vault-id was included in the potential matches, so
the vault id actually used to decrypt was not).
If that list was empty, there would be an IndexError when trying
to encrypted the changed file. This would result in the displayed
error:
ERROR! Unexpected Exception, this is probably a bug: list index out of range
Fix is two parts:
1) use the vault id when encrypting from edit
2) when matching the secret to use for encrypting after edit,
include the vault id that was used for decryption and not just
the vault id (or lack of vault id) from the envelope.
add unit tests for #30575 and intg tests for 'ansible-vault edit'
Fixes#30575
* Fix 'distribution' fact for ArchLinux
Allow empty wasn't breaking out of the process_dist_files
loop, so a empty /etc/arch-release would continue searching
and eventually try /etc/os-release. The os-release parsing
works, but the distro name there is 'Arch Linux' which does
not match the 2.3 behavior of 'Archlinux'
Add a OS_RELEASE_ALIAS map for the cases where we need to get
the distro name from os-release but use an alias.
We can't include 'Archlinux' in SEARCH_STRING because a name match on its keys
but without a match on the content causes a fallback to using the first
whitespace seperated item from the file content as the name.
For os-release, that is in form 'NAME=Arch Linux'
With os-release returning the right name, this also supports the
case where there is no /etc/arch-release, but there is a /etc/os-release
Fixes#30600
* pep8 and comment cleanup
* Fix pkg_mgr fact on OpenBSD
Add a OpenBSDPkgMgrFactCollector that hardcodes pkg_mgr
to 'openbsd_pkg'. The ansible collector will choose the
OpenBSD collector if the system is OpenBSD and the 'Generic'
one otherwise.
This removes PkgMgrFactCollectors depenency on the
'system' fact being in collected_facts, which also
avoids ordering issues (if the pkg mgr fact is collected
before the system fact...)
Fixes#30623
This is to catch vault secrets from config and
cli. Previously vault_password_file in config was
missed since it was added by setup_vault_secrets,
so check after setup_vault_secrets.
* Don't ask for password confirm on 'ansible-vault edit'
This is to match the 2.3 behavior on:
ansible-vault edit encrypted_file.yml
Previously, the above command would consider that a 'new password'
scenario and prompt accordingly, ie:
$ ansible-vault edit encrypted_file.yml
New Password:
Confirm New Password:
The bug was cause by 'create_new_password' being used for
'edit' action. This also causes the previous implicit 'auto prompt'
to get triggered and prompt the user.
Fix is to make auto prompt explicit in the calling code to handle
the 'edit' case where we want to auto prompt but we do not want
to request a password confirm.
Fixes#30491
Generate rst docs for plugins
Based on rst generated for modules. But generated plugin
docs go into docs/docsite/rst/plugins/$PLUGIN_TYPE/plugin_name.rst
( docs/docsite/rst/plugins/connection/ssh.py for ex)
* move plugins docs to rst/*_plugins/ subdirs for namespace
* Only gen support pages for modules for now.
* Add generated plugin docs to gitignore* add list_*_plugins templates
* support MODULES/PLUGINS filters for make htmldocs
Add a 'PLUGINS=ssh' filter env var like MODULES to filter plugins to build docs for.
* fixup 'historical' version_added, skip plugins/loader.py
* Fix plugins_by_support ref link to new plugins/*/ location
* use :ref: for common_return_values, allow empty version_added
* warnings on missing doc info
* add a prefix to _random_choice
It was colliding with the target for random_choice plugin
* Fix refs for local_facts and various cli :option:
* Fix dev_guide/testing_pep8 refs
* remove ref to non-existing 'developing_test_pr'
* Fix ref to ansible-vault encrypt_string
* Removed hard-to-localize colloquialism.
* Rename '_ansible-pull' in playbooks_intro.
It was conflicting with rst/ansible-pull.rst. Nothing
seems to reference it.
* Add explicit targets for and update refs
Replace some ':doc:' use with ':ref:'.
Replace some :ref: to section names with explicit targets
(:doc:`Dynamic vs. Static` -> :ref:`dynamic_vs_static` etc)
* The 'YAML+Jinja' syntax lex fails here, so just use yaml
Since the yaml+jinja highlight fails, code wasnt highlighted
at all, but 'yaml' works more or less.
* just use no lexer for the < python2.6 examples
py3 will fail highlighting them, and 'python2' throws
a lexer warning, and nothing actually highlights it, so
just disable.
* Use a rst glossary for playbooks_keywords docs
* Add a 'Task' and 'Tasks' to glossary.
* Update keywords desciptions,
* use :term: rst ref, some quoting
* Make it more obvious that 'retries' and 'until' need to be used in combination.
* let generate_man also gen rst pages for cli tools
* make template-file, output-dir, output format cli options for generate_man
* update main Makefile to use generate_man.py for docs (man pages and rst)
* update vault docs that use :option:
* Edits based on
https://github.com/alikins/ansible/commit/6e34ea62429c417939bd96b6de5bf7e6ab1ff765 and
https://github.com/alikins/ansible/commit/a3afc785357878da354e21a709cf3b9f16c3f146
* add a optparse 'desc' to lib/ansible/cli/config.py
The man page needs a short desc for the 'NAME' field
which it gets from the option parse 'desc' value.
Fixes building ansible-config man page.
* add trim_docstring from pep257 to generate_man
use pep258 docstring trim function to fix up any indention
weirdness inherit to doc strings (ie, lines other than
first line being indented.
* Add refs to cli command actions
To reference ansible-vaults --vault-id option, use:
:option:`The link text here <ansible-vault --vault-id>`
or:
:option:`--vault-id <ansible-vault --vault-id>`
To reference ansible-vault's 'encrypt' action, use:
:ref:`The link text here <ansible_vault_encrypt>`
or most of the time:
:ref:`ansible-vault encrypt <ansible_vault_encrypt>`
* start rst docs for ansible-vault encrypt_string
* wip, vault format docs
(sorta markdown ish atm)
* wip - formatting
* wip, vault docs
* Fix refs to other docs for now
fixing default_role in conf.py will remove need
for this
* add 'ref' to cli names
* more vault docs
* wip, misc fixes
* add some encrypt_string examples
* Fix up rstcheck warnings
The code blocks in question included the output that would
be echo'ed from running the command, which isnt valid bash.
* fix formatting and rstcheck warnings about code-block
* Add envvar ref for ANSIBLE_VAULT_PASSWORD_FILE
* fix doc title
* Fixed title underline
* wip, gen docs from config/base.yml
* wip
* dont change conf.py here
* cleanup, add dump_config --template-file cli opt
* some desc are string, some are lists...
TODO: fix base.yml so it is consistent
* Filter out TODO and empty descriptions
* Handle errors in jmespath in json_query better
Catch any exceptions raised from jmespath and raise
an AnsibleFilterError instead.
Avoid a traceback.
Fixes#20379
* pep8
* prompt for new pass on create/encrypt if none specified
Make 'ansible-vault' edit or encrypt prompt for a password
if none or provided elsewhere.
Note: ansible-playbook does not prompt if not vault password
is provided
* dont show vault password prompts if not a tty
* Add config option for a default list of vault-ids
This is the vault-id equilivent of ANSIBLE_DEFAULT_PASSWORD_FILE
except ANSIBLE_DEFAULT_VAULT_IDENTITY_LIST is a list.
* Better handling of empty/invalid passwords
empty password files are global error and cause an
exit. A warning is also emitted with more detail.
ie, if any of the password/secret sources provide
a bogus password (ie, empty) or fail (exception,
ctrl-d, EOFError), we stop at the first error and exit.
This makes behavior when entering empty password at
prompt match 2.3 (ie, an error)
The prompt_formats dict didn't get the 'prompt_ask_vault_pass'
item added for interactive --ask-vault-pass, which
caused "KeyError: u'prompt_ask_vault_pass'"
Fixes#27885
* Fix module failure with pacemaker_cluster: state=cleanup
If state=cleanup was used, set_cluster() was being called
with 'cleanup' state which it doesn't handle. Instead
use existing clean_cluster() method.
Fixes#27799
* get and return cluster_state on state=clean as well
* rm unneeded parens following assert
* rm unused parse_vaulttext_envelope from yaml.constructor
* No longer need index/enumerate over vault_ids
* rm unnecessary else
* rm unused VaultCli.secrets
* rm unused vault_id arg on VaultAES.decrypt()
pylint: Unused argument 'vault_id'
pylint: Unused parse_vaulttext_envelope imported from ansible.parsing.vault
pylint: Unused variable 'index'
pylint: Unnecessary parens after 'assert' keyword
pylint: Unnecessary "else" after "return" (no-else-return)
pylint: Attribute 'editor' defined outside __init__
* use 'dummy' for unused variables instead of _
Based on pylint unused variable warnings.
Existing code use '_' for this, but that is old
and busted. The hot new thing is 'dummy'. It
is so fetch.
Except for where we get warnings for reusing
the 'dummy' var name inside of a list comprehension.
* Add super().__init__ call to PromptVaultSecret.__init__
pylint: __init__ method from base class 'VaultSecret' is not called (super-init-not-called)
* Make FileVaultSecret.read_file reg method again
The base class read_file() doesnt need self but
the sub classes do.
Rm now unneeded loader arg to read_file()
* Fix err msg string literal that had no effect
pylint: String statement has no effect
The indent on the continuation of the msg_format was wrong
so the second half was dropped.
There was also no need to join() filename (copy/paste from
original with a command list I assume...)
* Use local cipher_name in VaultEditor.edit_file not instance
pylint: Unused variable 'cipher_name'
pylint: Unused variable 'b_ciphertext'
Use the local cipher_name returned from parse_vaulttext_envelope()
instead of the instance self.cipher_name var.
Since there is only one valid cipher_name either way, it was
equilivent, but it will not be with more valid cipher_names
* Rm unused b_salt arg on VaultAES256._encrypt*
pylint: Unused argument 'b_salt'
Previously the methods computed the keys and iv themselves
so needed to be passed in the salt, but now the key/iv
are built before and passed in so b_salt arg is not used
anymore.
* rm redundant import of call from subprocess
pylint: Imports from package subprocess are not grouped
use via subprocess module now instead of direct
import.
* self._bytes is set in super init now, rm dup
* Make FileVaultSecret.read_file() -> _read_file()
_read_file() is details of the implementation of
load(), so now 'private'.
* Add 'cacheable' param to set_fact action and module.
Used just like set_fact, except facts set with cacheable: true
will be stored in the fact cache if fact caching is enabled.
set_fact normally only sets facts in the non_persistent_fact_cache, so they
are lost between invocations.
* update set_facts docs
* use 'ansible_facts_cacheable' in module/actions result
* pop fact cacheable related items out of args/results
We dont want to use 'ansible_facts_cacheable' result item
or 'cacheable' arg as actual facts, so pop them out of the
dicts.
previously gather_subset=['!all'] would still gather the
min set of facts, and there was no way to collect no facts.
The 'min' specifier in gather_subset is equilivent to
exclude the minimal_gather_subset facts as well.
gather_subset=['!all', '!min'] will collect no facts
This also lets explicitly added gather_subsets override excludes.
gather_subset=['pkg_mgr', '!all', '!min'] will collect only the pkg_mgr
fact.
When parsing a vaulttext blob, use .splitlines()
instead of split(b'\n') to handle \n newlines and
windows style \r\n (CRLF) new lines.
The vaulttext enevelope at this point is just the header line
and a hexlify()'ed blob, so CRLF is a valid newline here.
Fixes#22914
Got removed in arg parsing updates. Now added back in
setup_vault_secrets().
The default value for DEFAULT_VAULT_PASSWORD_FILE was also
set to '~' for some reason, change to to no default.
Add integration tests.
If we don't use more than one vault-id, and we use
--ask-vault-pass, instead of using the new vault prompt
format ('Vault password (my_vault_id): ') we use the old
one ('Vault password: ').
This avoids confusing Tower when it needs to detect an
interactive vault password prompt.
This also potentially could allow vault password prompts
to be customized per vault_id.
* Add 2.0-2.3 facts api compat (ansible_facts(), get_all_facts())
These are intended to provide compatibilty for modules that
use 'ansible.module_utils.facts.ansible_facts' and
'ansible.module_utils.facts.get_all_facts' from 2.0-2.3 facts
API.
Fixes#25686
Some related changes/fixes needed to provide the compat api:
* rm ansible.constants import from module_utils.facts.compat
Just use a hard coded default for gather_subset/gather_timeout
instead of trying to load it from non existent config if the
module params dont include it.
* include 'external' collectors in compat ansible_facts()
* Add facter/ohai back to the valid collector classes
facter/ohai had gotten removed from the default_collectors
class used as the default list for all_collector_classes by
setup.py and compat.py
That made gather_subset['facter'] fail.
Fixes#13243
** Add --vault-id to name/identify multiple vault passwords
Use --vault-id to indicate id and path/type
--vault-id=prompt # prompt for default vault id password
--vault-id=myorg@prompt # prompt for a vault_id named 'myorg'
--vault-id=a_password_file # load ./a_password_file for default id
--vault-id=myorg@a_password_file # load file for 'myorg' vault id
vault_id's are created implicitly for existing --vault-password-file
and --ask-vault-pass options.
Vault ids are just for UX purposes and bookkeeping. Only the vault
payload and the password bytestring is needed to decrypt a
vault blob.
Replace passing password around everywhere with
a VaultSecrets object.
If we specify a vault_id, mention that in password prompts
Specifying multiple -vault-password-files will
now try each until one works
** Rev vault format in a backwards compatible way
The 1.2 vault format adds the vault_id to the header line
of the vault text. This is backwards compatible with older
versions of ansible. Old versions will just ignore it and
treat it as the default (and only) vault id.
Note: only 2.4+ supports multiple vault passwords, so while
earlier ansible versions can read the vault-1.2 format, it
does not make them magically support multiple vault passwords.
use 1.1 format for 'default' vault_id
Vaulted items that need to include a vault_id will be
written in 1.2 format.
If we set a new DEFAULT_VAULT_IDENTITY, then the default will
use version 1.2
vault will only use a vault_id if one is specified. So if none
is specified and C.DEFAULT_VAULT_IDENTITY is 'default'
we use the old format.
** Changes/refactors needed to implement multiple vault passwords
raise exceptions on decrypt fail, check vault id early
split out parsing the vault plaintext envelope (with the
sha/original plaintext) to _split_plaintext_envelope()
some cli fixups for specifying multiple paths in
the unfrack_paths optparse callback
fix py3 dict.keys() 'dict_keys object is not indexable' error
pluralize cli.options.vault_password_file -> vault_password_files
pluralize cli.options.new_vault_password_file -> new_vault_password_files
pluralize cli.options.vault_id -> cli.options.vault_ids
** Add a config option (vault_id_match) to force vault id matching.
With 'vault_id_match=True' and an ansible
vault that provides a vault_id, then decryption will require
that a matching vault_id is required. (via
--vault-id=my_vault_id@password_file, for ex).
In other words, if the config option is true, then only
the vault secrets with matching vault ids are candidates for
decrypting a vault. If option is false (the default), then
all of the provided vault secrets will be selected.
If a user doesn't want all vault secrets to be tried to
decrypt any vault content, they can enable this option.
Note: The vault id used for the match is not encrypted or
cryptographically signed. It is just a label/id/nickname used
for referencing a specific vault secret.
Based on issue 23642, add some info about the used python
executable and version to the error message when ssl connection
fail in a way that may be related to the version.
Module was importing '*' from facts to get to TimeoutError
but that has moved to facts.timeout, so import is updated.
Also rm old style imports to new style imports at the start
of the module.
'signal' py module was used and referenced but never imported,
presumably it was using the 'signal' previously imported into
module_utils.facts. Now imported directly.
'AnsibleModule' was also from a * import, so now imported directly.
A ref to 'module' was in _delete_disks_when_detached(), so now it
is updated to raise an AzureException() with its message, and
let its caller catch it and call module.fail_json()
* Rm check for unused 'name' arg for restarted.
The module docs claim 'name' is not required for
restarted state, and the code doesn't seem to use
it is.
* Better error msg for linode 'restarted' state.
* Remove the eval() and loop over args.
* Fix use of eval(args), and cleanups.
* linode 'stopped' state doesnt need name either
Fixes#3873
It was in lib/ansible/modules/system/setup.py since it
was the only thing using it, but move it back to module_utils
and add a ansible_collector.get_ansible_collector() to build
a facts collector just like the one used by setup.py
mv test_setup.py -> test_ansible_collector.py
All the code it was testing is now in ansible_collector
rm code to create 'ansible_facts' subkey from namespace
Just leave it up to the caller to do, and just return a
flat dictionary from AnsibleFactCollector.collect()
At present, the available facts around block devices are not sufficient to be able to find stable names guaranteed to work across reboots, or to identify block devices by label (UUID, etc).
This patch provides a list of observed links for each device. It relies on functionality specific to Linux (as does the existing sysfs-based code which it extends), but should not cause issues on other platforms.
Moreover, it prevents virtual devices from being excluded, and links such devices to the physical devices to which they are attached.
* Add more mount point statvfs info including sizes
Based on https://github.com/ansible/ansible/pull/12073
facts.utils.get_mount_size() now returns a dict of most
of the posix statvfs data, including block_size and inode
counts.
Update the facts.hardware classes that use get_mount_size() to
use the new info by mount_info.update(mount_statvfs_inof) to merge.
* add back unit tests for LinuxHardware mount/fs facts
* add test cases for facts.utils.get_mount_size
* show original exception for yaml (and other) errors
In places where we need to catch a yaml error and raise
an AnsibleError, add the orig yaml exc to the AnsibleError
via the orig_exc arg.
When the AnsibleError is displayed it will now include the
AnsibleError (AnsibleParserError for example) and the type
and message from the original yaml exception.
This provides more detail to the error messages related to
yaml errors.
This also improves errors from dataloader (for example,
previously if a wrong password was used for a vault encrypted
yaml file, the error was very vague and suggested yaml errors,
but now the message includes the original exception from vault
indicating the password was incorrect or missing).
Add a text note to playbook helper asserts. For playbook
syntax/layout errors that aren't yaml errors, but errors
indicating invalid data structures for a playbook/task/role/block,
we now include some info about where the assert was and
why it was raised.
In places we raise an AnsibleParserError in an except
clause, pass the original exception to AnsibleParserError via
orig_exc arg.
Make assorted error messages a little more specific (like
the playbook helper load methods)
* Revert "Include the original YAML error in syntax error messages"
This reverts commit 781bb44b02.
* Support NetBSD 7.1+ style ifconfig -a output
network facts on NetBSD after 7.1 cvs would fail
because of format changes in 'ifconfig -a' output.
update code to support new and old format.
add unit tests for both based on
examples from Bruce V Chiarelli.
* wrap use of interfaces.keys() in list() for py3 compat
* sort interface ids for stability
* Fix ansible_cmdline initrd fact for UEFI
UEFI cmdline paths use \ path sep which would
get munged by cmdline fact collection.
* Make CmdLineFactCollector easier to test
extract the parsing of the /proc/cmdline content to
_parse_proc_cmdline()
add a wrapper method for get_file_content _get_proc_cmdline()
Add unit tests of _parse_proc_cmdline based on examples
from issue #23647Fixes#23647
Fix 'hostname' module Facts is not defined by updating
'hostname' module to use it.
is_systemd_managed() was previously on the module_utils.facts.Facts
class that no longer exists.
Fixes#25289
Facts Refresh (2.4 roadmap)
This commit implements most of the 2.4 roadmap 'Facts Refresh'
- move facts.py to facts/__init__.py
- move facts Distribution() to its own class
- add a facts/utils.py
- move get_file_content and get_uname_version to facts/utils.py
- move Facts() class from facts/__init__ to facts/facts.py
- mv get_file_lines to facts/utils.py
- mv Ohai()/Facter() class to facts/ohai.py and facter.py
- Start moving fact Hardware() classes to facts/hardware/*.py
- mv HPUX() hardware class to facts/hardware/hpux.py
- move SunOSHardware() fact class to facts/hardware/sunos.py
- move OpenBSDHardware() class to facts/hardware/openbsd.py
- mv FreeBsdHardware() and DragonFlyHardware() to facts/hardware/
- mv NetBSDHardware() to facts/hardware/netbsd.py
- mv Darwin() hardware class to facts/hardware/darwin.py
- pep8/etc cleanups on facts/hardware/*.py
- Mv network facts classes to facts/network/*.py
- mv Virtual fact classes to facts/virtual
- mv Hardware.get_sysctl to facts/sysctl.py:get_sysctl
- Also mv get_uname_version from facts/utils.py -> distribution.py
since distribution.py is the only thing using it.
- add collector.py with new BaseFactCollector
- add a subclass for AnsibleFactCollector
- hook up dict key munging FactNamespaces
- add some test cases for testing the names of facts
- mv timeout stuff to facts.timeout
- rm ansible_facts()/get_all_facts() etc
- Instead of calling facts.ansible_facts(), fact collection
api used by setup.py is now to create an AnsibleFactCollector()
and call it's collect method.
- replace Facts.get_user_facts with UserFactCollector
- add a 'systems' facts package, mv UserFactCollector there
- mv get_dns_facts to DnsFactCollector
- mv get_env_facts to EnvFactCollector
- include the timeout length in exception message
- modules and module_utils that use AnsibleFactCollector
can now theoretically set the 'valid_subsets'
May be useful for network facts module that currently have
to reimplement a good chunk of facts.py to get gather_subsets
to work.
- get_local_facts -> system/LocalFactCollector
- get_date_time -> system/date_time.py
- get_fips_facts -> system/fips.py
- get_caps_facts() -> system/caps.py
- get_apparmor_facts -> system/apparmor.py
- get_selinux_facts -> system/selinux.py
- get_lsb_facts -> system/lsb.py
- get_service_mgr_facts -> system/service_mgr.py
- Facts.is_systemd_managed -> system/service_mgr.py
- get_pkg_mgr_facts -> system/pkg_mgr.py
- Facts()._get_mount_size_facts() -> facts.utils.get_mount_size()
- add unit test for EnvFactCollector
- add a test case for minimal gather_subsets
- add test case for collect_ids
- Make gather_subset match existing behavior or '!all'
If 'gather_subset' is provided as '!all', the existing behavior
(in 2.2/2.3) is that means 'dont collect any facts except those
from the Facts() class'. So 'skip everything except
'apparmor', 'caps', 'date_time', 'env', 'fips', 'local', 'lsb',
'pkg_mgr', 'python', 'selinux', 'service_mgr', 'user', 'platform', etc.
The new facts setup was making '!all' mean no facts at all, since
it can add/exclude at a finer granularity. Since that makes more
sense for the ansible collector, and the set of minimal facts to
collect is really more up to setup.py to decide we do just that.
So if setup.py needs to always collect some gather_subset, even
on !all, setup.py needs to have the that subset added to the
list it passes as minimal_gather_subset.
This should fix some intg tests that assume '!all' means that
some facts are still collected (user info and env for example).
If we want to make setup.py collect a more minimal set, we can do that.
- force facts_dicts.keys() to a list so py3 works
- split fact collector tests to test_collectors.py
- convert Facter(Facts) -> other/facter.py:FacterFactCollector
- add FactCollector.collect_with_namespace()
regular .collect() will return a dict with the key names
using the base names ('ip_address', 'service_mgr' etc)
.collect_with_namespace() will return a dict where the key names
have been transformed with the collectors namespace, if there is
one. For most, this means a namespace that adds 'ansible_' to the
start of the key name.
For 'FacterFactCollector', the namespace transforms the key to
'facter_*'.
- add test cases for collect_with_namespace
- move all the concrete 'which facts does setup.py' stuff to setup.py
The caller of AnsibleFactCollector.from_gather_subset() needs to
pass in the list of collector classes now.
- update system/setup.py to import all of the fact classes and pass
in that list.
- split the Distribution fact class up a bit
extracted the 'distro release' file handling (ie, linux
boxes with /etc/release, /etc/os-release etc) into its
own class.
- extract get_cmdline_facts -> cmdline.py
- extract get_public_ssh_host_keys -> system/ssh_pub_keys.py
- extract get_platform_facts -> system/platform.py
platform.py may be a good candidate for further splitting.
- rm test for plain Facts() base class
- let the base class for Collector unit tests provide collected_facts
some Collectors and/or their migrated Facts() subsclasses need
to look at facts collected by other modules ('ansible_architecture'
the main one...).
Collector.collect() has the collected_facts arg for this, so add
a class variable to BaseFactsTest so we can specify it.
- mv Ohai to other/ohai.py and convert to Collector
- update hardware/*.py to return facts (no side effects)
- mv AnsibleFactCollector to setup.py
- extra collector class gathering to module method in
facts/__init__.py (collector_classes_from_gather_subset)
- add a CollectorMetaDataCollector collector used to provide
the 'gather_setup' fact
- add unit test module for 'setup' module
(test/units/modules/system/setup.py)
- Collector init now doesnt need a module, but collect does
An instance of a FactCollector() isnt tied to a AnsibleModule
instance, but the collect() method can be, so optionally pass
in module to FactCollector.collect() (everywhere)
- add a default_collectors for list of default collectors
import and use it from setup.py module
eventually, would like to replace this with a plugin loader
style class finder/loader
- unit tests for module_utils/facts/__init__.py
- add unit tests for ohai facts collector
- remove self.facts side effect on populate() in hardware/sunos.py
- convert OpenBSDHardware() to rm side effects on self.facts
- try to rm some self.facts side effects in Network()
plumb in collected_facts from populate() where it is needed.
stop passing collected_facts into Network() [via cached_facts=,
where it eventually becomes self.facts]
- nothing provides Fact() cached_facts arg now, rm it
Facts() should be internal only implementation so nothing
should be using it.
Of course, now someone will.
- add a Collector.name attr to build a map of name->_fact_ids
To properly exclude a gather_subset spec like '!hardware', we
need to know that 'hardware' also means 'devices', 'dmi', etc.
Before, '!hardware' would remove the 'hardware' collector name
but not 'devices'. Since both would end up in id_collector_map,
we would still end up with the HardwareCollector in the collector
list. End result being that '!hardware' wouldn't stop hardware
from being collected.
So we need to be able to build that map, so add the Collector.name
attribute that is the primary name (like 'hardware') and let
Collector._fact_ids be the other fact ids that a collector is
responsible for.
Construct the aliases_map of Collector.name -> set of _fact_ids
in fact/__init__.py get_collector_names, and use it when we are
populating the exclude set.
- refactor of distribution.py
make the big OS_FAMILY literal a little easier to read
Also keys can now be any string instead of python literals
99% sure the test for 'KDE Neon' was wrong
I don't see how/where it should or could get 'Neon' instead
of 'KDE Neon' as provided in os-release NAME=
Use 'distribution' string for key to OS_MAP
ie, we dont need to make it a valid python label anymore so dont.
move _has_dist_file to module as _file_exists
easier to mock without mucking with os.path
mv platform.system() calls to within get_distribution_facts() instead
of Distribution() init.
- remove _json compat module
The code in here was to support:
-a 'json' python module that was not the standard one included
with python since 2.6.
- potentially fallback to simplejson if 'json' was not available.
'json' is available for all supported python versions now so
no longer needed.
- mv get_collector_names -> facts.collector
- mv collector_classes_from_gather_subset -> facts.collector
- mv collector tests from test_facts -> test_collector
- Use six's reduce() in sunos/netbsd hardware facts
- rm extraneous get_uname_version in utils
only system/distribution.py uses it
- Remove Facts() subclass metaclass usage
- using fact_id and a platform id for matching collectors
gut most of Facts() subclasses
rm Facts() subclasses with weird metaclass
only add collectors that match the fact_ids and the platform_info
to the list of collectors used.
atm, a collectors platform_id will default to 'Generic', and
any platform matches 'Generic'
goal is to select collector classes including matching the
systems platform in collector.py, instead of relying on
metaclasses in hardware/*. To finish this, the various
Facts() subclasses will need to be replaced entirely with
Collector() subclasses.
use collector classmethod platform_match() to match the platform
This lets the particular class decide if it is compatible with
a given platform_info. platform_info is a dict like obj, so it could be
expanded in the future.
Add a default platform_match to BaseFactCollector that matches
platform_info['system'] == cls._platform
They were needed previously to trigger a module
load on all the collector classes when we import
facts/hardare so that the Hardware() and related
classes that used __new__ and find_all_subclasses()
would work.
Now that is done in collectors based on platform matching
at runtime we dont need to do it py module import/parse
time. So the non empty __init__.pys are no longer needed
and their is a more flexible mechanism for selection
platform specific stuff.
facts/facts.py is no longer used, rm'ed
- if we dont find an implement class for gather spec.. just ignore it.
Would be useful to add a warn to warn about this case.
- Fix SD-UX typo (should be HP-UX)
- Port fix for #21893 (0 sockets) to this branch
This readds the change from 8ad182059d
that got lost in merge/rebase
Fixes#21893
- port sunos fact locale fix for #24542 to this branch
based on e558ec19cdFixes#24542
Solaris fact fix (#24793)
ensure locale for solaris fact gathering
fixes issue with locale interfering with proper reading of decimals
- raise exceptions in the air like we just dont care.
Pretty much ignore any not exit exception in facts
collection. And add some test cases.
- added new selinux fact to clarify python lib
the selinux fact is boolean false when the library is not installed,
a dictionary/hash otherwise, but this is ambigous
added new fact so we can eventually remove the type dichtomy and normalize it as a dict
Re-add of devel commit 85c7a7b844 to
the new code layout, since it got removed in merge/rebase
setup.py will not install package_data this is just an empty
directory, even if the globs in package_data match it.
So the role skeleton that was being installed to galaxy/data/*/
was excluding the files/ and templates/ directories since they were
empty.
Since the skeleton dir doesnt include those dirs
'ansible-galaxy init' would not set them up.
So this adds a .* glob to those directories so the .git_keep
will be included as well, so that setup.py will create the otherwise
empty directoty. ansible-galaxy init already knows to ignore those
files, so no other changes are needed.
(Including the .git_keep files is a little odd, but the alternative
would be creating our own placeholder files. Since ansible-galaxy
already understands .git_keep files we just reuse that)
Fixes#23597
The makefile in docs/api/ had a top level check for the
'sphinx-build' tool, but the check was getting triggered
on 'make clean' when it is not needed and causing 'make rpm'
and similar builds to fail.
So this commit removes the check.
Fixes#24163
Use the default repr of AnsibleVaultEncryptedUnicode.data instead
of a custom one, since jinja templating ends up using the repr()
results.
Fixes#23846, #24175
* Use sys.stdout.buffer to write vault bytes to stdout on py3
We need sys.stdout.buffer on py3 so we can write bytes to it since the plaintext
of the vaulted object could be anything/binary/etc
Before, attempting to write bytes to stdout on py3 would cause:
TypeError: write() argument must be str, not bytes
* Fix vault reading from stdin (avoid realpath() on non-links)
os.path.realpath() is used to find the target of file paths that
are symlinks so vault operations happen directly on the target.
However, in addition to resolving symlinks, realpath() also returns
a full path. when reading from stdin, vault cli uses '-' as a special
file path so VaultEditor() will replace with stdin.
realpath() was expanding '-' with the CWD to something like
'/home/user/playbooks/-' causing errors like:
ERROR! [Errno 2] No such file or directory: u'/home/user/ansible/-'
Fix is to specialcase '-' to not use realpath()
Fixes#23567
* to_text decrypt output when writing to stdout
