* Add more kubernetes Service tests
Services can often go wrong, and one of the main motivations for
apply was being able to correctly modify them, so more tests are good
* Remove a port from a service for k8s testing
* Add a Service check mode to k8s tests
There are too many lingering uncertainties about the correctness of
apply behaviour. All tests seem to suggest it works, and it's definitely
performed well in real world application, but it may be a breaking
and unexpected change to default to apply
We will let apply bed in during 2.9 and allow people to opt in, we
may default to it in future (or not)
* Ensure k8s apply works with check mode
Update the new predicted object with fields from the previous object
before applying in check mode
Don't log output of `file` with `state: absent` on huge virtualenvs!
Fixes#60510
* Use openshift client fix to improve apply for check mode
Use new apply_object method to get a better approximation
of the expected object in check mode.
Requires released upgrade to openshift
* Add changelog fragment for k8s apply check mode fix
* Update changelogs/fragments/60510-k8s-apply-check-mode.yml
Co-Authored-By: Felix Fontein <felix@fontein.de>
I no longer desire to be mentioned on most AWS changes.
If there are modules I'm on the authors list then I'll remain
notified.
Also remove me from openshift and kubectl plugins, I don't use
those.
Preserve tag key case by only calling camel_dict_to_snake_dict once,
before the tags are added.
Don't call assert_policy_shape as it seems to fail
Use aws_caller_info in the test suite now that it exists rather
than running `aws sts get_caller_identity`
Ensure that calls using `grant_types` can also use key aliases
* Add apply to k8s module
Use apply method for updating k8s resources.
* Improve apply documentation
* k8s: Make apply and merge_type mutually exclusive
* Add purge_tags to s3_bucket to allow preservation of existing tags
Adding `purge_tags` with default `True` to maintain existing behaviour
allows users to set it to `False` to preserve existing tags
Fixes#29366
* s3_bucket: Add further tests and improve tag handling further
Additional tests for purge_tags: False suggested some incorrect
logic and thus further improvements
Increase wait timeout on bucket deletion as it wasn't always completing
in the default 100 seconds
* Update testing policies to ensure all required permissions are present
* Tidy up security policies to reduce duplicate permissions
* Make roles static so that they can be present before CI is run,
meaning that role creation permission is not required by the CI
itself, only by someone setting up the roles prior to testing
* Move contents to cloudfront policy to network policy to ensure policy
count (maximum of 10) stays low
* Maintain compute policy below 6144 bytes
Not waiting outputs results in a format that will never
be matched by the tests
Ensure instances get tidied up
Allow ec2:ReportInstanceStatus
ec2_instance: Improve test cleanup on failing tests
Improve describe/modify attribute error handling
Address feedback on PR
* new module uses modern ansible AWS standards
* adds additional tests for snapshots
* Update return_skeleton_generator for python3 - should
set type to `str`, not `string`.
* Update testing policy to be correct for RDS test suite
* Create read replica in same region to avoid more permissions being
required
* Ensure modifying DB doesn't try to downgrade engine version
* Add tags to main test suite to limit number of tests run for problem
solving
* Combine testing policies
Because of the maximum of 10 policies per group, need to
consolidate testing policies as best we can.
* Tidy put-account-setting tasks and add permission
Using `environment` and `command` rather than `shell` avoids the
need for `no_log` and means that people can fix the problem
* refactor ecs_cluster test suite
move from runme.sh technique to virtualenv
use ec2_instance rather than ec2 module to
avoid need for boto
* Add test case for non-cascading deletes
Deleting a DaemonSet does not delete associated pods,
even though it should
* Add coverage module when using pip
Otherwise tests seemingly fail
Ensure `wait_condition`s with `Status: Unknown` actually
complete
Return k8s object after wait rather than k8s object before
wait when object is patched.
* Allow python_requirements_facts to cope with packages with dashes
```
python_requirements_facts:
dependencies:
- kubernetes-validate
```
should work as expected
* Ensure tests run for python_requirements_facts
* Make ansible adhoc work with include_role
Fix logic condition so that include_role works
without
```
ERROR! 'async_val' is not a valid attribute for a IncludeRole
The error appears to be in 'None': line 0, column 0, but may
be elsewhere in the file depending on the exact syntax problem.
(could not open file to display line)
```
* Add include_role test for adhoc
* Fix TagName and TagValue in aws_kms
Fixes#53061
* Improve test suite to check for tags
Also fixed some obvious failures, need to run the test suite from time
to time!
* Allow creation and deletion of keys (deletion just schedules for
deletion, recreating an old key is just cancelling its deletion)
* Allow grants to be set, thus enabling encryption contexts to be
used with keys
* Allow tags to be added and modified
* Add testing for KMS module
* Tidy up aws_kms module to latest standards
* k8s wait - don't try to call to_dict on None
Need to ensure that the resource dict is correctly returned for absent
resources
* Cope with resources without a namespace
Some resources (e.g. ClusterRoleBinding) do not have a Namespace.
Handle the lack of a namespace key.
So many reports of missing modules could be solved
more quickly with the info from `python_requirements_facts`
Replace the legacy issue template with a link to the
new interface
Warnings get printed at the end of loops, which means that if you're
running validation against a bunch of resources, the warning message
gets printed after a number of potentially unrelated resources.
Adding extra info about the resource failing validation will help
find the validation issues.
* Add append_hash functionality to k8s module
append_hash adds a hash based on the contents of a ConfigMap
or Secret to the name - this enables immutable ConfigMaps and
Secrets.
* Provide k8s_config_resource_name plugin
The k8s_config_resource_name filter plugin provides a means of determining
the name of ConfigMaps and Secrets created with append_hash
* Add changelog fragment
* fix failing tests
* Update openshift version needed for append_hash
* Provide Kubernetes resource validation to k8s module
Use kubernetes-validate to validate Kubernetes resource
definitions against the published schema
* Additional tests for kubernetes-validate
* Improve k8s error messages on exceptions
Parse the response body for the message rather than returning
a JSON blob
If we've validated and there are warnings, return those too - they
can be more helpful
```
"msg": "Failed to patch object: {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},
\"status\":\"Failure\",\"message\":\"[pos 334]: json: decNum: got first char 'h'\",\"code\":500}\n",
```
vs
```
"msg": "Failed to patch object: [pos 334]: json: decNum: got first char 'h'\nresource
validation error at spec.replicas: 'hello' is not of type u'integer'",
```
* Update versions used
In particular openshift/origin:3.9.0
* Add changelog for k8s validate change
kubeconfig should be loaded if *either* or both of context
or kubeconfig is set (this allows picking a context and default
kubeconfig or picking a kubeconfig with default context)
Fixes#47149
* Improve iam_group exception handling
Use AnsibleAWSModule for iam_group and handle BotoCoreErrors
as well as ClientErrors. Use fail_json_aws to improve error messages
* Add minimal iam_group test suite
Update some of the read-only IAM permissions (this is not sufficient
to run the test suite but it gets further than it did until it tries
to add a (non-existent) user)
* Clean up after tests
* Use a sensible default for k8s merge_type
The sensible default for merge_type is `['strategic-merge', 'merge'].
However, we can't make this the default default, as we need to support
users who are using openshift 0.6.0, where the merge_type parameter is
unsupported.
* Refactor k8s test suite for merge_type tests
Allow tests with pre-merge-type openshift and post-merge-type
openshift.
* Make merge_type a list and apply merge_type in order
Allow use case of preferring strategic-merge and failing
back to merge, or just preferring a different merge type
* Improve k8s module test coverage
`items` is a *terrible* return key for ansible as Jinja
will often think it refers to the `items()` function.
Even though you can typically work around this with
`results['items']`, sometimes even that doesn't work:
```
- name: Resources should exist
assert:
that: item.status.phase == 'Active'
loop: "{{ k8s_namespaces['items'] }}"
```
```
fatal: [testhost]: FAILED! => {"msg": "Invalid data passed to 'loop', it
requires a list, got this instead: <built-in method items of dict object
at 0x109dc9c58>. Hint: If you passed a list/dict of just one element,
try adding wantlist=True to your lookup invocation or use q/query
instead of lookup."}
```
Change it now while we still can.
Allows patching of custom Kubernetes resources that
don't support strategic merge patching
Check that openshift module supports content_type param
(requires version newer than 0.6.0)
Ensure that FieldLevelEncryptionId is properly handled - passing it if
set, and keeping it if returned by GetDistribution
Update cloudfront_distribution tests to remove references to
test_identifier so test suite actually works
Fixes#40724
Move dict_merge from azure_rm_resource module to
module_utils.common.dict_transformations and add tests.
Use dict_merge to provide a fairly realistic, reliable
diff output when k8s-based modules are run in check_mode.
Rename unit tests so that they actually run and reflect
the module_utils they're based on.
While the HTML produced is perfectly valid, without the `trim` filter,
a lot of warnings are emitted (700 lines of warnings out of 2812 are
eliminated by this change)
* Test case for missing permissions
* Update aws_s3 module to latest standards
* Use AnsibleAWSModule
* Handle BotoCoreErrors properly
* Test for BotoCoreErrors
* Check for XNotImplemented exceptions (#38569)
* Don't prematurely fail if user does not have s3:GetObject permission
* Allow S3 drop-ins to ignore put_object_acl and put_bucket_acl
* aws_eks: New module for managing AWS EKS
aws_eks module is used for creating and removing EKS clusters.
Includes full test suite and updates to IAM policies to enable it.
* Clean up all security groups
* appease shippable
* Rename aws_eks module to aws_eks_cluster
* Allow the use of 'aws:kms' as an encryption method
* Allow the use of a non standard KMS key
* Deduce whether AWS Signature Version 4 is required rather than specifying with a parameter
The compute policy was exceeding maximum size and contained
policies that already exist in ecs-policy.
Look up suitable AMIs rather than hardcode
We don't want to maintain multiple image IDs for multiple regions
so use ec2_ami_facts to set a suitable image ID
Improve exception handling
Remove VPC permissions from network-policy.json as they mostly duplicate
compute-policy.json permissions - separating the VPC and compute permissions
would likely lead to further confusion.
RDS instance and snapshot facts modules
Move common code into module_utils/rds.py
This work is a precursor to new rds_snapshot
and rds_instance modules
Improve rds_snapshot_facts to latest AWS standards
Improve exception handling, pagination etc.
Version added
Add clustered snapshot facts
Add proper RETURN information for snapshots and cluster_snapshots
pep8
Co-authored-by: Michael De La Rue <jababubic.margoteen@spamgourmet.com>
Create rds_instance_facts module using boto3.
Create common rds.py module_util library to be used by
rds_instance_facts and future RDS modules.
rds_instance_facts: use latest AWS helper methods
* Use `module.client` with `retry` to create AWS connection
* Return tags, using AWSRetry helper to backoff
* Use paginator to describe DB instances
* Further improve exception handling
* Fix RETURN documentation
update version_added
Co-authored-by: Michael De La Rue <jababubic.margoteen@spamgourmet.com>
* Add helpful failure message if target_type=ip is not supported
Create test case for target_type=ip not supported
* Update elb_target_group module to latest standards
Use AnsibleAWSModule
Improve exception handling
Improve connection handling
Improve naming of one of the cloudfront tasks
Change test_identifier back to resource_prefix now it's always
lower case.
More tests around using distribution_id and default_root_object
Too much to do, too little time. Happy to remain on the WG,
be consulted for advice by the WG, and retain the notifications
for my existing authorships and the remaining notifications
(and have added some more explicit ECS notifications)
Enable awsvpc network mode for ECS services and tasks and
their underlying task definitions
Improve test suite to thoroughly test the changes
Use runme.sh technique to run old and new versions of botocore to
ensure that the modules work with older botocore and older network modes
and fail gracefully if awsvpc network mode is used with older botocore
* Improve ec2_ami tests
Ensure that ec2_ami_image_id fact gets set immediately after AMI
creation so that they get torn down even if tests fail
Use YAML anchor to simplify AWS credential passing
Use aws_connection_info to reduce AWS credential boilerplate
Improve exception handling when updating image attributes
Error messages weren't correctly formatted to show image ids.
The docs suggest that `port` parameter is the default port upon
which targets listen. As such, a target need only provide a `Port`
key to override the default.
* Fix the function_name handling logic for lambda_policy
Switch the logic handling function_names that are ARNs
so that ARNs are correctly handled and detected
* Add tests for lambda_policy function_arn
Ensure that function_arn works.
Needs a reasonable ansible_lambda_role.
* Reflect updated names of the aws_batch_job modules
* Remove mention of `lambda_invoke` module which has yet to make it to
Ansible.
* Update broken rst link in win_dsc module
Cloudfront needs CreateOriginAccessIdentity
Add profile parameter to setup-iam.yml. Could arguably just use
AWS_PROFILE but given that other tasks are using profile, should
be consistent.
* Use AnsibleAWSModule to simplify AWS connection
* Add Exception handling, pagination, retries and backoff
* Allow events to be switched off
* Allow details to be obtained without having to specify services
SGs created when a VPC ID was not specified would not necessarily
get the default egress rule, even when no explicit egress rules
were set.
Add some checks for egress rules in results from existing tests
Change assign_public_ip to default to None rather than False so
that we can detect whether the value is being explicitly set or
not, and only warn if it is explicitly set to False for something
with a public_dns_name
Fixes#37985
* New generator inventory plugin
Allows construction of hosts and groups through cartesian
product of various group combinations
* Add generator plugin documentation
[cloud aws_s3] Most modes require `object` parameter, and this is easy to
get wrong (e.g. through confusion with the `dest` parameter). As
it's as easy to enforce, let's do that.
There should no longer be a need for boto documentation - it
just adds significant extra clutter to the guidelines.
AnsibleAWSModule greatly improves the codebase to such an extent
that we should mandate it for new code unless there is a
documented good reason.
ecs_taskdefinition_facts should not populate ansible_facts with
so much information.
Better to just return the contents of the dict directly.
Reduce line lengths, use AnsibleAWSModule, don't reimplement
camel_dict_to_snake_dict.
`wait_for_instances: no` means do not wait for instances to become
InService/Healthy before terminating them. It does not mean don't
wait for the ASG to delete.
Not waiting for the ASG to delete can cause problems when recreating it.
Ensure that waiting for the ASG to delete respects `wait_timeout`.
It is not possible to modify the load balancer configuration
for ECS Service.
As it is possible to detect this, it's nicer to fail gracefully
than return AWS's less meaningful failure message.
Fix PEP8 compliance
People expect to be able to upload files to s3 using standard
locations for files.
Providing an action plugin that effectively rewrites the `src`
key to the result of finding such a file is a great help.
Tests added, and IAM permissions corrected
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug
* Improvements to cloudfront_distribution
* Reduce the scope of the cloudfront_distribution module
* Remove presigning
* Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines
* Make requested changes
Fix tests
Use built-in waiter
Update copyright
* describe_images is very slow if not filtered to owner accounts
*or* if the Owners parameter is passed (unless the Owners parameter
is `self`). Convert Owners parameters to `owner-id` and `owner-alias`
filters where possible. Tests with CLI show that `--owners self` is
fast, `--owners 123456789012` is slow (with or without owner-id filter).
* describe_image_attributes fails against accounts other than your
own. Launch permissions are useful information, but not critical.
_name attribute is used when providing generic error messages
(such as connection problems). As AnsibleAWSModule does not inherit
things from AnsibleModule by default, need to provide it.
* Allow protection of certain keys during camel_to_snake
Create an `ignore_list` parameter that preserves the case
of the contents of certain dictionaries. Most valuable
for `tags` but other uses might arise.
* Port ec2_vpc_route_table to boto3
Update tests to reflect fixes in boto3.
* Add RETURN documentation to ec2_vpc_route_table
* Update DOCUMENTATION to be valid yaml
* Add check mode tests
