Non-idempotency when creating quota rules

[eliasp]

Summary

When applying a quota rule a 2nd time, an error is returned:

Error on creating quotas rule: calling: storage/quota/rules: got {'message': 'duplicate entry', 'code': '1', 'target': 'uuid'}

Component Name

netapp.ontap.na_ontap_quotas

Ansible Version

$ ansible --version
ansible [core 2.15.0]
  config file = /localworkspaces/elprobst/code-repositories/netapp/ansible.cfg
  configured module search path = ['/home/elprobst/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/lib/python3.10/site-packages/ansible
  ansible collection location = /home/elprobst/.ansible/collections:/usr/share/ansible/collections
  executable location = /nix/store/dqg1q8i9nj5y03cm971710c77jjjaw1z-python3.10-ansible-core-2.15.0/bin/ansible
  python version = 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.3.0] (/nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/bin/python3.10)
  jinja version = 3.1.2
  libyaml = True

ONTAP Collection Version

$ ansible-galaxy collection list
# /nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    5.2.0
ansible.netcommon             4.1.0
ansible.posix                 1.5.1
ansible.utils                 2.9.0
ansible.windows               1.13.0
arista.eos                    6.0.0
awx.awx                       21.11.0
azure.azcollection            1.14.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.3.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.3
cisco.intersight              1.0.23
cisco.ios                     4.3.1
cisco.iosxr                   4.1.0
cisco.ise                     2.5.12
cisco.meraki                  2.15.0
cisco.mso                     2.2.1
cisco.nso                     1.0.3
cisco.nxos                    4.0.1
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.4
community.aws                 5.2.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.10.0
community.digitalocean        1.23.0
community.dns                 2.5.0
community.docker              3.4.0
community.fortios             1.0.0
community.general             6.3.0
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         4.1.0
community.hrobot              1.7.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             5.0.0
community.okd                 2.2.0
community.postgresql          2.3.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.7.0
community.sap                 1.0.0
community.sap_libs            1.4.0
community.skydive             1.0.0
community.sops                1.6.0
community.vmware              3.3.0
community.windows             1.12.0
community.zabbix              1.9.1
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.17
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
dellemc.powerflex             1.5.0
dellemc.unity                 1.5.0
f5networks.f5_modules         1.22.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.2
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.1.2
grafana.grafana               1.1.0
hetzner.hcloud                1.9.1
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.11.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.3.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.2.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.10.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.16.2
purestorage.flashblade        1.10.0
purestorage.fusion            1.3.0
sensu.sensu_go                1.13.2
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.32.0
theforeman.foreman            3.8.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.7.0
vyos.vyos                     4.0.0
wti.remote                    1.0.4

# /nix/store/nrrwqzn43grz2nmb57cdjf2l4zhiyflk-python3.10-ansible-7.2.0/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    5.2.0
ansible.netcommon             4.1.0
ansible.posix                 1.5.1
ansible.utils                 2.9.0
ansible.windows               1.13.0
arista.eos                    6.0.0
awx.awx                       21.11.0
azure.azcollection            1.14.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.3.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.3
cisco.intersight              1.0.23
cisco.ios                     4.3.1
cisco.iosxr                   4.1.0
cisco.ise                     2.5.12
cisco.meraki                  2.15.0
cisco.mso                     2.2.1
cisco.nso                     1.0.3
cisco.nxos                    4.0.1
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.4
community.aws                 5.2.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.10.0
community.digitalocean        1.23.0
community.dns                 2.5.0
community.docker              3.4.0
community.fortios             1.0.0
community.general             6.3.0
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         4.1.0
community.hrobot              1.7.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             5.0.0
community.okd                 2.2.0
community.postgresql          2.3.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.7.0
community.sap                 1.0.0
community.sap_libs            1.4.0
community.skydive             1.0.0
community.sops                1.6.0
community.vmware              3.3.0
community.windows             1.12.0
community.zabbix              1.9.1
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.17
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
dellemc.powerflex             1.5.0
dellemc.unity                 1.5.0
f5networks.f5_modules         1.22.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.2
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.1.2
grafana.grafana               1.1.0
hetzner.hcloud                1.9.1
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.11.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.3.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.2.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.10.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.16.2
purestorage.flashblade        1.10.0
purestorage.fusion            1.3.0
sensu.sensu_go                1.13.2
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.32.0
theforeman.foreman            3.8.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.7.0
vyos.vyos                     4.0.0
wti.remote                    1.0.4

ONTAP Version

netapp01::> version
NetApp Release 9.11.1P8: Fri Apr 07 00:02:50 UTC 2023

Playbook

- name: add quota
    netapp.ontap.na_ontap_quotas:
     state: present
     vserver: testsvm
     volume: test1
     quota_target: "757"
     type: user
     disk_limit: 300MB
     set_quota_status: True
     use_rest: always
     <<: *login

Steps to Reproduce

Save the following playbook (e.g. test-quota.yml):

---
- hosts: localhost
  name: Create Mirror
  gather_facts: false
  vars:
    login: &login
     hostname: "{{ clusterip }}"
     username: "{{ destination_cluster_username }}"
     password: "{{ destination_cluster_password }}"
     https: "{{ https_option }}"
     validate_certs: "{{ validate_certs_option }}"
  vars_files:
  - variables.yml
  tasks:
  - name: add quota
    netapp.ontap.na_ontap_quotas:
     state: present
     vserver: testsvm
     volume: test1
     quota_target: "757"
     type: user
     disk_limit: 300MB
     set_quota_status: True
     use_rest: always
     <<: *login

Apply it for the 1st time:

ansible-playbook -vvvv test-quota.yml

The quota seems to be created as desired, but the output contains this warning:

[WARNING]: Quota policy rule create opertation succeeded. However quota resize failed due to an internal error. To make quotas active, reinitialize(disable and enable again) the quota for volume test1 in SVM testsvm.

As I first believed, this could be connected to the issue, I followed the instructions and manually disabled/enabled the quota for the volume, but it didn't change the actual outcome.

Apply the playbook for the 2nd time:

ansible-playbook -vvvv test-quota.yml

The result is:

fatal: [localhost]: FAILED! => {
  "changed": false,
  "invocation": {
    "module_args": {
      "activate_quota_on_change": null,
      "cert_filepath": null,
      "disk_limit": "300MB",
      "feature_flags": null,
      "file_limit": null,
      "force_ontap_version": null,
      "hostname": "[REDACTED]",
      "http_port": null,
      "https": true,
      "key_filepath": null,
      "ontapi": null,
      "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
      "perform_user_mapping": null,
      "policy": null,
      "qtree": "",
      "quota_target": "757",
      "set_quota_status": true,
      "soft_disk_limit": null,
      "soft_file_limit": null,
      "state": "present",
      "threshold": null,
      "type": "user",
      "use_rest": "always",
      "username": "ansible",
      "validate_certs": false,
      "volume": "test1",
      "vserver": "testsvm"
    }
  },
  "msg": "Error on creating quotas rule: calling: storage/quota/rules: got {'message': 'duplicate entry', 'code': '1', 'target': 'uuid'}."
}

Expected Results

I expect the Ansible playbook to show a proper idempotent behavior:

• apply desired changes on 1st run
• don't change/error out on any consecutive runs (unless something has changed)

Actual Results

ansible-playbook [core 2.15.0]
  config file = /localworkspaces/elprobst/code-repositories/netapp/ansible.cfg
  configured module search path = ['/home/elprobst/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/lib/python3.10/site-packages/ansible
  ansible collection location = /home/elprobst/.ansible/collections:/usr/share/ansible/collections
  executable location = /nix/store/dqg1q8i9nj5y03cm971710c77jjjaw1z-python3.10-ansible-core-2.15.0/bin/ansible-playbook
  python version = 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.3.0] (/nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/bin/python3.10)
  jinja version = 3.1.2
  libyaml = True
Using /localworkspaces/elprobst/code-repositories/netapp/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Loading collection netapp.ontap from /nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/lib/python3.10/site-packages/ansible_collections/netapp/ontap
Loading callback plugin default of type stdout, v2.0 from /nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/lib/python3.10/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: test-quota.yml *******************************************************
Positional arguments: test/test-quota.yml
verbosity: 6
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in test/test-quota.yml
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/vars/variables.yml"
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/variables.yml"
Read vars_file 'variables.yml'
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/vars/variables.yml"
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/variables.yml"
Read vars_file 'variables.yml'
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/vars/variables.yml"
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/variables.yml"
Read vars_file 'variables.yml'

PLAY [Create Mirror] ***********************************************************
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/vars/variables.yml"
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/variables.yml"
Read vars_file 'variables.yml'
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/vars/variables.yml"
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/variables.yml"
Read vars_file 'variables.yml'
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/vars/variables.yml"
looking for "variables.yml" at "/localworkspaces/elprobst/code-repositories/netapp/test/variables.yml"
Read vars_file 'variables.yml'

TASK [add quota] ***************************************************************
task path: /localworkspaces/elprobst/code-repositories/netapp/test/test-quota.yml:18
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: elprobst
<127.0.0.1> EXEC /bin/sh -c 'echo ~elprobst && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/elprobst/.ansible/tmp `"&& mkdir "` echo /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675 `" && echo ansible-tmp-1689750669.9888132-958041-268572850802675="` echo /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675 `" ) && sleep 0'
Including module_utils file ansible/__init__.py
Including module_utils file ansible/module_utils/__init__.py
Including module_utils file ansible/module_utils/_text.py
Including module_utils file ansible/module_utils/basic.py
Including module_utils file ansible/module_utils/common/_json_compat.py
Including module_utils file ansible/module_utils/common/__init__.py
Including module_utils file ansible/module_utils/common/_utils.py
Including module_utils file ansible/module_utils/common/arg_spec.py
Including module_utils file ansible/module_utils/common/file.py
Including module_utils file ansible/module_utils/common/locale.py
Including module_utils file ansible/module_utils/common/parameters.py
Including module_utils file ansible/module_utils/common/collections.py
Including module_utils file ansible/module_utils/common/process.py
Including module_utils file ansible/module_utils/common/sys_info.py
Including module_utils file ansible/module_utils/common/text/converters.py
Including module_utils file ansible/module_utils/common/text/__init__.py
Including module_utils file ansible/module_utils/common/text/formatters.py
Including module_utils file ansible/module_utils/common/validation.py
Including module_utils file ansible/module_utils/common/warnings.py
Including module_utils file ansible/module_utils/compat/selectors.py
Including module_utils file ansible/module_utils/compat/__init__.py
Including module_utils file ansible/module_utils/compat/_selectors2.py
Including module_utils file ansible/module_utils/compat/selinux.py
Including module_utils file ansible/module_utils/distro/__init__.py
Including module_utils file ansible/module_utils/distro/_distro.py
Including module_utils file ansible/module_utils/errors.py
Including module_utils file ansible/module_utils/parsing/convert_bool.py
Including module_utils file ansible/module_utils/parsing/__init__.py
Including module_utils file ansible/module_utils/pycompat24.py
Including module_utils file ansible/module_utils/six/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/netapp.py
Including module_utils file ansible/module_utils/ansible_release.py
Including module_utils file ansible_collections/__init__.py
Including module_utils file ansible_collections/netapp/__init__.py
Including module_utils file ansible_collections/netapp/ontap/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/__init__.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/netapp_module.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/rest_generic.py
Including module_utils file ansible_collections/netapp/ontap/plugins/module_utils/rest_response_helpers.py
Using module file /nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/lib/python3.10/site-packages/ansible_collections/netapp/ontap/plugins/modules/na_ontap_quotas.py
<127.0.0.1> PUT /home/elprobst/.ansible/tmp/ansible-local-958035gsdh0hg2/tmprplc73t0 TO /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675/AnsiballZ_na_ontap_quotas.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675/ /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675/AnsiballZ_na_ontap_quotas.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/nix/store/9vv68nmc7z4a96bkdxmk4gabmvqpsqqp-python3-3.10.12-env/bin/python3.10 /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675/AnsiballZ_na_ontap_quotas.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/elprobst/.ansible/tmp/ansible-tmp-1689750669.9888132-958041-268572850802675/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "activate_quota_on_change": null,
            "cert_filepath": null,
            "disk_limit": "300MB",
            "feature_flags": null,
            "file_limit": null,
            "force_ontap_version": null,
            "hostname": "[REDACTED]",
            "http_port": null,
            "https": true,
            "key_filepath": null,
            "ontapi": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "perform_user_mapping": null,
            "policy": null,
            "qtree": "",
            "quota_target": "757",
            "set_quota_status": true,
            "soft_disk_limit": null,
            "soft_file_limit": null,
            "state": "present",
            "threshold": null,
            "type": "user",
            "use_rest": "always",
            "username": "ansible",
            "validate_certs": false,
            "volume": "test1",
            "vserver": "testsvm"
        }
    },
    "msg": "Error on creating quotas rule: calling: storage/quota/rules: got {'message': 'duplicate entry', 'code': '1', 'target': 'uuid'}."
}

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[carchi8py]

@eliasp are you running this on a FSx System?

[eliasp]

@eliasp are you running this on a FSx System?

No, 2x on-premises AFF-A400

[carchi8py]

@eliasp we are trying to replicate this issue, but so far have not been able to.

In your case the create REST API https://library.netapp.com/ecmdocs/ECMLP2885799/html/index.html#/storage/quota_rule_create
Is returning a 5308568. I would reach out to the ONTAP team to ask what would cause that.

[eliasp]

Thank you!
I've opened a support case now and will get back to you once there are any updates which could further help solving this issue.

[eliasp]

Strangely, the issue (for which I opened the support case) regarding the initial error message disappeared and I am now completely unable to reproduce it.

The actual issue reported here (lack of idempotency, "msg": "Error on creating quotas rule: calling: storage/quota/rules: got {'message': 'duplicate entry', 'code': '1', 'target': 'uuid'}.") is still present.