OPERATION FAILED. REASON: PRE-CHECK ‘TEST_CATALOG_WORKFLOWS

Nutanix Leave a comment

OPERATION FAILED. REASON: PRE-CHECK ‘TEST_CATALOG_WORKFLOWS’ FAILED (FAILURE REASON: FAILED TO CREATE CATALOG ITEM. ERROR: TASK 23D6F5B48D4D92BF8BFA14C52483 DID NOT SUCCEED)

Issue 1

Description :-

When we perform the LCM (Life cycle Manager) operation inventory, the LCM framework downloads the required files and modules from the Nutanix web server or Nutanix portal with catalog service support. If there is an issue which is preventing downloading those files, like firewall scanning or port blocking, the LCM pre-check “test_catalog_workflows fails.”

Nutanix LCM issue :-

As per the error below, it seems the download is failing with the below error. We need to open the ports as per the link below, and then we need to retry the LCM inventory.

Nutanix Life Cycle Manager

Port requirement:

Please follow the Nutanix port requirement

https://portal.nutanix.com/page/documents/details?targetId=Acropolis-Upgrade-Guide-v5_17%3Awc-support-firewall-wc-c.html

Set the following setting on the firewall for LCM updates

CVM/ClusterIP download.nutanix.com HTTP – 80,TTPS – 443 ALLOW

Error:

2020-07-19 01:06:59 ERROR exception.py:54 LCM Exception [LcmRecoverableError]: Pre-check 'test_catalog_workflows' failed (Failure reason: Failed to create catalog item.
Error: Task e87ace2a97c24c29803b0fc85bdf8502 did not succeed)
Refer to KB 4584 for details.
Traceback (most recent call last):
File "/home/ubuntu/lcm-framework/build/python-tree/bdist.linux-x86_64/egg/framework/main/exception.py", line 209, in wrapper
 
"percentage_complete": 100,
"request": {
"arg": {
"spec": {
"annotation": "Test item created by test_catalog_workflows",
"global_catalog_item_uuid": "2be74087-da48-41be-a22c-378457355c0f",
"item_type": 5,
"name": "test_catalog_workflows",
"source_group_spec_list": [
{
"source_spec_list": [
{
"import_spec": {
"remote_import_list": [
{
"container_uuid": "e0f2cea0-0589-4b66-a328-eea8e522c6e8",
"url": "http://download.nutanix.com/lcm/2.0/master_manifest.tgz.sign"
}
]
}
}
]
}
]
}
},
"method_name": "CatalogItemCreate"
},
"requested_state_transition": 20,
"response": {
"error_code": 7,
"error_detail": "File 02822a92-fe74-406e-8081-0b161e6740f0 does not exist"
},
"sequence_id": 198,
"start_time_usecs": "2020-07-19T01:06:59.067752",
"status": "kFailed",
"subtask_sequence_id": 0,
"subtask_uuid_list": [
"3805c766-372a-46ae-8884-f2727d6abbfb"
],
"uuid": "e87ace2a-97c2-4c29-803b-0fc85bdf8502"

Solution:-

Once you open the port HTTP – 80,TTPS – 443 for the hostname re run the perform inventory

download.nutanix.com

=============================================================================

Issue 2:-

Description :-

The LCM inventory fails with error Operation failed. Reason: LCM failed staging to env ‘cvm-‘ at ip address x.x.x.78. Failure during step ‘Transfer for file xxx’, error ‘Failed to stage catalog item’

LCM Error Snippet as follows.

2019-10-12 14:43:29 WARNING prism_authentication.py:130 (x.x.x.71) Couldn't connect to prism, reason: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=9080): Read timed out. (read timeout=1)",),). Retrying after 8 

2019-10-12 14:43:29 WARNING prism_authentication.py:130 (x.x.x.73) Couldn't connect to prism, reason: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=9080): Read timed out. (read timeout=1)",),). Retrying after 8 

2019-10-12 14:43:29 WARNING prism_authentication.py:130 (x.x.x.78) Couldn't connect to prism, reason: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=9080): Read timed out. (read timeout=1)",),). Retrying after 8

2019-10-12 14:43:30 WARNING prism_authentication.py:130 (x.x.x.72) Couldn't connect to prism, reason: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=9080): Read timed out. (read timeout=1)",),). Retrying after 8 

2019-10-12 14:45:32 INFO catalog_utils.py:414 (x.x.x.78) Download of https://urldefense.proofpoint.com/v2/url?u=https-3A__x.x.x.78-3A9440_file-5Frepo_ef439244-2D6eaa-2D4a8a-2Da605-2D9bf780ca090b&d=DwICaQ&c=s883GpUCOChKOHiocYtGcg&r=KyygECLjgkrDVXV6-i5v4ypNAgiOHGsWBPptP47yyh8&m=J0I6z6_GUGhSIAOtjEeaFD0WvdhX8y511WgUDtTa6Sc&s=J-JzrfiL_ED1s-fgGrNS0jNq00s9e08gDDMsDc2W8u0&e=  is 

2019-10-12 14:45:33 ERROR prism_authentication.py:135 (x.x.x.72) Failed to access prism to get an auth 

2019-10-12 14:45:33 INFO catalog_utils.py:414 (x.x.x.72) Download of https://urldefense.proofpoint.com/v2/url?u=https-3A__x.x.x.78-3A9440_file-5Frepo_1ceb9c0c-2Ddb3c-2D46e4-2Dbe5b-2Dbfbdcf028e12&d=DwICaQ&c=s883GpUCOChKOHiocYtGcg&r=KyygECLjgkrDVXV6-i5v4ypNAgiOHGsWBPptP47yyh8&m=J0I6z6_GUGhSIAOtjEeaFD0WvdhX8y511WgUDtTa6Sc&s=eo-JO1uTwE2QmCJu81-7Srf-xTyZsDFYS6jfHvrNECE&e=  is 

2019-10-12 14:45:33 ERROR prism_authentication.py:135 (x.x.x.76) Failed to access prism to get an auth 

2019-10-12 14:45:33 INFO catalog_utils.py:414 (x.x.x.76) Download of https://urldefense.proofpoint.com/v2/url?u=https-3A__x.x.x.78-3A9440_file-5Frepo_1ceb9c0c-2Ddb3c-2D46e4-2Dbe5b-2Dbfbdcf028e12&d=DwICaQ&c=s883GpUCOChKOHiocYtGcg&r=KyygECLjgkrDVXV6-i5v4ypNAgiOHGsWBPptP47yyh8&m=J0I6z6_GUGhSIAOtjEeaFD0WvdhX8y511WgUDtTa6Sc&s=eo-JO1uTwE2QmCJu81-7Srf-xTyZsDFYS6jfHvrNECE&e=  is 

2019-10-12 14:45:33 ERROR prism_authentication.py:135 (x.x.x.74) Failed to access prism to get an auth 

2019-10-12 14:45:33 INFO catalog_utils.py:414 (x.x.x.74) Download of https://urldefense.proofpoint.com/v2/url?u=https-3A__x.x.x.78-3A9440_file-5Frepo_1ceb9c0c-2Ddb3c-2D46e4-2Dbe5b-2Dbfbdcf028e12&d=DwICaQ&c=s883GpUCOChKOHiocYtGcg&r=KyygECLjgkrDVXV6-i5v4ypNAgiOHGsWBPptP47yyh8&m=J0I6z6_GUGhSIAOtjEeaFD0WvdhX8y511WgUDtTa6Sc&s=eo-JO1uTwE2QmCJu81-7Srf-xTyZsDFYS6jfHvrNECE&e=  is finished

Solution :-

This is a known issue and the fix is to upgrade AOS to version 5.10.3 or 5.11.

Workaround :-

Performing a rolling restart of Prism service

LCM :-

  1. During an LCM framework update, LCM sometimes incorrectly polls an HTTP endpoint even when you select Enable HTTPS.
  2. Depending on your firewall settings, the update can fail as a result. Change this setting as per firewall policy.
  3. LCM support for two-node clusters is available only on clusters that are running versions of AOS 5.10.5 and later version.
  4. LCM cannot perform firmware updates on single-node Nutanix clusters that may be any OEM hardware.
  5. Nutanix recommends that you perform all updates through Life Cycle Manager (LCM).

Leave a Reply