OpenStack新增节点报错Failed to create resource provider怎么办

这篇文章将为大家详细讲解有关OpenStack新增节点报错Failed to create resource provider怎么办，小编觉得挺实用的，因此分享给大家做个参考，希望大家阅读完这篇文章后可以有所收获。

1、背景

一台计算节点故障宕机导致该计算节点不能继续工作，先将其移出集群，节点恢复后再重新加入集群发现报错ResourceProviderCreationFailed: Failed to create resource provider

2、报错信息

# vim nova-compute.log2019-07-16 16:27:55.441 1166754 ERROR nova.scheduler.client.report [req-c50f65e8-ffd8-4a10-8d5e-0ec8d408a3c8 - - - - -] [req-9e5aad63-21d1-4297-be27-92ba9b8bfe9f] Failed to create resource provider record in placement API for UUID 4d9ed4b4-f3a2-4e5d-9d8e-2f657a844a04. Got 409: {"errors": [{"status": 409, "request_id": "req-9e5aad63-21d1-4297-be27-92ba9b8bfe9f", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: bdc2 already exists.  ", "title": "Conflict"}]}.2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager [req-c50f65e8-ffd8-4a10-8d5e-0ec8d408a3c8 - - - - -] Error updating resources for node bdc2.: ResourceProviderCreationFailed: Failed to create resource provider bdc22019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager Traceback (most recent call last):2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7426, in update_available_resource_for_node2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 688, in update_available_resource2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     self._update_available_resource(context, resources)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     return f(*args, **kwargs)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 712, in _update_available_resource2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     self._init_compute_node(context, resources)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 561, in _init_compute_node2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     self._update(context, cn)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 886, in _update2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     inv_data,2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid,2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1104, in set_inventory_for_provider2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 665, in _ensure_resource_provider2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 64, in wrapper2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     return f(self, *a, **k)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 612, in _create_resource_provider2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager     raise exception.ResourceProviderCreationFailed(name=name)2019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider bdc22019-07-16 16:27:55.442 1166754 ERROR nova.compute.manager

3、处理过程

3.1、问题一 uuid冲突

具体报错信息，重点看到Conflicting resource provider name: bdc2 already exists. 移除bdc2的时候，确定nova库中的service和computer-node都清除了，包括元数据也delete了，但是这里还是有元数据信息

发现cell库中并没有删除

# nova-manage cell_v2 list_hosts                          /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported  exception.NotSupportedWarning+-----------+--------------------------------------+----------+| Cell Name |              Cell UUID               | Hostname |+-----------+--------------------------------------+----------+|   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc1   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc2   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc3   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc4   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc5   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc6   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc7   ||   cell1   | df0d7c04-52b3-454d-a295-4f4ad836526b |   bdc8   |+-----------+--------------------------------------+----------+

于是手动删除再添加,发现报错并没有改变

# su -s /bin/sh -c "nova-manage cell_v2 delete_host --cell_uuid df0d7c04-52b3-454d-a295-4f4ad836526b --host bdc2 " nova# su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported  exception.NotSupportedWarningFound 2 cell mappings.Skipping cell0 since it does not contain hosts.Getting compute nodes from cell 'cell1': df0d7c04-52b3-454d-a295-4f4ad836526bFound 0 unmapped computes in cell: df0d7c04-52b3-454d-a295-4f4ad836526b

报错中提示的uuid为4d9ed4b4-f3a2-4e5d-9d8e-2f657a844a04和bdc2冲突，检查元数据库，除了nova库，还有nova_api库

MariaDB [nova_api]> select uuid,name from resource_providers where name='bdc2';+--------------------------------------+------+| uuid                                 | name |+--------------------------------------+------+| e131e7c4-f7db-4889-8c34-e750e7b129da | bdc2 |+--------------------------------------+------+MariaDB [nova_api]>  select uuid,host from nova.compute_nodes where host='bdc2';+--------------------------------------+------+| uuid                                 | host |+--------------------------------------+------+| 4d9ed4b4-f3a2-4e5d-9d8e-2f657a844a04 | bdc2 |+--------------------------------------+------+

看到症结所在，确实uuid冲突了，e131e7c4-f7db-4889-8c34-e750e7b129da应该是旧bdc2的uuid 手动更新表resource_providers中的uuid

MariaDB [nova_api]> update resource_providers set uuid='4d9ed4b4-f3a2-4e5d-9d8e-2f657a844a04' where name='bdc2' and uuid='e131e7c4-f7db-4889-8c34-e750e7b129da';

3.2、问题二 分配冲突

到这里冲突的问题解决了，但是新增的计算节点还是有异常，创建的新云主机居然不在这台新计算节点上创建，但是可以迁移一个小资源的云主机，不能迁移大资源占用的云主机 nova-compute的日志一直在刷warning

2019-07-16 19:10:02.684 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 6446a84d-cdfd-4cfe-bcd2-2d1d75db229f has been moved to another host bdc3(bdc3). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 8, u'MEMORY_MB': 16384, u'DISK_GB': 50}}.2019-07-16 19:10:02.738 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance e0d8d6df-4b48-402b-aa33-97c4a6166c5b has been moved to another host bdc6(bdc6). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 6, u'MEMORY_MB': 12288, u'DISK_GB': 50}}.2019-07-16 19:10:02.791 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 9d860729-597a-4420-bb8f-e9415587d808 has been moved to another host bdc3(bdc3). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 4, u'MEMORY_MB': 8192, u'DISK_GB': 50}}.2019-07-16 19:10:02.860 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 8e42328d-fd1c-4abc-acac-5c6e09623af6 has been moved to another host bdc5(bdc5). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 8, u'MEMORY_MB': 16384, u'DISK_GB': 50}}.2019-07-16 19:10:02.912 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 1d59e7db-bf1b-478c-a6bd-10287365cb65 has been moved to another host bdc3(bdc3). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 8, u'MEMORY_MB': 16384, u'DISK_GB': 50}}.2019-07-16 19:10:02.960 1192779 INFO nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 61223c2d-0b0c-4729-85e6-741c88e6e476 has allocations against this compute host but is not found in the database.2019-07-16 19:10:03.014 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 50f71a07-306d-4d2c-8f4a-6eaa11fbd233 has been moved to another host bdc6(bdc6). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 8, u'MEMORY_MB': 16384, u'DISK_GB': 50}}.: InstanceNotFound_Remote: Instance 61223c2d-0b0c-4729-85e6-741c88e6e476 could not be found.2019-07-16 19:10:03.068 1192779 WARNING nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Instance 0e7c21d4-a5fb-4059-aa47-bad47700e827 has been moved to another host bdc1(bdc1). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 8, u'MEMORY_MB': 16384, u'DISK_GB': 50}}.: InstanceNotFound_Remote: Instance 61223c2d-0b0c-4729-85e6-741c88e6e476 could not be found.2019-07-16 19:10:03.069 1192779 INFO nova.compute.resource_tracker [req-7c022b9b-7659-4a4d-9b53-30366a7fd150 - - - - -] Final resource view: name=bdc2 phys_ram=131037MB used_ram=512MB phys_disk=115480GB used_disk=0GB total_vcpus=24 used_vcpus=0 pci_stats=[]

Warning信息说明当前计算节点上有几个实例信息与其它节点上的信息冲突，查看元数据库

MariaDB [nova_api]> select * from allocations where resource_provider_id=7;+---------------------+------------+------+----------------------+--------------------------------------+-------------------+-------+| created_at          | updated_at | id   | resource_provider_id | consumer_id                          | resource_class_id | used  |+---------------------+------------+------+----------------------+--------------------------------------+-------------------+-------+| 2019-07-09 09:10:27 | NULL       | 1471 |                    7 | 9d860729-597a-4420-bb8f-e9415587d808 |                 0 |     4 || 2019-07-08 14:58:36 | NULL       | 1444 |                    7 | 61223c2d-0b0c-4729-85e6-741c88e6e476 |                 0 |     6 || 2019-07-09 10:09:33 | NULL       | 1510 |                    7 | e0d8d6df-4b48-402b-aa33-97c4a6166c5b |                 0 |     6 || 2019-07-09 09:18:30 | NULL       | 1477 |                    7 | 1d59e7db-bf1b-478c-a6bd-10287365cb65 |                 0 |     8 || 2019-07-09 09:26:26 | NULL       | 1483 |                    7 | 6446a84d-cdfd-4cfe-bcd2-2d1d75db229f |                 0 |     8 || 2019-07-09 09:36:40 | NULL       | 1486 |                    7 | 0e7c21d4-a5fb-4059-aa47-bad47700e827 |                 0 |     8 || 2019-07-09 09:46:02 | NULL       | 1492 |                    7 | 8e42328d-fd1c-4abc-acac-5c6e09623af6 |                 0 |     8 || 2019-07-09 10:02:57 | NULL       | 1504 |                    7 | 50f71a07-306d-4d2c-8f4a-6eaa11fbd233 |                 0 |     8 || 2019-07-09 09:10:27 | NULL       | 1472 |                    7 | 9d860729-597a-4420-bb8f-e9415587d808 |                 1 |  8192 || 2019-07-08 14:58:36 | NULL       | 1445 |                    7 | 61223c2d-0b0c-4729-85e6-741c88e6e476 |                 1 | 12288 || 2019-07-09 10:09:33 | NULL       | 1511 |                    7 | e0d8d6df-4b48-402b-aa33-97c4a6166c5b |                 1 | 12288 || 2019-07-09 09:18:30 | NULL       | 1478 |                    7 | 1d59e7db-bf1b-478c-a6bd-10287365cb65 |                 1 | 16384 || 2019-07-09 09:26:26 | NULL       | 1484 |                    7 | 6446a84d-cdfd-4cfe-bcd2-2d1d75db229f |                 1 | 16384 || 2019-07-09 09:36:40 | NULL       | 1487 |                    7 | 0e7c21d4-a5fb-4059-aa47-bad47700e827 |                 1 | 16384 || 2019-07-09 09:46:02 | NULL       | 1493 |                    7 | 8e42328d-fd1c-4abc-acac-5c6e09623af6 |                 1 | 16384 || 2019-07-09 10:02:57 | NULL       | 1505 |                    7 | 50f71a07-306d-4d2c-8f4a-6eaa11fbd233 |                 1 | 16384 || 2019-07-08 14:58:36 | NULL       | 1446 |                    7 | 61223c2d-0b0c-4729-85e6-741c88e6e476 |                 2 |    50 || 2019-07-09 09:10:27 | NULL       | 1473 |                    7 | 9d860729-597a-4420-bb8f-e9415587d808 |                 2 |    50 || 2019-07-09 09:18:30 | NULL       | 1479 |                    7 | 1d59e7db-bf1b-478c-a6bd-10287365cb65 |                 2 |    50 || 2019-07-09 09:26:26 | NULL       | 1485 |                    7 | 6446a84d-cdfd-4cfe-bcd2-2d1d75db229f |                 2 |    50 || 2019-07-09 09:36:40 | NULL       | 1488 |                    7 | 0e7c21d4-a5fb-4059-aa47-bad47700e827 |                 2 |    50 || 2019-07-09 09:46:02 | NULL       | 1494 |                    7 | 8e42328d-fd1c-4abc-acac-5c6e09623af6 |                 2 |    50 || 2019-07-09 10:02:57 | NULL       | 1506 |                    7 | 50f71a07-306d-4d2c-8f4a-6eaa11fbd233 |                 2 |    50 || 2019-07-09 10:09:33 | NULL       | 1512 |                    7 | e0d8d6df-4b48-402b-aa33-97c4a6166c5b |                 2 |    50 |+---------------------+------------+------+----------------------+--------------------------------------+-------------------+-------+24 rows in set (0.00 sec)MariaDB [nova_api]> select * from allocations where consumer_id='9d860729-597a-4420-bb8f-e9415587d808';+---------------------+------------+------+----------------------+--------------------------------------+-------------------+------+| created_at          | updated_at | id   | resource_provider_id | consumer_id                          | resource_class_id | used |+---------------------+------------+------+----------------------+--------------------------------------+-------------------+------+| 2019-07-09 09:10:27 | NULL       | 1468 |                    6 | 9d860729-597a-4420-bb8f-e9415587d808 |                 0 |    4 || 2019-07-09 09:10:27 | NULL       | 1469 |                    6 | 9d860729-597a-4420-bb8f-e9415587d808 |                 1 | 8192 || 2019-07-09 09:10:27 | NULL       | 1470 |                    6 | 9d860729-597a-4420-bb8f-e9415587d808 |                 2 |   50 || 2019-07-09 09:10:27 | NULL       | 1471 |                    7 | 9d860729-597a-4420-bb8f-e9415587d808 |                 0 |    4 || 2019-07-09 09:10:27 | NULL       | 1472 |                    7 | 9d860729-597a-4420-bb8f-e9415587d808 |                 1 | 8192 || 2019-07-09 09:10:27 | NULL       | 1473 |                    7 | 9d860729-597a-4420-bb8f-e9415587d808 |                 2 |   50 |+---------------------+------------+------+----------------------+--------------------------------------+-------------------+------+

可以看到resource_provider_id为7也就是新增的bdc2上有资源分配信息，摘出其中一个实例id查看发现它不仅在7也在6上，联想到先前旧节点上的实例疏散，新增节点是改了uuid继承了旧节点的信息，所以日志会报冲突Warining。并且 (8192+12288+12288+16384+16384+16384+16384+16384)/1024=112G

内存资源占了112G，即将达到当前机器的内存上限，所以创建云主机不会优先选择这个节点，迁移也只能迁移小的

既然已经修改了元数据，那就走到黑，继续清除

MariaDB [nova_api]> delete from allocations where resource_provider_id=7;Query OK, 24 rows affected (0.00 sec)MariaDB [nova_api]> select * from allocations where resource_provider_id=7;Empty set (0.00 sec)

4、验证

创建三个需要大资源的虚拟机，发现都创建在了bdc2上，并且nova-compute日志中没有刷类似的Warning,说明问题解决。

5、备注

OpenStack基础组件的有数据的元数据表

MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'nova' and table_rows<>0 ORDER BY table_rows DESC; +--------------------------+------------+| table_name               | table_rows |+--------------------------+------------+| instance_actions_events  |       2335 || instance_system_metadata |       2324 || instance_actions         |       1959 || virtual_interfaces       |        474 || block_device_mapping     |        451 || instance_info_caches     |        267 || instances                |        267 || instance_id_mappings     |        260 || instance_faults          |        217 || instance_extra           |        206 || migrations               |        122 || s3_images                |         17 || services                 |         13 || compute_nodes            |          8 || security_groups          |          6 |+--------------------------+------------+15 rows in set (0.00 sec)MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'nova_api' and table_rows<>0 ORDER BY table_rows DESC; +--------------------+------------+| table_name         | table_rows |+--------------------+------------+| consumers          |        339 || instance_mappings  |        283 || request_specs      |        213 || allocations        |        195 || traits             |        164 || quotas             |         57 || inventories        |         23 || flavors            |          9 || projects           |          8 || key_pairs          |          8 || users              |          8 || resource_providers |          8 || host_mappings      |          8 || cell_mappings      |          2 |+--------------------+------------+14 rows in set (0.01 sec)MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'nova_cell0' and table_rows<>0 ORDER BY table_rows DESC; +--------------------------+------------+| table_name               | table_rows |+--------------------------+------------+| instance_system_metadata |        112 || instance_id_mappings     |         16 || block_device_mapping     |         16 || instance_faults          |         16 || instance_extra           |         16 || instances                |         16 || instance_info_caches     |         16 || s3_images                |          2 |+--------------------------+------------+MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'cinder' and table_rows<>0 ORDER BY table_rows DESC; +------------------------+------------+| table_name             | table_rows |+------------------------+------------+| reservations           |        573 || volume_admin_metadata  |        478 || volume_attachment      |        451 || volumes                |        200 || volume_glance_metadata |         64 || quotas                 |         21 || quota_usages           |         15 || quota_classes          |          6 || services               |          2 || workers                |          1 |+------------------------+------------+10 rows in set (0.07 sec)MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'glance' and table_rows<>0 ORDER BY table_rows DESC; +------------------+------------+| table_name       | table_rows |+------------------+------------+| images           |         19 || image_locations  |         19 || image_properties |          9 || alembic_version  |          1 |+------------------+------------+4 rows in set (0.03 sec)MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'keystone' and table_rows<>0 ORDER BY table_rows DESC; +-----------------+------------+| table_name      | table_rows |+-----------------+------------+| endpoint        |         18 || assignment      |         17 || user            |         14 || password        |         14 || local_user      |         14 || project         |         12 || service         |          6 || migrate_version |          4 || role            |          2 |+-----------------+------------+9 rows in set (0.08 sec)MariaDB [nova_api]> SELECT table_name,table_rows FROM information_schema.tables WHERE TABLE_SCHEMA = 'neutron' and table_rows<>0 ORDER BY table_rows DESC; +---------------------------+------------+| table_name                | table_rows |+---------------------------+------------+| ml2_vxlan_allocations     |       1000 || standardattributes        |        149 || ports                     |         69 || ipamallocations           |         69 || ipallocations             |         69 || ml2_port_bindings         |         69 || portsecuritybindings      |         69 || securitygroupportbindings |         68 || ml2_port_binding_levels   |         66 || securitygrouprules        |         59 || quotas                    |         56 || quotausages               |         17 || agents                    |         11 || default_security_group    |          8 || segmenthostmappings       |          8 || securitygroups            |          8 || allowedaddresspairs       |          4 || provisioningblocks        |          4 || alembic_version           |          2 |+---------------------------+------------+19 rows in set (0.18 sec)

6、总结

1、元数据操作非常危险，尽量不动或者少动，如果要动，先备份数据库;2、删除不掉的云主机和卷，不要直接修改元数据的deleted字段，这是自欺欺人的办法，只是dashboard上看不到而已，实际资源并不释放而且后端存储中还存在文件；3、硬件操作要小心，总之任何危险的操作都要再三确认。

关于"OpenStack新增节点报错Failed to create resource provider怎么办"这篇文章就分享到这里了，希望以上内容可以对大家有一定的帮助，使各位可以学到更多知识，如果觉得文章不错，请把它分享出去让更多的人看到。