| |
| Current cluster status: |
| RemoteNode database-0: UNCLEAN (offline) |
| RemoteNode database-2: UNCLEAN (offline) |
| Online: [ controller-0 controller-1 controller-2 ] |
| RemoteOnline: [ database-1 messaging-0 messaging-1 messaging-2 ] |
| Containers: [ galera-bundle-1:galera-bundle-docker-1 rabbitmq-bundle-0:rabbitmq-bundle-docker-0 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-0:redis-bundle-docker-0 redis-bundle-2:redis-bundle-docker-2 ] |
| |
| database-0 (ocf::pacemaker:remote): Stopped |
| database-1 (ocf::pacemaker:remote): Started controller-2 |
| database-2 (ocf::pacemaker:remote): Stopped |
| messaging-0 (ocf::pacemaker:remote): Started controller-2 |
| messaging-1 (ocf::pacemaker:remote): Started controller-2 |
| messaging-2 (ocf::pacemaker:remote): Started controller-2 |
| Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] |
| rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started messaging-0 |
| rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started messaging-1 |
| rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started messaging-2 |
| Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] |
| galera-bundle-0 (ocf::heartbeat:galera): FAILED Master database-0 (UNCLEAN) |
| galera-bundle-1 (ocf::heartbeat:galera): Master database-1 |
| galera-bundle-2 (ocf::heartbeat:galera): FAILED Master database-2 (UNCLEAN) |
| Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] |
| redis-bundle-0 (ocf::heartbeat:redis): Slave controller-0 |
| redis-bundle-1 (ocf::heartbeat:redis): Stopped |
| redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 |
| ip-192.168.24.11 (ocf::heartbeat:IPaddr2): Stopped |
| ip-10.0.0.104 (ocf::heartbeat:IPaddr2): Stopped |
| ip-172.17.1.19 (ocf::heartbeat:IPaddr2): Started controller-2 |
| ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Stopped |
| ip-172.17.3.13 (ocf::heartbeat:IPaddr2): Stopped |
| ip-172.17.4.19 (ocf::heartbeat:IPaddr2): Started controller-2 |
| Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] |
| haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 |
| haproxy-bundle-docker-1 (ocf::heartbeat:docker): Stopped |
| haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 |
| openstack-cinder-volume (systemd:openstack-cinder-volume): Stopped |
| stonith-fence_ipmilan-525400244e09 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-525400cdec10 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-525400c709f7 (stonith:fence_ipmilan): Stopped |
| stonith-fence_ipmilan-525400a7f9e0 (stonith:fence_ipmilan): Started controller-0 |
| stonith-fence_ipmilan-525400a25787 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-5254005ea387 (stonith:fence_ipmilan): Stopped |
| stonith-fence_ipmilan-525400542c06 (stonith:fence_ipmilan): Stopped |
| stonith-fence_ipmilan-525400aac413 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-525400498d34 (stonith:fence_ipmilan): Stopped |
| |
| Transition Summary: |
| * Fence (reboot) galera-bundle-2 (resource: galera-bundle-docker-2) 'guest is unclean' |
| * Fence (reboot) galera-bundle-0 (resource: galera-bundle-docker-0) 'guest is unclean' |
| * Start database-0 ( controller-0 ) |
| * Start database-2 ( controller-1 ) |
| * Recover galera-bundle-docker-0 ( database-0 ) |
| * Start galera-bundle-0 ( controller-0 ) |
| * Recover galera:0 ( Master galera-bundle-0 ) |
| * Recover galera-bundle-docker-2 ( database-2 ) |
| * Start galera-bundle-2 ( controller-1 ) |
| * Recover galera:2 ( Master galera-bundle-2 ) |
| * Promote redis:0 ( Slave -> Master redis-bundle-0 ) |
| * Start redis-bundle-docker-1 ( controller-1 ) |
| * Start redis-bundle-1 ( controller-1 ) |
| * Start redis:1 ( redis-bundle-1 ) |
| * Start ip-192.168.24.11 ( controller-0 ) |
| * Start ip-10.0.0.104 ( controller-1 ) |
| * Start ip-172.17.1.11 ( controller-0 ) |
| * Start ip-172.17.3.13 ( controller-1 ) |
| * Start haproxy-bundle-docker-1 ( controller-1 ) |
| * Start openstack-cinder-volume ( controller-0 ) |
| * Start stonith-fence_ipmilan-525400c709f7 ( controller-1 ) |
| * Start stonith-fence_ipmilan-5254005ea387 ( controller-1 ) |
| * Start stonith-fence_ipmilan-525400542c06 ( controller-0 ) |
| * Start stonith-fence_ipmilan-525400498d34 ( controller-1 ) |
| |
| Executing cluster transition: |
| * Resource action: database-0 start on controller-0 |
| * Resource action: database-2 start on controller-1 |
| * Pseudo action: redis-bundle-master_pre_notify_start_0 |
| * Resource action: stonith-fence_ipmilan-525400c709f7 start on controller-1 |
| * Resource action: stonith-fence_ipmilan-5254005ea387 start on controller-1 |
| * Resource action: stonith-fence_ipmilan-525400542c06 start on controller-0 |
| * Resource action: stonith-fence_ipmilan-525400498d34 start on controller-1 |
| * Pseudo action: redis-bundle_start_0 |
| * Pseudo action: galera-bundle_demote_0 |
| * Resource action: database-0 monitor=20000 on controller-0 |
| * Resource action: database-2 monitor=20000 on controller-1 |
| * Pseudo action: galera-bundle-master_demote_0 |
| * Resource action: redis notify on redis-bundle-0 |
| * Resource action: redis notify on redis-bundle-2 |
| * Pseudo action: redis-bundle-master_confirmed-pre_notify_start_0 |
| * Pseudo action: redis-bundle-master_start_0 |
| * Resource action: stonith-fence_ipmilan-525400c709f7 monitor=60000 on controller-1 |
| * Resource action: stonith-fence_ipmilan-5254005ea387 monitor=60000 on controller-1 |
| * Resource action: stonith-fence_ipmilan-525400542c06 monitor=60000 on controller-0 |
| * Resource action: stonith-fence_ipmilan-525400498d34 monitor=60000 on controller-1 |
| * Pseudo action: galera_demote_0 |
| * Pseudo action: galera_demote_0 |
| * Pseudo action: galera-bundle-master_demoted_0 |
| * Pseudo action: galera-bundle_demoted_0 |
| * Pseudo action: galera-bundle_stop_0 |
| * Resource action: galera-bundle-docker-0 stop on database-0 |
| * Resource action: galera-bundle-docker-2 stop on database-2 |
| * Pseudo action: stonith-galera-bundle-2-reboot on galera-bundle-2 |
| * Pseudo action: stonith-galera-bundle-0-reboot on galera-bundle-0 |
| * Pseudo action: galera-bundle-master_stop_0 |
| * Resource action: redis-bundle-docker-1 start on controller-1 |
| * Resource action: redis-bundle-1 monitor on controller-1 |
| * Resource action: ip-192.168.24.11 start on controller-0 |
| * Resource action: ip-10.0.0.104 start on controller-1 |
| * Resource action: ip-172.17.1.11 start on controller-0 |
| * Resource action: ip-172.17.3.13 start on controller-1 |
| * Resource action: openstack-cinder-volume start on controller-0 |
| * Pseudo action: haproxy-bundle_start_0 |
| * Pseudo action: galera_stop_0 |
| * Resource action: redis-bundle-docker-1 monitor=60000 on controller-1 |
| * Resource action: redis-bundle-1 start on controller-1 |
| * Resource action: ip-192.168.24.11 monitor=10000 on controller-0 |
| * Resource action: ip-10.0.0.104 monitor=10000 on controller-1 |
| * Resource action: ip-172.17.1.11 monitor=10000 on controller-0 |
| * Resource action: ip-172.17.3.13 monitor=10000 on controller-1 |
| * Resource action: haproxy-bundle-docker-1 start on controller-1 |
| * Resource action: openstack-cinder-volume monitor=60000 on controller-0 |
| * Pseudo action: haproxy-bundle_running_0 |
| * Pseudo action: galera_stop_0 |
| * Pseudo action: galera-bundle-master_stopped_0 |
| * Resource action: redis start on redis-bundle-1 |
| * Pseudo action: redis-bundle-master_running_0 |
| * Resource action: redis-bundle-1 monitor=60000 on controller-1 |
| * Resource action: haproxy-bundle-docker-1 monitor=60000 on controller-1 |
| * Pseudo action: galera-bundle_stopped_0 |
| * Pseudo action: galera-bundle_start_0 |
| * Pseudo action: galera-bundle-master_start_0 |
| * Resource action: galera-bundle-docker-0 start on database-0 |
| * Resource action: galera-bundle-0 monitor on controller-1 |
| * Resource action: galera-bundle-docker-2 start on database-2 |
| * Resource action: galera-bundle-2 monitor on controller-1 |
| * Pseudo action: redis-bundle-master_post_notify_running_0 |
| * Resource action: galera-bundle-docker-0 monitor=60000 on database-0 |
| * Resource action: galera-bundle-0 start on controller-0 |
| * Resource action: galera-bundle-docker-2 monitor=60000 on database-2 |
| * Resource action: galera-bundle-2 start on controller-1 |
| * Resource action: redis notify on redis-bundle-0 |
| * Resource action: redis notify on redis-bundle-1 |
| * Resource action: redis notify on redis-bundle-2 |
| * Pseudo action: redis-bundle-master_confirmed-post_notify_running_0 |
| * Pseudo action: redis-bundle_running_0 |
| * Resource action: galera start on galera-bundle-0 |
| * Resource action: galera start on galera-bundle-2 |
| * Pseudo action: galera-bundle-master_running_0 |
| * Resource action: galera-bundle-0 monitor=60000 on controller-0 |
| * Resource action: galera-bundle-2 monitor=60000 on controller-1 |
| * Pseudo action: redis-bundle-master_pre_notify_promote_0 |
| * Pseudo action: redis-bundle_promote_0 |
| * Pseudo action: galera-bundle_running_0 |
| * Resource action: redis notify on redis-bundle-0 |
| * Resource action: redis notify on redis-bundle-1 |
| * Resource action: redis notify on redis-bundle-2 |
| * Pseudo action: redis-bundle-master_confirmed-pre_notify_promote_0 |
| * Pseudo action: redis-bundle-master_promote_0 |
| * Pseudo action: galera-bundle_promote_0 |
| * Pseudo action: galera-bundle-master_promote_0 |
| * Resource action: redis promote on redis-bundle-0 |
| * Pseudo action: redis-bundle-master_promoted_0 |
| * Resource action: galera promote on galera-bundle-0 |
| * Resource action: galera promote on galera-bundle-2 |
| * Pseudo action: galera-bundle-master_promoted_0 |
| * Pseudo action: redis-bundle-master_post_notify_promoted_0 |
| * Pseudo action: galera-bundle_promoted_0 |
| * Resource action: galera monitor=10000 on galera-bundle-0 |
| * Resource action: galera monitor=10000 on galera-bundle-2 |
| * Resource action: redis notify on redis-bundle-0 |
| * Resource action: redis notify on redis-bundle-1 |
| * Resource action: redis notify on redis-bundle-2 |
| * Pseudo action: redis-bundle-master_confirmed-post_notify_promoted_0 |
| * Pseudo action: redis-bundle_promoted_0 |
| * Resource action: redis monitor=20000 on redis-bundle-0 |
| * Resource action: redis monitor=60000 on redis-bundle-1 |
| * Resource action: redis monitor=45000 on redis-bundle-1 |
| |
| Revised cluster status: |
| Online: [ controller-0 controller-1 controller-2 ] |
| RemoteOnline: [ database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 ] |
| Containers: [ galera-bundle-0:galera-bundle-docker-0 galera-bundle-1:galera-bundle-docker-1 galera-bundle-2:galera-bundle-docker-2 rabbitmq-bundle-0:rabbitmq-bundle-docker-0 rabbitmq-bundle-1:rabbitmq-bundle-docker-1 rabbitmq-bundle-2:rabbitmq-bundle-docker-2 redis-bundle-0:redis-bundle-docker-0 redis-bundle-1:redis-bundle-docker-1 redis-bundle-2:redis-bundle-docker-2 ] |
| |
| database-0 (ocf::pacemaker:remote): Started controller-0 |
| database-1 (ocf::pacemaker:remote): Started controller-2 |
| database-2 (ocf::pacemaker:remote): Started controller-1 |
| messaging-0 (ocf::pacemaker:remote): Started controller-2 |
| messaging-1 (ocf::pacemaker:remote): Started controller-2 |
| messaging-2 (ocf::pacemaker:remote): Started controller-2 |
| Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] |
| rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started messaging-0 |
| rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started messaging-1 |
| rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started messaging-2 |
| Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] |
| galera-bundle-0 (ocf::heartbeat:galera): Master database-0 |
| galera-bundle-1 (ocf::heartbeat:galera): Master database-1 |
| galera-bundle-2 (ocf::heartbeat:galera): Master database-2 |
| Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] |
| redis-bundle-0 (ocf::heartbeat:redis): Master controller-0 |
| redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1 |
| redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 |
| ip-192.168.24.11 (ocf::heartbeat:IPaddr2): Started controller-0 |
| ip-10.0.0.104 (ocf::heartbeat:IPaddr2): Started controller-1 |
| ip-172.17.1.19 (ocf::heartbeat:IPaddr2): Started controller-2 |
| ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Started controller-0 |
| ip-172.17.3.13 (ocf::heartbeat:IPaddr2): Started controller-1 |
| ip-172.17.4.19 (ocf::heartbeat:IPaddr2): Started controller-2 |
| Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] |
| haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 |
| haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-1 |
| haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 |
| openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-0 |
| stonith-fence_ipmilan-525400244e09 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-525400cdec10 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-525400c709f7 (stonith:fence_ipmilan): Started controller-1 |
| stonith-fence_ipmilan-525400a7f9e0 (stonith:fence_ipmilan): Started controller-0 |
| stonith-fence_ipmilan-525400a25787 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-5254005ea387 (stonith:fence_ipmilan): Started controller-1 |
| stonith-fence_ipmilan-525400542c06 (stonith:fence_ipmilan): Started controller-0 |
| stonith-fence_ipmilan-525400aac413 (stonith:fence_ipmilan): Started controller-2 |
| stonith-fence_ipmilan-525400498d34 (stonith:fence_ipmilan): Started controller-1 |
| |
