OpenStack の ダッシュボードにアクセスできなくなった時の対処メモ

OpenStack の ダッシュボードにアクセスできなくなったのでその時の対処メモ。

■ 概要

OS を立ち上げ直したらなぜか OpenStack dashboard にアクセスできなくなった。
結論から言えば下記サイトの answer 1 の通りにやったら直った。
https://ask.openstack.org/en/question/95308/dashboard-uncontactable/

それだけの話だけどその時のログを残す。

環境:
 CentOS 7
 OpenStack Mitaka

■ いくつかのコンポーネントが上手く立ち上がらなかった

VMware で動かしている CentOS をいつも通り起動して OpenStack のダッシュボードにアクセスしたら入れなかった。
特に前回シャットダウンするときも変なことをした覚えはなく、恐らく単にコンポーネントの立ち上げに失敗しただけと思われる。
もう一度OS再起動したら直りそうな気もするけど、せっかくなのでなんとかしてみる。

とりあえず状況を確認する。

# openstack-status 
== Nova services ==
openstack-nova-api:                     active
openstack-nova-compute:                 active
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-cert:                    active
openstack-nova-conductor:               active
openstack-nova-console:                 inactive  (disabled on boot)
openstack-nova-consoleauth:             active
openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active
== Keystone service ==
openstack-keystone:                     inactive  (disabled on boot)
== Horizon service ==
openstack-dashboard:                    uncontactable
== neutron services ==
neutron-server:                         failed
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    active
neutron-lbaasv2-agent:                  inactive  (disabled on boot)
neutron-openvswitch-agent:              active
neutron-metering-agent:                 active
== Swift services ==
openstack-swift-proxy:                  active
openstack-swift-account:                active
openstack-swift-container:              active
openstack-swift-object:                 active
== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                active
openstack-cinder-backup:                active
== Ceilometer services ==
openstack-ceilometer-api:               inactive  (disabled on boot)
openstack-ceilometer-central:           active
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         active
openstack-ceilometer-notification:      active
== Heat services ==
openstack-heat-api:                     active
openstack-heat-api-cfn:                 inactive  (disabled on boot)
openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
openstack-heat-engine:                  active
== Support services ==
mariadb:                                active
openvswitch:                            active
dbus:                                   active
target:                                 active
rabbitmq-server:                        active
memcached:                              active
== Keystone users ==
Warning keystonerc not sourced
#

openstack-dashboard: が uncontactable になってるし
neutron-server: も failed になってる。
そりゃダッシュボードにアクセスできんわ。

■ failed と uncontactable のコンポーネントの状態を直す

まずは neutron-server を restart してみる。

# systemctl restart  neutron-server.service
# openstack-status 
== Nova services ==
openstack-nova-api:                     active
openstack-nova-compute:                 active
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-cert:                    active
openstack-nova-conductor:               active
openstack-nova-console:                 inactive  (disabled on boot)
openstack-nova-consoleauth:             active
openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active
== Keystone service ==
openstack-keystone:                     inactive  (disabled on boot)
== Horizon service ==
openstack-dashboard:                    uncontactable
== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    active
neutron-lbaasv2-agent:                  inactive  (disabled on boot)
neutron-openvswitch-agent:              active
neutron-metering-agent:                 active
== Swift services ==
openstack-swift-proxy:                  active
openstack-swift-account:                active
openstack-swift-container:              active
openstack-swift-object:                 active
== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                active
openstack-cinder-backup:                active
== Ceilometer services ==
openstack-ceilometer-api:               inactive  (disabled on boot)
openstack-ceilometer-central:           active
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         active
openstack-ceilometer-notification:      active
== Heat services ==
openstack-heat-api:                     active
openstack-heat-api-cfn:                 inactive  (disabled on boot)
openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
openstack-heat-engine:                  active
== Support services ==
mariadb:                                active
openvswitch:                            active
dbus:                                   active
target:                                 active
rabbitmq-server:                        active
memcached:                              active
== Keystone users ==
Warning keystonerc not sourced
#

neutron-server は active になった。
openstack-dashboard もそれに引きずられて自動的に直ることを期待したけどそんなこともなく。

openstack-dashboard も同じように restart してみる。

# systemctl restart  openstack-dashboard.service
Failed to restart openstack-dashboard.service: Unit openstack-dashboard.service failed to load: No such file or directory.
# 

...もしかして openstack-dashboard は単体でどうこうするものではないのか ?
その辺もまだ理解していない。

その後色々試して格闘したがどうにもならず。。
ググっていった結果、このページに辿り着いた。
思考停止で answer 1 の通りにやってみる。

# systemctl restart rabbitmq-server
# systemctl restart openstack-keystone
You have new mail in /var/spool/mail/root
#
# ls -l /etc/systemd/system/
total 16
drwxr-xr-x. 2 root root   53 Nov  3 17:34 basic.target.wants
drwxr-xr-x. 2 root root   30 Nov  3 15:39 bluetooth.target.wants
lrwxrwxrwx. 1 root root   41 Nov  3 15:39 dbus-org.bluez.service -> /usr/lib/systemd/system/bluetooth.service
lrwxrwxrwx. 1 root root   44 Nov  3 15:40 dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx. 1 root root   44 Nov  3 15:41 dbus-org.freedesktop.ModemManager1.service -> /usr/lib/systemd/system/ModemManager.service
lrwxrwxrwx. 1 root root   46 Nov  3 15:39 dbus-org.freedesktop.NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root   57 Nov  3 15:39 dbus-org.freedesktop.nm-dispatcher.service -> /usr/lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx. 1 root root   36 Nov  3 15:44 default.target -> /lib/systemd/system/graphical.target
drwxr-xr-x. 2 root root   85 Nov  3 15:38 default.target.wants
drwxr-xr-x. 2 root root   37 Nov  3 15:40 dev-virtio\x2dports-org.qemu.guest_agent.0.device.wants
lrwxrwxrwx. 1 root root   35 Nov  3 15:40 display-manager.service -> /usr/lib/systemd/system/gdm.service
drwxr-xr-x. 2 root root   31 Nov  3 15:38 getty.target.wants
drwxr-xr-x. 2 root root   63 Nov  3 15:39 graphical.target.wants
drwxr-xr-x. 2 root root 4096 Nov  5 19:27 multi-user.target.wants
drwxr-xr-x. 2 root root   38 Nov  3 18:03 openstack-cinder-volume.service.requires
drwxr-xr-x. 2 root root   25 Nov  3 15:39 printer.target.wants
drwxr-xr-x. 2 root root   24 Nov  3 17:40 rabbitmq-server.service.d
drwxr-xr-x. 2 root root   23 Nov  3 18:21 redis-sentinel.service.d
drwxr-xr-x. 2 root root   23 Nov  3 18:21 redis.service.d
drwxr-xr-x. 2 root root   30 Nov  3 15:39 remote-fs.target.wants
drwxr-xr-x. 2 root root 4096 Nov  5 19:16 sockets.target.wants
drwxr-xr-x. 2 root root   35 Nov  3 15:40 spice-vdagentd.target.wants
drwxr-xr-x. 2 root root   25 Nov  3 16:34 sssd.service.d
drwxr-xr-x. 2 root root 4096 Nov  5 19:16 sysinit.target.wants
drwxr-xr-x. 2 root root   43 Nov  3 15:38 system-update.target.wants
# 
# 
# ln -s /usr/lib/systemd/system/httpd.service /etc/systemd/system/openstack-keystone.service
#
# ls -l /etc/systemd/system/open*
lrwxrwxrwx. 1 root root 37 Nov 12 15:38 /etc/systemd/system/openstack-keystone.service -> /usr/lib/systemd/system/httpd.service

/etc/systemd/system/openstack-cinder-volume.service.requires:
total 0
lrwxrwxrwx. 1 root root 49 Nov  3 18:03 openstack-losetup.service -> /usr/lib/systemd/system/openstack-losetup.service
# systemctl daemon-reload
# systemctl stop httpd.service
# systemctl stop openstack-keystone.service
# systemctl start openstack-keystone.service
# 
# openstack-status 
== Nova services ==
openstack-nova-api:                     active
openstack-nova-compute:                 active
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-cert:                    active
openstack-nova-conductor:               active
openstack-nova-console:                 inactive  (disabled on boot)
openstack-nova-consoleauth:             active
openstack-nova-xvpvncproxy:             inactive  (disabled on boot)
== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active
== Keystone service ==
openstack-keystone:                     active
== Horizon service ==
openstack-dashboard:                    active
== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    active
neutron-lbaasv2-agent:                  inactive  (disabled on boot)
neutron-openvswitch-agent:              active
neutron-metering-agent:                 active
== Swift services ==
openstack-swift-proxy:                  active
openstack-swift-account:                active
openstack-swift-container:              active
openstack-swift-object:                 active
== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                active
openstack-cinder-backup:                active
== Ceilometer services ==
openstack-ceilometer-api:               inactive  (disabled on boot)
openstack-ceilometer-central:           active
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         active
openstack-ceilometer-notification:      active
== Heat services ==
openstack-heat-api:                     active
openstack-heat-api-cfn:                 inactive  (disabled on boot)
openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
openstack-heat-engine:                  active
== Support services ==
mariadb:                                active
openvswitch:                            active
dbus:                                   active
target:                                 active
rabbitmq-server:                        active
memcached:                              active
== Keystone users ==
Warning keystonerc not sourced
#

openstack-dashboard: が active になってる!
ダッシュボードにもアクセスできるようになった。

httpd の停止と httpd.service へリンクする openstack-keystone.service の作成がポイントだったようだ。
#これも何となくしか理解できていないが......。

しかし、今までこんな問題が発生しなかったのになんで起きたのかは分からず終い。