Local Database Server: initialized and stopped

Hello,

Contractors set up our instance of MIQ and left us some OVA files. Using these OVA files, I created a dev instance of MIQ. When I ssh into the VM, run the appliance_console, and choose to view the summary information, this is what I get:

IPv4 Address:            10.###.##.##/255.###.##
IPv4 Gateway:            10.###.##.#
IPv6 Address:
IPV6 Gateway:
Primary DNS:             10.###.##.#
Secondary DNS:
Search Order:            chq.com  c
MAC Address:             00:##:##:##:##:##
Timezone:                America/New_York
Local Database Server:   initialized and stopped
ManageIQ Server:         not running
ManageIQ Database:       localhost
Database/Region:         vmdb_production / 0
External Auth:           not configured
ManageIQ Version:        ivanchuk-2

the initialized and stopped and not running are what bothers me.

Does anyone have any pointers where I can being to sort this out? I do recall last time I worked on these machines I was trying to configure the database connections.

By the way, this is the summary info from the db server:

Hostname:                localhost.localdomain
IPv4 Address:            10.##.##.##/255.###.###.#
IPv4 Gateway:            10.###.##.#
IPv6 Address:
IPV6 Gateway:
Primary DNS:             10.###.#.##
Secondary DNS:
Search Order:            chq.com
MAC Address:             00:##:##:##:#:##
Timezone:                America/New_York
Local Database Server:   running (primary)
ManageIQ Server:         not running
ManageIQ Database:       localhost
Database/Region:         vmdb_production / 0
External Auth:           not configured
ManageIQ Version:        ivanchuk-2

It says "Local database server: running (primary) but the app server says the database server is not running / initialized and stopped?

Any pointers? Thank you!

If you try to start the server, do you see any errors in production.log or evm.log?

Hello @pemcg,

Thanks for asking. I have discovered that the issue appears to be with just one of the app servers. (In total there are two app servers and two db servers). Here’s what I found in logs when I try to start the server process:

production.log is completely empty. evm.log printed the following upon starting the server process:

[----] I, [2020-08-13T15:00:30.116750 #12385:a145f0]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:00:30.116951 #12385:a145f0]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:00:30.241916 #12385:a145f0]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:00:30.594402 #12385:a145f0]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12385, ENV['EVMSERVER']
:
[----] I, [2020-08-13T15:00:32.290699 #12385:a145f0]  INFO -- : MIQ(EvmApplication.stop) EVM Shutdown initiated
[----] I, [2020-08-13T15:00:50.346680 #12405:7545f4]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:00:50.347011 #12405:7545f4]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:00:50.484317 #12405:7545f4]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:00:50.836621 #12405:7545f4]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12405, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:01:00.621650 #12420:a88608]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:01:00.621864 #12420:a88608]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:01:00.751907 #12420:a88608]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:01:01.186779 #12420:a88608]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12420, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:01:02.991715 #12420:a88608]  INFO -- : MIQ(EvmApplication.stop) EVM Shutdown initiated
[----] I, [2020-08-13T15:01:39.241737 #12472:a125e8]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:01:39.242036 #12472:a125e8]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:01:39.494372 #12472:a125e8]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:01:39.927650 #12472:a125e8]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: bin/rails, PID: 12472, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:01:42.078332 #2070:7d25f8]  INFO -- : EvmWatchdog - Detected that the database is now available.
[----] I, [2020-08-13T15:01:42.078438 #2070:7d25f8]  INFO -- : EvmWatchdog - Starting EVM server...
[----] I, [2020-08-13T15:01:50.394052 #12487:676600]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:01:50.394336 #12487:676600]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:01:50.528367 #12487:676600]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:01:50.921905 #12487:676600]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12487, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:02:00.948119 #12502:78e600]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:02:00.948332 #12502:78e600]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:02:01.077164 #12502:78e600]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:02:01.437356 #12502:78e600]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12502, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:02:03.115645 #12502:78e600]  INFO -- : MIQ(EvmApplication.stop) EVM Shutdown initiated
[----] I, [2020-08-13T15:03:09.597812 #12522:9e6600]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:03:09.598058 #12522:9e6600]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:03:09.821192 #12522:9e6600]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:03:10.173320 #12522:9e6600]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: bin/rails, PID: 12522, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:03:11.788518 #2070:7d25f8]  INFO -- : EvmWatchdog - Detected that the database is now available.
[----] I, [2020-08-13T15:03:11.788596 #2070:7d25f8]  INFO -- : EvmWatchdog - Starting EVM server...
[----] I, [2020-08-13T15:03:19.038525 #12537:eba5f0]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:03:19.038763 #12537:eba5f0]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:03:19.162511 #12537:eba5f0]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:03:19.505436 #12537:eba5f0]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12537, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:03:28.777681 #12552:98a5f8]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
[----] I, [2020-08-13T15:03:28.777899 #12552:98a5f8]  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for vim.log has been changed to [WARN]
[----] I, [2020-08-13T15:03:28.904663 #12552:98a5f8]  INFO -- : MIQ(SessionStore) Using session_store: ActionDispatch::Session::MemCacheStore
[----] I, [2020-08-13T15:03:29.231708 #12552:98a5f8]  INFO -- : MIQ(Vmdb::Initializer.init) - Program Name: /usr/local/bin/rake, PID: 12552, ENV['EVMSERVER']:
[----] I, [2020-08-13T15:03:30.901365 #12552:98a5f8]  INFO -- : MIQ(EvmApplication.stop) EVM Shutdown initiated


I don’t see anything particularly enlightening in those entries. I suspect that a more verbose log level might help. In the docs it says one can set debug trace to true, but that one should only do this when directed by Red Hat support. Am I gonna break my environment if I try it?