How to discard / flush queued messages

euwe

#1

Hi,

TL;DR: I’m trying to recover from an outage in C&U which left our environment with ~500K messages in ready state. As this is a dev / test environment, we can choose to discard this 3 month worth of C&U data. Can anybody has a hint on how to discard these messages?

Environment: Euwe (cfme 5.7.2 to be precise), RHV 4.1 provider with 12 Hosts and ~200 VMs. 1 Master appliance with DB, 2 worker appliances on RHV zone.

History: A network policy left us with no connection to RHV’s ovirt_engine_history for 3 months, and after the connection was re-established, we found out we have way too much (and increasing) messages for our metrics collector workers to process:

# cat evm.log | grep 'count for state=\["ready"\]' | grep -Eho '"RHV Zone"=>{"ems_metrics_collector"=>[[:digit:]]*'
"RHV Zone"=>{"ems_metrics_collector"=>465696
"RHV Zone"=>{"ems_metrics_collector"=>465828

# cat evm.log | grep perf_capture_health_check
[----] I, [2017-11-07T22:09:15.463774 #14914:5b5130]  INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 460854 "realtime" captures on the queue for zone [RHV Zone] - oldest: [2017-06-22T16:02:53Z], recent: [2017-11-07T19:06:48Z]
[----] I, [2017-11-07T22:09:15.464286 #14914:5b5130]  INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 5484 "hourly" captures on the queue for zone [RHV Zone] - oldest: [2017-11-07T00:17:09Z], recent: [2017-11-07T19:06:36Z]
[----] I, [2017-11-07T22:09:15.464453 #14914:5b5130]  INFO -- : MIQ(Metric::Capture.perf_capture_health_check) 0 "historical" captures on the queue for zone [RHV Zone]

We’ve increased workers and number of appliances (9 collector workers with 1 GB mem. on 2 appliances each), but this did not help much on the increase rate. As the environment is non-prod and past C&U data can be discarded, I’m considering to flush these C&U messages and start from scratch, instead waiting for the workers to eat up these messages.

Any suggestions?

Thanks,
Ekin.