No Disk IO or CPU Graphs are shown on the Cluster, and Host level and no graphs are shown for the datastore

Ashraf · August 15, 2018, 10:54am

I have integrated ManageIQ with RHEVM, and configured the C&U for the region to collect everything for clusters and datastore as per the image below.
I can see all the perfromance graphs are available on the VM level, but the CPU, and the DISK I/O are missing on the Cluster, and host level, also most of the perfromance graphs on the datastore level is missing as per the screenshots below, any advice please to see all the available graphs?
I have this issue on both gaprindashvili 2 and 3
C&U:

VM:

Cluster:

Host:

Datastore:

pemcg · August 17, 2018, 5:10pm

Hi Ashraf

Try using the perf_process_timings.rb script from here to look at your performance timings, it’ll also give you an idea if any are failing. You should be able to see whether any host performance records are being returned from the data collection (they’ll be Object type: Host)

Ashraf · August 20, 2018, 5:01pm

Here is the output I do not see any error, here is a sample of the output for one host, and storage, what do you think:

---
Worker PID:                    2029
Message ID:                    1000002058649
Message fetch time:            2018-08-20T05:00:36.115984
Message time in queue:         8.216479262 seconds
Provider:                      
Object type:                   Storage
Object name:                   Host1
Metrics processing start time: 2018-08-20T05:00:36.117243
Time range:                    Hourly
Rows added:                    
Rows updated:                  
Capture state:                 storage_capture_complete
Capture timings:
  db_find_storage_files:               0.005377 seconds
  capture_state:                       0.021347 seconds
  init_attrs:                          0.005770 seconds
  db_find_prev_perfs:                  0.004630 seconds
  process_perfs:                       0.197792 seconds
  process_bottleneck:                  1.188092 seconds
  total_time:                          1.448956 seconds
Metrics processing end time:   2018-08-20T05:00:37.566350
Message delivered time:        2018-08-20T05:00:37.569216
Message state:                 ok
Message delivered in:          1.453088007 seconds
---

---
Worker PID:                    2029
Message ID:                    1000002058651
Message fetch time:            2018-08-20T05:00:37.578996
Message time in queue:         9.668919094 seconds
Provider:                      
Object type:                   Storage
Object name:                   Storage-01
Metrics processing start time: 2018-08-20T05:00:37.580164
Time range:                    Hourly
Rows added:                    
Rows updated:                  
Capture state:                 storage_capture_complete
Capture timings:
  db_find_storage_files:               0.007766 seconds
  capture_state:                       0.022904 seconds
  init_attrs:                          0.005522 seconds
  db_find_prev_perfs:                  0.005683 seconds
  process_perfs:                       0.844841 seconds
  process_bottleneck:                  1.205171 seconds
  total_time:                          2.118012 seconds
Metrics processing end time:   2018-08-20T05:00:39.698334
Message delivered time:        2018-08-20T05:00:39.698528
Message state:                 ok
Message delivered in:          2.11939757 seconds
---

pemcg · August 21, 2018, 4:23pm

that looks like the output from perf_rollup_timings?

Ashraf · August 24, 2018, 2:52pm

Here is the output from perf_rollup_timings.rb

---
Worker PID:                    2072
Message ID:                    1000002133716
Message fetch time:            2018-08-24T04:00:51.887927
Message time in queue:         16.122682264 seconds
Rollup processing start time:  2018-08-24T04:00:51.889965
Object Type:                   ManageIQ::Providers::Redhat::InfraManager::Host
Object Name:                   host08
Rollup Type:                   hourly
Time:                          2018-08-24T01:00:00Z
Rollup timings:
  db_find_prev_perf:                   0.002761 seconds
  rollup_perfs:                        0.075810 seconds
  db_update_perf:                      0.010886 seconds
  process_bottleneck:                  2.202615 seconds
  total_time:                          2.314269 seconds
Rollup processing end time:    2018-08-24T04:00:54.204370
Message delivered time:        2018-08-24T04:00:54.204611
Message state:                 ok
Message delivered in:          2.316553695 seconds
---

Here is the output from perf_process_timings.rb

---

Worker PID:                    1991
Message ID:                    1000002133865
Message fetch time:            2018-08-24T04:03:29.383398
Message time in queue:         12.238611697 seconds
Provider:                      Redhat::InfraManager
Object type:                   Host
Object name:                   host08
Metrics processing start time: 2018-08-24T04:03:29.385852
Time range:                    2018-08-24T02:01:00Z - 2018-08-24T02:03:40Z
Rows added:                    
Rows updated:                  
Capture state:                 perf_capture_complete
Capture timings:
  capture_state:                       0.002164 seconds
  rhevm_connect:                       0.003417 seconds
  collect_data:                        0.330949 seconds
  total_time:                          0.340823 seconds
Process timings:
  process_counter_values:              0.000399 seconds
  db_find_prev_perfs:                  0.007709 seconds
  preload_vim_performance_state_for_ts: 0.002426 seconds
  process_perfs:                       0.004404 seconds
  process_build_ics:                   0.000517 seconds
  process_perfs_db:                    0.019837 seconds
  total_time:                          0.049761 seconds
Metrics processing end time:   2018-08-24T04:03:29.779161
Message delivered time:        2018-08-24T04:03:29.779352
Message state:                 ok
Message delivered in:          0.395820182 seconds
---

What do you think the problem is?

pemcg · August 29, 2018, 8:48am

It looks like your metrics collectors are working fine, but your results have pointed out to me a bug in perf_process_timings - you should have some values in the ‘Rows added’ or Rows updated’ lines which indicate how many metrics were updated for the object.

I’ve just updated the script for CFME 5.9/MIQ Gaprindashvili, could you possibly re-run perf_process_timings.rb to see if these lines contain valid numbers?

Thanks.

Ashraf · August 29, 2018, 2:03pm

Here is the output of the new script, I see the Rows Added and Update are empty in storage but not in case of the host, so why I am getting empty graph for “Virtual Machine CPU State” if I am collecting for VMs and Hosts, and why I am not collecting for storages (Rows added and updated are empty for all storages):

---
Worker PID:                    1991
Message ID:                    1000002229882
Message fetch time:            2018-08-29T06:17:03.422379
Message time in queue:         21.394569386 seconds
Provider:                      Redhat::InfraManager
Object type:                   Host
Object name:                   Host01
Metrics processing start time: 2018-08-29T06:17:03.424745
Time range:                    2018-08-29T03:27:00Z - 2018-08-29T04:17:40Z
Rows added:                    153
Rows updated:                  0
Capture state:                 perf_capture_complete
Capture timings:
  capture_state:                         0.022899 seconds
  rhevm_connect:                         0.003277 seconds
  collect_data:                          1.108931 seconds
  total_time:                            1.139823 seconds
Process timings:
  process_counter_values:                0.005072 seconds
  db_find_prev_perfs:                    0.007332 seconds
  preload_vim_performance_state_for_ts:  0.002061 seconds
  process_perfs:                         0.025994 seconds
  process_build_ics:                     0.005153 seconds
  process_perfs_db:                      0.116226 seconds
  total_time:                            0.185213 seconds
Metrics processing end time:   2018-08-29T06:17:04.752619
Message delivered time:        2018-08-29T06:17:04.752919
Message state:                 ok
Message delivered in:          1.330389329 seconds
---
---
Worker PID:                    1991
Message ID:                    1000002227912
Message fetch time:            2018-08-29T03:43:54.575061
Message time in queue:         26.594707605 seconds
Provider:                      Redhat::InfraManager
Object type:                   Vm
Object name:                   VM01
Metrics processing start time: 2018-08-29T03:43:54.578473
Time range:                    2018-08-29T00:53:01Z - 2018-08-29T01:43:40Z
Rows added:                    150
Rows updated:                  0
Capture state:                 perf_capture_complete
Capture timings:
  capture_state:                         0.002255 seconds
  rhevm_connect:                         0.006818 seconds
  collect_data:                          0.111801 seconds
  total_time:                            0.125744 seconds
Process timings:
  process_counter_values:                0.011921 seconds
  db_find_prev_perfs:                    0.006397 seconds
  preload_vim_performance_state_for_ts:  0.001991 seconds
  process_perfs:                         0.047823 seconds
  process_build_ics:                     0.006231 seconds
  process_perfs_db:                      0.113464 seconds
  total_time:                            0.220854 seconds
Metrics processing end time:   2018-08-29T03:43:54.928439
Message delivered time:        2018-08-29T03:43:54.928631
Message state:                 ok
Message delivered in:          0.35345337 seconds
---
---
Worker PID:                    1991
Message ID:                    1000002228278
Message fetch time:            2018-08-29T04:01:46.596654
Message time in queue:         13.500020045 seconds
Provider:                      
Object type:                   Storage
Object name:                   Storage01
Metrics processing start time: 2018-08-29T04:01:46.599988
Time range:                    Hourly
Rows added:                    
Rows updated:                  
Capture state:                 storage_capture_complete
Capture timings:
  db_find_storage_files:                 0.004048 seconds
  capture_state:                         0.027160 seconds
  init_attrs:                            0.007111 seconds
  db_find_prev_perfs:                    0.002839 seconds
  process_perfs:                         0.010460 seconds
  process_bottleneck:                    1.251064 seconds
  total_time:                            1.340289 seconds
Metrics processing end time:   2018-08-29T04:01:47.940460
Message delivered time:        2018-08-29T04:01:47.940671
Message state:                 ok
Message delivered in:          1.343833078 seconds
---

pemcg · August 30, 2018, 1:45pm

The reason that Rows added and updated are empty for all storages was down to laziness on my part and not filtering out those lines then they’re nil . The rows added/updated aren’t included in the log file for storage metrics which is why there not available. I’ve further updated the script to be less lazy.

It does look however like the C&U components in ManageIQ are all working as they are supposed to however, and looking closely at your graphs (and comparing them to my systems) I think you are getting valid results. If you look at the ‘y’ axis for the memory graph for the host it starts at about 60GB rather than zero. I think perhaps the graph scaling is the confusion.

Similarly on my system the CPU utilisation barely registers above the ‘x’ axis but I think that’s because the CPUs aren’t all that loaded. I think perhaps the same is for your host?

Ashraf · August 30, 2018, 3:40pm

Actually I have some traffic which should be reflected on the CPU, and DISK, because even in the tables below in the screenshots you see that all records are zeros

Ashraf · August 30, 2018, 3:42pm

Do you think that the RHVM is not reporting these 2 values, that is why these 2 graphs are empty?