Openfhift provider hawkular validate success but cannot get data


#1

Hi,
I create a openshift origin cluster + hawkular metricse, I add this into manageIQ but the hawkular cannot get data.
I can read the metrics from openshift console


I can get the hawkular info on manageIQ host through cli.

The ManageIQ hawkular verify is success, but it cannot get any data, evm log show hawkular Time out to connecting to server.

My environment
ManageIQ fine-3
openshift origin 3.6


#2

cc @bazulay @simon3z


#3

Hi,

To better understand what is happening, you can do two more tests using curl:

In this tests:
HAWKULAR_HOST=[ THE HAWKULAE PATH ] # e.g. metrics.example.com:443
HAWKULAR_TENANT=_system
HAWKULAR_TOKEN=[ THE OPENSHIFT TOKEN ]

a. Status query that does not require authentication:

curl -k \
-H “Hawkular-Tenant: $HAWKULAR_TENANT” \
https://$HAWKULAR_HOST/hawkular/metrics/status

b. Mettrics query that does require authentication:

curl -k \
-H “Hawkular-Tenant: $HAWKULAR_TENANT” \
-H “Authorization: Bearer $HAWKULAR_TOKEN” \
https://$HAWKULAR_HOST/hawkular/metrics/metrics

This tests may give us more data to understand the problem. [ If you think adding -v to the curl is useful please do, but I do not think it will add useful data. ]


#4

Hi, Thanks your reply.
There are the result.
env var

[root@localhost vmdb]# echo $HAWKULAR_TENANT 
_system
[root@localhost vmdb]# echo $HAWKULAR_TOKEN
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4taHJ2c2giLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijk2NzJjYTUyLWM5MTEtMTFlNy1iOTc1LTA4MDAyNzFiYzA2OCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYW5hZ2VtZW50LWluZnJhOm1hbmFnZW1lbnQtYWRtaW4ifQ.pOG9vkz2qaMzF7PKbH-s8APUn7Oo7vv0-yxmWZE_bOOK-J6-Gdq3Q87_BJ7OS9-pmwi7QqnFu1uXzkY7eV6l1COWWZ3udvq2ovf-v2DRSup_2S26dO0-RO1ozItISSFHBKJ636CqF3cePQYm_o8bAeg3LUirTx5uBDYC9AcxrwF3uX9v8ZbqtFHc7lb3MtOP_wWFFUIIdbJ7IIjILNFXsv_cbbOjWSWQm6ZZT1LPvooP3waf3wpkCj-n74CeuSGWwpUP7aI4u7unIjEO_OwewwiZAyH4KrKTMGyyTui01q2KkigcbKhboEAlQbewLncrPIAPhieTW6FaVruoxP42MA
[root@localhost vmdb]# echo $HAWKULAR_HOST 
hawkular-metrics.apps.oc.com

Curl result
a. curl -k -H "Hawkular-Tenant: $HAWKULAR_TENANT" https://$HAWKULAR_HOST/hawkular/metrics/status

{"MetricsService":"STARTED","Implementation-Version":"0.26.1.Final","Built-From-Git-SHA1":"45b148c834ed62018f153c23187b4436ae4208fe","Cassandra":"up"}

b. curl -k -H "Hawkular-Tenant: $HAWKULAR_TENANT" -H "Authorization: Bearer $HAWKULAR_TOKEN" https://$HAWKULAR_HOST/hawkular/metrics/metrics


[{"id":"cluster/cpu/limit","tags":{"descriptor_name":"cpu/limit","group_id":"/cpu/limit","type":"cluster"},"dataRetention":7,"type":"gauge","tenantId":"_system"},{"id":"cluster/cpu/request","tags":{"descri
ptor_name":"cpu/request","group_id":"/cpu/request","type":"cluster"},"dataRetention":7,"type":"gauge","tenantId":"_system"},{"id":"cluster/cpu/usage_rate","tags":{"descriptor_name":"cpu/usage_rate","group_
id":"/cpu/usage_rate","type":"cluster"},"dataRetention":7,"type":"gauge","tenantId":"_system"},{"id":"cluster/memory/limit","tags":{"descriptor_name":"memory/limit","group_id":"/memory/limit","type":"clust
er","units":"bytes"},"dataRetention":7,"type":"gauge","tenantId":"_system"},{"id":"cluster/memory/request","tags":{"descriptor_name":"memory/request","group_id":"/memory/request","type":"cluster","units":"
bytes"},"dataRetention":7,"type":"gauge","tenantId":"_system"},{"id":"cluster/memory/usage","tags":{
...skip

After test, I found add the ip and hostname map in /etc/hosts let manageIQ can get the data.

#/etc/hosts
# the openshift router host
172.22.2.188 hawkular-metrics.apps.oc.com

#5

Did that solve the problem ?


#6

Yes, but manageiq should use DNS to resolve the hostname not use /etc/hosts.


#7

a. Thanks for the issue !
b. @bazulay who do you think should look into the dns issue ?


#8

Well DNS service is a part of the environment,
In case the DNS is not set properly on the DNS server than /etc/hosts is the solution.