How to success get Kubernetes provider metric by hawklur?


#1

Hi all,
I add kubernetes provider in manageiq and enable C&U role, but the metric is nil.
In the evm.log show

[----] E, [2017-10-30T03:52:41.825814 #16603:9fd134] ERROR -- : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::MetricsCapture#perf_collect_metrics) Hawkular metrics service unavailable: Failed to o
pen TCP connection to 172.22.131.55:5000 (Connection refused - connect(2) for "172.22.131.55" port 5000)

It need hawuklar, but there is not enough document for it, there only openshift document.


#2

@bazulay @simon3z


#3

Hi,

I do not know of any supported way to install Hawkualr + Heapster on Kubernetes, The methods I know assume you have roles and permissions created when installing openshift.

But you can install Hawkular + Heapster on a Kubernetes cluster. The setup we use in ManageIQ metrics is Heapster with source kubernetes.summary_api and sink hawkular-metrics.

a. Do you have Hawkular + Heapster configured and you want to access this metrics via ManageIQ ?
b. Do you need help setting up Hawkualr + Heapster on your Kubernetes cluster ?


#4

Hi, Yaacov_Zamir
a. No, I don’t have hawkular and heapster configured.
b. Yes, I need help.

Thanks your help.


#5

Do you need this for production or for proof of concept ?

We do not support metrics on k8s for production

If we are on none production land :slight_smile: -

Hawkular is a big thing ( uses Cassandra ) that has a lot of features you may not need …

@MilkBotttle : Do you need Hawkular, or you can settle for something smaller with less features ? ( e.g. no persistence, no alerts, only metrics for ManageIQ )

A smaller metrics engine like prometheus or mohawk may be easier to install, but will have less features.

@abonas : Do you know who can help with installing a TSDB (Hawkular/Prometheus/Mohawk) on k8s, all the methods I know relay on secrets / rolls installed using the openshift-ansible playbook that install openshift ?

We do not have any ready made template for that, it will be time consuming …


#6

@theute afaik there’s no doc/template for k8s, is that right?


#7

Correct, only for OpenShift


#8

@Yaacov_Zamir

I use for proof of concept.
The other metric method is fine.

Thanks


#9

Great,

I do not have a k8s setup, so I can not at this time check if it works … I will spin up one, and try to help.

p.s.

The easiest way ( that I know of ) is to install Heapster with Mohawk.

Below is a link to a Openshift yaml file for Heapster + Mohawk:

Changes needed:

  • remove the template part, k8s does not have templates :slight_smile:
  • remove the route , k8s does not have routes
  • create a secret with two files tls.key and tls.crt and use it instead of the openshift secret.

If you can translate it to a k8s yaml file, please send me a link to the translation :slight_smile:


#10

Hi,

I used minikube, credentials and routing may different when using a “real” k8s deploy.
I used my key and cert to deploy Mohawk, you may want to use your credential files for that.

a. Spinning up minikube:

~]# minikube start --vm-driver=kvm

b. Creating the Mahawk + Heapster pods:
The k8s list file I used is ( note: the credential files are missing ):
https://github.com/MohawkTSDB/mohawk-container/blob/master/mohawk-k8s.yaml

~]# kubectl create -f Projects/mohawk-container/mohawk-k8s.yaml

c. Check that the pods and endpoints are up:

~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
kube-system   heapster-jgzvx                1/1       Running   0          41m
kube-system   kube-addon-manager-minikube   1/1       Running   0          58m
kube-system   kube-dns-6fc954457d-m5rkf     3/3       Running   0          58m
kube-system   kubernetes-dashboard-qxd4j    1/1       Running   0          58m
kube-system   mohawk-b2qhn                  1/1       Running   0          41m

d. Check endpoints:

~]# kubectl get ep --all-namespaces
NAMESPACE     NAME                      ENDPOINTS                     AGE
default       kubernetes                192.168.122.89:8443           49m
kube-system   kube-controller-manager   <none>                        49m
kube-system   kube-dns                  172.17.0.3:53,172.17.0.3:53   49m
kube-system   kube-scheduler            <none>                        49m
kube-system   kubernetes-dashboard      172.17.0.2:9090               49m
kube-system   mohawk                    172.17.0.6:8443               32m

e. Get the k8s token:
Use the correct secret name.

~]# kubectl get secret default-token-f59kh -o yaml | grep token | head -n 1 - |  cut -d' ' -f 4 | base64 -d -
eyJhbG ... YkwIpww

f. Set provider and metrics on ManageIQ:

provider url: the default endpoint, in this case 192.168.122.89
provider port: the default endpoint port, in this case 8443
provider token: the token

metrics url: the same as the provider
mertics port: the mohawk endpoint port 30064

In this setup use ssl-without-validation.