Kubernetes provider displays no info because of missing cluster-reader ClusterRole

Hi everyone, ManageIQ newbie here.

I have a remote k8s (not OpenShift) cluster and I want to manage it through ManageIQ.

I performed the steps described at the section` Prepare cluster for use with ManageIQ listed here. Notice how there is a ClusterRoleBinding to a ClusterRole called cluster-reader. Then I successfully added a corresponding k8s containers provider to ManageIQ:

The problem is that the dashboard reports 0 services, 0 nodes, etc…, and this of course is not correct, I do have services and associated pods running in the cluster:

image

I looked at the contents of /var/www/miq/vmdb/log/evm.log in the virtual appliance and found this message:

[----] E, [2018-06-21T10:06:40.397410 #13333:6bc9e80] ERROR – : [KubeException]: events is forbidden: User “system:serviceaccount:management-infra:management-admin” cannot list events at the cluster scope: clusterrole.rbac.authorization.k8s.io “cluster-reader” not found Method:[block in method_missing]

So what’s happening is that the ClusterRole cluster-reader is not defined in my cluster. I double checked with “kubectl get …” and could not find it. Bear in mind that I did not set up the cluster, I was only given access to it. Also, at the section Prepare cluster for use with ManageIQ I used to prepare the cluster to interact with ManageIQ (here)
, step nbr. 3 says:

image

But executing the listed command grants the cluster-reader cluster role to a subject of Kind User, not ServiceAccount. Again, I double checked with “kubectl get …” and that’s correct (only the relevant portion of the output is reported):

image

Is this normal?

I think one possible solution is to create cluster-reader manually, but I don’t know what are the permissions it grants or where to find the .yaml where it’s defined.

So does anyone have any helpful hints on how to solve this?
Thanks.

2 Likes

Same problem here, no info anywhere

Same problem here, no info anywhere

help pleaze

@agrare Are you familiar with this, or know who is?

Cluster-reader must be an openshift concept. @cben Do you know what permissions are required for kubernetes?

Try with this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-reader
rules:
- apiGroups: [""]
resources: [“services”, “endpoints”, “pods”, “events”, “deployments”, “nodes”]
verbs: [“get”, “list”, “watch”]

which file I add this information
more détails plaise

i can’t… restricted by new user…

I have the same problem
in the kubernetes dashboard is added
but without any details of the nodes
you wrote add these lines so I wonder or we add it

An update. With this, the container portal began to show information

@koheiren
thank you
just I would like to telle me where I doing this update
in which file . yaml in my kubernetes
just to be on sure

stp is what i need to create a new .yaml file in my kubernetes folder or i add these line in my kub-apiserver.yaml file
thank you for clarifying that
and when I create a new file normally I have to make another step the execute or restart the cluster

??

i have created a new file miq.yaml next i deplyed
but no more détaill an dashbord MIQ

Please, paste the log file for ManageIQ

I wondered if k8s’s default ClusterRole view would work? => I think not.
https://dev.to/mhausenblas/on-some-defaults-in-kubernetes-rbac-270l
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings

If it works, it sounds more future-proof, as it’s supposed to cover pretty much every resource type in k8s, except roles, rolebindings, secrets.
However, I diffed it to openshift’s cluster-reader:
https://diff-online.com/view/5d188700adf65140f0c47cda
and view seems to miss essential stuff like nodes :slightly_frowning_face:.

(P.S. in openshift, the view clusterrole is wider than in k8s, but still misses stuff like nodes: https://diff-online.com/view/5d18af79adf65140f0c47cdd)

You are sure. If you add kubernets some containers provider in manageiq you can see the nodes
In officiel documentation i don’t see this details
Please share your experiences in this point
I need it more detail or steps

Show us the evm.log on manageiq folder log. If you have a error of connection or other thing, we can see in this file.

i have this error in evm.log and second journal in dashbord MIQ

[----] I, [2019-07-02T03:29:06.406862 #10328:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::MetricsCollectorWorker.emses_in_zone) Skipping [MasterK8s] since it has no metrics endpoint

MIQ(AuthToken#validation_successful) [ExtManagementSystem] [10], previously valid/invalid on: [2019-07-02 08:31:56 UTC]/[2019-07-02 08:01:58 UTC], previous status: [Valid]
[----] I, [2019-07-02T04:32:04.067457 #56544:b9cf58] INFO – : MIQ(MiqQueue.put) Message id: [20150], id: , Zone: [default], Role: , Server: , MiqTask id: , Ident: [generic], Target id: , Instance id: , Task id: , Command: [MiqEvent.raise_evm_event], Timeout: [600], Priority: [100], State: [ready], Deliver On: , Data: , Args: [[“ManageIQ::Providers::Kubernetes::ContainerManager”, 10], “ems_auth_valid”, {}]
[----] I, [2019-07-02T04:32:04.399797 #56530:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#do_work) EMS [172.16.3.10] as Event Monitor Thread gone. Restarting…
[----] I, [2019-07-02T04:32:04.400053 #56530:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Validating Connection/Credentials
[----] I, [2019-07-02T04:32:04.400163 #56530:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager#with_provider_connection) Connecting through ManageIQ::Providers::Kubernetes::ContainerManager: [MasterK8s]
[----] I, [2019-07-02T04:32:04.510468 #56530:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Starting Event Monitor Thread
[----] I, [2019-07-02T04:32:04.513979 #56530:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Started Event Monitor Thread
[----] E, [2019-07-02T04:32:04.601182 #56530:1277378] ERROR – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Event Monitor Thread aborted because [events is forbidden: User “system:serviceaccount:management-infra:management-admin” cannot list resource “events” in API group “” at the cluster scope]
[----] E, [2019-07-02T04:32:04.601295 #56530:1277378] ERROR – : [Kubeclient::HttpError]: events is forbidden: User “system:serviceaccount:management-infra:management-admin” cannot list resource “events” in API group “” at the cluster scope Method:[block (2 levels) in class:LogProxy]
[----] E, [2019-07-02T04:32:04.601358 #56530:1277378] ERROR – : /usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:125:in rescue in handle_exception' /usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:115:inhandle_exception’
/usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:311:in get_entities' /usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:206:inblock (2 levels) in define_entity_methods’
/usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-kubernetes-19da66b40c68/app/models/manageiq/providers/kubernetes/container_manager/kubernetes_event_monitor.rb:31:in each' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-kubernetes-19da66b40c68/app/models/manageiq/providers/kubernetes/container_manager/event_catcher_mixin.rb:40:inmonitor_events’
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:161:in `block in start_event_monitor’
[----] I, [2019-07-02T04:32:05.588778 #56530:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#do_work) EMS [172.16.3.10] as Event Monitor Thread gone. Restarting…

and


[----] I, [2019-07-02T08:49:56.554922 #15082:13dedc4] INFO – Success: MIQ(EmsCommon.block in process_emss) userid: [admin] - [MasterK8s] Record delete initiated
[----] I, [2019-07-02T08:57:06.887821 #15082:13e07dc] INFO – Success: MIQ(EmsCommonAngular.create_ems_button_add) userid: [admin] - [MasterK8s] Record created (name:[MasterK8s], provider_region:, hostname:, azure_tenant_id:, keystone_v3_domain_id:, port:, api_version:, security_protocol:[ssl-without-validation], provider_id:, zone:[default])


[----] I, [2019-07-02T09:24:15.123703 #16894:b9cf58] INFO – : MIQ(ManageIQ::Providers::Openstack::NetworkManager::EventCatcher::Runner#start_event_monitor) EMS [auth.cloud.ovh.net] as [rx2DBpwZ5e7H] Starting Event Monitor Thread
[----] I, [2019-07-02T09:24:15.123940 #16894:b9cf58] INFO – : MIQ(ManageIQ::Providers::Openstack::NetworkManager::EventCatcher::Runner#start_event_monitor) EMS [auth.cloud.ovh.net] as [rx2DBpwZ5e7H] Started Event Monitor Thread
[----] I, [2019-07-02T09:24:15.409779 #29467:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#do_work) EMS [172.16.3.10] as Event Monitor Thread gone. Restarting…
[----] I, [2019-07-02T09:24:15.410149 #29467:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Validating Connection/Credentials
[----] I, [2019-07-02T09:24:15.410222 #29467:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager#with_provider_connection) Connecting through ManageIQ::Providers::Kubernetes::ContainerManager: [MasterK8s]
[----] I, [2019-07-02T09:24:15.494199 #29467:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Starting Event Monitor Thread
[----] I, [2019-07-02T09:24:15.495565 #29467:b9cf58] INFO – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Started Event Monitor Thread
[----] E, [2019-07-02T09:24:15.563564 #29467:9cc66dc] ERROR – : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [172.16.3.10] as Event Monitor Thread aborted because [events is forbidden: User “system:serviceaccount:management-infra:management-admin” cannot list resource “events” in API group “” at the cluster scope]
[----] E, [2019-07-02T09:24:15.563657 #29467:9cc66dc] ERROR – : [Kubeclient::HttpError]: events is forbidden: User “system:serviceaccount:management-infra:management-admin” cannot list resource “events” in API group “” at the cluster scope Method:[block (2 levels) in class:LogProxy]
[----] E, [2019-07-02T09:24:15.563842 #29467:9cc66dc] ERROR – : /usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:125:in rescue in handle_exception' /usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:115:inhandle_exception’
/usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:311:in get_entities' /usr/local/lib/ruby/gems/2.4.0/gems/kubeclient-4.1.2/lib/kubeclient/common.rb:206:inblock (2 levels) in define_entity_methods’
/usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-kubernetes-19da66b40c68/app/models/manageiq/providers/kubernetes/container_manager/kubernetes_event_monitor.rb:31:in each' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-kubernetes-19da66b40c68/app/models/manageiq/providers/kubernetes/container_manager/event_catcher_mixin.rb:40:inmonitor_events’
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:161:in `block in start_event_monitor’
[----] I, [2019-07-02T09:24:15.899308 #17064:b9cf58] INFO – : MIQ(ManageIQ::Providers::Openstack::StorageManager::CinderManager::EventCatcher::Runner#do_work) EMS [auth.cloud.ovh.net] as [rx2DBpwZ5e7H] Event Monitor Thread gone. Restarting…


haw i can resolve this

@koheiren
@agrare
@Fryguy

thank you

Hi, please try this:

kubectl create ns management-infra
kubectl create sa -n management-infra management-admin
kubectl create sa -n management-infra inspector-admin
kubectl create clusterrolebinding management-infra-cluster-reader --clusterrole=cluster-reader --user=system:serviceaccount:management-infra:management-admin

And apply this file .yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: cluster-reader
rules:
- apiGroups: [""]
resources: [“services”, “endpoints”, “pods”, “events”, “deployments”, “nodes”, “replicationcontrollers”, “namespaces”, “resourcequotas”, “limitranges”, “persistentvolumes”, “persistentvolumeclaims”]
verbs: [“get”, “list”, “watch”]

like:

kubectl create -f <file.yaml>

And get the authentication key with this:

kubectl describe secret -n management-infra $(kubectl get secrets -n management-infra | grep management-admin | cut -f1 -d ’ ‘) | grep -E ‘^token’ | cut -f2 -d’:’ | tr -d ‘\t’

and tell us.

tahnks
i just create the .yaml file and is ok
but i dont see cpu RAM