How to consolidate CF logging using something like Splunk or EFK

I am specifically looking for a more developed project or something out there to collect/index/analyze VMDB logs using Splunk other than this old Splunk app: https://splunkbase.splunk.com/app/1052/

Not much newer but there is logstash from the ELK stack as another solution.
Here is a blog on how to implement log consolidation:
Cloudforms / ManageIQ automation and centralized logging with logstash kibana and elasticsearch

But that is 2015.

The problem I found was the the evm.log file is a bunch of an entry + dump. One can also find ruby hash dumps… Which a programmer really shouldn’t be doing. Instead a simple JSON serialize would have been nice.

ruby -e 'require "json"; puts JSON.generate( ruby_hash )'

Does it have to be Splunk?
Are you looking for anything specific to analyze?

The logging question has been asked a couple of times already and as basically everybody has to deal with centralized logging sooner or later, there might be an opportunity for a community project here.

Is anyone else interested in starting a logging project for manageiq?
If not I can still try to put a post/repo/whatever together with our current config to get up and running with the basics

I think this is a great idea.

It might be worth investigating whether we could use the containerised “common logging” platform, which is really just the OpenShift EFK logging stack with an external route for Elasticsearch.

oVirt/RHV have deployed their logging using https://github.com/linux-system-roles/logging/tree/master/roles/rsyslog. We might be able to utilise some of these playbooks.

pemcg

I am sorry for the late reply here. I work with a client who is heavily invested in Splunk and is not open to explore another logging stock. That is the reason we are specifically looking to leverage Splunk for CF logging.

Yes!! I would appreciate it if you can share (post/repo/wht) your current config.

Thanks,

Mamadou

I copy-pasted our current config here: https://gist.github.com/ThomasBuchinger/1bd636526a198cdd5af58740b12bed60

If I am less busy, I could probably do a post with screenshots

1 Like

Thanks, Thomas for sharing this. I appreciate it.

Backend-wise we use a gem we’ve created called manageiq-loggers. I could see adding support for Splunk there.

Note that in there we have a thing called ContainerLogger, which outputs the log to STDOUT in a pre-parsed JSON format suitable for elasticsearch (it was original built for the EFK stack).

We also recently added Amazon CloudWatch format.

At the moment, we don’t have a mechanism to choose a different logger in ManageIQ, but if you could make a PR to add that, I think we could accomodate that.