Log file vim.log in /var/www/miq/vmdb/log

Support
#1

Hi,

I am having an issue with the vim.log growing rapidly, and I mean rapidly! :slight_smile:

There is an error message in my vim.log that is just “flying” away… I filled up 30GB of logs in about 2-3 hours.

When I look at the log output there is anerror of some kind that I can not understand.

starts with:
[----] E, [2014-09-23T13:10:32.114238 #21921:122f4100] ERROR – : RuntimeError: Response <#<Handsoap::Response:0x00000021c24608 @http_body="<?xml version=“1.0” enco
ding=“UTF-8”?>\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encodin

and ens with:

@fault=nil>> has no XML document

Between that there is a 10MB text message of information regarding the hosts within my environment.

Since we have… some hosts… I guess that’s why the message gets so big. It has gotten the nice “side effect” that about 300 mb/s of writes to our SAN was generated so we have been forced to IOPS-cap the appliance :slight_smile:

#2

Starting the old “Trial and error”. I have removed all infrastructures but one small one. Now the vim.log has stopped beeing totally spammed. So I guess I will have to re-add each infrastructure trying to figure out which infrastructure triggered the error message…

#3

@maxton, unfortunately, I think that’s the best approach for now. The fact that the error message is an unwieldy 10MB chunk of text makes debugging this a bit more difficult.

Let us know what you find with slowly adding back the infrastructure providers. Hopefully that helps nail down something a little more concrete.

#4

Yeah, totally agree with you. So since the “garbage” in the logfile is less now, I can concentrate on my initial question in another topic regarding the “Guest OS Information” not showing up in the dashboard. Once that’s solved I will proceed adding back my other vcenters again.

Testing against a fairly big environment with 3500 VM’s and 170+ hosts… So it becomes a bit jamed in the log files when error and debug info pops out… :slight_smile:

#5

@maxton the key to solving this is isolating the issue as you have already done. Once you get back to it, you should note that our log messages are prefixed with useful info such as the timestamp but also the process id and thread id. In your sample:
#21921:122f4100

21921 is the process id and 122f4100 is the thread. You can then grep your logs to only see this process’ messages and determine what it was doing just before it received a bad response. For example grep “21921” log/evm.log.

If it’s communicating to vmware, the vim.log might also have some useful information.