Use Process.fork to create workers


#1

Hi all,

I wanted to share with everyone a summary of the changes that are to be merged on master very soon in this forking workers PR.

Summary:

When launching rake evm:start (via evmserverd systemctl or directly), workers will be created via Process.fork instead of Process.spawn:

  • bypassing all of the rails and vmdb initialization since it’s already loaded in the parent process
  • workers will share some memory with the server process

Benefits:

  • Faster worker startup including much less IO contention
  • Potential for future memory savings by preloading code, gems, libraries, etc. before we fork

What do I have to do?

  • git pull
  • rake db:migrate

What changes should I expect?

  • miq_workers command_line column is removed
  • MiqEnvironment::Process is removed
  • lib/vmdb/initializer.rb worker init code has been moved to the applicable worker models
  • top and system utilities that measure RSS will report more process memory
    usage since they don’t accurately report shared memory
  • the existing worker memory limits code now uses PSS instead of RSS (linux only)

Please ask any questions in this thread.

Thanks!
Joe


#2

The forking workers PR was merged several hours ago so please do ask questions here or report issues in the github issue tracker.

Note: We know that memory usage per worker is expected to increase a bit with this code. This is because we are yet not loading much of the application and gems into the memory of the server process before we fork children. On the other hand, worker startup will be significantly faster especially in systems with slower IO such as non-SSD drives.

Thanks!
Joe