MiQ systemd support

Developers
#1

PR #1569 adds preliminary support for systemd which is now present on many Linux systems. This topic is intended to explore continuing to add support for this.

Background: Systemd is a mechanism used to manage various aspects (called units) of modern Linux systems including the services running on them. Units are individually configured and activated based on enabled targets. Targets include but are not limited to the sysvinit runlevels and more.

Units and targets are configured on both a system and per-user basis. Configuration files may reside in one of several locations:

  • /etc/systemd/system
  • /etc/systemd/user
  • /usr/lib/systemd/system
  • /usr/lib/systemd/user
  • ~/.config/systemd/user/

Systemd configuration may be retrieved by scanning these directories for the appropriate unit and target files and parsing them.

Each unit file may list the targets which it is ‘RequiredBy’ and those which it is ‘WantedBy’. The former is a hard dependency, if a requirement fails to launch / be configured properly then the component which requires it will be deactivated. The later is a softer dependency, if the wanted unit fails to launch, the unit/target which references it will still continue.

Implementation: The current patch scans the directories specified above (minus the user home directories) and extracts service units into the existing MiQ service xml structure. The enable/disable run_level fields are not currently populated as the required / wanted by model does not fit into this.

To resolve this the system_services table could be expanded with new ‘required_by’ and ‘wanted_by’ columns to store this information in and the corresponding service screen updated to display this information.

Thoughts? Alternate solutions?

#2

Nice explanation @mmorsi. It sounds like requiredby services are basically what our existing services are. The wantedby services are a new concept. Is this correct?

So, what would cause wantedby services to not start? Is this information we can get when the system is on or off? Does the type(wanted/required) of other services help us determine if a wantedby service is likely to start?

#3

@jrafanie sort of. From my understanding ‘units’ (which services are classified under) and "targets’ (which runlevels have been converted to) can depend on other units via ‘Require’ / ‘Wanted’ directives.

Units may also specify the targets which require them by ‘RequiredBy’ and ‘WantedBy’ directives in the unit configuration files. The enabled_run_level sort of fits this, but again Targets encapsulate more than just run levels (eg a target may activate other targets).

Services could not start for any number of reasons, they same they wouldn’t start today (internal errors, external env factors, etc). Yes the active services can be deduced when the system is running but I don’t think this helps us for analysis purporses (since the vms are offline).

Note I’m also in the process of coming up to speed with systemd so while this should all be accurate, there may be more to the picture still to be accounted for.

#4

Good writeup @mmorsi.

Can you give some examples of possible values for RequiredBy and WantedBy? For example, targets that refer to runlevels and how the runlevels have been converted to targets. In addition, to runlevels, to what else can targets refer?

How are the old enable/disable runlevels handled? If it’s not RequiredBy or WantedBy is it assumed to be disabled at that level?

#5

One thing to note is the systemd targets are units themselves. So this essentially boils down to units may “require” or “want” other units and/or may be “required by” or “wanted by” other units.

Some examples from my local system (stripped of comments for conciseness):

/usr/lib/systemd/system/httpd.service

[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
Environment=LANG=C

ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/multi-user.target

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

/usr/lib/systemd/system/nfs.target

[Unit]
Description=Network File System Server
Requires=var-lib-nfs-rpc_pipefs.mount proc-fs-nfsd.mount rpcbind.service
After=network.target named.service 

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/iptables.service

[Unit]
Description=IPv4 firewall with iptables
After=syslog.target
ConditionPathExists=/etc/sysconfig/iptables

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/libexec/iptables/iptables.init start
ExecStop=/usr/libexec/iptables/iptables.init stop
Environment=BOOTUP=serial
Environment=CONSOLETYPE=serial
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=basic.target

/usr/lib/systemd/system/cups-browsed.service

[Unit]
Description=Make remote CUPS printers available locally
After=cups.service avahi-daemon.service
Wants=cups.service avahi-daemon.service

[Service]
ExecStart=/usr/sbin/cups-browsed

[Install]
WantedBy=multi-user.target

There are many more for various services, targets, and other units

#6

You can read more about targets here. Note units can ‘want’ additional units outside of their config file via a “unit.wants” directory (eg httpd.service.wants) residing in the same dir as the unit conf. Inside there should be symlinks to the other unit conf files the unit wants.

From my understanding, yes if a service unit is not ‘required by’ or ‘wanted by’ a target unit, and that target unit does not ‘require’ or ‘want’ that service then the service is not enabled for that target (unless it is an indirect dependency, eg the target requires a service which depends on the service in question)

#7

So, it sounds like “required by” and “wanted by” can have multiple values. If so, it seems they can be implemented as has_and_belongs_to_many associations within the system_services table, is that correct?

Would that be overkill for the given use case? Would a simple list of target names suffice? If all we need is a list of names, can we just implement them via text columns?

@chessbyte, @Fryguy, @gmccullough, any opinions/thoughts on this?

#8

@rpo Yes ‘required by’ and ‘wanted by’ can have multiple values as can ‘requires’ and ‘wants’. Defining those columns as references to the system_sevices table wouldn’t suffice though as these values may refer to either services or targets (any unit type really but for the purpose of system service detection those are are the only ones we need to be concerned with).

Depending on how we wanted to do it, there are multiple ways to implement the storage. Since enable_run_levels and disable_run_levels are just two data columns (eg not references) the simplest approach would be to add a few more columns required_by and wanted_by which would list the corresponding units which we could parse / lookup in the system_services table to determine if they are services or target (service if present, else a target).

Alternatively the “proper” way to do it would be to fully normalize all the run level & target references, defining new table(s) for those and bridge tables as appropriate. May be overkill for our needs though, so perhaps a the simpler approach would be best.

#9

@mmorsi So, do we need 4 new columns for: ‘required by’, ‘wanted by’, ‘requires’ and ‘wants’?

I think these fields will be used primarily for display in the UI and for reporting. Should they just be text columns containing csv data? @dclarizio, @Fryguy, any opinions?

#10

If you do decide on just stuffing it in text columns, which I’m ok with, I would suggest

  • not use CSV…prefer a serialized Array of values in a column
  • try to avoid having multiple serialized columns. You can easily create accessors over a single serialized column to make it appear like multiple columns, which makes callers easier. If enable_run_levels and disable_run_levels are already there, perhaps we can merge them into this new column, but that can be done in a follow up phase. Typically we have been naming these kinds of “shared” columns, :settings
#11

To be clear what I meant, the column content would look something like

---
:required_by:
- value1
- value2
:wanted_by:
- value1
- value2
:requires:
- value1
- value2
:wants:
- value1
- value2

Then in the model:

class SystemServices < ActiveRecord::Base
  serialize :settings, Hash

  def required_by
    settings[:required_by]
  end

  def required_by=(values)
    settings[:required_by] = values
  end
end
#12

If we want certain formatting for these in the UI, we may want some virtual columns, like required_by_display or something.

#13

@rpo I’m thinking columns for ‘requires’ and ‘wants’ are probably not needed as they can be represented through the inverse relation (‘requiredby’ / ‘wantedby’) on the foreign entity. eg the mutli-user targer requires the cups service, one could say the cups service is requiredby the multi-user target. Obviously this is not how it’s represented on disk but would alleviate some db complexity.

@Fryguy one column works for me, though in general what are the guidelines for combining columns?

@dclarizio not quite sure how that all plays out at the ui level so can’t comment but most likely the systemd services will be rendered in the same manner the initprocs are

#14

I’m a visual guy… I need a drawing or an example or something…