Graph Refresh...What is it and why?

providers

#1

Graph Refresh

You might have heard discussion about “Graph Refresh” or save strategies but weren’t sure what any of that actually means. Officially the module is called ManagerRefresh and the code can be found here, but pretty much everyone calls it graph refresh.

Why another refresh?

To understand the reason behind why graph refresh was developed we have to look at how “classic” EmsRefresh works.

Lets say that we have a VM vm1 which is on a Host host1. We relate these two by setting vm1.host_id = host1.id. But what if that host hasn’t been saved to the database yet? Currently EmsRefresh handles this by:

  1. Use the same ruby object to represent associated inventory
  2. After saving an object, take the ID and add it to the hash
  3. Pull the ID out of the associated hash

Refresh Parser:

host = {:ems_ref => "host-123", :name => "host1"}
hosts << host

vm = {:ems_ref => "vm-234", :name => "vm1", :host =>host}
vms << vm

Save Inventory:

# vm[:host] looks like {:ems_ref => "host-123", :name => "host1"}

found = ems.hosts.find_by(:ems_ref => host[:ems_ref])
found.update_attributes(host)
found.save!
host[:id] = found.id

# Now vm[:host] is {:id => 1, :ems_ref => "host-123", :name => "host1"}
vm[:host_id] = vm.delete(:host).try(:id)
found = ems.vms.find_by(:ems_ref => vm[:ems_ref])
found.update_attributes(vm)

This works but there are some problems with it…

Strict ordering defined statically in code

The save order is defined in EmsRefresh.save_inventory and cannot be updated “on-the-fly”. This means that when a new provider type is added which has different types of inventory or ordering rules, a new save_inventory method has to be written.

Cyclic Dependencies

Cyclic dependencies are difficult to reconcile, you have to save one item, then the second, then go back and update the first

Targeted refresh

Having to have the ruby object of the associated inventory isn’t a big issue when you are doing a full refresh and you have the entire inventory in memory, but if you only want to update 1 VM without clearing out all of its associations you have to build a sparse tree of everything that is related to the one item you’re trying to update

Deletions

How do you know to delete something? If a VM is in the database but not in the returned hashes does that mean it was deleted from the provider or just wasn’t updated? Deletions can basically only be done safely if all inventory is present a.k.a you’ve done a full refresh.

How does Graph Refresh solve these issues?

Essentially what graph refresh does is take the inventory returned by a RefreshParser and represent it as a graph data structure with the inventory objects as nodes and the associations as edges. This is done by ManagerRefresh::Graph ref.

This means that the save order doesn’t have to be defined in code but rather is dynamically determined by introspecting the inventory data.

A key difference between EmsRefresh.save_inventory and graph refresh for refresh authors is that instead of storing inventory as arrays of hashes inventory is stored as InventoryObjects in InventoryCollections. This allows additional metadata to be added to these objects for example what scope does a collection cover. This allows easier deletions of sub-collections without requiring a full refresh.

Another important tool is the concept of a “lazy link” represented by an InventoryObjectLazy. This is essentially a stub reference that is evaluated at save-time instead of parse time as in our previous example.

The way we would write the vm1 example above using lazy references would be:

vm = vms_inventory_collection.build(:ems_ref => "vm-234", :name => "vm1")
vm.host = hosts_inventory_collection.lazy_find("host-123")

This means that we can define cyclic dependencies easier in the parser, and we don’t need to have associated inventory objects parsed and in memory in order to make this association.

What are save strategies?

Traditionally save_inventory has only one strategy but deletions behave differently depending on if the target is an ExtManagementSystem or not.

By contrast Graph Refresh provides a number of different strategies for inventory saving depending on what you want to do. These strategies are here app/models/manager_refresh/save_collection/saver.

A strategy that will be used for InventoryCollection persisting into the DB.
Allowed saver strategies are ref:

  • :default => Using Rails saving methods, this way is not safe to run in multiple workers concurrently, since it will lead to non consistent data.
  • :batch => Using batch SQL queries, this way is not safe to run in multiple workers concurrently, since it will lead to non consistent data.
  • :concurrent_safe => This method is designed for concurrent saving. It uses atomic upsert to avoid data duplication and it uses timestamp based atomic checks to avoid new data being overwritten by the the old data.
  • :concurrent_safe_batch => Same as :concurrent_safe, but the upsert/update queries are executed as batched SQL queries, instead of sending 1 query per record.

The :concurrent_safe and :concurrent_safe_batch strategies are where things get interesting as these are intended to allow for multiple parallel persister workers operating on the same EMS for greatly improved refresh performance. These take advantage of database atomic operations (e.g. https://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT), unique indexes, and timestamps to ensure that creates and updates run in parallel don’t corrupt the resulting data stored in the database.

Needless to say if you are just starting stick with the :default strategy.

Conclusion

Hopefully this helped give a little background into what graph refresh is, why it was done, and what benefits it provides. There is a lot more to cover so this will be a multi-part series!