Documentation for creating a new provider

providers

#1

Hi all,

I’m attempting to build my own provider for a custom in-house cloud-like service, but I’m definitely struggling a bit to get going. I saw the developer guides and this discussion, but using the OpenStack and AWS providers to try to reverse-engineer my own provider is not really a great solution, and the developer guides seem to be missing quite a bit of necessary info. The guide/blog from the Hawker docs is much more inline with what I’m looking for, but still lacks quite a bit of detail.

What I’m really looking for is some documentation that specifies:

  • What each type of manager is (more in depth than just an example of each)
  • Define the classes/modules & methods each manager type must implement and what the return values should look like
  • How to insert custom providers into the deployment process of MIQ. I can think of a few ways, but I’m sure there are gotchas and best-practices.
  • Ideally, an article walking through creating a new provider

Anyways, I know that’s a big ask that will take some time, but I’m hoping to get the conversations started and some initial help.

Thanks in advance,
Jared


#2

I’ve been looking around for this as well. I’ve seen this question asked a few times, but no answer seems to be forthcoming


#3

Hi @jsmartt and @jzuilkowski

Would you be interested to use google doc to document this subject ?
I like to learn how to create a new provider for system monitoring tools like zabbix or NagiosCore.

See R1 for a gdoc I created for “MIQ VMWare Appliance Version Upgrade”

Let me know if you guys are interested to help out and I can start a new gdoc on “how to create a miq provider” subject.

R1: https://docs.google.com/document/d/1wRN2uxkKespKsWeAuCryHhAsQFf7BJfatqeGnKs4rtM/edit#


#4

Sure, I’d love to. But we need someone that already knows how to do this to give us at least an outline of the process.

I tried to create UML diagrams of the rails code yesterday, but the resulting graph was 22 feet long, so not all that helpful.


#5

It looks like the place to begin investigating is here https://github.com/ManageIQ/manageiq/blob/master/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb

The first part of the provider to tackle, as told to me by Sergio, is inventory. This appears to happen via ems_refresh.

If you look in the file I linked above, you’ll see some clues:

def refresh_targets_for_ems(ems, targets)
# handle a 4-part inventory refresh process
# 1. collect inventory
# 2. parse inventory
# 3. save inventory
# 4. post process inventory (only when using InventoryCollections)

A little further down we find:

    # TODO: implement this method in all refreshers and remove from here
    # legacy refreshers collect inventory during the parse phase so the
    # inventory component of the return value is empty
    # TODO: Update the docs/comment here to show the *real* bell-shaped
    # targeted inventory
    #
    # override this method and return an array of:
    #   [[target1, inventory_for_target1], [target2, inventory_for_target2]]

So, I think that this is the place to start. Once this method is able to pull the data, then we can parse it and all that comes after.

Thoughts anyone?


#6

I used to think that making a provider could be self-contained in a single provider gem, but after looking through more of the code bases, it looks like there is quite a bit of provider-specific code in the manageiq-ui-classic repo (and possibly the main manageiq backend repo?) as well that would need updated to support a new provider. This seems to make it extremely difficult to make a provider that won’t be added upstream, since you’ll be trying to maintain separate forks of those repos. Can anyone verify I’m on the right/wrong track here?


#7

That would be a great goal for UI plugability - that a “provider repo” could have both the backend and ui needed code, and the ability plug either at runtime vs. build time.


#8

@jsmartt indeed, at the moment in order to add a provider you will need to add code to the following 5 repositories:

  • manageiq
  • manageiq-ui-classic
  • manageiq-schema (for db scripts)
  • manageiq-api (for REST api)
  • a specific provider repo with refresher, parser, specific models and etc. (check manageiq-providers-hawkular, manageiq-providers-openshift and a few others for example)

#9

Thanks for the info @abonas. That’s what I feared; it’s probably doable for providers that can be merged upstream, but trying to manage forks of all those repos for an in-house provider is a bit of a non-starter.


#10

@jsmartt I am not sure I understand the “in-house” vs “upstream”, do you mean by “in-house” closed source?
Anyhow, doing it upstream is very challenging as well from my experience.


#11

The goal is to make it more plugable. can you describe your use case in
a bit more details of what you are trying to do?


#12

Yes, by in-house, I mean a closed-source, custom cloud management system that has enough assumptions made about its environment that it wouldn’t even make sense to open-source at this point. It does all the provisioning and management work for compute, networking and storage, and we were really hoping to plug ManageIQ into that system for the RBAC, reporting and cost-analytics (and maybe more later).


#13

ok, so I would classify a bit broader than just creating a new provider
to how do one creates an out-of-tree, pluggable provider.
@chessbyte - any thoughts on this?


#14

Could you elaborate a bit on what this means?
“you will need to add code to the following 5 repositories”

I’ve spent the better part of a week looking over these repos, but without a diagram or explanation, I can’t seem to make a whole lot of sense of this.

Is there an “easier” provider we could look at just to get the basic mechanics understood?


#15

Well, I did find this - https://github.com/ManageIQ/guides/blob/master/providers/hawkular-dev-guide.adoc

That looks like a basic roadmap to creating a provider


#16

@blomquisg @agrare @bronaghs @durandom Would you please document this in https://github.com/ManageIQ/guides