MIQ Enterprise Architecture - multiple Regions implementation

euwe

#1

Good afternoon all.

I am looking for help on how o configure MIQ into several Regions, each Region containing several zones. For some reason, I have been browsing the documentation for some days now but I can’t find the step-by-step approach on how to achieve such an implementation.

Can anyone hint me, direct me to documentation, etc …

Thank you in advance


#2

Hi @codebeaver22,

ManageIQ fits in many different environments, from very small to very large, single site to distributed, and each of them has their own considerations. So there is not a one-size-fits-all architecture.

The good news is that @pemcg is working on a reference architecture that explains some of the considerations that go into such a design. I’ve seen a draft of it and it’s turning into a very nice doc. I am not sure if he is able to share a draft at this point though, for community feedback, that is up to him.


#3

Hi @geertj,

Thanks for the information. I am looking for this kind of doc that cover networks behind firewalls or networks across continents.
Looking forward to see @pemcg architecture doc/diagrams.


#4

Thank you @geertj for your response. It is great to hear that @pemcg will be coming out with such a guide. However, I need to progress on this project and I need to get things rolling. I would be interested to get in touch with @pemcg to see how I could contribute to his effort. Some of the things that come to mind is that I could validate the implementation of 1 or more architectures and perform additional testing that I am sure would benefit the end goals. If you know how to reach @pemcg please let me know.

Regards,


#5

The challenge with trying to document how to scale ManageIQ/CloudForms is that the answer to nearly every question posed is “it depends”. The reference architecture (perhaps that’s the wrong term for it) that I’m putting together is more of an overview of the architectural components that affect scaling, and a set of guidelines on what to consider and monitor when scaling.

As an example, maximum practical region size is largely determined by database performance, but VMDB load is determined by:

  • The number of managed objects (VMs, hosts, datastores, etc.) in the region
  • The number and type of providers added to the region; the more fully-featured providers such as VMware or RHEV have more out-of-the-box events to detect and handle
  • The overall “busyness” of the external management systems (such as vCenters), which determines the rate at which events are received and processed, and thus the rate at which inventory refreshes are requested and loaded
  • The frequency of “event storms” from the external management systems
  • Whether or not Capacity and Utilization (C&U) metric collection is enabled for the region
    ** Whether this is for all clusters and datastores or a subset of each
    ** The frequency of collection
  • Whether or not SmartState Analysis is enabled for the region
    ** The frequency of collection
    ** The amount of data collected in the SmartState Analysis profile
  • The complexity of reports and widgets, and frequency of generation
  • The frequency of VM lifecycle operations
    ** Provisioning
    ** Retirement
  • The frequency of running automate requests and tasks, including service requests
  • The number of control or compliance policies in use
  • The number of concurrent users accessing the “classic” WebUI, especially displaying large numbers of objects such as VMs
  • The frequency and load profile of connections to the RESTful API (including the Self-Service UI)
  • The number of MIQ/CFME appliances (more accurately, worker processes) in the region

If there’s anything particular that you’d like to see discussed then please make a list of topics in this thread, it would be useful for me to see what’s needed that I hadn’t considered. I’ll see if I can publish the ‘Regions and Zones’ chapter as a draft somewhere which might help.

Cheers,
pemcg


#6

Good morning @pemcg.

The list you presented will really help leap MIQ into serious enterprise models. I am interested in all of these and I very much understand that many variants will arose from the different architecture design deployed to meet the business needs. With time I am certain I can contribute to some of those initiatives but to do so I need to establish a basic distributed architecture implementation in a lab and that is the main reason I started this tread. After several attempts at browsing the documentation and several blogs, I find myself still trying the hard way (i.e. trials and errors) in coming up with the recipe to implement multi-region architecture (e.g. how to configure MIQ into several Regions, each Region containing several zones). For example:

  1. To change the region ID via the appliance_console I discovered I had to select 3) Configure Database, then 4) Reset Configured Database, but before stop the EVM server in order to change the ID.
  2. Then the question about replication of databases. Seems intuitive from within a region but yet how do configure it. Then the harder question, how to you replicate regions database with the super region database so you can use the super region database as your CMDB, what’s the recipe?
  3. etceterra

These are my initial challenges. I have a rough time believing I am the first one attempting such an implementation so this is why I was asking to be pointed to the documentation that will help me achieve the implementation. Once I have the implementation establish then I will venture into many functional testing of the environment that I am certain will assist in tweaking many of the points from your list in the previous post. For example:

  1. Scaling a region (i.e. adding appliance to optimize)
  2. Evaluating the replication of policies across the architecture
  3. Evaluating the replication of catalogs
  4. Workload placements/migrations
  5. Integration of an external federated user authentication infrastructure (i.e. access any portal with the same credentials)
  6. Etceterra

It goes without saying that being privy to look at your draft would certainly assist in understanding many of the associated challenges.

I hope this clarifies my request and I certainly hope you can see that the work can also contribute to your objectives as well as the MIQ community.

Regards,


#7

Hi @codebeaver22

Thanks for your comments and suggestions.

I will agree that unfortunately some of the documentation is not necessarily easy to find. The description of how to setup database replication is here for CloudForms 4.2 (I’m not sure where the corresponding ManageIQ guide is I’m afraid): https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.2/html-single/general_configuration/#configuring_database_replication

The CloudForms 4.2 HA Guide is here: https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.2/html/configuring_high_availability/

The ManageIQ guide to setting up external authentication is here: http://manageiq.org/docs/reference/ (‘Authentication’ in the side bar)

Hope this helps,
pemcg


#8

Thank you @pemcg this looks good. I’ll let you know once I get through the references you are suggesting.

Have a great day

27 Apr 2017 update - The implementation is progressing where I have Regions 999, 001, 002, and 003 implemented and configured with an OpenStack Private cloud attach to each region 001, 002, and 003. Still going thru configuration steps and testing some functions going along (i.e. configuring both MIQ and OS). I am planning to provide detailed info of the implementation in the coming weeks. Cheers


#9

The reference architecture that @geertj mentioned has just been published here:

https://access.redhat.com/documentation/en-us/reference_architectures/2017/html/deploying_cloudforms_at_scale/

The content is also applicable to ManageIQ.

Hope this helps,
pemcg


#10

Thank you @pemcg.