Version control integration

Developers
#1

Here at CBTS, we are a managed services provider. We already have a multitude of CloudForms appliances that we are managing.

Thoughts about how to implement changes across the board to the various appliances (adding buttons, state machines, etc)

Ease of implementation and replication between environments, how to roll out changes, new features to multiple different environments. Not necessarily to the same customer and/or environment

We are currently using Git, but it is still incredibly manual.

Perhaps a plugin type system would work best?

Automation Engine Integration with VCS
#2

To add to what Eric said, currently we have a cron job that runs on the appliance that does a couple of things:

. /root/.bash_profile

pushd /var/www/miq/vmdb >/dev/null
if [ -d /tmp/FOO ]; then
  rm -rf /tmp/FOO
fi    

bin/rake evm:automate:export DOMAIN=FOO EXPORT_DIR=/tmp
bin/rake dialogs:export['/tmp/foo_dialogs.yml']
script/rails runner tools/export_tags.rb /tmp/foo_tags.yml >/dev/null 2>&1
popd >/dev/null

Then I have another cron job that uses scp to get that data into a git working dir and commit it with a generic message that includes the date. Great for keeping track of changes over time, but not so great for useful metadata.

Matt

#3

It’s useful to see the user pain points so posts like this are really helpful.

I know @mkanoor is working on automate versioning using git to help solve some of those pain points so perhaps he can comment here.

Please do post other areas that are tedious for users as there might be other solutions others have come up with or something we can try to help automate for most use cases.

#4

Would Chef or Puppet make sense here? It sounds like to are trying to manage configuration and deployments of manageiq appliances, which is a good fit for a config management tool. @purpleidea might be able to help you out with this as he’s looked into it a lot.

1 Like
#5

It might, but we’re really talking about the actual code in the Automate model.

CBTS is a Service Provider, so we’re a little different than most people using ManageIQ/Cloud Forms. When a customer comes on board, we need to import an Automation model that does most of what we need, and then we can customize methods specifically for the customer with the new Domain hierarchy. So think Customer -> CBTS -> Red Hat (locked) -> MIQ (locked).

What happens when we have 20 customers and we need to update the CBTS domain on 80 appliances or whatever it ends up being? I’m not sure puppet helps us here (but if it does I’d be happy to be proven wrong and start using it!).

Matt

#6

Are all the appliances going to be fetching the same CBTS domain or would they be using different commit SHA’s.
We are researching adding the automate model as a git repository and keeping track of the changes made by the editor/imported. Each domain would be a separate git repository, a domain can be internal or external. The external domains (ManageIQ, CBTS) would typically be read only and the internal domains e.g CUSTOMER would be read/write and would be tracked locally using GIT and synced across the zones. The External domains would have the ability to be synchronized on a schedule or on demand from the external sites.

#7

That pretty much sounds like exactly what we would do. We would implement 90% of the integrations with our standard monitoring, CMDB, etc. in the CBTS domain and distribute that to all customer appliances, then the last 10% that is customer specific would be implemented in the CUSTOMER domain and be specific to that customer’s zone(s).

#8

Number 1 pain point we currently have as a customer. We want to be able to develop automate code against a Dev environment of MIQ and “promote” that code to production environments without having to manually copy and paste from browser windows manually.

1 Like
#9

should this conversation then be moved into GitHub rather than here?

#10

Eventually. If we’re at a point that we can scope it out, then yes, although I suspect that there may be a feature request already that we can piggyback on - @mkanoor?

#11

The dev/prod thing is integral to the GIT design, without a GIT backend support this will be a difficult feature to add.

#12

@mkanoor - can you link to the github issue or elsewhere this is being worked on?

#13

https://github.com/ManageIQ/manageiq/issues/1199

#14

We are in the process of not only adding it to git version control, but integrating with a CI (with jenkins) to help us with the promotion from Dev -> Test -> Pre-Prod -> Prod. This is not an automatic promotion but utilizing Jenkins Build Pipelines plugin to authorize the promotion process from appliance to appliance utilizing git at the center of it all. It gets a bit hairy at times, its still a work in progress and once we are finished we are hoping to present it out to the world for others to start using the set of scripts and review the docs. We can export service dialogs, catalogs, catalog items, tags and roles but currently need to finalize the code for exporting/importing the custom buttons. The important thing to note is that we try to separate out everything in their own files (dialogs, tags, roles, catalogs) for better version control of individual items. We saw that dumping each type in one massive yaml becomes a nightmare to deal with in git versioning when you add new items to the mix as when the exporting happens it will sort them. Once we complete the button export/import we should be ready to share with all of you. I can provide more details if needed, just send me a PM.

3 Likes
#15

One of my customers has his own dev2prod pipeline with export/import scripts. And he keeps track of his changes through Git. His main pain point is password encryption. When you create a password field, it is encrypted using the instance key. So when you import the automation from another instance, the passwords can’t be decrypted.

@jsimonelli - How do you handle this ?

#16

I did mention it was a work in progress right :wink:

For now there are a couple ways to do this but they are complete workarounds/hacks. You could use a “config” domain where it doesn’t get replicated. The code promotion happens at another domain. Then load some environmental variables from the config domain with a helper method.

Secondly, which we did for one customer was that the “password” fields in instances themselves where strings but the passwords entered there where encrypted using an external key that was accessible on all appliances. So in a sense, still encrypted but not utilizing the built in encryption of the appliance. The code was written that it would load this “key” to unencrypt the encrypted hash “string” passwords. This process was also not for code import/export promotability but they had their own security reasons why they wanted to have passwords that could be passed from cloudforms to other systems and since they would have the key as well they could also unencrypt these fields being passed. Same concept could apply here.

Either way both still don’t really fix the underlying problem that in the promotion process we are in need to also keeping track of a region “key” to be able to unencrypt these fields and then re-encrypt using the new key for the new region.

The question is, how will it live in git. encrypted from old region or what? but where are you keeping the key. Surely having that key in git is just asking for trouble.

I feel like the “right” answer would mean to add some features to /var/www/miq/vmdb/lib/tasks/evm_automate.rake in the import logic to be able to input a key during the import so it knows how to unencrypt the password fields to the import using the current regions key.

1 Like
#17

@jsimonelli - I see you got your hands dirty on that matter. I am glad you found workarounds that I can share with my customer. He will be happy :slight_smile:

Maybe, we could also consider that all instances from dev to prod are a global cluster and share the encryption key on all instances… It would be in the devops way, further more if we integrate CI to validate that the code we write is clean.

#18

well, hmmm… Never thought about just using the same v2_key across the different environments. This would make things MUCH easier to implement. I would be interested in hearing some of the engineering folks on this matter. Why could we NOT (or just shouldn’t) just use the same key across regions within each customer. So a customer owns a specific key that can be used across all of their environments. Just seems too easy. Even if they are using different region numbers this should not affect their security. In fact, I think it would make things much easier to manage. This is of course is unless their security restrictions limit the organization from using the same key across dev, test, pre-prod and prod. I will run through some of these scenarios in our internal lab and go from there. The whole un-encrypt to then encrypt would seem like it would only benefit a certain amount of people. The communication between the client and appliance will be encrypted anyways, and the password fields would still be encrypted in the database just happens to be using the same exact key across the regions. We could just write up a “DevOps” requirement document that just borrows some of the steps outlined in https://access.redhat.com/articles/1124753 to copy the “organizational” key across the regions, then run the fix_auth steps on all of the appliances. As long as the appliances are pointing to their own databases then it shouldn’t matter.

Our end goal here is to have a complete DevOps-y version control with CI for the promotion process. @jhardy @bascar What do you guys think about this?

#19

@jsimonelli I’m a developer on manageiq and only have my own opinion, but…

I think it depends on the environment. The v2_key can and SHOULD work cross region/(database).

I can think of many reasons why you would want to do this, note some of these are related:

  1. Simulate production migration and upgrades in test/development environments
  2. Practice disaster recovery in test/development using production data
  3. Practice production v2_key resets in test/development
  4. Test automation/policy/schedule using production using production data

The only downside I envision is what you already mentioned. If people have access to test/development environments but not production, they would have access to the production v2_key and possibly production data if you’re using the v2_key to test/develop. Additionally, there is an increased number of places that need to be touched to reset the v2_key in case of the key becoming compromised.

My personal thought is that the same v2_key across regions allows you to practice many things including what would happen if that key is compromised so it’s benefits might outweigh any risks. This is something each organization would have to consider as there is some risk both ways.

#20

I agree with @jrafanie, i think the risks might be outweighed as long as the customer doesn’t have hard requirements for different keys for different environments. Having a single v2 key per customer makes all of the promotion, testing, cert invalidation and recovery much cleaner. It is still unique per customer.

Another way to do this same thing I have seen in usage, without all of these issues is to have an entire automate domain at the highest level which is just configuration data. Anything like an endpoint, credential or password is in that highest domain and never gets replicated. Less chance of moving something from one env to another and cross pointing dev/qa/prod credentials or the endpoints. Since that doesn’t move, key doesn’t come into play.