[SOLVED] Git-flow for Automate Code


#1

Hello,
I am trying to upgrade our workflow to develop automate code to a git-flow workflow (https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow) and I was wondering if someone already did it and can point me to some resources on their solution.

I have a rough idea on how it could work, but it would require quite a bit of scripting to do it, therefore I thought I’d ask the community first, to validate my assumptions and get another opinion. The main problems I am facing are:

  1. There is no way to export or import on a more granular level than a single automate domain.
  2. Since higher priority domains overwrite everything from lower priority domains, any domain modeling a feature-branch MUST only contain changed files.
  3. I cannot use the git-import feature for feature-branches, because I need to import a dirty working copy.

A bit of Background:
At the moment we have a DEV-Cloudforms and a PROD-Cloudforms appliance. Every week we do an export from DEV, commit the whole export to git and import the same commit PROD.
All the development happens in the WebUI and we want to have the possibility to do small changes in the UI (espacially changes in dialogs, etc.)

The problems:

  • At the moment git is just a documentation tool. There is no save way to contribute via git and not have that change overwritten or producing a merge-conflict.
  • All the code is pushed to PROD all the time, If a change is not yet ready for production, it has do be manually disabled in code ( e.g. with an if-statement) before the export and enabled again after the export. => Doing a production deployment at the wrong time could break everything
  • This process is against all coding best practices :slight_smile:

Implementation:

  • Have a complicated git repo, with at least 3 types of branches.
    • release-branches: one for PROD and one for DEV
    • helper-branches: this is where automated commits go
    • feature-branches: this is where humans work
  • Export everything from DEV
    • do really small commits (e.g. for each class) to the helper-branches
    • at this point all the manual changes are in the helper-branches
  • merge the helper-branches to the relese branches
  • Import PROD
    • release-branches should work fine
    • import feature-branches, after removing all unchanged files
  • do_magic()
    • create new branches as needed and prune old ones
    • automatically maintain all the branches
    • keep all the merges free from any possible conflicts

As I said, the goal for this post is to validate my assumptions and to see if someone has already done something similar already. The rest is just background information and my thoughts on the topic.


#2

Not exactly the same but we are using git repo with branch too.
We have one Prod manageiq that rely on prod branch, and one dev manageiq which use development branch. When we are ready to push new functionalities, we merge branch and got a git import.
You can also do webhook in order to trigger the git import on your dev server when someone do a push.


#3

Hi @buc
Currently we are using such branching model, but big problem for us is that configuration for domain stored with code:

If it wasn’t for this problem, we would work in the following way:

  1. in my own tenant on the development appliance I connect a feature-branch domain from the repository.
  2. I’m working on it.
  3. Doing PR to the devel branch, which is connected on the same appliance, but applies to all tenants.
  4. After running the tests, the code goes to the master branch.

Devel-branch and Master-branch domains have the same configuration in _domain_.yaml file, but feautre-branch domains have different configuration for tenant_id. For this reason, we do cherry-picks from feature-domains to devel-domain.

Storing configuration with code is known anti-pattern and I think we should pay attention of MIQ developers on this.


#4

@LorkScorguar: That is exactly what I plan to do with the release-branches.

@igortiunov: I was not aware that automate domains can be scoped to apply to just one tenant (reference: https://pemcg.gitbooks.io/mastering-automation-in-cloudforms-4-2-and-manage/content/tenancy_and_automate/chapter.html). It is definitely a much cleaner solution, than my approach.

Regarding the __domain__.yaml:
I guess it would break the import if you just add it to .gitignore or something similar? :slight_smile:
However the rake-import-task has an option to specify the tenant (https://github.com/rhtconsulting/cfme-rhconsulting-scripts/blob/master/rhconsulting_miq_ae_datastore.rake). It looks like it does, what you want and having that function in the git import is probably just a matter of exposing the parameter(?)

Thanks for the feedback


#5

@buc
ssh on appliance is not a solution. I am using domain refresh via API: http://manageiq.org/docs/reference/latest/api/reference/automate_domains

I am sure that the domain configuration should not be together with the code. This configuration need to specify during the import from the git repository or at the time of updating from the repository.


#6

/push

I am currently working on a ruby script, that prepares “restricted” views of the Automate Code using the information in git and imports them with the evm:automate:import rake-task. The basic concept is the same as outlined above, I just trashed the 3-way-merge-with-branches-thing and simplified some details.

The workflow I am aiming for would be like this:

  • Create a new PR (and the associated branch)
    • CI executes the script and creates a new emtpy Automate domain
  • Copy the relevant methods in the new feature domain (via the WebUI)
  • Work on the Code
    • There is one Jenkins Job to export the current code from the UI to git (using evm:automate:export)
    • There is another Job to deploy the current code from git to ManageIQ (could be a push trigger)
  • Once it is ready for release, merge the PR and remove the branch
    • The script will detect, that there is a feature-domain without a feature-branch to go with it and destroy the domain (triggered periodically)

Limitations

  • 2 people working on the same file will shadow each other
  • You can alternate between git and WebUI but not both at the same time
  • I am expecting the following naming convention <prefix>-<issue_id>-<domain_name>-* (e.g. feature-1-feature1-this-text-doesnt-matter-but-it-is-usually-the-issue-title)

Advantages over a single develop-branch

  • You can use PRs for Code Review
  • There is no “change-freeze” for deployment to PROD
  • You don’t accidentally leak half finished code to PROD

#7

/update

I finally feel confident enough to actually post a link to what I am working on, so here it is: https://github.com/ThomasBuchinger/automate-gitflow

State:
I finished the deploy (aka import) part and a bunch of stuff around error handling, configuration options and so on. I am pretty sure it won’t crash with an exception if you call it with a wrong parameter.

  • The script can identify, which files have been changed on the feature-branch, copy them and the associated YAML files to a temporary directory and deploy (import) them to ManageIQ.
  • The repository is cloned in the background and removed again before the script finishes. Alternatively you can point it to a local repository.
  • The imported Domain will only contain the changed Automate methods, thus not overriding lower priority domains

I am thinking about not implementing export, since I think the demand is not that big and it’s pretty straight forward to do in bash.

I have been using it the script the last 2 weeks without any major problems (granted I am the author)

How-to use:

  • Clone the git repository onto your ManageIQ appliance.
    Unfortunately it seems, that there is no way to import Automate Domains via the REST-API, therefore I am using the rake-tasks.

  • The repository includes a file called example_custom.rb, rename it to custom.rb.
    $git_url/git_path configure the git repository to use

  • Create a feature-branch in your repository.
    I am using the naming convention feature-1234-<import-domain-name>-whatever. If your branches are named differently, just pass --domain to the deploy-command.
    You can find the example repo I am using for testing here

  • … DO YOUR CODE CHANGES HERE …
    When you are ready to test your changes, commit and push them to the feature-branch

  • On the appliance, run ./bin/cli.rb deploy <branch-name> --provider local [--domain <name-for-domain>]
    <branch-name> The name of the feature branch created earlier
    --provider local Th is configures, how the script talks to ManageIQ (local = run rake evm:automate:import). Another provider would be noop which does everything, except talking to ManageIQ

  • There should be a new Domain in Automate-Explorer, which contains the changed files

Next:
I am planing at least 2 more features:

  • Delete feature-domains, which have no associated branch in git anymore (PR merged and the branch deleted)
  • Create Automate Domains for feature-branches if they don’t exist (currently each one has to be created explicitly)

After that, it is mostly quality improvements like tests, packaging (gem and/or rpm), way more documentation and other stuff expected from an open source project.

Any ongoing bugs/enhancements on existing features should be tracked on the issues page


I though I keep updating this post, as long as the script is still in development and do a proper introduction post once I am ready for release.
However if you want to try it out, let me know! I would be happy to see it being useful for other people too.


#8

Important Changes:

  • The Script can now handle multiple domains for each feature branch
  • I am now actually discovering AeDomains from the filesystem, instead of relying on configuration
  • Added list and inspect commands for feature branches (I think you can guess, what they do)

Notes:
I just merged a PR refactoring a bunch of Automate domain related code. Previously the code could only handle one Automate domain and it relied on default parameters and configuration to find and import the domain.
With this PR the script actually looks for the __domain__.yaml and reads the information it needs from the filesystem.

The new list command lists remote branches in the repository and finds domains with changed methods. Domains with 0 changed methods are (of course) not imported.

./bin/cli.rb list  
origin/feature-1-f1: feature_f1_buc: 1 feature_f1_foo: 0

The new inspect command displays some git related information: branch name, the current commit in the branch and the last common commit with master and lists all changed files

./bin/cli.rb inspect feature-1-f1
Feature: f1 on branch feature-1-f1 
 Branch: 64ee11bd2293a6f6ee4b1734fe7facab3b9b5003: another change
   Base: 68b8995ea91c3d1857be40d17c11e0eb351f3152: Merge pull request #7 from ThomasBuchinger/feature-1-f1

feature_f1_buc
  automate/buc/Cloud/VM/Retirement/Email.class/__methods__/vm_retirement_emails.rb
feature_f1_foo

Another thing I am working on is the requirement to have the domains editable in the WebUI.

This requires an export, that does not blindly overwrite changes committed to git (which is what happens out of the box), since the last import. Currently it is not part of this script, unless someone else needs that feature


#9

Code Stuff
There are no functional changes this time around, however there are improvements regarding quality (in increasing order of excitement):

  • rubocop and bundler
    I started using bundler to manage my 3 dependencies and rubocop will keep me from carelessly mixing up my formatting
  • First automated test on Travis!
    It is only a shell script, that test 1 (one) pretty basic function for now. But there will be more down the road
  • YAML Config
    I refactored the way I handle configuration to properly handle default values, YAML config and CLI flags (including a --config parameter)

Random Ramblings
As you may have noticed I have not referenced the project by name yet. I would like to pick something vaguely related to chess to stick with the ManageIQ theme, however the glossary of chess apparently is no great source for inspiration. Any suggestions are welcome.