[SOLVED] Git-flow for Automate Code

#1

Hello,
I am trying to upgrade our workflow to develop automate code to a git-flow workflow (https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow) and I was wondering if someone already did it and can point me to some resources on their solution.

I have a rough idea on how it could work, but it would require quite a bit of scripting to do it, therefore I thought I’d ask the community first, to validate my assumptions and get another opinion. The main problems I am facing are:

  1. There is no way to export or import on a more granular level than a single automate domain.
  2. Since higher priority domains overwrite everything from lower priority domains, any domain modeling a feature-branch MUST only contain changed files.
  3. I cannot use the git-import feature for feature-branches, because I need to import a dirty working copy.

A bit of Background:
At the moment we have a DEV-Cloudforms and a PROD-Cloudforms appliance. Every week we do an export from DEV, commit the whole export to git and import the same commit PROD.
All the development happens in the WebUI and we want to have the possibility to do small changes in the UI (espacially changes in dialogs, etc.)

The problems:

  • At the moment git is just a documentation tool. There is no save way to contribute via git and not have that change overwritten or producing a merge-conflict.
  • All the code is pushed to PROD all the time, If a change is not yet ready for production, it has do be manually disabled in code ( e.g. with an if-statement) before the export and enabled again after the export. => Doing a production deployment at the wrong time could break everything
  • This process is against all coding best practices :slight_smile:

Implementation:

  • Have a complicated git repo, with at least 3 types of branches.
    • release-branches: one for PROD and one for DEV
    • helper-branches: this is where automated commits go
    • feature-branches: this is where humans work
  • Export everything from DEV
    • do really small commits (e.g. for each class) to the helper-branches
    • at this point all the manual changes are in the helper-branches
  • merge the helper-branches to the relese branches
  • Import PROD
    • release-branches should work fine
    • import feature-branches, after removing all unchanged files
  • do_magic()
    • create new branches as needed and prune old ones
    • automatically maintain all the branches
    • keep all the merges free from any possible conflicts

As I said, the goal for this post is to validate my assumptions and to see if someone has already done something similar already. The rest is just background information and my thoughts on the topic.

0 Likes

#2

Not exactly the same but we are using git repo with branch too.
We have one Prod manageiq that rely on prod branch, and one dev manageiq which use development branch. When we are ready to push new functionalities, we merge branch and got a git import.
You can also do webhook in order to trigger the git import on your dev server when someone do a push.

0 Likes

#3

Hi @buc
Currently we are using such branching model, but big problem for us is that configuration for domain stored with code:

If it wasn’t for this problem, we would work in the following way:

  1. in my own tenant on the development appliance I connect a feature-branch domain from the repository.
  2. I’m working on it.
  3. Doing PR to the devel branch, which is connected on the same appliance, but applies to all tenants.
  4. After running the tests, the code goes to the master branch.

Devel-branch and Master-branch domains have the same configuration in _domain_.yaml file, but feautre-branch domains have different configuration for tenant_id. For this reason, we do cherry-picks from feature-domains to devel-domain.

Storing configuration with code is known anti-pattern and I think we should pay attention of MIQ developers on this.

0 Likes

#4

@LorkScorguar: That is exactly what I plan to do with the release-branches.

@igortiunov: I was not aware that automate domains can be scoped to apply to just one tenant (reference: https://pemcg.gitbooks.io/mastering-automation-in-cloudforms-4-2-and-manage/content/tenancy_and_automate/chapter.html). It is definitely a much cleaner solution, than my approach.

Regarding the __domain__.yaml:
I guess it would break the import if you just add it to .gitignore or something similar? :slight_smile:
However the rake-import-task has an option to specify the tenant (https://github.com/rhtconsulting/cfme-rhconsulting-scripts/blob/master/rhconsulting_miq_ae_datastore.rake). It looks like it does, what you want and having that function in the git import is probably just a matter of exposing the parameter(?)

Thanks for the feedback

0 Likes

#5

@buc
ssh on appliance is not a solution. I am using domain refresh via API: http://manageiq.org/docs/reference/latest/api/reference/automate_domains

I am sure that the domain configuration should not be together with the code. This configuration need to specify during the import from the git repository or at the time of updating from the repository.

0 Likes

#6

/push

I am currently working on a ruby script, that prepares “restricted” views of the Automate Code using the information in git and imports them with the evm:automate:import rake-task. The basic concept is the same as outlined above, I just trashed the 3-way-merge-with-branches-thing and simplified some details.

The workflow I am aiming for would be like this:

  • Create a new PR (and the associated branch)
    • CI executes the script and creates a new emtpy Automate domain
  • Copy the relevant methods in the new feature domain (via the WebUI)
  • Work on the Code
    • There is one Jenkins Job to export the current code from the UI to git (using evm:automate:export)
    • There is another Job to deploy the current code from git to ManageIQ (could be a push trigger)
  • Once it is ready for release, merge the PR and remove the branch
    • The script will detect, that there is a feature-domain without a feature-branch to go with it and destroy the domain (triggered periodically)

Limitations

  • 2 people working on the same file will shadow each other
  • You can alternate between git and WebUI but not both at the same time
  • I am expecting the following naming convention <prefix>-<issue_id>-<domain_name>-* (e.g. feature-1-feature1-this-text-doesnt-matter-but-it-is-usually-the-issue-title)

Advantages over a single develop-branch

  • You can use PRs for Code Review
  • There is no “change-freeze” for deployment to PROD
  • You don’t accidentally leak half finished code to PROD
1 Like

#7

/update

I finally feel confident enough to actually post a link to what I am working on, so here it is: https://github.com/ThomasBuchinger/automate-gitflow

State:
I finished the deploy (aka import) part and a bunch of stuff around error handling, configuration options and so on. I am pretty sure it won’t crash with an exception if you call it with a wrong parameter.

  • The script can identify, which files have been changed on the feature-branch, copy them and the associated YAML files to a temporary directory and deploy (import) them to ManageIQ.
  • The repository is cloned in the background and removed again before the script finishes. Alternatively you can point it to a local repository.
  • The imported Domain will only contain the changed Automate methods, thus not overriding lower priority domains

I am thinking about not implementing export, since I think the demand is not that big and it’s pretty straight forward to do in bash.

I have been using it the script the last 2 weeks without any major problems (granted I am the author)

How-to use:

  • Clone the git repository onto your ManageIQ appliance.
    Unfortunately it seems, that there is no way to import Automate Domains via the REST-API, therefore I am using the rake-tasks.

  • The repository includes a file called example_custom.rb, rename it to custom.rb.
    $git_url/git_path configure the git repository to use

  • Create a feature-branch in your repository.
    I am using the naming convention feature-1234-<import-domain-name>-whatever. If your branches are named differently, just pass --domain to the deploy-command.
    You can find the example repo I am using for testing here

  • … DO YOUR CODE CHANGES HERE …
    When you are ready to test your changes, commit and push them to the feature-branch

  • On the appliance, run ./bin/cli.rb deploy <branch-name> --provider local [--domain <name-for-domain>]
    <branch-name> The name of the feature branch created earlier
    --provider local Th is configures, how the script talks to ManageIQ (local = run rake evm:automate:import). Another provider would be noop which does everything, except talking to ManageIQ

  • There should be a new Domain in Automate-Explorer, which contains the changed files

Next:
I am planing at least 2 more features:

  • Delete feature-domains, which have no associated branch in git anymore (PR merged and the branch deleted)
  • Create Automate Domains for feature-branches if they don’t exist (currently each one has to be created explicitly)

After that, it is mostly quality improvements like tests, packaging (gem and/or rpm), way more documentation and other stuff expected from an open source project.

Any ongoing bugs/enhancements on existing features should be tracked on the issues page


I though I keep updating this post, as long as the script is still in development and do a proper introduction post once I am ready for release.
However if you want to try it out, let me know! I would be happy to see it being useful for other people too.

1 Like

#8

Important Changes:

  • The Script can now handle multiple domains for each feature branch
  • I am now actually discovering AeDomains from the filesystem, instead of relying on configuration
  • Added list and inspect commands for feature branches (I think you can guess, what they do)

Notes:
I just merged a PR refactoring a bunch of Automate domain related code. Previously the code could only handle one Automate domain and it relied on default parameters and configuration to find and import the domain.
With this PR the script actually looks for the __domain__.yaml and reads the information it needs from the filesystem.

The new list command lists remote branches in the repository and finds domains with changed methods. Domains with 0 changed methods are (of course) not imported.

./bin/cli.rb list  
origin/feature-1-f1: feature_f1_buc: 1 feature_f1_foo: 0

The new inspect command displays some git related information: branch name, the current commit in the branch and the last common commit with master and lists all changed files

./bin/cli.rb inspect feature-1-f1
Feature: f1 on branch feature-1-f1 
 Branch: 64ee11bd2293a6f6ee4b1734fe7facab3b9b5003: another change
   Base: 68b8995ea91c3d1857be40d17c11e0eb351f3152: Merge pull request #7 from ThomasBuchinger/feature-1-f1

feature_f1_buc
  automate/buc/Cloud/VM/Retirement/Email.class/__methods__/vm_retirement_emails.rb
feature_f1_foo

Another thing I am working on is the requirement to have the domains editable in the WebUI.

This requires an export, that does not blindly overwrite changes committed to git (which is what happens out of the box), since the last import. Currently it is not part of this script, unless someone else needs that feature

1 Like

#9

Code Stuff
There are no functional changes this time around, however there are improvements regarding quality (in increasing order of excitement):

  • rubocop and bundler
    I started using bundler to manage my 3 dependencies and rubocop will keep me from carelessly mixing up my formatting
  • First automated test on Travis!
    It is only a shell script, that test 1 (one) pretty basic function for now. But there will be more down the road
  • YAML Config
    I refactored the way I handle configuration to properly handle default values, YAML config and CLI flags (including a --config parameter)

Random Ramblings
As you may have noticed I have not referenced the project by name yet. I would like to pick something vaguely related to chess to stick with the ManageIQ theme, however the glossary of chess apparently is no great source for inspiration. Any suggestions are welcome.

0 Likes

#10

/update
Happy 12019 for those of you following the holocene calendar and back my usual semi-regular schedule.

The next milestone I am targeting is a full release with:

  • A downloadable gem package
  • Version numbers and a real Changelog (not whatever this post turned into)
  • A proper introduction post

I will need a few more of my 2-week-ish development cycles to get there, but it is getting closer

Changes

  • Added support to query the Automate Model via /api/automate-Endpoint
  • Added list miq command to list Automate domains
  • Renamed list command to list git
  • Refectored the CLI-Interface code
  • Renamed the binary miq-workflow
  • Added dependency test to travis and updated ManageIQ Container images used

How to use it?
Prerequisites:

  • You want to run the script on a ManageIQ appliance, because it needs access to the evm:automate:import rake task
  • Your code is stored in a git repository and you have used rake evm:automate:export or rhtconsulting scripts to export the code.
  • Your feature branch names start with feature or fix

Install miq-workflow

  • Clone the repository
  • Open config.yaml
    • Configure git.url (or git.path) to point to the repository with your automate code.
      • Credentials to clone are configured with git.user and git.password
      • Cloning via SSH is not supported on ManageIQ Appliances
    • Configure miq.url to point to the API Endpoint of ManageIQ (currently only used for list miq-command)
  • Change your working directory to the repository root (otherwise it won’t pick up the configuration in config.yaml)

miq-workflow commands

./bin/miq-workflow list git
origin/feature-1-f1: feature_f1_buc: 1 feature_f1_foo: 0
origin/master: feature_origin/master_buc: 0 feature_origin/master_foo: 0

List Automate Domains and the number of changed files, for every feature branch in git.
The displayed name the default name used by the deploy-command


./bin/miq-workflow list miq
feature_f1_buc: ID=77 Enabled=yes Prio=1 Last Update=1 minute ago
ManageIQ (Base domain): ID=1 Enabled=yes Prio=0 Last Update=4 days ago

List Automate Domains currently in ManageIQ


./bin/miq-workflow inspect BRANCH
Feature: f1 on branch feature-1-f1 
  Branch: 64ee11bd2293a6f6ee4b1734fe7facab3b9b5003: another change
  Base: 68b8995ea91c3d1857be40d17c11e0eb351f3152: Merge pull request #7 from ThomasBuchinger/feature-1-f1

feature_f1_buc
  automate/buc/Cloud/VM/Retirement/Email.class/__methods__/vm_retirement_emails.rb
feature_f1_foo

Displays additional information for a given feature branch. This includes the latest commit in the branch, the commit you branch branched of from, as well as the Automate Domains and a list of changed files


./bin/miq-workflow deploy BRANCH --provider local
** ManageIQ hammer-1-rc1, codename: Hammer
Importing automate domain: buc from directory /tmp/miq_import_20181224-5333-1aesqwi/import/automate
...
{"@timestamp":"2018-12-24T01:27:53.536573","hostname":"08359126461f","pid":20557,"tid":"764f80","level":"info","message":"Your database has been updated."}

This command imports a feature branch into ManageIQ.
This is the (one and only) command that actually does something.

1 Like

#11

/update
This cycle took me a little bit longer, because I tried to get everything ready and finalized for the first release. All the code changes I want for the first release are in, so I published a release candidate over on github.

The next and final step is going to be documentation. Right now I am aiming for a short reference document for the different config parameters and a longer How-to/concept guide explaining prerequisites, primary use-case, …

Download Link: miq-flow-1.0.0-rc1
If you want to give it a try, don’t hesitate to shoot me a message, I would really love to get some feedback. (and it really isn’t black magic anyway)
Side note: Does anybody prefer RPM packaging over GEM? I am still divided over that question


Changes: Exit Codes, Gem, Config

  • The program now always exits with a meaningful exit code.
    List of exit codes: here
  • Fixed a problem, where I did not check the exit code of the import task, causing the program to appear successful, even when the import was not.
  • All the CLI commands are now covered with automatic tests.
  • Refactored the code to work with bundler as packaging tool
  • Updated my test-docker images to latest-hammer and latest
  • Renamed binary to miq-flow
  • Made sure every configuration option is configurable via CLI parameters, environment variables and a config file (with this precedence)
  • Set configuration searchpath to /etc/miqflow.yaml, ~/.miqflow.yaml or ~/.miqflow/config.yaml
  • Add support for configurable branch naming conventions.
    Previously the code assumed branches to follow a type-ref-name-optional-description naming scheme (which is common). there was a option to override that, but that was not always honored correctly.

Export Support

I mentioned, that one of my requirements is to keep the export based workflow (described in the first post) working. to pull this off, you need to remember the last time the code was imported (easy) and

  • blindly overwrite any conflicting changes between the last import and current master (easy)
  • or dedect conflicts. Save the conflicting commit on another branch and notify a human (ideally by opening a PR) (not so easy)

I am currently working on option 2 as a standalone project. It is basically a ansible module (written in ruby), but can also be called as a rake task (along with a shell script for option 1).
It is not going to get the same level of polish as the main project (as long as there is no demand for it), but in case you need that too, there is someone else working on the problem.
Repo: https://github.com/ThomasBuchinger/miq-flow-export


Current CLI commands

miq-flow deploy BRANCH-NAME --provider=<local|docker|noop> [--name import-name]
miq-flow branch list
miq-flow branch inspect BRANCH-NAME
miq-flow domain list

Options:
  [--verbose], [--no-verbose]      # Turn on verbose logging
  [--quiet], [--no-quiet]          # Only show errors and warnings
  [--silent], [--no-silent]        # Do not output anything
  [--cleanup], [--no-cleanup]      # Clean up the working dir before exiting
  [--workdir=WORKDIR]              # Override the working directory
  [--config=CONFIG]                # Specify config file to load
  [--git-url=GIT_URL]              # Git clone URL for remote repositories
  [--git-path=GIT_PATH]            # path to a local git repositories
  [--git-user=GIT_USER]            # Username for remote repositories
  [--git-password=GIT_PASSWORD]    # Password/token for remote repositories
  [--git-separator=GIT_SEPARATOR]  # List of characters separating part of your branch naming convention
  [--git-index=N]                  # Index the NAME par of your branch naming convenion
  [--miq-url=MIQ_URL]              # ManageIQ API URL. (e.g. https://localhost/api)
  [--miq-user=MIQ_USER]            # ManageIQ API User. (default: admin)
  [--miq-password=MIQ_PASSWORD]    # Passwork/login-token for the ManageIQ API User
0 Likes

#12

/update

No (code) changes this time. However I finally worked on that documentation thing I’ve been talking about since the 2nd post :slight_smile:

Docs

README
The README file has been out of date for the last 2 weeks, that is now updated.

User Guide
I added a user guide, that starts from a fresh installation of ManageIQ, walks you through setting up a repository for your Automate code, installs and configures miq-flow, describes how one would develop code with miq-flow and outlines a few options how to release code developed in this proposed workflow.

It turned out a bit long, but it is a step-by-step guide and I had a lot to talk about…

Note: some (most) of the links do not work, because they point to some additional documenation, that I am still working on.

Link: https://github.com/ThomasBuchinger/miq-flow/blob/master/doc/user_guide.md

Proposed Workflow

Since I haven’t updated the proposed workflow here, the current version in a nutshell:
Setup:

  • Install the miq-flow gem
  • Pick one of 0-2 options, how to merge the feature branches back to master. The setup isn’t that hard, but you have to work around some restrictions in ManageIQ.

Dev cycle:

  • Create a PR (and the associated branch)
  • Work on the code
    • The CI executes miq-flow for every push
    • miq-flow creates those “restricted views” of the Automate domain based on the changed files in git
  • Test your code in Automate
  • Once the code is ready for release merge the PR
    • Currently you have to remove the feature domain miq-flow created for you manually

As always, feedback is very much appreciated.

0 Likes

#13

/update
I am currently working on some smaller side projects, therefore the changes to master are only a couple of bug fixes and minor improvements.

Changes

  • miq-flow has a changelog now!
  • --verbose flag now prints the full stack trace if uncatched error occurs. previously the stack trace was always silently ignored
  • Domain priorities are now honored during the import.
  • Fixed (display) bug, when two domains had similar names (e.g. buc and buc_lib)
  • If the metadata for a inline method changes (e.g. by adding an embedded method) but the body remains unchanged, the import would not import the method body

Provider
I am not happy, that we need to install miq-flow on a ManageIQ appliance, I have some ideas to work around that, but that needs more experimenting

Diff command
The current usage pattern requires you to make all code changes (including typos, experiments, …) locally and push them to git, if you modify the code in the WebUI, you need redo everything in git.

What you really want to do is: Deploy an initial version using miq-flow. Fix typos and do your debugging in the WebUI (because it is faster), diff the current version in ManageIQ with the initially deployed version. Commit the final version of the code in git.

I got the initial version working, however I am not happy with it at all and need to rewrite the whole thing again.

Dead Code dedection
Another experiment I am working on is finding unreachable code in Automate.
Basic idea:

  • Start with a list of known entry points to automate
  • Build a tree of all referenced Automate Methods from this list
  • Compare the tree with an Automate export
  • Remove the unreferenced files
  • PROFIT

If that sound like a problem you have, feel free to contact me

0 Likes