UI hangs if errors are encountered during Git Import

michaelbutak · June 9, 2020, 10:12pm

Hello -

In 2017 there was a bugzilla bug (see here) about a bug I’m experiencing now in 2020.

When I try to import from Github under Automation>Automate>Import/Export, the UI hangs and then gives the following error:

@dm00 @eclarizio @gmccullough you all chimed in on that bugzilla bug. Any pointers? I’ve stopped/started my app servers, logged out/in. The console log has the following errors:

gmccullough · June 10, 2020, 1:24pm

@lfu @Fryguy Can you review this issue or assign someone to look into it? Thanks.

michaelbutak · June 10, 2020, 4:42pm

@gmccullough @lfu @Fryguy here is what the logs printed:

michaelbutak · June 10, 2020, 4:44pm

@gmccullough @lfu @Fryguy Wait, my application is behind a corporate firewall. I have a request to configure the proxy server, which I haven’t started. I suspect this is the cause of the connection time out. Or would that instead be a connection refused?

Fryguy · June 10, 2020, 5:26pm

@michaelbutak IMO the UI should not hang as this should be an async process, so I’d expect the spinner to keep spinning. Please open a bug at github.com/ManageIQ/manageiq-ui-classic with repro steps. Thanks!

michaelbutak · June 10, 2020, 6:02pm

Will do @Fryguy! Thank you.

michaelbutak · June 10, 2020, 6:26pm

@Fryguy @lfu @gmccullough New issue created here.

michaelbutak · June 11, 2020, 10:26pm

@Fryguy @gmccullough @himdel Seems like the issue is that I’m behind a corporate firewall. Would the fix be to add my proxy server to the advanced configuration file:

git_repository_proxy:
:host:
:password:
:port:
:user:
:scheme:
:path:

Fryguy · June 14, 2020, 4:11pm

If you have a proxy server I would give that a try.

michaelbutak · June 15, 2020, 6:08pm

@Fryguy tried this, still doesn’t work, even after cycling the appliances:

If anyone has any suggestions, I would really appreciate it!

Fryguy · June 16, 2020, 1:44am

@NickLaMuro Do you know? I thought we got proxies working but I can’t remember offhand.

NickLaMuro · June 16, 2020, 1:41pm

@Fryguy I know that @carbonin added the changes to rugged to add proxy support, but I didn’t really get a chance to check it myself.

I kinda assumed it worked, but didn’t really have a test environment to test it with. I can maybe see if I can test it a bit today, but no promises.

NickLaMuro · June 16, 2020, 2:00pm

For reference, this is the PR that adds proxy support to rugged that has yet to be merged by the team over at github:

Still in limbo after ~9 months, so we end up using our fork of it in prod:

github.com

ManageIQ/manageiq/blob/4701b115e424701719c12d2931c720819c80fda6/Gemfile#L83


gem "ruby-progressbar",               "~>1.7.0",       :require => false
gem "rubyzip",                        "~>2.0.0",       :require => false
gem "snmp",                           "~>1.2.0",       :require => false
gem "sprockets",                      "~>3.7.2",       :require => false
gem "sqlite3",                        "~>1.3.0",       :require => false
gem "sync",                           "~>0.5",         :require => false
gem "sys-filesystem",                 "~>1.3.4"
gem "terminal",                                        :require => false

# Modified gems (forked on Github)
gem "rugged",                         "=0.28.2.2", :source => "http://rubygems.manageiq.org", :require => false
gem "ruport",                         "=1.7.0.3",  :source => "http://rubygems.manageiq.org"

# In 1.9.3: Time.parse uses british version dd/mm/yyyy instead of american version mm/dd/yyyy
# american_date fixes this to be compatible with 1.8.7 until all callers can be converted to the 1.9.3 format prior to parsing.
# See miq_expression_spec Date/Time Support examples.
# https://github.com/jeremyevans/ruby-american_date
gem "american_date"

# Make sure to tag your new bundler group with the manageiq_default group in addition to your specific bundler group name.
# This default is used to automatically require all of our gems in processes that don't specify which bundler groups they want.

michaelbutak · June 16, 2020, 2:27pm

Thank you @NickLaMuro, @Fryguy

Fryguy · June 16, 2020, 3:52pm

Only thing I can think of is that you may also need to specify the scheme (e.g. https). username and password should be optional unless you have a password on your proxy.

NickLaMuro · June 16, 2020, 4:13pm

So, for other reasons, I have been messing around with mitmproxy, and was just able to get a repo clone through that working for this on a Vagrant Appliance VM I was testing with.

Steps I went through:

Downloaded and extracted the linux binaries for mitmproxy: mitmproxy.org
Ran ./mitmproxy (this was just extracted in to the home dir)

Updated my git proxy settings to:

:git_repository_proxy:
  :host: 127.0.0.1
  :password: 
  :port: 8080
  :user: 
  :scheme: http
  :path:

When adding the repo, uncheck Verify Peer Certificate since that tries to verify the peer of the proxy, and not of the endpoint of the git remote.

This allowed me to fetch the repo and the tags just fine. However, I did end up getting the following error:

Error: import failed: unknown attribute 'scope' for MiqAeInstance.

So seems like there is something wrong with the repo contents itself, but that is not related to the proxy or the git import having troubles.

My guess is there is something wrong with the proxy/firewall that isn’t allowing the connections to go through.

I was able to also test that things were working with the proxy by doing the following in a Rails console:

$ vmdb
$ bin/rails c
irb(main):001:0> GitRepository.where(:url => "https://github.com/rhtconsulting/miq-Utilities.git").first.update_repo
=> true
irb(main):002:0> # if you need to turn off ssl verification, you can do the following and re-run the above
irb(main):003:0> GitRepository.where(:url => "https://github.com/rhtconsulting/miq-Utilities.git").first.update(:verify_ssl => 0)
=> true

If the proxy doesn’t work, you will probably get an error like:

Traceback (most recent call last):
       16: from bin/rails:4:in `<main>'
       15: from bin/rails:4:in `require'
       14: from railties (5.1.7) lib/rails/commands.rb:16:in `<top (required)>'
       13: from railties (5.1.7) lib/rails/command.rb:44:in `invoke'
       12: from railties (5.1.7) lib/rails/command/base.rb:63:in `perform'
       11: from thor (1.0.1) lib/thor.rb:392:in `dispatch'
       10: from thor (1.0.1) lib/thor/invocation.rb:127:in `invoke_command'
        9: from thor (1.0.1) lib/thor/command.rb:27:in `run'
        8: from railties (5.1.7) lib/rails/commands/console/console_command.rb:97:in `perform'
        7: from railties (5.1.7) lib/rails/commands/console/console_command.rb:17:in `start'
        6: from railties (5.1.7) lib/rails/commands/console/console_command.rb:62:in `start'
        5: from (irb):1
        4: from app/models/git_repository.rb:89:in `update_repo'
        3: from app/models/git_repository.rb:83:in `with_worktree'
        2: from app/models/git_repository.rb:188:in `handling_worktree_errors'
        1: from app/models/git_repository.rb:191:in `rescue in handling_worktree_errors'
MiqException::MiqUnreachableError (SSL connections to proxy are not supported)

michaelbutak · June 17, 2020, 4:10pm

You were right @Fryguy. Specifying the scheme for my proxy made it work. Thank you!

michaelbutak · June 19, 2020, 2:19pm

Thank you @NickLaMuro