Lost encryption key and unable to connect to OLD external DB

Hi, we have completely lost (due to some ceph/openstack issue) our test appliance cluster with 2 hammer instances and there is also no available backup of our former encryption key.

Our external PostgreSQL DB survived (which hopefully contains our complete datastore and forms?), but we are not able to connect to it from our new hammer appliances.

Is there any procedure how to connect to existing region again with new encryption key?

  1. I connected to clean appliance and run appliance_console

  2. selected number 7 (Configure Database), than 3 (Join Region in External Database) which asked for new encryption key, so I selected “create new one”

  3. I filled connection data and than I got:

    Activating the configuration using the following settings…
    Host: PT01301
    Username: miq_own
    Database: miq
    Port: 5432

``Configuration failed`

this is in evm.log:

[----] I, [2019-04-16T13:41:36.008365 #16489:52af58]  INFO -- : MIQ(EvmDatabase.seed) Seeding...
[----] I, [2019-04-16T13:41:36.051307 #16489:52af58]  INFO -- : MIQ(EvmDatabase.seed) Seeding MiqDatabase
[----] E, [2019-04-16T13:41:36.093273 #16489:52af58] ERROR -- : [ManageIQ::Password::PasswordError]: can not decrypt v2_key encrypted string  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2019-04-16T13:41:36.093461 #16489:52af58] ERROR -- : /usr/local/lib/ruby/gems/2.4.0/gems/manageiq-password-0.3.0/lib/manageiq/password.rb:43:in `rescue in decrypt'
/usr/local/lib/ruby/gems/2.4.0/gems/manageiq-password-0.3.0/lib/manageiq/password.rb:40:in `decrypt'
/usr/local/lib/ruby/gems/2.4.0/gems/manageiq-password-0.3.0/lib/manageiq/password.rb:72:in `decrypt'
/var/www/miq/vmdb/app/models/mixins/password_mixin.rb:24:in `block in encrypt_column'

I have also tried fix_auth.rb with no luck

Am I doing it wrong?
Thank you very much

The database contains all the persistent state in ManageIQ. So yes, the Automate Code and Dialogs are in there too.

fix_auth.rb is a tool to re-encrypt the passwords in the database, if you know the new and the old v2_keys.

If you don’t have the password/encryption key, your best chance is start with a new appliance and database, dump the old database to a file and import everything in the new database.
Most of the stuff in there isn’t encrypted and should work just fine, but you will have to recreate everything that involved passwords/credentials. Maybe it works in the UI, if not you will most likely have to use the big guns (i.e. use rails console :slight_smile: )

@kbrock Can you help out here?

We tried to import pg dump into empty region, but got almost nothing back (just catalog items)

Luckily I have found one manual datastore export zip file created just few days before the incident. Also most dialogs were recovered from my recently exported yaml files. We’ve also found in pgdump timestamped changes so we were able to pick latest datastore changes

We need to be able to read from the database. If there are records in the database that we can not read, it gets confused and it blows up, as you have seen.

The job of fix_auth is to fix problems with authentications in a database. From your stack trace it looks like this is your problem, so lets fix this problem.

In a perfect world, you know the old encryption key and the new key, and you can upgrade from the old key (--legacy) to the new one.

However, in your case, you do not know the old key. Without your old key, there is no way to recover the old passwords. So, we can tell the tool to change any invalid password to a place holder. You can pick any placeholder, but for our example we’ll use the word "bogus":

fix_auth --db --invalid bogus

Next, start up the app and update system passwords like the ems password. They are currently the place holder you picked (which will probably not work on the various systems)

1 Like

So we have loaded the original pgdump and tried:
# ruby fix_auth.rb --db --invalid bogus

output:

fixing authentications.password, auth_key
processed 0 records
fixing miq_databases.registration_http_proxy_server, session_secret_token, csrf_secret_token
processed 1 records
fixing miq_ae_values.value
processed 0 records
fixing miq_ae_fields.default_value
processed 0 records
fixing settings_changes.value
processed 0 records
fixing miq_requests.options
processed 0 records
fixing miq_request_tasks.options
processed 0 records

But same PasswordError as in my initial question was thrown during server start again…

Not sure your version, but some versions want ruby fix_auth.rb --db --v2 --invalid bogus (can’t remember when the --v2 became no longer necessary)

Do you have multiple appliances? If you do, do all of them have the same v2_key?

We left others apliances offline. We also use hammer-4 so I suppose there was no need for --v2
argument?

I can’t try it anymore because we already have whole TEST environment up and running again. Ultimately we’ve lost like 2 days work but already managed to make it good…

Hopefully this thread will help someone in the future

Thank you very much

you know, your database.yaml may have been the culprit.

fix_auth --databaseyml --password smartvm

pick what ever your database password you have.

Best of luck.