AWS Tag Synchronization

Developers
#1

We need tag synchronization between AWS and MIQ! Here are the details on why and how we leverage tagging in AWS.

AWS Tags

Tagging resources in AWS is not optional. Each team is responsible for tagging their resources, whether it is a short term (ephemeral) environment for testing, or a long standing (static, non-ephemeral) environment.

At the very least it allows us locate the team or individual responsible for a particular resource in AWS.
It also allows the consolidated bill to be reconciled against the projects as well as provide administrative access.

Tags enable you to categorize your AWS resources in different ways, for example, by purpose, owner, or environment. There are some set tags that AWS provides like stack name.

Each custom tag consists of a key and a value, both of which you define. For example, you could define a set of tags for your account’s Amazon EC2 instances that helps you track each instance’s owner and stack level.

Projects are expected to Tag every single resource inside of the AWS account and risk the resource being removed if the appropriate tags are not implemented.

The tags that are required are:

Project – Name of the project
Environment – Dev, QA. PProd, Prod, DR
Component - REST, web, scaler, worker - a deployable chunk of a project/system - could deploy a fix to one without having to redeploy the other pieces
Version - Deployed version
OwnerEmail - the email address of owner of the deploying resources.

Tags should be applied programmatically so to prevent any untagged resources entering AWS and also to provide a consistency in the use of them.

Thanks

Sync Vmware and ManageIQ tags (VM tags)
#2

I thought we already did this by saving them into the manageiq advanced_settings table, but apparently not. The plan was to treat them exactly the way we treat advanced settings on vSphere. I don’t know why they are not there. We do pull the name from the tags, and store that in the name column, but it seems that’s the only one we reference.

Tagging resources in AWS is not optional

Technically this is not correct, you can very well have entities that have no tags, even no name.

This seems more of a specific restriction due to customer requirements, not something that ManageIQ should enforce for every user. Most of this can, and probably should, be done in Automate. We may have to expose setting those tags however, as I think that cannot be done through automate directly (though one could use the aws-sdk gem in automate to get it working).

#3

Yes you are correct. I should have clarified that the info posted is our internal IT tagging requirements,

#4

It seems that the notion of tags, as it was introduced by ManageIQ way back when, is now extremely pervasive. Not only the notion of tag. but most if not all of the providers have also embraced that “categorization” side of the tag (Key-Value pair). See the following public links confirming this:

AWS: Tag Basics
VMware: Tagging Objects in the vSphere Web Client
Microsoft: SCVMM (virtual) environments

Handling those “provider tags” as attributes was a great first step in making them available for reporting or used in an expression gating a policy, but having them tightly integrated in the ManageIQ tag concept could allow to get to the next level of provider integration.

With that in mind, wouldn’t it make sense to expand the concepts of “Tag Source” and “Tag Function” where as, as part of the tag definition the system could capture the source (Provider Vs System) as well as what functions this/these tag(s) and tag(s) categories may be used for in the concept of ManageIQ Engine (RBAC, Automation, Policy,…).

Thoughts?

1 Like
#5

I wouldn’t go that far. Tags were around way before ManageIQ. GMail had tagging before we did, tag clouds were common, so I could argue it was pervasive before we even got to it. What we did was introduce the concept of categorization, but that is not the same as a key-value pair.

Tags and key->value pairs are two completely different things. “Real” tags are like just a key that you assign to an object. What AWS calls “Tags” is equivalent to Custom Attributes on vSphere. AWS and others should not call them tags…it’s confusing the terminology.

The advanced_settings and custom_attributes tables are meant for these purposes. I totally agree with keeping those in sync, and in fact, I thought we already did that for AWS. Note that advanced_settings for vSphere is like a zillion entries…we would never want those to all become tags. In fact, we had to change refresh to not collect those, because it was so large…they are only available via SmartState Analysis™.

As far as “expanding” tags go, code-wise I don’t see how they would mix together further, except to perhaps make custom_attributes more entwined with functions like RBAC. Policy and Automate should already be able to see custom_attributes right now. If the concerns is converting those to MIQ tags, then you can kind of already do that now with some sort of automate-post-refresh-create-tags-based-on-custom-attributes thing.

#6

Tags offer ease-of-use to the end-user. I agree that anything could virtually be done through custom attributes BUT the tagging model and how the tags are being used in ManageIQ is what makes ManageIQ so efficient.

By normalizing the concept of tags inherited from the providers and how those are being treated within ManageIQ, we would basically immediately provide all the current benefits associated with the tags to those end-user.

It is not trivial to create an expression or a report based on custom attributes. Those custom attributes don’t participate in things like chargeback, capacity management amongst other things.

Also, per link above, VMWare now has the same tag definition as AWS. Specially see the section on Migrate Custom Attributes to Tags .

My point was merely to point out that the “Tags” as they are defined in MIQ also match that definition in the managed providers and there is a gap on how we handle them from provider to provider (for those where they are available). With that in mind, I think there is merit in normalizing the way they are being handled by ManageIQ.

#7

I don’t agree. For example, take the “Name” tag. Amazon uses its “tags” for the name of an instance. Every instance has a different “Name” value, or potentially none at all. If you had 1000 Amazon Instances, are you saying that should be equated to a ManageIQ “Name” category and 1000 individual tags? That’s not how tags are supposed to work, and that seems nuts to me.

If custom_attributes are not sufficient maybe we need to beef that up?

1 Like
#8

@Fryguy
Hi! Currently I have similar task - setting tags for aws instances, but I want to set it from Automation during provision or during day 2 operations. Is there some automation methods for this ?

#9

Found vm.ems_custom_set(key, value)