4

Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.

My understanding is that I have a few ways to pass those secret data to Spot instances:

  • Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)
  • Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)
  • Use Secrets Manager, but that could quickly cost a lot...
  • Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)

And I am not sure I could individually identify spot instances?

Am I missing something there? How do you solve those challenges?

Philippe
  • 155
  • 4
  • A variant of instance Ebs is to share an EFS file system – jdog Feb 22 '19 at 06:26
  • 1
    "lot of engineering there" Are you sure? https://docs.aws.amazon.com/kms/latest/developerguide/services-parameter-store.html – Michael - sqlbot Feb 22 '19 at 08:03
  • @Michael-sqlbot Thanks for your feedback, I rephrased. – Philippe Feb 22 '19 at 11:51
  • But maybe I am just wrong to think about node-specific data and I should accept that all nodes are the same? It's very hard for me as I cannot, for example, allow myself to blindly trust an unknown SSH host key. – Philippe Feb 22 '19 at 11:52
  • A quick one with the ssh keys, you don't need to trust unknown ssh keys. I've used the approach linked below a lot and it works really well, it's simple to implement and secure enough. Also it's worlds easier to maintain than a key per instance or user. https://code.fb.com/security/scalable-and-secure-access-with-ssh/ – Briansbum Feb 22 '19 at 14:02
  • @Philippe I see what you mean, now. Thank you for the clarification. But it does feel like you might be overcomplicating the issue. SSM Parameter Store allows granular access control of which parameters (by path prefix) any given set of credentials can access. If your instances are doing the same work, there should be no need for them to have individual "personalities" -- they can share one IAM role that allows them to access the relevant parameters, by path, and SSMPS/KMS integration can use the shared KMS key. You can't decrypt what you can't access. – Michael - sqlbot Feb 22 '19 at 15:11

1 Answers1

3

When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.

The two most common methods I've seen for doing this are:

  1. If your provisioning tool (terraform, cloudformation, salt cloud) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.

  2. You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use Vault. It's not that hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.

Briansbum
  • 1,112
  • 9
  • 13