Create multiple EC2 instances using python script (in loop iteration)

Question

I Have JSON file in which email address and count of instances are specified.I'm using python script to iterate through that file and get username/count number (count number will be used as number of EC2 instances), output from JSON file:

DJukes
1
JWilson
2
eflame
3

So, for DJukes one instance will be created,JWilson 2,eFlame 3, also these names will be usernames for new instances.

Idea is to create as many instances as it's specified in JSON file based on username and count values

When creating manually i got

Terraform will perform the following actions:

-/+ aws_instance.win-example (new resource required)
      id:    

                   "i-073bda5b30f10d3ba" => <computed> (forces new resource)

I passed these variables to python-terraform wrapper (https://github.com/beelit94/python-terraform)

If only one user/count exists in JSON file, no issues, but as soon i add more entries in JSON, then machine for DJukes is created then terminated,same for JWilson, only last created machine remains running. Can someone point out what's wrong with this code ?

#!/bin/python
import json
from pprint import pprint
from python_terraform import *




def myfunc():

  tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':count,'INSTANCE_USERNAME':user})
  tf.plan(no_color=IsFlagged, refresh=False, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.plan())
  print(tf.apply(**approve))
  return

    json_data=open('./my.json')
    data = json.load(json_data)

    json_data.close()

    for i in range (0, len (data['customers'])):
        #print data['customers'][i]['email']
        k=data['customers'][i]['email']
        #print(k.split('@')[0])
        user=k.split('@')[0]
        #print(user)
        count=data['customers'][i]['instances']
        #print(count)
        #enter = int(input('Enter number of instances: '))
        myfunc()

windows.tf:

resource "aws_key_pair" "mykey" {
  key_name = "mykey"
  public_key = "${file("${var.PATH_TO_PUBLIC_KEY}")}"
}

resource "aws_instance" "win-example" {
  ami = "${lookup(var.WIN_AMIS, var.AWS_REGION)}"
  instance_type = "t2.medium"
  count="${var.count}"

  key_name = "${aws_key_pair.mykey.key_name}"
  user_data = <<EOF
<powershell>
net user ${var.INSTANCE_USERNAME} '${var.INSTANCE_PASSWORD}' /add /y
net localgroup administrators ${var.INSTANCE_USERNAME} /add

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

netsh advfirewall firewall add rule name="WinRM 5985" protocol=TCP dir=in localport=5985 action=allow
netsh advfirewall firewall add rule name="WinRM 5986" protocol=TCP dir=in localport=5986 action=allow

net stop winrm
sc.exe config winrm start=auto
net start winrm
</powershell>
EOF

  provisioner "file" {
    source = "test.txt"
    destination = "C:/test.txt"
  }
  connection {
    type = "winrm"
    timeout = "10m"
    user = "${var.INSTANCE_USERNAME}"
    password = "${var.INSTANCE_PASSWORD}"
  }

tags {
Name="${var.INSTANCE_USERNAME}"
}
}

This is pretty impossible to debug without seeing your .tf file, but my best guess is that you are not creating unique EC2 instances. Have you tried running a .json file with only DJukes then only JWilson? Are the instances still terminated then? — Preston Martin, Feb 14 '18 at 19:37
yes, it got terminated, i ran it manually,without script) and get "forces new resources-new resource required"-in red and with - sign — Milister, Feb 14 '18 at 19:49
If you by chance append the ${var.INSTANCE_USERNAME} to mykey and winexample resources (to uniquely identify them), does that create a new instance? — Preston Martin, Feb 14 '18 at 20:10
no :), same as before, read it can be done with count, i tried this but no help:tags { Name="${format("test-%01d",count.index+1)}" } — Milister, Feb 14 '18 at 20:30
Can you check the instance-id of the ec2 instance? Does it change after you run it for each username? — Preston Martin, Feb 14 '18 at 20:52
id is the same, no change, i ran into this code, but no idea how to apply it to my case, i'm new to terrafom:https://groups.google.com/forum/#!topic/terraform-tool/ALwvMGv9-fc — Milister, Feb 14 '18 at 21:12
sorry PrestonM, it seems creating new keypair for every user solves the issue, is this the only way to solve it ? — Milister, Feb 14 '18 at 22:02
Nice! I imagine there are other ways to solve it. It doesn't quite make sense to me why that would allow it to create new ones, but maybe that is how terraform identifies different ec2 instances. Don't be sorry. You found a solution! :) — Preston Martin, Feb 14 '18 at 22:05
thanks for pointing me to right direction, now need to modify python script to pass username as keypair name — Milister, Feb 14 '18 at 22:07
Yep. I would add your answer as well. That way anyone looking at this answer in the future will know of a solution. — Preston Martin, Feb 14 '18 at 22:10

score 1 · Answer 1 · answered Apr 04 '18 at 18:20

The main issue I see is that you are calling Terraform at the end of every python loop. This is creating new "state" file. That will make it really hard to manage what you are creating. This is how I would do it.

users = []
count = 0
for customer in data['customers']:
    username = customer['email'].split('@')[0]
    instances = customer['instances']
    count += instances
    users.extend([username] * instances)
    #print(Added {} to build list with {} instances.".format(username, instances))
myfunc(count, users)

then I would change...

def myfunc(count, users):

  tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':count,'INSTANCE_USERS':users})
  tf.plan(no_color=IsFlagged, refresh=False, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.plan())
  print(tf.apply(**approve))
  return

and finally the Terraform.

resource "aws_key_pair" "mykey" {
  key_name = "mykey"
  public_key = "${file("${var.PATH_TO_PUBLIC_KEY}")}"
}

resource "aws_instance" "win-example" {
  ami = "${lookup(var.WIN_AMIS, var.AWS_REGION)}"
  instance_type = "t2.medium"
  count="${var.count}"

  key_name = "${aws_key_pair.mykey.key_name}"
  user_data = <<EOF
<powershell>
net user ${var.INSTANCE_USERS[count.index]} '${var.INSTANCE_PASSWORD}' /add /y
net localgroup administrators ${var.INSTANCE_USERS[count.index]} /add

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

netsh advfirewall firewall add rule name="WinRM 5985" protocol=TCP dir=in localport=5985 action=allow
netsh advfirewall firewall add rule name="WinRM 5986" protocol=TCP dir=in localport=5986 action=allow

net stop winrm
sc.exe config winrm start=auto
net start winrm
</powershell>
EOF

  provisioner "file" {
    source = "test.txt"
    destination = "C:/test.txt"
  }
  connection {
    type = "winrm"
    timeout = "10m"
    user = "${var.INSTANCE_USERNAME[count.index]}"
    password = "${var.INSTANCE_PASSWORD}"
  }

tags {
Name="${var.INSTANCE_USERS[count.index]}"
}
}

What I did was change your code so we now create a list of the users, that match the length of the count. That way we can call Terraform just once, and use the count.index on our users list to make sure we create the correct amount of instances per user.

You can read more about count.index here. I have always found reading and understanding more complicated code makes me better. Try reading some of the modules from the Terraform Registry. The vpc module gives good examples of using count.

If you still need help drop me a comment.

Create multiple EC2 instances using python script (in loop iteration)

1 Answers1