Posted on by brock

AWS Provisioning and Deployment with Linux EC2 instances using PowerShell

I didn’t expect to find myself needing to learn PowerShell for automation purposes, but I must admit I really like it. It seems sort of like an amalgam of Bash, Perl and Python. It’s an unexpectedly impressive creation from Microsoft. I’ve been using PowerShell on macOS but it can also be used easily on Linux, and Windows of course.

I created three simple PowerShell scripts for automated provisioning of Linux EC2 instances within AWS. Running these will provision an Amazon Linux 2 EC2 instance with SSH key pair and Security Group, with a webapp deployed thereon, plus an associated DNS record in Route 53.

You can find these scripts and related config on my GitHub.

Requirements

  1. An AWS account with a VPC set up, and with a DNS domain/zone set up in Route 53.
  2. PowerShell needs to be installed if you don’t already have it, as per Microsoft’s instructions for installing on macOS, Linux, Windows, etc.
  3. AWS Tools for PowerShell needs to be installed, as per the installation instructions for macOS/Linux or for Windows. I installed the AWS.Tools version on macOS, as opposed to the AWSPowerShell.NetCore version.
  4. SSH needs to be installed for the deployment phase. This is only likely to be an issue on Windows, so follow Microsoft’s instructions for getting OpenSSH installed and configured.

Installing necessary AWS modules in PowerShell

I’m using the AWS.Tools version of AWS Tools for PowerShell on macOS, so I found it necessary to install the necessary AWS modules for shared services, EC2 and Route 53, as follows. (Start Powershell with pwsh if needed before running these commands):

Install-Module -Name AWS.Tools.Common
Install-Module -Name AWS.Tools.EC2
Install-Module -Name AWS.Tools.Route53

With the AWSPowerShell.NetCore version of AWS Tools for PowerShell, installing these additional modules does not appear to be necessary.

Setting up AWS credentials and config

On macOS I found that AWS Tools for PowerShell used my existing AWS credentials and config set up in ~/.aws/credentials and ~/.aws/config (originally set up for use with the AWS CLI) without any additional setup needed.

If you need help setting up your AWS credentials and config (e.g. default AWS region) for use with PowerShell, refer to the Getting Started documentation.

Setting up variables

Copy etc/variables_template.ps1 to etc/variables.ps1 and edit the file as per your requirements. The $ZoneDnsName in particular will need to be changed for the DNS domain you want to use in Route 53.

Usage

Once you’ve grabbed my GitHub repo using whichever method you prefer, run pwsh to enter PowerShell if you’re on macOS or Linux, or start the Windows PowerShell app if you’re on Windows, then change to this PowerShell_AWS_Linux directory to run the scripts.

Provisioning

The ./provision.ps1 script provisions the instance and DNS. This will provision an SSH key pair and Security Group, then launch the EC2 instance from the latest Amazon Linux 2 AMI, then set up a CNAME in the DNS pointing www.mydomain.com to the public DNS of the EC2 instance:

# Source variables file
. etc/variables.ps1

# Create EC2 keypair if it doesn't already exist
# and output private key to local file
if (!((Get-EC2KeyPair).KeyName | Select-String -Pattern "$KeyName")) {
  (New-EC2KeyPair -KeyName "$KeyName").KeyMaterial | Out-File -Encoding ascii -FilePath etc/$KeyName.pem
  Write-Output "Keypair $KeyName created and private key saved to etc/$KeyName.pem"
}

# SSH deployment cannot work unless the private key
# file has the correct permissions. I cannot find a
# standard way of achieving this in PowerShell.
# This command will work on macOS, Linux and Unix
# but won't work on Windows. On Windows this should
# be changed or removed as needed.
chmod 600 etc/$KeyName.pem

# Create EC2 Security Group if it doesn't already exist
# and add inbound rules for SSH and HTTP
if (!((Get-EC2SecurityGroup).GroupName | Select-String -Pattern "$SecurityGroupName")) {
  $SecurityGroupID = (New-EC2SecurityGroup -GroupName "$SecurityGroupName" -GroupDescription "$SecurityGroupDescription")
  $PermitSSH = @{ IpProtocol="tcp"; FromPort="22"; ToPort="22"; IpRanges="0.0.0.0/0" }
  $PermitHTTP = @{ IpProtocol="tcp"; FromPort="80"; ToPort="80"; IpRanges="0.0.0.0/0" }
  $SecurityGroupRules = (Grant-EC2SecurityGroupIngress -GroupId "$SecurityGroupId" -IpPermission @( $PermitSSH, $PermitHTTP ))
  Write-Output "Security Group $SecurityGroupName created"
}

# Find latest official Amazon Linux 2 Kernel 5 AMI image
$AL2Image = (Get-EC2Image -Owners amazon -Filters @{Name = "name"; Values = "amzn2-ami-kernel-5*"}, @{Name = "architecture"; Values = "x86_64"} | Sort-Object CreationDate | Select-Object ImageId | Select-Object -Last 1).ImageId

# Create EC2 instance tag specification for Name tag
$NameTag = [Amazon.EC2.Model.Tag]@{ Key = "Name"; Value = "$InstanceName" }
$TagSpecification = [Amazon.EC2.Model.TagSpecification]::new()
$TagSpecification.ResourceType = "Instance"
$TagSpecification.Tags.Add($NameTag)

# Launch instance from Amazon Linux 2 AMI
# if it's not already running
if (!((Get-EC2Instance -Filter @( @{name='tag:Name'; values="$InstanceName"})).Instances | Select-Object InstanceId)) {
  $EC2Instance = (New-EC2Instance -ImageId "$AL2Image" -KeyName $KeyName -SecurityGroups $SecurityGroupName -InstanceType $InstanceType -TagSpecification $TagSpecification)
  $ReservationID = ($EC2Instance).ReservationId
  while (((Get-EC2Instance -Filter @{Name = "reservation-id"; Values = "$ReservationId"}).Instances.State.Name.Value) -ne "running") { 
    Start-Sleep -s 15
  }
  $InstanceID = (Get-EC2Instance -Filter @{Name = "reservation-id"; Values = "$ReservationId"}).Instances.InstanceId
  Write-Output "Instance $InstanceName launched with ID $InstanceId"
}

# Get instance Public DNS from Name
$Instance = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="$InstanceName"}))
$InstancePublicDNS = ($Instance).Instances.publicdnsname

# Create DNS entry definition
$DnsChange = New-Object Amazon.Route53.Model.Change
$DnsChange.Action = "CREATE"
$DnsChange.ResourceRecordSet = New-Object Amazon.Route53.Model.ResourceRecordSet
$DnsChange.ResourceRecordSet.Name = "www.$ZoneDnsName"
$DnsChange.ResourceRecordSet.Type = "CNAME"
$DnsChange.ResourceRecordSet.TTL = 300
$DnsChange.ResourceRecordSet.ResourceRecords.Add(@{Value=$InstancePublicDns})

# Get Zone ID from DNS Domain Name
$ZoneId = (Get-R53HostedZonesByName -DNSName "$ZoneDnsName").Id

# Create DNS Record Set if it doesn't already exist
if (!((Get-R53ResourceRecordSet -HostedZoneId $ZoneId).ResourceRecordSets.Name | Select-String "www.$ZoneDnsName")) {
  $Output = (Edit-R53ResourceRecordSet -HostedZoneId $ZoneId -ChangeBatch_Change $DnsChange)
  Write-Output "DNS entry created (CNAME www.$ZoneDnsName -> $InstancePublicDns)"
}

Run the script as follows:

./provision.ps1

N.B. This script saves the private SSH key to etc/$AppName.pem. For the SSH deployment to work (see below) this file needs to have its permissions set to mode 0600, but there does not seem to be a standard way of achieving this in PowerShell. I’ve used a non-PowerShell chmod command in the script which will work on macOS and Linux. In Windows PowerShell this line will need to be commented out because chmod does not exist in that environment. If the permissions of this file prevent SSH deployment on Windows, you’ll need to manually set the permissions before running deploy.ps1. If anyone knows of a better solution for solving this across all operating systems, let me know.

Deployment

The deploy.ps1 script deploys the webapp on the EC2 instance. This does the following via SSH connections:

  • Downloads my basic Python webapp simple-webapp to the instance and unzips it to the necessary location.
  • Installs the systemd service script for the webapp, then enables it and starts it as a service running on port 8080.
  • Installs nginx and downloads an nginx config file which proxies all incoming requests on port 80 to the webapp on port 8080.
  • Enables nginx and starts it as a service running on port 80.

Here’s how the script looks:

# Source variables file
. etc/variables.ps1

# Get instance Public DNS from Name
$Instance = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="$InstanceName"}))
$InstancePublicDNS = ($Instance).Instances.publicdnsname

# List of commands to run, to install and 
# configure simple-webapp and nginx
$Commands = @(
  'sudo curl -L -s -o /opt/simple-webapp.zip https://github.com/mattbrock/simple-webapp/archive/refs/heads/master.zip'
  'sudo unzip /opt/simple-webapp.zip -d /opt'
  'sudo mv -fv /opt/simple-webapp-master /opt/simple-webapp'
  'sudo cp -fv /opt/simple-webapp/simple-webapp.service /usr/lib/systemd/system/simple-webapp.service'
  'sudo systemctl daemon-reload'
  'sudo systemctl enable simple-webapp'
  'sudo systemctl start simple-webapp'
  'sudo amazon-linux-extras install nginx1'
  'sudo curl -s -o /etc/nginx/default.d/simple-webapp.conf https://raw.githubusercontent.com/mattbrock/mattbrock/master/Ansible_Docker_EC2/etc/simple-webapp.conf'
  'sudo systemctl enable nginx'
  'sudo systemctl start nginx'
)

# Function to send command via SSH
function Send-Command {
  param ($Command)
  ssh -o CheckHostIP=no -o StrictHostKeyChecking=no -i etc/$KeyName.pem ec2-user@$InstancePublicDNS "$Command"
}

# Iterate through commands and send/run
# each in turn via SSH
$Commands | % { Send-Command $_ }

Run the script as follows:

./deploy.ps1

You can modify this script to deploy a different webapp if so desired.

Deprovisioning

The destroy.ps1 script deletes the DNS entry, terminates the EC2 instance, removes the Security Group and SSH key pair, and deletes the private key file:

# Source variables file
. etc/variables.ps1

# Get instance Public DNS from Name
$Instance = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="$InstanceName"}))
$InstancePublicDNS = ($Instance).Instances.publicdnsname

# Create DNS entry definition
$DnsChange = New-Object Amazon.Route53.Model.Change
$DnsChange.Action = "DELETE"
$DnsChange.ResourceRecordSet = New-Object Amazon.Route53.Model.ResourceRecordSet
$DnsChange.ResourceRecordSet.Name = "www.$ZoneDnsName"
$DnsChange.ResourceRecordSet.Type = "CNAME"
$DnsChange.ResourceRecordSet.TTL = 300
$DnsChange.ResourceRecordSet.ResourceRecords.Add(@{Value=$InstancePublicDns})

# Get Zone ID from DNS Domain Name
$ZoneId = (Get-R53HostedZonesByName -DNSName "$ZoneDnsName").Id

# Delete DNS Record Set
if ((Get-R53ResourceRecordSet -HostedZoneId $ZoneId).ResourceRecordSets.Name | Select-String "www.$ZoneDnsName") {
  $Output = (Edit-R53ResourceRecordSet -HostedZoneId $ZoneId -ChangeBatch_Change $DnsChange)
  Write-Output "DNS entry deleted (CNAME www.$ZoneDnsName -> $InstancePublicDns)"
}

# Terminate EC2 instance if it's running, and remove Name tag
$Instance = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="$InstanceName"}))
$InstanceId = ($Instance).Instances.InstanceId
$InstanceState = ($Instance).Instances.State.Name.Value
if ( ($InstanceId) -and ($InstanceState -ne "terminated") ) {
  $Output = (Remove-EC2Instance -InstanceId $InstanceId -Force)
  while (((Get-EC2Instance -Filter @{Name = "instance-id"; Values = "$InstanceId"}).Instances.State.Name.Value) -ne "terminated") {
    Start-Sleep -s 15
  }
  Remove-EC2Tag -Resource $InstanceId -Tag Name -Force
  Write-Output "Instance $InstanceName ($InstanceId) terminated and Name tag removed"
}

# Delete Security Group if it exists
if ((Get-EC2SecurityGroup).GroupName | Select-String -Pattern "$SecurityGroupName") {
  Remove-EC2SecurityGroup -GroupName "$SecurityGroupName" -Force
  Write-Output "Security Group $SecurityGroupName deleted"
}

# Delete EC2 keypair if it exists
if ((Get-EC2KeyPair).KeyName | Select-String -Pattern "$KeyName") {
  Remove-EC2KeyPair -KeyName "$KeyName" -Force
  Write-Output "Keypair $KeyName deleted"
}

# Delete local key file if it exists
if ((Test-Path "etc/$KeyName.pem" -PathType Leaf)) {
  Remove-Item -Path "etc/$KeyName.pem"
  Write-Output "Private key file etc/$KeyName.pem deleted"
}

Run the script as follows:

./destroy.ps1

N.B. Run this script with extreme caution as it removes AWS infrastructure and has the potential for serious damage if, for example, you accidentally run it in the wrong environment.

Checking the deployment

To check the webapp via the EC2 instance Public DNS (change simple-webapp to the name of your app, if different):

$InstancePublicDNS = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="simple-webapp"})).Instances.publicdnsname ; curl http://$InstancePublicDNS/

To check the webapp via the Route 53 DNS:

curl http://www.mydomain.com/

Checking the logs

To check the nginx access log (change simple-webapp to the name of your app, if different):

$InstancePublicDNS = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="simple-webapp"})).Instances.publicdnsname ; ssh -o CheckHostIP=no -o StrictHostKeyChecking=no -i etc/simple-webapp.pem ec2-user@$InstancePublicDNS "sudo tail -50 /var/log/nginx/access.log"

To check the nginx error log (change simple-webapp to the name of your app, if different):

$InstancePublicDNS = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="simple-webapp"})).Instances.publicdnsname ; ssh -o CheckHostIP=no -o StrictHostKeyChecking=no -i etc/simple-webapp.pem ec2-user@$InstancePublicDNS "sudo tail -50 /var/log/nginx/error.log"

To check the webapp log (change simple-webapp to the name of your app, if different):

$InstancePublicDNS = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="simple-webapp"})).Instances.publicdnsname ; ssh -o CheckHostIP=no -o StrictHostKeyChecking=no -i etc/simple-webapp.pem ec2-user@$InstancePublicDNS "sudo grep simple-webapp /var/log/messages | tail -50"

Connecting to the EC2 instance

To get an interactive session on the EC2 instance (change simple-webapp to the name of your app, if different):

$InstancePublicDNS = (Get-EC2Instance -Filter @( @{name='tag:Name'; values="simple-webapp"})).Instances.publicdnsname ; ssh -o CheckHostIP=no -o StrictHostKeyChecking=no -i etc/simple-webapp.pem ec2-user@$InstancePublicDNS

Final thoughts

If, like me, you’ve previously been unfamiliar with PowerShell and want or need to use it for cloud infrastructure automation, I hope this helps to provide a useful starting point. If you are looking for help with any of the issues raised here, or with any other infrastructure, DevOps or SysAdmin issues, please feel free to contact me to discuss whether I could assist.