Awless tutorial: Try a smarter CLI for AWS

Awless tutorial: Try a smarter CLI for AWS

When the cloud was just about virtual machines, tools like Chef or Puppet helped us easily prepare our VMs. The only thing that mattered was to provision instances that contained all of the code and data required. But now that Amazon Web Services has ballooned to more than 90 services, interacting with the AWS API becomes the major part of the work.

How should we manage AWS infrastructure, and what interfaces should we use? Most beginners start with AWS Console, the default GUI, while seasoned sysadmins generally prefer a command-line interface (CLI). Problem is, the AWS CLI is not user friendly. Because it integrates the entire AWS API, it exposes an enormous surface area in terms of commands, flags, and options.

Awless is born from our need for a fast, powerful, and easy-to-use CLI to manage AWS. With Awless, you can create and run an AWS infrastructure, starting from scratch, and always get readable output (for both humans and programs), explore and query all cloud resources (even offline), connect to instances, and create, update, and delete cloud resources. Beyond single command lines, Awless supports templates that enable higher levels of automation. Last, but not least, Awless aims to ensure smart defaults and security best practices.

Because there are so many AWS services, it is often important to find and display a hierarchy of services from the command line. We can group services by functionality—such as compute and database. But going through each of them exhaustively is tedious as there are, as of this writing, no fewer than 15 services around storage and database, not counting four data migration services and nine analytics services that are directly related to data usage.

We find it easier to group services by cloud readiness. In this article, we will detail how to use Awless to create and manage cloud resources for a real use case, the deployment of production-ready WordPress instances. We will use the following AWS resources:

1. VM services EC2 (Elastic Compute Cloud) and ELB (Elastic Load Balancing);
2. High-level services that run in VMs but are managed by AWS, such as RDS (Relational Database Service) or ElastiCache (for queues);
3. “Serverless” services that run in multi-tenant VMs, such as S3 (object storage) or Lambda (single function execution).

Wallix

Cloud readiness levels start with VMs and extend to managed services and serverless services. We will use all layers in this article.

Get started with Awless

Sign up for AWS and create a first account with AdministratorAccess rights. Carefully note your access key and secret key.

Install Awless

Awless is available at GitHub. We provide pre-built binaries and Homebrew packages for MacOS:

>brew tap wallix/awless 
>brew install awless

You can check that Awless is properly installed by running:

>awless version

Awless is modeled after popular command-line tools such as Git. Most commands are in the form of:

>awless verb [entity] [parameter=value ...]

This article will give a 360-degree overview of real production workloads on AWS, starting from scratch. For clarity, we omit all confirmation and some output steps, as Awless always asks to confirm commands that create, update, or delete resources.

First steps with Awless

We can issue our first Awless command by listing our Virtual Private Clouds (VPCs). Because this is our first run, we will need to input some necessary data to configure Awless:

>awless list vpcs 
Welcome to awless! Resolving environment data... 
Please choose an AWS region: 
ap-northeast-1, ap-northeast-2, ap-south-1, ap-southeast-1, ap-southeast-2, ca-central-1, cn-north-1, eu-central-1, eu-west-1, eu-west-2, sa-east-1, us-east-1, us-east-2, us-gov-west-1, us-west-1, us-west-2 
Value ? > us-west-2 
Syncing region ‘us-west-2’... 
Cannot resolve AWS credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) Please enter access keys and choose a profile name (stored at /Users/john/.aws/credentials): 
AWS Access Key ID? AKIAIINZQI7WIEXAMPLE 
AWS Secret Access Key? hYWZBVOusePEPSr5PkscplskB84fjbgUEXAMPLE 
Choose a profile name? admin 
✓ /Users/john/.aws/credentials created 
✓ Credentials for profile ‘admin’ stored successfully 
All done. Enjoy! 
You can review and configure awless with `awless config`. 
Now running: awless list vpcs 
|     ID ▲     | NAME | DEFAULT |   STATE   |     CIDR      | 
|--------------|------|---------|-----------|---------------| 
| vpc-1d1df679 |      | true    | available | 172.31.0.0/16 |

Create an AWS user

We will now use Awless to create a new AWS user and give him sufficient rights using the admin profile. We create the user John and his access key:

>awless create user name=john 
>awless create accesskey user=john aws_access_key_id = AKIAIOSFODNN7EXAMPLE 
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY 
Do you want to save in your .aws/credentials? (y/n) y 
Entry name in .aws/credentials? [default] john

Now that John exists, he needs a set of permissions. We will give John full access to the EC2, RDS, Auto Scaling, CloudFront, and S3 services that we will use in this article:

>awless attach policy service=ec2 access=full user=john 
>awless attach policy service=rds access=full user=john 
>awless attach policy service=s3 access=full user=john 
>awless attach policy service=autoscaling access=full user=john 
>awless attach policy service=cloudfront access=full user=john

Now that John is a fully functional user, we’ll switch to his profile for the next steps:

>awless config set aws.profile john

We will use AWS to set up a highly available, managed WordPress deployment, combining VMs, managed, and serverless services. Our main goal is pictured below. We will have to address three “devops challenges” to reach it, making use of AWS infrastructure services, managed services, and serverless services, respectively.

Wallix

We will deploy a production WordPress blogging application that relies on managed services and serverless improvements.

Challenge 1: Lift and shift an application to EC2

Lift and shift is the quickest to migrate legacy applications to the cloud and benefit from the flexibility and cost advantages of cloud platforms. In this case, we will start by deploying a WordPress engine and its database in a single VM. Clients will connect directly to the VM.

Wallix

The first step is deploying our blog application in a single EC2 instance.

Create a VPC

Before we proceed with the VM creation, we first need to create network resources:

• A private network (or VPC)
• An Internet gateway for this VPC
• A subnet using the Internet gateway

Awless will prompt for any missing parameters with autocompletion. Here we use a mix of both provided (param=value) and prompted parameters:

>awless create vpc cidr=10.0.0.0/16 name=wordpress-vpc 
>awless create internetgateway 
[OK] id=igw-1234567 
>awless attach internetgateway 
Please specify (Ctrl+C to quit, Tab for completion): 
internetgateway.id? [Tab] 
internetgateway.id? igw-1234567 
internetgateway.vpc? @wo[Tab] 
internetgateway.vpc? @wordpress-vpc

Awless puts forward the best practice to use names rather than resource IDs. As such, @resource-name is the identifier of the resource named “resource-name.”

Let’s create a public subnet to host our WordPress instance, and attach a route table that routes the Internet traffic to the VPC’s Internet gateway:

>awless create subnet cidr=10.0.0.0/24 vpc=@wordpress-vpc name=wordpress-public-subnet 
>awless update subnet id=@wordpress-public-subnet public=true 
>awless create routetable vpc=@wordpress-vpc 
>awless attach routetable subnet=@wordpress-public-subnet
         Please specify (Ctrl+C to quit, Tab for completion):
         routetable.id?[tab]
         *select the ID of the routetable you created above* 
>awless create route cidr=0.0.0.0/0
         Please specify (Ctrl+C to quit, Tab for completion): 
route.gateway? *the ID of the internet gateway you attached to the VPC above* 
route.table? *the ID of the routetable you created above*

Note that each action in Awless is about as simple as it can get. Although we follow a comprehensive step-by-step approach, Awless allows us to get through the tedious process of setting up an infrastructure much faster than with the graphical console or the AWS CLI.

Create an SSH keypair and a security group

The cloud network is now ready. Before creating the instance, we need an SSH key pair, to connect to the instance later. In a single command, Awless generates an SSH key pair locally and registers it on AWS:

>awless create keypair name=johnkey

A best practice is to give minimal access to any resource, so we will only accept HTTP connections from all the Internet and SSH from our outgoing IP address. To do that, we create and configure a security group:

>awless create securitygroup vpc=@wordpress-vpc description=”HTTP public + SSH access” name=wordpress-secgroup 
>MY_IP=$(awless whoami —ip-only) 
>awless update securitygroup id=@wordpress-secgroup inbound=authorize cidr=$MY_IP/32 portrange=22 
>awless update securitygroup id=@wordpress-secgroup inbound=authorize cidr=0.0.0.0/0 portrange=80

Provision the application with AWS user data

We will now provision our WordPress instance through AWS user data. Here we will use the script available on GitHub:

>awless create instance subnet=@wordpress-public-subnet keypair=johnkey name=wordpress-instance userdata=https://raw.githubusercontent.com/zn3zman/AWS-WordPress-Creation/16a52aef4f618d558d61197df4e4eca4c015277f/WP-Setup.sh securitygroup=@wordpress-secgroup

You can use awless show to get information about any resource, such as the public IP address of our WordPress instance:

>awless show wordpress-instance

You can connect to the IP address from the command output to access your WordPress service (though you might have to wait a few minutes for the instance to be provisioned properly).

WordPress Foundation

Our initial launch of WordPress.

By default, Awless will create a type t2.micro (1 vCPU, 1GB RAM) using Amazon Linux. You can update default values by using awless config set:

>awless config set instance.type m4.large 
>UBUNTU_AMI=$(awless search images canonical:ubuntu —id-only —silent) 
>awless config set instance.image $UBUNTU_AMI

To this point, we have built several resources. Using awless list, we can list users, instances, subnets, and all other types of resources (provided your AWS profile has sufficient rights of course). For example, we can list instances:

>awless list instances 
|       ID ▲        |   ZONE   |        NAME        | UPTIME  | 
|-------------------|----------|--------------------|---------| 
|i-00268db26b0d0393c|us-west-1c| wordpress-instance | 57 mins | 
[...]

Awless provides a powerful feature that enables easy connections to instances with SSH. Behind the scenes, Awless will automatically get the instance IP address, guess the username, and connect with the keypair we created earlier:

>awless ssh wordpress-instance

If you want to delete the WordPress instance, you can run awless delete instance id=@wordpress-instance. You can do it now, as we will create a more advanced deployment in the next challenge.

How to use Awless templates

All the steps in this challenge can be described as a sequence of Awless commands, where the results of previous commands (for instance, the ID of the Internet gateway) are used as inputs to subsequent commands. Because Awless provides a built-in templating system, you could encapsulate all of Challenge 1 in a template and run it with:

>awless run https://raw.githubusercontent.com/wallix/awless-templates/bcd0dd41b1524eeac1e53d12b2998bc56689c517/simple_wordpress_infra.aws

Awless offers a powerful feature that enables you to revert most changes applied to an AWS infrastructure. For instance, you can delete the whole infrastructure created by a template in a single command: awless revert revert-id. To find a given revert-id, awless log lists all of the commands previously applied to the cloud infrastructure, with both their output and their ID:

>awless log # find the ID to revert >awless revert 01BM6D1YRZ5SSN5Z09VEEGN0HV

Challenge 2: Use AWS managed services

Our previous deployment is functional, but pretty artisanal. Our blog is powered by a single instance in a single Availability Zone (AZ). We now want to build a highly available blog, with a load balancer, two instances in different AZs, and a replicated database that is shared by our instances. Instead of running our own database in an instance, we will use AWS RDS, Amazon’s managed service for SQL databases. Using a managed service provides many advantages including clustering, managed security, and backups.

Wallix

Our goal for Challenge 2 is to set up a highly available infrastructure with a managed and replicated database.

In order to have highly available resources, we need to distribute them in subnets in different availability zones (AZs) and balance the load through Elastic Load Balancing.

Wallix

We want our WordPress instances to be able to fail gracefully (one at a time).

For this challenge, we will create the following:

• One load balancer to distribute the load between the instances
• Two public subnets to associate with the Internet-facing load balancer
• Two private subnets in different AZs (e.g. us-east-1a, us-east-1e) to host the instances
• One auto scaling group to manage the scaling of WordPress instances
• One NAT gateway in one public subnet to enable outbound calls for instances provisioning
• One public fixed IP (Elastic IP) for the NAT gateway
• One RDS for MariaDB instance automatically replicated in the private subnets

We will build this infrastructure by running Awless templates. The first template creates subnets and routing. The {hole} notation allows parameters to be filled dynamically during the running of the template. The $reference notation enables back references of created resources.

# Filename: “subnets_public_and_private_with_NAT.aws” 
# 2 public subnets in different AZs attached to internet gateway 
pubSub1 = create subnet cidr=10.0.100.0/24 vpc=@wordpress-vpc name=wordpress-public-subnet-1 availabilityzone={az1} 
update subnet id=$pubSub1 public=true 
pubSub2 = create subnet cidr=10.0.101.0/24 vpc=@wordpress-vpc name=wordpress-public-subnet-2 availabilityzone={az2} 
update subnet id=$pubSub2 public=true rt = create routetable vpc=@wordpress-vpc 
create route table=$rt cidr=0.0.0.0/0 gateway={vpc.internetgateway} 
attach routetable id=$rt subnet=$pubSub1 
attach routetable id=$rt subnet=$pubSub2
# 2 private subnets in different AZs 
privSub1 = create subnet cidr=10.0.10.0/24 vpc=@wordpress-vpc name=wordpress-private-subnet-1 availabilityzone={az1} 
privSub2 = create subnet cidr=10.0.11.0/24 vpc=@wordpress-vpc name=wordpress-private-subnet-2 availabilityzone={az2}   
# NAT Gateway in public subnet with a fixed IP 
ip = create elasticip 
natgw = create natgateway elasticip-id=$ip subnet=@wordpress-public-subnet 
check natgateway id=$natgw state=available timeout=180   
# Routing between private subnets and NAT gateway 
natgw_rtable = create routetable vpc=@wordpress-vpc 
attach routetable id=$natgw_rtable subnet=$privSub1 
attach routetable id=$natgw_rtable subnet=$privSub2 
create route cidr=0.0.0.0/0 gateway=$natgw table=$natgw_rtable

>awless list availabilityzones 
|   NAME ▲   |   STATE   |  REGION   | MESSAGES | 
|------------|-----------|-----------|----------| 
| us-east-1a | available | us-east-1 |          | 
| us-east-1b | available | us-east-1 |          | 
[...] 
>awless run subnets_public_and_private_with_NAT.aws az1=us-east-1a az2=us-east-1e

# Filename: “provision-lb.aws” 
# Create the load balancer security group 
lbsecgroup = create securitygroup vpc=@wordpress-vpc description="authorize HTTP from the internet" name=wordpress-lb-securitygroup 
update securitygroup id=$lbsecgroup inbound=authorize protocol=tcp cidr=0.0.0.0/0 portrange=80 
# Provision the load balancer listening in the public subnets, with its target group and HTTP listener 
tg = create targetgroup name=wordpress-workers port=80 protocol=HTTP vpc=@wordpress-vpc 
update targetgroup id=$tg stickiness=true 
lb = create loadbalancer name=wordpress-loadbalancer subnets={public.subnets} securitygroups=$lbsecgroup 
create listener actiontype=forward loadbalancer=$lb port=80 protocol=HTTP targetgroup=$tg

>PUBLIC_SUBNETS=$(awless show wordpress-public-subnet-1 —values-for ID),$(awless show wordpress-public-subnet-2 —values-for ID) >awless run provision-lb.aws public.subnets=$PUBLIC_SUBNETS

# Filename: “rds_database.aws” 
# Database security group (authorize access from private subnets to port 3306) 
dbsec = create securitygroup name=wordpress-database-securitygroup vpc=@wordpress-vpc description="authorize mysql port 3306 from private subnets" 
update securitygroup id=$dbsec inbound=authorize cidr=10.0.10.0/24 portrange=3306 
update securitygroup id=$dbsec inbound=authorize cidr=10.0.11.0/24 portrange=3306 
# Distribute the MariaDB database in the 2 subnets 
dbsub = create dbsubnetgroup name=wordpress-dbsubnets description="subnets for wordpress databases" subnets={private.subnets} 
create database engine=mariadb id=wordpress-database password={dbpassword} size=8 type=db.t2.micro username=wordpress dbname=wordpress multiaz=true subnetgroup=$dbsub vpcsecuritygroups=$dbsec

>PRIVATE_SUBNETS=$(awless show wordpress-private-subnet-1 —values-for ID),$(awless show wordpress-private-subnet-2 —values-for ID) 
>awless run rds_database.aws private.subnets=$PRIVATE_SUBNETS dbpassword=mysqlpassword

>awless check —force database id=wordpress-database state=available timeout=600

# wordpress.aws : 
dbname={dbname} 
dbhost={dbhost} 
dbuser={dbuser} 
dbpassword={dbpassword} 
# Create the security group and launch configuration for the instances 
instSecGroup = create securitygroup vpc=@wordpress-vpc description="HTTP + SSH within VPC" name=wordpress-private-secgroup 
update securitygroup id=$instSecGroup inbound=authorize cidr=10.0.0.0/16 portrange=22 
update securitygroup id=$instSecGroup inbound=authorize cidr=10.0.0.0/16 portrange=80 
launchconf = create launchconfiguration image={instance.image} keypair={keypair} name=wordpress-launch-configuration type=t2.micro userdata=https://raw.githubusercontent.com/wallix/awless-templates/bcd0dd41b1524eeac1e53d12b2998bc56689c517/userdata/wordpress.sh securitygroups=$instSecGroup 
create scalinggroup desired-capacity=2 launchconfiguration=$launchconf max-size=2 min-size=2 name=wordpress-scalinggroup subnets={private.subnets} targetgroups=@wordpress-workers

>DBHOST=$(awless show wordpress-database —values-for PublicDNS) 
>awless run wordpress.aws private.subnets=$PRIVATE_SUBNETS keypair=johnkey dbname=wordpress dbhost=$DBHOST dbuser=wordpress dbpassword=mysqlpassword

>awless check loadbalancer id=@wordpress-loadbalancer state=active timeout=180 
>awless show wordpress-loadbalancer —values-for PublicDNS

>awless revert 01BM9DBTH1Q1579HXHVF377GF7 # Revert of wordpress.aws
>awless revert 01BM9CYN3TAYGK5FYQMHWCH83Y # Revert of rds_database.aws

>awless create bucket name=awless-wordpress-bucket 
>awless create distribution origin-domain=awless-wordpress-bucket.s3.amazonaws.com

>PRIVATE_SUBNETS=$(awless show wordpress-private-subnet-1 —values-for ID),$(awless show wordpress-private-subnet-2 —values-for ID) 
>awless run rds_database.aws private.subnets=$PRIVATE_SUBNETS dbpassword=mysqlpassword

>awless -p admin create policy name=AllowCreateAttachPolicy effect=Allow resource=* description=”Authorize the creation of policies” action = iam:CreatePolicy, iam:DeletePolicy, iam:AttachRolePolicy, iam:DetachRolePolicy, iam:CreateRole, iam:DeleteRole, iam:CreateInstanceProfile, iam:DeleteInstanceProfile, iam:AddRoleToInstanceProfile, iam:RemoveRoleFromInstanceProfile, iam:PassRole, iam:ListInstanceProfiles 
>POLICY_ARN=$(awless -p admin show AllowCreateAttachPolicy —values-for arn) 
>awless -p admin attach policy arn=$POLICY_ARN user=john

>DBHOST=$(awless show wordpress-database —values-for PublicDNS) 
>PRIVATE_SUBNETS=$(awless show wordpress-private-subnet-1 —values-for ID),$(awless show wordpress-private-subnet-2 —values-for ID) 
>URL=$(awless show wordpress-loadbalancer —values-for PublicDNS) 
>CLOUDFRONT_URL=$(awless list distributions —filter origins=wordpress —sort modified —format tsv |tail -n1 |cut -f2) 
>awless run https://gist.githubusercontent.com/fxaguessy/c09ad9d9452afb5e1885e7963fca7987/raw/5d55aac230d88c11ff9b44cfcf1f2feb13ce6c33/wordpress-with-S3-plugin.aws dbhost=$DBHOST private.subnets=$PRIVATE_SUBNETS wordpress.url=$URL cdn.URL=$CLOUDFRONT_URL 
Please specify (Ctrl+C to quit, Tab for completion): 
dbpassword? mysqlpassword 
keypair? johnkey 
wordpress.email? test@test.com 
wordpress.password? wordpresspassword

Source: InfoWorld – Cloud Computing