How can I use AWS SQS?

Introduction #

To securely access external resources from DevBoxes, we typically use Tailscale VPN. Combined with some basic EC2 and VPC configuration in your AWS account, you can leverage a development (or production!) SQS queue from your remote dev environment. While these instructions are Tailscale-specific, any VPN will do.

AWS setup #

Existing VPC requirements #

If you do not have a VPC to use, skip to the Create a VPC if needed section below to get started.

If you an existing VPC, ensure it meets these requirements:

• DNS Hostnames enabled
• A public subnet
• A VPC Gateway endpoint (with the default routes for your public subnet)

Create a VPC if needed #

Most of the steps below come directly from Tailscale’s instructions for connecting to an AWS VPC. We’ve copied and tailored the applicable ones below.

1. Go to the AWS’s “create VPC” experience. In the toolbar, update the region to match up with where your SQS queue is located.
2. Select the VPC and more option. Enter a tag name to auto-generate VPC and Subnet Names. Customize the IPv4 CIDR block if needed, and select ‘In 1 AZ’ in the Nat gateways section.

Leave everything else as the default and click “Create VPC”.

Configure your SQS queue to allow VPC access #

1. Edit your queue’s Access Policy (AWS docs).
2. Include the following statement, adding your account and VPC IDs:

    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-west-2:M<account-id>:<sqs-queue-name>",
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": "your-vpc-id"
        }
      }
    }

Create a new EC2 “relay” instance #

Launch an instance with the following configuration:

1. Use one of the supported linux distros.
2. In the Network Settings, edit the VPC to assign the one you created above.
3. Assign the instance to a public subnet of the VPC, and assign it a public IP address.
4. In the security groups configuration, edit the Security Group to allow inbound ssh (Port range = 22, Source = 0.0.0.0/0). We’ll need this during initial setup but will close the firewall later.

Tailscale Setup #

Install Tailscale on your EC2 relay instance #

1. ssh into the EC2 instance and install Tailscale by following the install instructions for your distro.
2. Advertise-routes requires IP forwarding to be enabled. Enable it with:‍

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

Enable Tailscale #

1. Enable the Tailscale systemd service with the commands below.
2. Then, authenticate the machine to your Tailscale network by visiting the link in your browser.

sudo systemctl enable --now tailscaled
sudo tailscale up --advertise-routes=10.0.0.0/24,10.0.1.0/24

Configure your Tailscale network #

Visit the admin console and perform the following actions:

1. Disable key expiry so that you don’t need to re-authenticate the server periodically.
2. Authorize subnet routes on the machine, so that Tailscale distributes the 10.0.0.0/24 and 10.0.1.0/24 routes to the rest of your Tailscale network.

Verify your connection #

Find your EC2 instance’s Tailscale IP address:

tailscale ip -4

Ping the IP address from your personal machine (Windows, macOS, etc):

ping <ip_address>

Generate an auth key #

1. Go to the Tailscale Keys page and click “Generate auth key…”
2. Check “Pre-authorized”, “Ephemeral”, and “Reusable” (if you want to use this key on multiple DevBoxes).
3. Generate the key, copy it, and save it somewhere safe.

Use the queue from a DevBox #

1. Launch a new instance from the template you edited.
2. When the machine is up, confirm you can perform commands on your SQS queue. The example below uses the AWS CLI to interact with a queue in the us-west-2 region, but you should update the region to match the region with your queue, VPC, EC2 instance, etc.

Use your queue from a DevBox #

# install and configure awscli if you have not already (Debian/ Ubuntu)
sudo apt install awscli -y
aws configure 
 
# Gget the URL of the SQS queue
aws sqs get-queue-url --queue-name your-queue-name --region us-west-2

# send a message (enqueue)
aws sqs send-message --region us-west-2 \
 --endpoint-url https://us-west-2.queue.amazonaws.com/ \
 --queue-url https://us-west-2.queue.amazonaws.com/your-aws-account-id>/your-queue-name \
 --message-body "Hello from your DevBox!"

# receive messages (dequeue)
aws sqs receive-message --queue-url https://us-west-2.queue.amazonaws.com/your-aws-account-id/your-queue-name

DevZero template setup #

To automatically configure each of your DevBoxes with Tailscale, update your applicable templates.

Add auth key to template #

1. Edit a template you want to enable SQS access for Save the auth key as a secret
2. Add an Env Var with:

• Key name: TAILSCALE_AUTH_KEY
• Value: the Tailscale auth key you just generated

Update template policy to run Tailscale #

Add this snippet to the scriptpolicy:

# installs and starts up Tailscale on your Devboxes
- script: |
    curl -fsSL https://tailscale.com/install.sh | sh
    sudo tailscale up --accept-routes --authkey ${{ TAILSCALE_AUTH_KEY }}
  runas: devzero

Optionally, add this softwarepolicy to always install the AWS CLI:

softwarepolicy:
- packagename: awscli

Close off your AWS firewall #

Edit your EC2 relay instance’s AWS Security Group settings to remove inbound ssh access. At this point, you are able to ssh to the EC2 instance securely over Tailscale, so you can close the hole in your public-facing firewall.