How can I connect to S3?

Introduction #

To securely access external resources from DevBoxes, we typically use Tailscale VPN. Combined with some basic EC2 and VPC configuration in your AWS account, you can securely access a development (or production!) S3 Bucket from your remote dev environment. While these instructions are Tailscale-specific, any VPN will do.

AWS setup #

Existing VPC requirements #

If you do not have a VPC to use, skip to the Create a VPC if needed section below to get started.

If you an existing VPC, ensure it meets these requirements:

• DNS Hostnames enabled
• A public subnet
• A VPC Gateway endpoint (with the default routes for your public subnet)

Create a VPC if needed #

Most of the steps below come directly from Tailscale’s instructions for connecting to an AWS VPC. We’ve copied and tailored the applicable ones below.

1. Go to the AWS’s “create VPC” experience. In the toolbar, update the region to match up with where your SQS queue is located.
2. Select the VPC and more option. Enter a tag name to auto-generate VPC and Subnet Names. Customize the IPv4 CIDR block if needed, and select ‘In 1 AZ’ in the Nat gateways section.

Leave everything else as the default and click “Create VPC”.

Configure your S3 Bucket to allow VPC access #

1. Edit your Bucket's Access Policy (AWS docs).
2. Include the following statement, adding your Bucket and VPC Endpoint ID:

    {
      "Sid": "Stmt12345",
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
      	"s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
      	"arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
			],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": "your-vpc-id"
        }
      }
    }

Create a new EC2 “relay” instance #

Launch an instance with the following configuration:

1. Use one of the supported linux distros.
2. In the Network Settings, edit the VPC to assign the one you created above.
3. Assign the instance to a public subnet of the VPC, and assign it a public IP address.
4. In the security groups configuration, edit the Security Group to allow inbound ssh (Port range = 22, Source = 0.0.0.0/0). We’ll need this during initial setup but will close the firewall later.‍

Tailscale Setup #

We need to advertise all the applicable routes on our Tailnet, directing all traffic from other machines to our relay host. Since S3 is fully-managed and has predictable IP address prefixes, this means that we need to advertise all the routes for S3 in the region our bucket, VPC, and relay instance reside in.

Install Tailscale on your EC2 relay instance #

1. Ssh into the EC2 instance and install Tailscale by following the install instructions for your distro.
2. Advertise-routes requires IP forwarding to be enabled. Enable it with:

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

Enable Tailscale #

Now, we need to enable Tailscale and advertise the S3 route prefixes.

1. Enable the Tailscale systemd service with the commands below.
2. Request the regional S3 IP prefixes from AWS and use JQ to create the command.
3. Then, run the command on your relay instance to advertise the routes.
4. Authenticate the machine to your Tailscale network by visiting the link in your browser.

# install jq if you don't have it
sudo yum install jq

# start Tailscale
sudo systemctl enable --now tailscaled

# get S3 prefixes for your region (in this case: 'us-west-2')
curl https://ip-ranges.amazonaws.com/ip-ranges.json | \
jq -r '.prefixes | map(select(.region == "us-west-2") | select(.service == "S3") | .ip_prefix) | join(",")' | \
 awk '{print "sudo tailscale up --advertise-routes="$1}'


# advertise the routes
sudo tailscale up \
--advertise-routes=3.5.76.0/22,18.34.244.0/22,18.34.48.0/20,3.5.80.0/21,52.218.128.0/17,52.92.128.0/17,35.80.36.208/28,35.80.36.224/28

Configure your Tailscale network #

Visit the admin console and perform the following actions:

1. Disable key expiry so that you don’t need to re-authenticate the server periodically.
2. Authorize all subnet routes on the machine, so that Tailscale distributes the S3 routes to the rest of your Tailscale network.

Verify your connection #

Find your EC2 instance’s Tailscale IP address:

tailscale ip -4

Ping the IP address from your personal machine (Windows, macOS, etc):

ping <ip_address>

Generate an auth key #

1. Go to the Tailscale Keys page and click “Generate auth key…”
2. Check “Pre-authorized”, “Ephemeral”, and “Reusable” (if you want to use this key on multiple DevBoxes).
3. Generate the key, copy it, and save it somewhere safe.

Access S3 from a DevBox #

If your S3 bucket is properly configured to only allow specific actions originating from your VPC endpoint, you should be able to confirm with the AWS CLI:

# set the awscli up with your account + region
aws configure

# confirm that you can connect to S3, e.g with telnet or ping
telnet s3.us-west-2.amazonaws.com 443
ping s3.amazonaws.com

# try to list the Bucket, should get AccessDenied
aws s3 ls my-bucket-name

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Start Tailscale using the key that you generated, then confirm access to the S3 bucket:

sudo tailscale up --accept-routes --authkey $TAILSCALE_AUTH_KEY

# should not get AccessDenied
aws s3 ls my-bucket-name

Close off your AWS firewall #

Edit your EC2 relay instance’s AWS Security Group settings to remove inbound ssh access. At this point, you are able to ssh to the EC2 instance securely over Tailscale, so you can close the hole in your public-facing firewall.

‍

DevZero Template Setup #

To automatically configure each of your DevBoxes with Tailscale, update your applicable templates.

Add auth key to template #

1. Edit a template you want to enable SQS access for Save the auth key as a secret
2. Add an Env Var with:

Key name: TAILSCALE_AUTH_KEY

Value: the Tailscale auth key you just generated

Update template policy to run Tailscale #

Add this snippet to the scriptpolicy:

# installs and starts up Tailscale on your Devboxes
- script: |
    curl -fsSL https://tailscale.com/install.sh | sh
    sudo tailscale up --accept-routes --authkey ${{ TAILSCALE_AUTH_KEY }}
  runas: devzero

Optionally, add this softwarepolicy to always install the AWS CLI:

softwarepolicy:
- packagename: awscli