The /bin/bash Theory ▮

Amazon S3 Automated Backups

Amazon S3 buckets can be a great place to store regular backups, using AWS API storing backups at S3 buckets can be done easily.

S3cmd is one of the free tools, that can send files to the S3 buckets as well as Google cloud storage. In this post i will demonstrate how to create automatic backup system that can send regular backups every day to S3 bucket using s3cmd tool.

Creating S3 Bucket And Generating API Key

First creating Access key to be used with aws-cli and s3cmd, after logging to the AWS console, click on security credentials and then Access Keys and finally Create New Access Keys:

aws-sec

In order to create s3 bucket using aws-cli, awscli should be installed, the easiest way to install awscli is using pip:

$ sudo pip install awscli

Now applying the keys we just created to the awscli configuration:

$ aws configure
AWS Access Key ID [None]: <your-access-key-id>
AWS Secret Access Key [None]:<your-secret-access-key>
Default region name [None]: <add-region-for-ex-us-west-1>
Default output format [None]: json

The previous command will create two files within the current directory, the first file is .aws/config which stores the configuration used with AWS, the second is .aws/credentials which used to store the access key id and secret key.

Now aws-cli is all set and you can use it to create s3 bucket, of course if this is too much trouble you can always use AWS console but what is the fun in that, right?

Create S3 Bucket

Using high level commands of aws-cli to create s3 bucket of name backup-binbash:

$ aws s3 mb s3://backup-binbash
make_bucket: s3://backup-binbash/

To list the buckets:

$ aws s3 ls
2015-01-24 16:24:15 backup-binbash

Now the first part is done, which is creating the bucket and configure the access keys.

Installing s3cmd Tool

You can find the tool here, to install s3cmd on Ubuntu:

$ wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add -
$ sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list
$ sudo apt-get update && sudo apt-get install s3cmd

s3cmd can be used also to create, remove, and list buckets. First you need to configure the tool to use the access keys for the AWS account:

# s3cmd --configure

This tool will also ask some questions:

  • Access Key: access-key
  • Secret Key: secret-key
  • Encryption Password: encryption password to encrypt files before sending
  • Path to GPG program [/usr/bin/gpg]: choose the encryption program
  • Use HTTPS protocol [No]: using https when connecting to s3
  • HTTP Proxy server name: if using proxy to connect to s3
  • Test access with supplied credentials? [Y/n] y

Now it should say you that it successfully authenticated. Great!

Creating The Backup Script

The final part is to create the backup script and add it as a cron job:

#!/bin/bash
TIMESTAMP=`date +"%Y-%m-%d-%H:%M"`
S3CMD_PATH="/usr/bin/s3cmd"
S3_BUCKET="backup-binbash"
BKP_FILE="/tmp/binbashBAK-${TIMESTAMP}.tar.gz"
# Compress the Backup Dirs
tar czf $BKP_FILE /var/www/binbash
# Upload the File to s3
S3CMD_PATH put $BKP_FILE s3://$S3_BUCKET
rm $BKP_FILE

Creating a cronjob to execute this job on a daily basis:

$ sudo crontab -e -u root
00 01 * * *     /root/S3_backup.sh 2>&1 >> /var/log/backups3.log

Make sure you got all the permission right. Thats it :)

January 24, 2015 ∙