How to backup sql and upload to AWS S3

How to backup sql and upload to AWS S3

  AWS    —  2021, May 12    

Prerequisite

Create IAM and grant permission

  • Access AWS IAM console and create new user named: dev_user (any name you want)
  • While creating new user, you should store in secret place the Access Key Id and Secret Access Key to configure aws profile later.
  • Set permission: AmazonS3FullAccess

Create S3 Bucket

  • Login to aws console
  • Access link to create new S3 Bucket named backup for example.

Setup cronjob

SSH to EC2

There are several ways to ssh to EC2 using pem key or username, password.

Here i’m using pemkey so i run cmd:

ssh -i /path/to/pemkey ec2-user@IP_ADDRESS

Switch to root user

sudo su

Configure aws profile

  • Check aws profile
aws configure list
  • Create new profile
aws configure

Prompt will require AWS Access Key Id and AWS Secret Access Key what you get above when create new IAM.

Create bash script

  • Create a script
touch /usr/local/bin/backup_script.sh
  • Edit script and input below content: vi /usr/local/bin/backup_script.sh
#!/bin/bash
# Saved in /usr/local/bin
# Basic configuration: datestamp e.g. YYYYMMDD
DATE=$(date +"%Y%m%d%H%M%S")

# Location of your backups (create the directory first!)
TMP_BACKUP_DIR="/tmp/backup"
BACKUP_DIR="${TMP_BACKUP_DIR}/${DATE}"

# Create a new directory into backup directory location for this date
mkdir -p $BACKUP_DIR

# ------------------------------
# ---- BEGIN BACKING UP SQL ----
# ------------------------------
SQL_DIR="sql"
mkdir -p $BACKUP_DIR/$SQL_DIR

# MySQL login details
MYSQL_USER=""
MYSQL_PASSWORD=""

# MySQL executable locations (no need to change this)
MYSQL=/usr/bin/mysql
MYSQLDUMP=/usr/bin/mysqldump

# MySQL databases you wish to skip
EXCLUDE_DB="|#mysql50#.rocksdb"
SKIPDATABASES="Database|information_schema|performance_schema|mysql$EXCLUDE_DB"

# ---- DO NOT CHANGE BELOW THIS LINE ------------------------------------------

# Retrieve a list of all databases
databases=$($MYSQL -u$MYSQL_USER -p$MYSQL_PASSWORD -e "SHOW DATABASES;" | grep -Ev "($SKIPDATABASES)")

# Dumb the databases in seperate names and gzip the .sql file
for db in $databases; do
    echo $(date +"%b %d, %Y %H:%M:%S Begin backing up $db")
    $MYSQLDUMP --force --opt --user=$MYSQL_USER -p$MYSQL_PASSWORD --skip-lock-tables --events --databases $db | gzip >"${BACKUP_DIR}/${SQL_DIR}/$db.sql.gz"
    echo $(date +"%b %d, %Y %H:%M:%S End backing up $db")
done

# ----------------------------
# ---- BACKUP UPLOADS DIR ----
# ----------------------------
DOC_DIR="documentation"
mkdir -p $BACKUP_DIR/$DOC_DIR

UPLOAD_DIR="/var/www/html/wp-content/uploads"

# Gzip folder /var/www/html/wp-content/uploads
echo $(date +"%b %d, %Y %H:%M:%S Begin zipping $UPLOAD_DIR")
cd 
tar czf "${BACKUP_DIR}/${DOC_DIR}/uploads.tar.gz" -P  $UPLOAD_DIR
echo $(date +"%b %d, %Y %H:%M:%S End zipping $UPLOAD_DIR")

# -------------------------------
# ---- BEGIN UPLOADING TO S3 ----
# -------------------------------
BACKUP_FILE="${TMP_BACKUP_DIR}/${DATE}.tar.gz"
tar czf $BACKUP_FILE -P $BACKUP_DIR

S3_BUCKET="s3://YOUR_S3_BUCKET_NAME/"

echo $(date +"%b %d, %Y %H:%M:%S Begin uploading $BACKUP_FILE to $S3_BUCKET")
aws s3 cp $BACKUP_FILE $S3_BUCKET
echo $(date +"%b %d, %Y %H:%M:%S End uploading to S3")

# Remove backup files after uploading to S3
echo $(date +"%b %d, %Y %H:%M:%S Remove $TMP_BACKUP_DIR after uploading to S3")
rm -rf $TMP_BACKUP_DIR

# --------------------------------------
# ---- DELETE OLD BACKUP FILE IN S3 ----
# --------------------------------------
flist=($(aws s3 ls ${S3_BUCKET} | awk '{print $4}'))
len=${#flist[@]}

# Loop to delete all files and keep 2 latest files
for ((i = 0; i < $((len-2)); i++)); do
    ele0="${S3_BUCKET}${flist[$i]}"
    echo $(date +"%b %d, %Y %H:%M:%S Remove ${ele0}")
    aws s3 rm ${ele0}
done

Setup crontab

  • Show cronjob list
crontab -l
  • Edit cronjob
crontab -e

Vim editor will be appeared and you can edit it.

Refer to crontab online to create cronjob you want

Here i will run script at 00:00 every Sunday and write log into file. So i input below content:

0 0 * * 7 /usr/bin/sh /usr/local/bin/backup_script.sh >> /var/log/backup_script.log 2>&1
  • Restart crontab
service crond restar