Backing up the Cluster Node Database

Cluster node data, including access keys and user data, is stored in the Scale KV database. Aspera recommends backing up this database so that, should you need to relaunch a cluster, the database can be restored. Aspera also recommends backing up your access keys (see Backing up and Restoring Access Keys) which cannot be completely restored from the backup described here.

  1. Log in to a cluster node using the SSH key that you configured in the Provider section of your cluster template.
  2. Create a backup by running the following command on any cluster node:
    # echo bgrewriteaof | /opt/aspera/bin/asredis -p 36001

    This creates one or more append-only files (.aof) in /cluster/scaledb on the nodes that are assigned to store the database.

  3. Determine which nodes contain your backups.
    The scalekv database is partioned between specific nodes, and the backup files are stored on the those nodes. To identify the nodes that contain the backup files, run the followign command on any node:
    # /opt/aspera/bin/ascalekvadmin info --full

    The output describes where the Scale KV database partitions are stored. For example:

    Name: scaledb-default
    ...
        # Master Partitions: 2
        # Replicas per Master: 1
    
        Partition Details:
            Master 0:
                Host: 124.0.0.24
                Port: 43001
                Storage: scaledb-default-0-0
                ...
                # Replicas: 1
                Replica Detail:
                    Slave 1:
                        Host: 124.13.0.25
                        ...
    
            Master 1:
                Host: 124.13.0.25
                Port: 43002
                Storage: scaledb-default-1-0
                ...
                # Replicas: 1
                Replica Detail:
                    Slave 1:
                        Host: 124.0.0.24
                        ...

    In this case, the Scale KV database is partitioned between two nodes, Master 0 that is located at 124.0.0.24 and Master 1 that is located at 124.13.0.25. Master 0 has the partition scaledb-default-0-0 and Master 1 has the partition scaledb-default-1-0. The partitions are replicated to the other master node.

  4. Log in to each of the nodes where the database partition is stored and copy the backup files to a bucket or path in your object storage.
    For example, to copy the backup file scaledb-default-1-0.aof from 124.13.0.25 to an S3 bucket, the command is similar to the following:
    # /opt/aspera/bin/axcloudcp scaledb-default-1-0.aof s3://EIFNWEIGUBNDSE:Se3rnf23neifh3@s3.amazonaws.com/mycompany-tmp/backups/