Setting riak to store data in an EBS volume in Amazon EC2

These days I've performed some tests in a riak cluster configuration, involving about a million write operations from many virtual machines hosted in EC2. Due to the database replication and the number and size of the records to be written, the default storage space coming with an Ubuntu Server 12.04LT instance was insufficient, so the solution was to add extra storage space through Elastic Storage Blocks (EBS).

The procedure is quite simple and will be described below in order to save the time for those who need to do this storage extension.

First of all a new volume must be created from AWS console and attach it to the desired instance.

See how the volume is attached in Ubuntu by isuing the command:

df

The result of the command is:

Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/xvda1       8256952 793960   7043564  11% /
udev              838328      8    838320   1% /dev
tmpfs             338520    172    338348   1% /run
none                5120      0      5120   0% /run/lock
none              846300      0    846300   0% /run/shm
/dev/xvdb      153899044 192068 145889352   1% /mnt

The new volume is /dev/xvdb and prior to use it it must be formatted appropriately. For doing this:

Unmount the /dev/xvdb volume:

sudo umount /dev/xvdb

Format the partition using ext4

sudo mkfs.ext4 -j /dev/xvdb

Mount back the volume:

sudo mount /dev/xvdb

Find out how was the EBS volume mounted:

df

The result of the command is:

Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/xvda1       8256952 860640   6976884  11% /
udev              838328      8    838320   1% /dev
tmpfs             338520    172    338348   1% /run
none                5120      0      5120   0% /run/lock
none              846300      0    846300   0% /run/shm
/dev/xvdb      153899044 191936 145889484   1% /mnt

Modify riak configuration file /etc/riak/app.config:

Find the riak_core setting and modify

{platform_data_dir, "/var/lib/riak"} to: 

{platform_data_dir, "/mnt/var/lib/riak"}

and bitcask setting

{data_root, "/var/lib/riak/bitcask"} to: 

{data_root, "/mnt/var/lib/riak/bitcask"}

On the new formatted volume mounted under /mnt prepare the directories that will hold the date on the new volume:

/mnt/var/lib/riak/bitcask/mnt/var/lib/riak/kv_node 

and make them writable.

Then, start the riak service by issuing the command:

sudo service riak start 

After these steps, riak will use the new EBS volume to store data instead the default storage space offered by the Ubuntu Server instance.

Delete all records from a riak database

Note: Applies to riak 1.2.1 installed on Ubuntu Server 12.04 LT (hosted in Amazon EC2)

The riak API allows to delete a bucket only if its content is empty. Sometimes (if you are performing various performance tests over the database, for example) the contained data is not important, so a script or program iterating through all the buckets and deleting all the content in a bucket is very time consuming.

A most convenient solution is to delete the directories holding the data on the riak server. In order to do this, follow the steps below:

1. Stop the riak server

sudo service riak stop

2. Change directory to the riak data

cd /var/lib/riak/bitcask
sudo rm -r *

cd /var/lib/riak/kv_nodes
sudo rm -r *

3. Restart riak service

sudo service riak start

After these steps, the database is empty and riak is up and running.

Setting a riak cluster in Amazon EC2

Just some words related to a riak 1.2.1 cluster installation inside the Amazon AWS network. The official information is very poor documented, some tips were discovered by trial and error, so maybe this post will help you if you are in the same position to build a riak cluster and fighting with poor documentation.

I used Ubuntu Server 12.04 LT micro instances, 64 bits, from cost reasons.

I use the following terminology in order to refer the nodes inside the cluster: Master node and a Slave nodes. The slave nodes will join the Master as they are needed.

The first step is to install riak on every node as it is described here.

First of all, do not use public IPs in configuring the individual nodes. Use the private IPs obtained inside the Amazon AWS network.

Even I followed exactly the steps, when tried to start the riak service, I've fallen in the following error:

Riak failed to start within 15 seconds,
see the output of 'riak console' for more information.
If you want to wait longer, set the environment variable
WAIT_FOR_ERLANG to the number of seconds to wait.

After examining the riak console, by issuing the

riak console

command, the following information is useful :

** Found 1 name clashes in code paths
08:06:51.374 [info] Application lager started on node 'riak@XX.XXX.XX.XXX'
08:06:51.418 [warning] No ring file available.
08:06:51.457 [error] CRASH REPORT Process <0 data-blogger-escaped-.165.0=".165.0"> with 0 neighbours exited with reason: eaddrnotavail in gen_server:init_it/6 line 320

This means that another node is running in the memory and there is a clash in what concerns the port number.

The solution is to issue a

ps aux | grep riak

command and kill all the running riak processes, then restart the service.

If the virtual machine hosting the riak service was stopped for various reasons, restarting it will change its IP so the configuration steps must be reiterated. However the ring information is no more valid anymore. In my situation, being a test installation, I didn't need the stored data so I have deleted the ring info by issuing the follow commands:

cd /var/lib/riak/ring
sudo rm -rf *.*

In case of valuable data, the ring must be redone by performing:

riak-admin reip <old_nodename> <new_nodename>

where  <old_nodename> and <new_nodename> are the names given in /etc/riak/vm.args

Restart then the riak service. It should work now.

If the settings were done on the Master slave, leave it as it is and go to the Slave node(s). For a Slave node, the next step is to join the current node to the Master node. This is done by issuing the command:

riak-admin cluster join <master_nodename>

This is a staged command, it is only planned but not committed since a specific commit command isn’t issued:

riak-admin cluster commit

Perform  these steps for every Slave node in the cluster.  

Hope this post will help you to set a riak cluster in Amazon much faster and smoothly than me.