Setting up a network distributed file system, SSL/TLS encrypted, with GlusterFS on the server and clients.

GlusterFS is a network-attached storage file system like NFS but better and perfect for i.e. Webserver data or any other data storage for cloud services. The basic installation and setup is pretty much straight forward using our package manager. By default GlusterFS clients and server will communicate without any encryption. This short tutorial will show you how to quickly setup a basic client-server model with SSL/TLS enabled. Please note that further information to GlusterFS SSL/TLS can be found here.

Requirements:

You need to have or install at least two machines with Debian 8.x. If you don’t have two test servers yet, just login to your AWS account and create two instances with Debian 8.7 (https://wiki.debian.org/Cloud/AmazonEC2Image/Jessie).

For this example I use ap-southeast-2 (SYDNEY) ami-881317eb, t2.micro with the options “Auto-assign Public IP = enabled” and 8GB HDD. Please ensure that the security group/firewall rules you assign to the servers has nothing blocked for internal traffic (debug only).

Example server setup:

Debian GlusterFS Master Server (gfs-master)

External IP: 1.1.1.1
Internal IP: 10.0.0.1
Connect command: ssh [email protected] -i mykey.pem
SuperUser command: sudo su

Debian GlusterFS Client (gfs-client)

External IP: 1.1.1.2
Internal IP: 10.0.0.2
Connect command: ssh [email protected] -i mykey.pem
SuperUser command: sudo su

Setup and install GlusterFS on the Master and Client without SSL/TLS

Installation on the master server

First we update our machine, and the install GlusterFS client and server software.

ssh [email protected] -i mykey.pem
sudo su
apt-get update && apt-get upgrade
apt-get install glusterfs-client glusterfs-server

Now we create our storage path on the server and a local mount directory

mkdir /srv/glusterfs/
mkdir /mnt/gfs-vol-test-1

Next we actually create a test volume called “gfs-vol-test-1”, start it and mount it.

gluster volume create gfs-vol-test-1 10.0.0.1:/srv/glusterfs/gfs-vol-test-1 force
gluster volume start gfs-vol-test-1
mount.glusterfs 127.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1

At last we create a test file in our new volume

echo 'hello gluster' > /mnt/gfs-vol-test-1/testglust.txt
ls -l /mnt/gfs-vol-test-1/

Installation on the client

First we update our machine, and the install the GlusterFS client software.

ssh [email protected] -i mykey.pem
sudo su
apt-get update && apt-get upgrade
apt-get install glusterfs-client

First we create a local mount path

mkdir /mnt/gfs-vol-test-1

Now we can mount the volume and view our previous created text file from the master server

mount.glusterfs 10.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1
ls -l /mnt/gfs-vol-test-1/
cat /mnt/gfs-vol-test-1/testglust.txt

Enabling SSL/TLS on the server and client

We start again with the master server and have at first a look into the volume information

gluster volume info gfs-vol-test-1
> Volume Name: gfs-vol-test-1
> Type: Distribute
> Volume ID: f3b5e3a2-a9a4-4099-b6cc-958c7d9572e0
> Status: Started
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.0.1:/srv/glusterfs/gfs-vol-test-1

This tells us that SSL/TLS is not yet setup.

For our example we create us our own local certificates:

openssl genrsa -out /etc/ssl/glusterfs.key 2048
openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj /CN=Anyone -out /etc/ssl/glusterfs.pem
cp /etc/ssl/glusterfs.pem /etc/ssl/glusterfs.ca
ls -l /etc/ssl/glusterfs*
> /etc/ssl/glusterfs.ca
> /etc/ssl/glusterfs.key
> /etc/ssl/glusterfs.pem

Next we are actually enabling SSL/TLS support for clients and server:

gluster volume set gfs-vol-test-1 client.ssl on
gluster volume set gfs-vol-test-1 server.ssl on
touch /var/lib/glusterd/secure-access

At last we need to copy our certificates over to the client. In this demo setup I will use our GlusterFS test volume. DO NOT DO THIS IF THIS IS A LIVE SYSTEM! In production service please securely transfer the keys!

cp -ar /etc/ssl/glusterfs* /mnt/gfs-vol-test-1/

Client SSL/TLS setup

On our client side we now have to copy the keys into our SSL folder, create a config file for our client and then we unmount the volume.

cp /mnt/gfs-vol-test-1/glusterfs* /etc/ssl/
touch /var/lib/glusterd/secure-access
umount /mnt/gfs-vol-test-1

Getting the master server finally serving via SSL/TLS

Unmount the volume

umount /mnt/gfs-vol-test-1

Reboot server and not just restart the service, unless you stop the service and use kill to remove remaining not stopped daemons as the command “service glusterfs-server restart” is faulty or may takes up to 15 minutes to kill old connections!

Once rebooted mount the local volume

mount.glusterfs 127.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1

Mounting the SSL/TLS encrypted volume on the client

On the client side we only need to execute the standard mount command

mount.glusterfs 10.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1

I hope that this short tutorial will help you setting up your own distributed file system. Please note that the above is absolutely simplified. I have written this after been asked a few times for to setup the SSL part so I thought I give everyone a very easy tutorial which actually works out of the box (and yes I tested it).