General Information

Description: GlusterFS is a scalable network filesystem. Uses common off the shelf hardware. It is free and open source.

• Official Site: http://www.gluster.org/
• Gluster Terminology: http://gluster.readthedocs.org/en/latest/Quick-Start-Guide/Terminologies/
• Gluster Architecture: http://gluster.readthedocs.org/en/latest/Quick-Start-Guide/Architecture/

Checklist

• Distro: Enterprise Linux 6

These examples all assume that there are:

• 2 servers: server1 and server2
• 1 client system

• It is recommended to dedicate a disk to GlusterFS storage on the server.
• XFS is the preferred file system as it can handle more files than others (such as EXT4)
• Gluster recommends the following naming convention setup:

  • /data/glusterfs/<volume-name>/<brick#>/brick

• Reasoning:
  • This allows for multiple disks (bricks) to be mounted under a Gluster volume and directory structure.
  • By creating an additional “/brick” directory under the “<brick#>” directory, the brick will fail to start if the underlying xfs mount is not available.

1. Install XFS utilities

   1. yum install xfsprogs

2. Create file system with GlusterFS recommended fileystem and inode size

   1. mkfs.xfs -i size=512 /dev/mapper/vgdata-lvdata1

3. Create directory structure that data will be stored in

   1. mkdir -p /data/glusterfs/myvol/brick1

4. /etc/fstab entry for mount point

   1. ## XFS Mount Used for GlusterFS
      /dev/mapper/vgdata-lvdata1 /data/glusterfs/myvol/brick1 xfs  defaults 1 2

5. Mount data directory

   1. mount -a

6. Create top level “brick” directory - All data will go under here

   1. mkdir /data/glusterfs/myvol/brick1/brick

Installation steps taken for the glusterfs servers and clients.

On the Gluster Servers:

1. Add the gluster repo

   1. wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo

2. Install glusterfs-server

   1. yum install glusterfs-server

1. Start gluster daemon

   1. service glusterd start

2. Ensure gluster daemon is enabled on startup

   1. chkconfig glusterd on

On the Gluster Clients:

1. Install Gluster repo

   1. wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo

2. Install glusterfs and fuse

   1. yum install glusterfs glusterfs-fuse

Configuration steps to setup gluster servers and clients.

1. Configure Trusted Pool
   1. From server1

      1. gluster peer probe server2

      2. Note: Once a trusted pool has been created, only existing pool members can probe new servers.
      3. New servers cannot probe a pool member.
2. Create a GlusterFS Volume

   1. gluster volume create myvol server1:/data/glusterfs/myvol/brick1/brick server2:/data/glusterfs/myvol/brick1/brick

   2. Distributed(default): No redundancy, files will round robin between bricks that are in the volume.
   3. Note: A volume cannot be created at a mount point; it must be a subdirectory within the mount point
3. Start the Gluster volume to make it usable by clients

   1. gluster volume start myvol

• Temporary mount

  • mount -t glusterfs server1:/myvol /mnt

• Persistent mount in /etc/fstab

  • vim /etc/fstab
    ## Gluster mounts
    server1:/myvol /data glusterfs defaults 0 0

• View peer status

  • gluster peer status

• View gluster volume info

  • gluster volume info

• Re-balance Data

  • gluster volume rebalance myvol start

• View re-balance status

  • gluster volume rebalance status

• Daemon control

  • service glusterd [stop|start|restart|status]

• Add server to the trusted storage pool

  • gluster peer probe <server-name>

• Remove server from trusted storage pool

  • gluster peer detach <server-name>

• Allow only clients from a specific network (networks comma separated for multiple)

  • gluster volume set myvol auth.allow 10.1.2.*

  • Note: Gluster servers do not need to be added to this list. The servers will use an auto-generated username/password when they are added to the trusted storage pool.