This is an old revision of the document!
Glusterfs
General Information
Description: GlusterFS is a scalable network filesystem. Uses common off the shelf hardware. It is free and open source.
- Official Site: http://www.gluster.org/
- Gluster Terminology: http://gluster.readthedocs.org/en/latest/Quick-Start-Guide/Terminologies/
- Gluster Architecture: http://gluster.readthedocs.org/en/latest/Quick-Start-Guide/Architecture/
Checklist
- Distro: Enterprise Linux 6
Servers and Client
These examples all assume that there are:
- 2 servers: server1 and server2
- 1 client system
Data Directory Config
Some important recommendations
- It is recommended to dedicate a disk to GlusterFS storage on the server.
- XFS is the preferred file system as it can handle more files than others (such as EXT4)
- Gluster recommends the following naming convention setup:
/data/glusterfs/<volume-name>/<brick#>/brick
- Reasoning:
- This allows for multiple disks (bricks) to be mounted under a Gluster volume and directory structure.
- By creating an additional “/brick” directory under the “<brick#>” directory, the brick will fail to start if the underlying xfs mount is not available.
Servers: Data Dir Config Steps
- Install XFS utilities
yum install xfsprogs
- Create file system with GlusterFS recommended fileystem and inode size
mkfs.xfs -i size=512 /dev/mapper/vgdata-lvdata1
- Create directory structure that data will be stored in
mkdir -p /data/glusterfs/myvol/brick1
- /etc/fstab entry for mount point
## XFS Mount Used for GlusterFS /dev/mapper/vgdata-lvdata1 /data/glusterfs/myvol/brick1 xfs defaults 1 2
- Mount data directory
mount -a
- Create top level “brick” directory - All data will go under here
mkdir /data/glusterfs/myvol/brick1/brick
Installation
Installation steps taken for the glusterfs servers and clients.
Server Install
On the Gluster Servers:
- Add the gluster repo
wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo
- Install glusterfs-server
yum install glusterfs-server
- Start gluster daemon
service glusterd start
- Ensure gluster daemon is enabled on startup
chkconfig glusterd on
Client Install
On the Gluster Clients:
- Install Gluster repo
wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo
- Install glusterfs and fuse
yum install glusterfs glusterfs-fuse
Configuration
Configuration steps to setup gluster servers and clients.
Server Config
- Configure Trusted Pool
- From server1
gluster peer probe server2
- Note: Once a trusted pool has been created, only existing pool members can probe new servers.
- New servers cannot probe a pool member.
- Create a GlusterFS Volume ⇒ How the volume is created depicts whether data is distributed, striped, replicated, or a combination.
- Note: A volume cannot be created at a mount point; it must be a subdirectory within the mount point
- Distributed(default): A normal “gluster volume create” with no arguments creates a distributed volume. This has no redundancy; files will round robin between bricks that are in the volume. The following alternates between server1 and server2 storage.
gluster volume create myvol server1:/data/glusterfs/myvol/brick1/brick server2:/data/glusterfs/myvol/brick1/brick
- Replicated: A copy of the data is kept on each brick. (Amount determined by the replica number, but should equal # of bricks) The following keeps a copy on server1 and server2.
gluster volume create myvol replica 2 transport tcp server1:/data/glusterfs/myvol/brick1/brick server2:/data/glusterfs/myvol/brick1/brick
- Distributed Replicated: Files distributed across replicated sets of bricks. (# bricks should be multiple of # of replicas and adjacent servers are the replicated sets) The following round robins between the “replicated sets” server1/server2 and server3/server4. Data will exist on two servers at all times.
gluster volume create myvol replica 2 transport tcp server1:/data/glusterfs/myvol/brick1/brick server2:/data/glusterfs/myvol/brick1/brick server3:/data/glusterfs/myvol/brick1/brick server4:/data/glusterfs/myvol/brick1/brick
- More types:
- Start the Gluster volume to make it usable by clients
gluster volume start myvol
Client Config
- Temporary mount
mount -t glusterfs server1:/myvol /mnt
- Persistent mount in /etc/fstab
vim /etc/fstab ## Gluster mounts server1:/myvol /data glusterfs defaults 0 0
Operation
Server Ops
Viewing Status
- View peer status
gluster peer status
- View gluster volume info
gluster volume info
Balance Data
- Re-balance Data
gluster volume rebalance myvol start
- View re-balance status
gluster volume rebalance status
Storage Pools
- Add server to the trusted storage pool
gluster peer probe <server-name>
- Remove server from trusted storage pool
gluster peer detach <server-name>
Volumes
- Add a brick to the volume (add to trusted storage pool first)
gluster volume add-brick myvol server5:/data/glusterfs/myvol/brick1/brick
- Remove a brick from the volume (and then remove from trusted storage pool)
gluster volume remove-brick myvol server2:/data/glusterfs/myvol/brick1/brick
- Replace a faulty brick
gluster volume replace-brick myvol server3:/data/glusterfs/myvol/brick1/brick server5:/data/glusterfs/myvol/brick1/brick commit force
Security
- Allow only clients from a specific network (networks comma separated for multiple)
gluster volume set myvol auth.allow 10.1.2.*
- Note: Gluster servers do not need to be added to this list. The servers will use an auto-generated username/password when they are added to the trusted storage pool.