Xen Virtualization on Linux and Solaris: Glusterfs volume based Havana 2013.2 instances on NFS-Like Standalone Storage Server With GlusterFS 3.4.1 Fedora 19

Following http://www.gluster.org/category/openstack/

This is a snapshot to show the difference between the Havanna and Grizzly releases with GlusterFS.

Grizzly	Havana
Glance – Could point to the filesystem images mounted with GlusterFS, but had to copy VM image to deploy it	Can now point to Cinder interface, removing the need to copy image
Cinder – Integrated with GlusterFS, but only with Fuse mounted volumes	Can now use libgfapi-QEMU integration for KVM hypervisors
Nova – No integration with GlusterFS	Can now use the libgfapi-QEMU integration
Swift – GlusterFS maintained a separate repository of changes to Swift proxy layer	Swift patches now merged upstream, providing a cleaner break between API and implementation

Please, view also nice article: http://www.mirantis.com/blog/openstack-havana-glusterfs-and-what-improved-support-really-means/

and https://wiki.openstack.org/wiki/CinderSupportMatrix

Actually, on Glusterfs F19 Server included in cluster procedure of cinder tuning should be the same. First step - set up Havana RC1 RDO on Fedora 19 per

http://bderzhavets.blogspot.ru/2013/10/neutron-basic-rdo-setup-havana-to-have.html

Post install fix (as of 10/31/2013) :

Also set ALLOWED_HOSTS = ['*'] in /etc/openstack-dashboard/local_settings and restart httpd

Next - installing GlusterFS Server on Cinder host

# yum install glusterfs glusterfs-server glusterfs-fuse
# systemctl status glusterd

glusterd.service - GlusterFS an clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)
   Active: active (running) since Fri 2013-10-18 13:47:51 MSK; 2h 37min ago
Process: 1126 ExecStart=/usr/sbin/glusterd -p /run/glusterd.pid (code=exited, status=0/SUCCESS)
Main PID: 1136 (glusterd)
   CGroup: name=systemd:/system/glusterd.service
           ├─1136 /usr/sbin/glusterd -p /run/glusterd.pid
           ├─8861 /usr/sbin/glusterfsd -s 192.168.1.135 --volfile-id cinder-volume.192.168.1.135.rhs-brick1-cinder-volume -...
           ├─8878 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/lo...
           └─8885 /sbin/rpc.statd

Oct 18 13:47:51 localhost.localdomain systemd[1]: Started GlusterFS an clustered file-system server.
Oct 18 13:58:19 localhost.localdomain rpc.statd[8885]: Version 1.2.7 starting
Oct 18 13:58:19 localhost.localdomain sm-notify[8886]: Version 1.2.7 starting
Oct 18 13:58:19 localhost.localdomain rpc.statd[8885]: Initializing NSM state

# mkdir -p /rhs/brick1/cinder-volume
# gluster volume create cinder-volume \
     192.168.1.135:/rhs/brick1/cinder-volume
# gluster volume start cinder-volume
# gluster volume info

Volume Name: cinder-volume
Type: Distribute
Volume ID: d52c0ba1-d7b1-495d-8f14-07ff03e7db95
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 192.168.1.135:/rhs/brick1/cinder-volume

A sample of utilizing stripe gluster volume may be viewed here

Configuring Cinder to Add GlusterFS

# openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver

# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config /etc/cinder/shares.conf

# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base /var/lib/cinder/volumes

# vi /etc/cinder/shares.conf

192.168.1.135:cinder-volume

:wq

# iptables-save > iptables.dump

Add to *filter section:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24007 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24008 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24009 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24010 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 24011 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 38465:38469 -j ACCEPT

# iptables-restore < iptables.dump
# service iptables restart

Restart openstack-cinder-volume service mounts glusterfs volume with no problems with ownership :

# for i in api scheduler volume ; do service openstack-cinder-${i} restart ;done

Redirecting to /bin/systemctl restart openstack-cinder-api.service
Redirecting to /bin/systemctl restart openstack-cinder-scheduler.service
Redirecting to /bin/systemctl restart openstack-cinder-volume.service
# df -h

Filesystem                   Size Used Avail Use% Mounted on
/dev/mapper/fedora-root      164G   29G 127G 19% /
devtmpfs                     3.9G     0 3.9G   0% /dev
tmpfs                       3.9G 148K 3.9G   1% /dev/shm
tmpfs                       3.9G 1.1M 3.9G   1% /run
tmpfs                       3.9G     0 3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G 800K 3.9G   1% /tmp
/dev/sda1                    477M   87M 362M 20% /boot
/dev/loop0                   928M 1.4M 860M   1% /srv/node/device1
tmpfs                           3.9G 1.1M 3.9G   1% /run/netns
192.168.1.135:cinder-volume 164G   29G 127G 19% /var/lib/cinder/volumes/f39d1b2d7e2a2e48af66eceba039b139

# ls -l /var/lib/cinder/volumes/f39d1b2d7e2a2e48af66eceba039b139

total 5528248

-rw-rw-rw-. 1 qemu qemu 5368709120 Oct 18 16:19 volume-0474ead2-61a8-41dd-8f8d-ef3000266403

-rw-rw-rw-. 1 qemu qemu 5368709120 Oct 18 16:19 volume-da344703-dcf9-450e-9e34-cafb331f80f6

Screen shots on another F19 instance dual booting with first

Creating via cinder command line Ubuntu 13.10 Server bootable volume in glusterfs storage :

References

http://www.gluster.org/community/documentation/index.php/GlusterFS_Cinder

Xen Virtualization on Linux and Solaris

Friday, October 18, 2013

Glusterfs volume based Havana 2013.2 instances on NFS-Like Standalone Storage Server With GlusterFS 3.4.1 Fedora 19