Tuesday, November 11, 2014

Tuning RDO Juno CentOS 7 TwoNode Gluster 3.5.2 Cluster for Qemu integration with libgfapi to work seamlessly

This post is  focused on tuning replica 2 gluster volume when building RDO Juno Gluster Cluster on CentOS 7. Steps undertaken come from Gluster 3.5.2 Release Notes (http://blog.nixpanic.net/2014_07_01_archive.html) and make integration Qemu (1.5.3) && libgfapi really working

- Controller node: Nova, Keystone, Cinder, Glance, Neutron (using Open vSwitch plugin  )
- Compute node: Nova (nova-compute), Neutron (openvswitch-agent)

juno1.localdomain   -  Controller (192.168.1.127)
juno2.localdomain   -  Compute   (192.168.1.137)

Download from http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.2/EPEL.repo/epel-7/SRPMS/
glusterfs-3.5.2-1.el7.src.rpm

$ rpm -iv glusterfs-3.5.2-1.el7.src.rpm

$ sudo yum install bison flex gcc automake libtool ncurses-devel readline-devel libxml2-devel openssl-devel libaio-devel lvm2-devel glib2-devel libattr-devel libibverbs-devel librdmacm-devel fuse-devel

$ rpmbuild -bb glusterfs.spec
. . . . . . . . . . . . . . . . . . . . . . .

Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-libs-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-cli-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-rdma-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-geo-replication-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-fuse-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-server-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-api-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-extra-xlators-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/noarch/glusterfs-resource-agents-3.5.2-1.el7.centos.noarch.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-devel-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-api-devel-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-regression-tests-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-debuginfo-3.5.2-1.el7.centos.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Sigc7l
+ umask 022
+ cd /home/boris/rpmbuild/BUILD
+ cd glusterfs-3.5.2
+ rm -rf /home/boris/rpmbuild/BUILDROOT/glusterfs-3.5.2-1.el7.centos.x86_64
+ exit 0

[boris@juno1 x86_64]$ cat install
sudo yum install glusterfs-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-api-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-api-devel-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-cli-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-devel-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-extra-xlators-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-fuse-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-geo-replication-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-libs-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-rdma-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-server-3.5.2-1.el7.centos.x86_64.rpm

$ sudo service glusterd start

1. First step is tuning /etc/sysconfig/iptables for IPv4 iptables firewall (service firewalld should be disabled) :-

Update /etc/sysconfig/iptables on both nodes:-

-A INPUT -p tcp -m multiport --dport 24007:24047 -j ACCEPT
-A INPUT -p tcp --dport 111 -j ACCEPT
-A INPUT -p udp --dport 111 -j ACCEPT
-A INPUT -p tcp -m multiport --dport 38465:38485 -j ACCEPT

Comment out lines bellow , ignoring instruction

# -A FORWARD -j REJECT --reject-with icmp-host-prohibited
# -A INPUT -j REJECT --reject-with icmp-host-prohibited

 Restart service iptables on both nodes

2. Second step:-


On juno1, run the following commands :

# ssh-keygen (Hit Enter to accept all of the defaults)
# ssh-copy-id -i ~/.ssh/id_rsa.pub  root@juno2

On both nodes run :-

# ./install
# service glusterd start

On juno1

#gluster peer probe juno2.localdomain

Should return "success"

[root@juno1 ~(keystone_admin)]# gluster peer status
Number of Peers: 1

Hostname: juno2.localdomain
Uuid: 3ca6490b-c44a-4601-ac13-51fec99e9caf
State: Peer in Cluster (Connected)

[root@juno1 ~(keystone_admin)]# ssh 192.168.1.137
Last login: Thu Aug 14 17:53:41 2014
[root@juno2 ~]# gluster peer status
Number of Peers: 1

Hostname: 192.168.1.127
Uuid: 051e7528-8c2b-46e1-abb6-6d84b2f2e45b
State: Peer in Cluster (Connected)


*************************************************************************
On Controller (192.168.1.127) and on Compute (192.168.1.137)
*************************************************************************

Verify ports availability:-

[root@juno1 ~(keystone_admin)]# netstat -lntp | grep gluster
tcp        0      0 0.0.0.0:49152           0.0.0.0:*               LISTEN      5453/glusterfsd 
tcp        0      0 0.0.0.0:2049             0.0.0.0:*               LISTEN      5458/glusterfs   
tcp        0      0 0.0.0.0:38465           0.0.0.0:*               LISTEN      5458/glusterfs   
tcp        0      0 0.0.0.0:38466           0.0.0.0:*               LISTEN      5458/glusterfs   
tcp        0      0 0.0.0.0:38468           0.0.0.0:*               LISTEN      5458/glusterfs   
tcp        0      0 0.0.0.0:38469           0.0.0.0:*               LISTEN      5458/glusterfs   
tcp        0      0 0.0.0.0:24007           0.0.0.0:*               LISTEN      2667/glusterd   
tcp        0      0 0.0.0.0:978               0.0.0.0:*               LISTEN      5458/glusterfs

************************************
Switching Cinder to Gluster volume
************************************

# gluster volume create cinder-volumes57 \
replica 2 juno1.localdomain:/data5/data-volumes   juno2.localdomain:/data5/data-volumes 

# gluster volume start cinder-volumes57

# gluster volume set cinder-volumes57  auth.allow 192.168.1.*

The following configuration changes are necessary for 'qemu' and '
samba vfs plugin' integration with libgfapi to work seamlessly:

1. First step

       gluster volume set cinder-volumes57 server.allow-insecure on

2. Restarting is required
   
    gluster volume stop cinder-volumes57
    gluster volume start cinder-volumes57

3. Edit /etc/glusterfs/glusterd.vol   to have a line :
    
     option rpc-auth-allow-insecure on

4. Restart glusterd is required :

     service glusterd restart
  

Nova.conf (on Compute Node)  should have entry :-

qemu_allowed_storage_drivers = gluster

[root@juno1 ~]# gluster volume info
Volume Name: cinder-volumes57
Type: Replicate
Volume ID: c1f2e1d2-0b11-426e-af3d-7af0d1d24d5e
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: juno1.localdomain:/data5/data-volumes
Brick2: juno2.localdomain:/data5/data-volumes
Options Reconfigured:
auth.allow: 192.168.1.*
server.allow-insecure: on

[root@juno1 ~]# gluster volume status
Status of volume: cinder-volumes57
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick juno1.localdomain:/data5/data-volumes        49152    Y    3346
Brick juno2.localdomain:/data5/data-volumes        49152    Y    3113
NFS Server on localhost                    2049    Y    3380
Self-heal Daemon on localhost                N/A    Y    3387
NFS Server on juno2.localdomain                2049    Y    3911
Self-heal Daemon on juno2.localdomain            N/A    Y    3916

Task Status of Volume cinder-volumes57
------------------------------------------------------------------------------
There are no active volume tasks


##############################
Create entries  in /etc/cinder/cinder.conf
############################## 

enabled_backends=gluster

[gluster]
volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config = /etc/cinder/shares.conf
glusterfs_mount_point_base = /var/lib/cinder/volumes
volume_backend_name=GLUSTER


# vi /etc/cinder/shares.conf
    192.168.1.127:/cinder-volumes57
:wq


[root@juno1 ~(keystone_admin)]# cinder type-create gluster
+--------------------------------------+---------+
|                  ID                  |   Name  |
+--------------------------------------+---------+
| 29917269-d73f-4c28-b295-59bfbda5d044 | gluster |
+--------------------------------------+---------+

[root@juno1 ~(keystone_admin)]# cinder type-key gluster  set volume_backend_name=GLUSTER

Next step is cinder services restart :-

[root@juno1 ~(keystone_demo)]# for i in api scheduler volume ; do service openstack-cinder-${i} restart ; done

[root@juno1 ~(keystone_admin)]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos01-root00      147G   43G  105G  29% /
devtmpfs                         3.9G     0  3.9G   0% /dev
tmpfs                            3.9G  152K  3.9G   1% /dev/shm
tmpfs                            3.9G   26M  3.8G   1% /run
tmpfs                            3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0                       1.9G  6.0M  1.7G   1% /srv/node/swift_loopback
/dev/sda3                        477M  146M  302M  33% /boot
/dev/mapper/centos01-data5        98G   15G   83G  16% /data5
192.168.1.127:/cinder-volumes57   98G   15G   83G  16% /var/lib/cinder/volumes/8478b56ad61cf67ab9839fb0a5296965
tmpfs                            3.9G   26M  3.8G   1% /run/netns


###################################################
How to verify implementation success. Boot nova instance 
( with instance-id say 00000049) based on cinder volume.
###################################################

On Compute Node grep /var/log/libvirt/qemu/instance-00000049.log looking for
"gluster" entry . You are supposed to find a string highlighted down here

# cd /var/log/libvirt/qemu
# [root@juno2 qemu]# cat instance-00000049.log | grep gluster
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00000049 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 92151b16-c7b4-48d1-b49f-1e310e005c80 -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2014.2-2.el7.centos,serial=5dff0de4-c27d-453d-85b4-b2d9af514fcd,uuid=92151b16-c7b4-48d1-b49f-1e310e005c80 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000049.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=gluster://192.168.1.127:24007/cinder-volumes57/volume-179b9782-d2b7-4891-ba89-5198b71c6188,if=none,id=drive-virtio-disk0,format=raw,serial=179b9782-d2b7-4891-ba89-5198b71c6188,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:8b:9f:6c,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/92151b16-c7b4-48d1-b49f-1e310e005c80/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


At the same time issue on Controller following commands  :-

[root@juno1 ~(keystone_boris)]# cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status |   Display Name  | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| 179b9782-d2b7-4891-ba89-5198b71c6188 | in-use | Win2012GLSVOL01 |  20  |   gluster   |   true   | 92151b16-c7b4-48d1-b49f-1e310e005c80 |
| ca0694ae-7e8d-4c84-aad8-3f178416dec6 | in-use |  VF20LVG520711  |  7   |     lvms    |   true   | 51a20959-0a0c-4ef6-81ec-2edeab6e3588 |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+

[root@juno1 ~(keystone_boris)]# nova list
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
| ID                                   | Name         | Status    | Task State | Power State | Networks                               |
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
| 51a20959-0a0c-4ef6-81ec-2edeab6e3588 | VF20RX520711 | SUSPENDED | -          | Shutdown    | private_boris=50.0.0.12, 192.168.1.175 |
| 92151b16-c7b4-48d1-b49f-1e310e005c80 | Win2012SRV05 | SUSPENDED | -          | Shutdown    | private_boris=50.0.0.25, 192.168.1.179 |
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+

[root@juno1 ~(keystone_boris)]# nova show 92151b16-c7b4-48d1-b49f-1e310e005c80 | grep 179b9782-d2b7-4891-ba89-5198b71c6188
| os-extended-volumes:volumes_attached | [{"id": "179b9782-d2b7-4891-ba89-5198b71c6188"}]         |



##############################################
Another way of verification - run on Compute Node:-
##############################################

[root@juno1 ~(keystone_boris)]# ssh 192.168.1.137
Last login: Tue Nov 11 17:12:04 2014 from juno1.localdomain

[root@juno2 ~]# . keystonerc_boris

[root@juno2 ~(keystone_boris)]# nova list
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+
| ID                                   | Name           | Status    | Task State | Power State | Networks                               |
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+
| 57640068-3ab7-466a-8eae-cf132359b233 | UbuntuUTRX1211 | ACTIVE    | -          | Running     | private_boris=50.0.0.26, 192.168.1.174 |
| 51a20959-0a0c-4ef6-81ec-2edeab6e3588 | VF20RX520711   | SUSPENDED | -          | Shutdown    | private_boris=50.0.0.12, 192.168.1.175 |
| 92151b16-c7b4-48d1-b49f-1e310e005c80 | Win2012SRV05   | SUSPENDED | -          | Shutdown    | private_boris=50.0.0.25, 192.168.1.179 |
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+

[root@juno2 ~(keystone_boris)]# virsh dumpxml 57640068-3ab7-466a-8eae-cf132359b233 | grep -E 'source (file|protocol)'

  <source protocol='gluster' name='cinder-volumes57/volume-bf448475-50c8-4491-92aa-77d36666f296'>

[root@juno2 ~(keystone_boris)]# nova show 57640068-3ab7-466a-
8eae-cf132359b233 | grep bf448475-50c8-4491-92aa-77d36666f296
| os-extended-volumes:volumes_attached | [{"id": "bf448475-50c8-4491-92aa-77d36666f296"}]         |