如何分析iscsi、nfs与ceph

这篇文章给大家介绍如何分析iscsi、nfs与ceph，内容非常详细，感兴趣的小伙伴们可以参考借鉴，希望对大家能有所帮助。

一、iscsi设置

1. 服务器端

[root@dlp ~]# yum -y install scsi-target-utils

[2] Configure iSCSI Target .
For example, create a disk image under the [/iscsi_disks] directory and set it as a shared disk.

# create a disk image[root@dlp ~]# mkdir /iscsi_disks[root@dlp ~]# dd if=/dev/zero of=/iscsi_disks/disk01.img count=0 bs=1 seek=80G[root@dlp ~]# vi /etc/tgt/targets.conf# add follows to the end# if you set some devices, add - and set the same way with follows# naming rule : [ iqn.yaer-month.domain:any name ]   # provided devicce as a iSCSI target    backing-store /iscsi_disks/disk01.img   # iSCSI Initiator's IP address you allow to connect    initiator-address 10.0.0.31   # authentication info ( set anyone you like for "username", "password" )    incominguser username password[root@dlp ~]# /etc/rc.d/init.d/tgtd startStarting SCSI target daemon:                   [  OK  ][root@dlp ~]# chkconfig tgtd on# confirm status[root@dlp ~]# tgtadm --mode target --op showTarget 1: iqn.2014-08.world.server:target00    System information:        Driver: iscsi        State: ready    I_T nexus information:    LUN information:        LUN: 0            Type: controller            SCSI ID: IET     00010000            SCSI SN: beaf10            Size: 0 MB, Block size: 1            Online: Yes            Removable media: No            Prevent removal: No            Readonly: No            Backing store type: null            Backing store path: None            Backing store flags:        LUN: 1            Type: disk            SCSI ID: IET     00010001            SCSI SN: beaf11            Size: 85899 MB, Block size: 512            Online: Yes            Removable media: No            Prevent removal: No            Readonly: No            Backing store type: rdwr            Backing store path: /iscsi_disks/disk01.img            Backing store flags:    Account information:        username    ACL information:        10.0.0.31

2. 客户端配置

[1] Configure iSCSI Initiator.

[root@www ~]# yum -y install iscsi-initiator-utils[root@www ~]# vi /etc/iscsi/iscsid.conf# line 53: uncommentnode.session.auth.authmethod = CHAP# line 57,58: uncomment and set username and password which set on iSCSI Targetnode.session.auth.username = usernamenode.session.auth.password = password# discover target[root@www ~]# iscsiadm -m discovery -t sendtargets -p 10.0.0.30Starting iscsid:                                         [  OK  ]10.0.0.30:3260,1 iqn.2014-08.world.server:target00# confirm status after discovery[root@www ~]# iscsiadm -m node -o show# BEGIN RECORD 6.2.0-873.10.el6node.name = iqn.2014-08.world.server:target00node.tpgt = 1node.startup = automaticnode.leading_login = No.........node.conn[0].iscsi.IFMarker = Nonode.conn[0].iscsi.OFMarker = No# END RECORD# login to target[root@www ~]# iscsiadm -m node --loginLogging in to [iface: default, target: iqn.2014-08.world.server:target00, portal: 10.0.0.30,3260] (multiple)Login to [iface: default, target: iqn.2014-08.world.server:target00, portal: 10.0.0.30,3260] successful.# confirm established session[root@www ~]# iscsiadm -m session -o showtcp: [1] 10.0.0.30:3260,1 iqn.2014-08.world.server:target00# confirm partitions[root@www ~]# cat /proc/partitionsmajor minor  #blocks  name   8        0  209715200 sda   8        1     512000 sda1   8        2  209202176 sda2 253        0  200966144 dm-0 253        1    8232960 dm-1   8       16   83886080 sdb# added new device provided from target as [sdb]

[2] It's possible to use iSCSI device like follows.

[root@www ~]# yum -y install parted# create a label[root@www ~]# parted --script /dev/sdb "mklabel msdos"# create a partition[root@www ~]# parted --script /dev/sdb "mkpart primary 0% 100%"# format with EXT4[root@www ~]# mkfs.ext4 /dev/sdb1# mount[root@www ~]# mount /dev/sdb1 /mnt[root@www ~]# df -hTFilesystem           Type   Size  Used Avail Use% Mounted on/dev/mapper/vg_dlp-lv_root                     ext4   189G  1.1G  179G   1% /tmpfs                tmpfs  1.9G     0  1.9G   0% /dev/shm/dev/sda1            ext4   485M   75M  385M  17% /boot/dev/sdb1            ext4    79G  184M   75G   1% /mnt

补充：

客户端安装后需要启动守护进程，然后进行发现：

/etc/init.d/iscsid startiscsiadm -m discovery -t st -p 12.123.0.51iscsiadm -m discovery -T iqn.2015-06.com.oracle:zjxl -p 12.123.0.51:3260 -l#修改配置文件vim /var/lib/iscsi/send_targets/12.123.0.51\,3260/iqn.2015-06.com.oracle\:oracle\:zjxl\,12.123.0.51\,3260\,1\,default/default

配置multipath：

一个机器至少要挂2个存储IP，一个坏了，另一个还可以连接。编辑/etc/multipath.conf

blacklist{devnode "^sda"}defaults{user_friendly_names yesudev_dir /devpath_grouping_policy multibusfailback immediateno_path_retry fail}

blacklist代表系统盘不做multipath。

然后service multipathd restart；执行cat /proc/partitions可以看到以dm-*开头的设备，这些就是多路径设备了。

二、ceph设置

其实对于ceph来讲，直接通过rbd create一个块，然后rbd map这个块，再rbd showmapped可以看到此块map的路径，如/dev/rbd0。然后在iscsi服务器端的/etc/tgt/targets.conf进行配置：

    direct-store /dev/rbd0

这样在iscsi客户端就能使用此块设备了。

但这样使用的代价比较大，因为它是通过ceph rbd kernel module的形式，挂载rbd。这样会有大量的内核态和用户态的切换，势必会影响性能。

这个问题已经有人解决了：

http://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/

http://ceph.com/dev-notes/updates-to-ceph-tgt-iscsi-support/

http://www.sebastien-han.fr/blog/2014/07/07/start-with-the-rbd-support-for-tgt/

在ubuntu上的iscsi已经支持rbd；fedora 20之后有支持rbd的iscsi rpm包。但在centos 6.5，无法配置带rbd支持的iscsi。

如果你也配置过此功能，欢迎交流。

三、利用支持rbd的scsi-target-utils

利用上述方法使ceph导出iscsi设备是没有问题的，但由于其利用了rbd的内核模块，导致内核态和用户态的频繁切换，势必会影响性能。可不可以使rbd块设备直接被导出成为iscsi设备呢？答案是肯定的。

先从以下地址下载支持rbd的scsi-target-utilsrpm包：

#wget http://ceph.com/packages/ceph-extras/rpm/centos6/x86_64/scsi-target-utils-1.0.38-48.bf6981.ceph.el6.x86_64.rpm  #rpm -ivh scsi-target-utils-1.0.38-48.bf6981.ceph.el6.x86_64.rpm

安装完成后，查看当前tgt对于rbd driver是否支持：

# tgtadm --lld iscsi --mode system --op show  System:      State: ready      debug: off  LLDs:      iser: error      iscsi: ready  Backing stores:      rbd (bsoflags sync:direct)      rdwr (bsoflags sync:direct)      ssc      null      bsg      sg      sheepdog  Device types:      passthrough      tape      changer      controller      osd      cd/dvd      disk  iSNS:      iSNS=Off      iSNSServerIP=      iSNSServerPort=3205      iSNSAccessControl=Off

创建一个rbd设备：

#rbd create iscsi/tgt1 -s 10240

修改/etc/tgt/targets.conf，导出刚才创建的rbd设备：

include /etc/tgt/conf.d/*.conf       driver iscsi     bs-type rbd     backing-store iscsi/tgt1   

重启tgt：

# /etc/init.d/tgtd restart  Stopping target framework daemon  Starting target framework daemon

在iscsi initiator端连接该iscsi target：

[root@ceph-osd-1 ~]# iscsiadm -m discovery -t sendtargets -p 10.10.200.165      Starting iscsid:                              [  OK  ]      10.10.200.165:3260,1 iqn.2014-11.rbdstore.com:iscsi  [root@ceph-osd-1 ~]# iscsiadm -m node -T iqn.2014-11.rbdstore.com:iscsi -l      Logging in to [iface: default, target: iqn.2014-11.rbdstore.com:iscsi, portal: 10.10.200.165,3260] (multiple)      Login to [iface: default, target: iqn.2014-11.rbdstore.com:iscsi, portal: 10.10.200.165,3260] successful.  [root@ceph-osd-1 ~]# fdisk -l            Disk /dev/sdb: 5788.2 GB, 5788206759936 bytes      255 heads, 63 sectors/track, 703709 cylinders      Units = cylinders of 16065 * 512 = 8225280 bytes      Sector size (logical/physical): 512 bytes / 512 bytes      I/O size (minimum/optimal): 512 bytes / 512 bytes      Disk identifier: 0x00000000                  Disk /dev/sda: 209.7 GB, 209715068928 bytes      255 heads, 63 sectors/track, 25496 cylinders      Units = cylinders of 16065 * 512 = 8225280 bytes      Sector size (logical/physical): 512 bytes / 512 bytes      I/O size (minimum/optimal): 512 bytes / 512 bytes      Disk identifier: 0x0009a9dd               Device Boot      Start         End      Blocks   Id  System      /dev/sda1   *           1         131     1048576   83  Linux      Partition 1 does not end on cylinder boundary.      /dev/sda2             131        1176     8392704   82  Linux swap / Solaris      /dev/sda3            1176       25497   195356672   8e  Linux LVM            Disk /dev/mapper/vg_swift-LogVol00: 200.0 GB, 200043134976 bytes      255 heads, 63 sectors/track, 24320 cylinders      Units = cylinders of 16065 * 512 = 8225280 bytes      Sector size (logical/physical): 512 bytes / 512 bytes      I/O size (minimum/optimal): 512 bytes / 512 bytes      Disk identifier: 0x00000000                  Disk /dev/mapper/docker-253:0-3539142-pool: 107.4 GB, 107374182400 bytes      255 heads, 63 sectors/track, 13054 cylinders      Units = cylinders of 16065 * 512 = 8225280 bytes      Sector size (logical/physical): 512 bytes / 512 bytes      I/O size (minimum/optimal): 512 bytes / 65536 bytes      Disk identifier: 0x00000000                  Disk /dev/sdc: 10.7 GB, 10737418240 bytes      64 heads, 32 sectors/track, 10240 cylinders      Units = cylinders of 2048 * 512 = 1048576 bytes      Sector size (logical/physical): 512 bytes / 4194304 bytes      I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes      Disk identifier: 0x00000000

从以上可以看到，导出成功。

四、nfs设置

在ceph的一个节点利用rbd map一个块设备，然后格式化并挂载在一个目录，如/mnt。在此节点上安装nfs的rpm包：

yum -y install nfs-utils

设置挂载目录：

[root@mon0 mnt]# cat /etc/exports /mnt 192.168.101.157(rw,async,no_subtree_check,no_root_squash)/mnt 192.168.108.4(rw,async,no_subtree_check,no_root_squash)

启动并导出：

service rpcbind startchkconfig rpcbind onservice nfs startchkconfig nfs onexportfs -r

客户端查看一下：

[root@osd2 /]# showmount -e mon0Export list for mon0:/mnt 192.168.108.4,192.168.101.157

然后挂载：

mount -t nfs mon0:/mnt /mnt

需要注意的是，NFS默认是用UDP协议，如果网络不稳定，换成TCP协议即可：

mount -t nfs mon0:/mnt /mnt -o proto=tcp -o nolock

五、rbd-fuse设置

在客户机上配置ceph.repo后安装rbd-fuse的rpm包，然后就可以挂载pool了：

rbd-fuse -p test /mnt

上面的示例是在没有cephx下将test pool挂载到客户机的/mnt。然后就可以在/mnt中看到test pool中的块了。此时可以利用losetup挂载这个img。

卸载直接利用fusermount -u /mnt。

关于如何分析iscsi、nfs与ceph就分享到这里了，希望以上内容可以对大家有一定的帮助，可以学到更多知识。如果觉得文章不错，可以把它分享出去让更多的人看到。