CentOS 7 iSCSI 多路径（Multipath）配置

1 设置精简预配置LVM逻辑卷

1.1 使用pvcreate命令创建物理卷

假设检测到新磁盘为/dev/sdb

pvcreate /dev/sdb

1.2 使用vgcreate命令创建卷组

vgcreate volgrp /dev/sdb

1.3 从卷组创建精简池

精简池就像使用lvcreate命令创建的逻辑卷

句法 ：

＃lvcreate –L <池的大小> -T <卷组> / <池名称>

其中–L用于指定池的大小，–T用于指定精简池

lvcreate -L 100G -T volgrp/lvpool

验证精简池大小:

[root@target ~]# lvs /dev/volgrp/lvpool
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lvpool volgrp twi-aotz-- 100.00g             0.01   0.43                            
[root@target ~]# lvs
  LV     VG     Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   centos -wi-ao---- 150.00g                                                      
  swap   centos -wi-ao----  <3.88g                                                      
  lvpool volgrp twi-aotz-- 100.00g               0.01   0.43                            
  node1  volgrp Vwi-aotz--   4.00g lvpool        0.26  

1.4 从精简池创建逻辑卷。

让我们创建两个逻辑卷，每个大小为4 GB。

句法 ：

＃lvcreate -V <逻辑卷大小> -T <卷组名> / <名称thinpool> -n <逻辑卷名>

lvcreate -V 4G -T volgrp/lvpool -n node1

1.5 格式化精简配置逻辑卷并挂载到/mnt，写入文件

mkfs.xfs /dev/volgrp/node1
mount /dev/volgrp/node1 /mnt
touch /mnt/wangjinxiong.txt

备注：lvm查看命令如下：

[root@target ~]# vgdisplay 
  --- Volume group ---
  VG Name               centos
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <153.88 GiB
  PE Size               4.00 MiB
  Total PE              39393
  Alloc PE / Size       39392 / <153.88 GiB
  Free  PE / Size       1 / 4.00 MiB
  VG UUID               90zR5I-cTJK-6NsO-MgUr-o9Pe-4W4b-e50bCh
   
  --- Volume group ---
  VG Name               volgrp
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <500.00 GiB
  PE Size               4.00 MiB
  Total PE              127999
  Alloc PE / Size       25650 / <100.20 GiB
  Free  PE / Size       102349 / 399.80 GiB
  VG UUID               MTrVFc-Ax3a-EXym-vViS-mqa5-RPfU-LzcHYf
   
[root@target ~]# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                wdZfj5-UKzO-uiq7-pjsr-OmT8-1hkL-Nb1WHl
  LV Write Access        read/write
  LV Creation host, time localhost, 2020-01-31 18:29:18 +0800
  LV Status              available
  # open                 1
  LV Size                150.00 GiB
  Current LE             38400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/centos/swap
  LV Name                swap
  VG Name                centos
  LV UUID                7swxQq-7Dsk-jWzI-uEcR-sI2x-hucx-OHe73v
  LV Write Access        read/write
  LV Creation host, time localhost, 2020-01-31 18:29:19 +0800
  LV Status              available
  # open                 2
  LV Size                <3.88 GiB
  Current LE             992
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1
   
  --- Logical volume ---
  LV Name                lvpool
  VG Name                volgrp
  LV UUID                VyTn0Q-SFMY-TExw-zjD9-aDZ9-x9Cg-HQyVhP
  LV Write Access        read/write
  LV Creation host, time target, 2020-09-11 09:50:17 +0800
  LV Pool metadata       lvpool_tmeta
  LV Pool data           lvpool_tdata
  LV Status              available
  # open                 2
  LV Size                100.00 GiB
  Allocated pool data    0.01%
  Allocated metadata     0.43%
  Current LE             25600
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:4
   
  --- Logical volume ---
  LV Path                /dev/volgrp/node1
  LV Name                node1
  VG Name                volgrp
  LV UUID                Yf9tsK-urFw-kbrP-kA0y-dWCA-qZ4e-fRMkhU
  LV Write Access        read/write
  LV Creation host, time target, 2020-09-11 09:57:47 +0800
  LV Pool name           lvpool
  LV Status              available
  # open                 1
  LV Size                4.00 GiB
  Mapped size            0.26%
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:6

参考文档：https://blog.csdn.net/allway2/article/details/103113057

2 target服务配置

2.1 安装前准备

关闭selinux

setenforce 0
sed -i '/^SELINUX=.*/ s//SELINUX=disabled/' /etc/selinux/config

关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

安装epel扩展包

yum install -y epel-release

2.2 target设置

安装 scsi-target-utils

yum --enablerepo=epel -y install iscsi-initiator-utils.x86_64

配置

vi /etc/tgt/targets.conf
<target iqn.1994-05.com.redhat:c1ec1f96a0b7>
    # 把/dev/sdb作为块设备共享出去
    backing-store /dev/volgrp/node1
    # 可选，iSCSI Initiator限制
    initiator-address 10.5.29.55
    # 可选，认证配置，username和password配置成你自己的信息
#    incominguser username password
</target>

iqn.1994-05.com.redhat:c1ec1f96a0b7为客户端的iqn号，initiator-address为客户端地址，以下是客户端查看iqn号：

# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:c1ec1f96a0b7

# ip add | grep ens
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.5.29.55/28 brd 10.5.29.63 scope global noprefixroute ens3

开启服务

systemctl enable tgtd.service
systemctl start tgtd.service

查看服务

tgtadm --mode target --op show
Target 1: iqn.1994-05.com.redhat:c1ec1f96a0b7
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1994-05.com.redhat:c1ec1f96a0b7 alias: docker1
            Connection: 0
                IP Address: 10.5.29.55
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 4295 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/volgrp/node1
            Backing store flags: 
    Account information:
    ACL information:
        10.5.29.55

2.3 Initiator配置（服务器及客户端都安装）

安装

yum -y install iscsi-initiator-utils

查看客户端iqn号，iqn号可以自定义，也可以采用默认。

 cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:c1ec1f96a0b7

如果有账号及密码，需要添加以下：（非必须）

# vi /etc/iSCSI/iSCSId.conf
# 以下配置，如果你target没有配置ACL限制，可以直接跳过，使用默认配置即可
# 57行:取消注释
node.session.auth.authmethod = CHAP
# 61,62行:取消注释，用户名和密码设置成前面在target里面设置好的
node.session.auth.username = username
node.session.auth.password = password

扫描设备并展示设备

iscsiadm -m discovery -t sendtargets -p 10.5.30.2
10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7

iscsiadm -m node -o show
# BEGIN RECORD 6.2.0.874-17
node.name = iqn.1994-05.com.redhat:c1ec1f96a0b7
node.tpgt = 1
node.startup = automatic
node.leading_login = No
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_ifacename = default
iface.net_ifacename = <empty>
iface.gateway = <empty>
iface.subnet_mask = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
iface.state = <empty>
iface.vlan_id = 0
iface.vlan_priority = 0
iface.vlan_state = <empty>
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.bootproto = <empty>
iface.dhcp_alt_client_id_state = <empty>
iface.dhcp_alt_client_id = <empty>
iface.dhcp_dns = <empty>
iface.dhcp_learn_iqn = <empty>
iface.dhcp_req_vendor_id_state = <empty>
iface.dhcp_vendor_id_state = <empty>
iface.dhcp_vendor_id = <empty>
iface.dhcp_slp_da = <empty>
iface.fragmentation = <empty>
iface.gratuitous_arp = <empty>
iface.incoming_forwarding = <empty>
iface.tos_state = <empty>
iface.tos = 0
iface.ttl = 0
iface.delayed_ack = <empty>
iface.tcp_nagle = <empty>
iface.tcp_wsf_state = <empty>
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.tcp_timestamp = <empty>
iface.redirect = <empty>
iface.def_task_mgmt_timeout = 0
iface.header_digest = <empty>
iface.data_digest = <empty>
iface.immediate_data = <empty>
iface.initial_r2t = <empty>
iface.data_seq_inorder = <empty>
iface.data_pdu_inorder = <empty>
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
iface.chap_auth = <empty>
iface.bidi_chap = <empty>
iface.strict_login_compliance = <empty>
iface.discovery_auth = <empty>
iface.discovery_logout = <empty>
node.discovery_address = 10.5.30.2
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.nr_sessions = 1
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.session.scan = auto
node.conn[0].address = 10.5.30.2
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD

登陆

iscsiadm -m node --login

确认信息：

iscsiadm -m session -o show
tcp: [1] 10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7 (non-flash)

确认分区信息:

cat /proc/partitions
major minor  #blocks  name

   8        0  209715200 sda
   8        1    1048576 sda1
   8        2  207494144 sda2
 253        0  199229440 dm-0
 253        1    8257536 dm-1
   8       16    4194304 sdb   <--------挂载的iscsi盘

可以看到，sdd和sde就是通过iSCSI挂载得到的磁盘，此时通过fdisk -l命令，你可以发现这两块磁盘和node01的/dev/sdb完全一样，为了正常使用这块磁盘，我们还需要配置磁盘多路径。

参考文档：https://blog.csdn.net/weixin_34082854/article/details/89369861

3 配置多路径软件(Multipath)

什么是多路径？

普通的电脑主机都是一个硬盘挂接到一个总线上，这里是一对一的关系。而到了有光纤组成的SAN环境，或者由iSCSI组成的IPSAN环境，由于主机和存储通过了光纤交换机或者多块网卡及IP来连接，这样的话，就构成了多对多的关系。也就是说，主机到存储可以有多条路径可以选择。主机到存储之间的IO由多条路径可以选择。每个主机到所对应的存储可以经过几条不同的路径，如果是同时使用的话，I/O流量如何分配？其中一条路径坏掉了，如何处理？还有在操作系统的角度来看，每条路径，操作系统会认为是一个实际存在的物理盘，但实际上只是通向同一个物理盘的不同路径而已，这样是在使用的时候，就给用户带来了困惑。多路径软件就是为了解决上面的问题应运而生的。

多路径的主要功能就是和存储设备一起配合实现如下功能：

故障的切换和恢复IO流量的负载均衡磁盘的虚拟化

为了主机可以使用 iSCSI 多路径访问存储设备，我们需要在主机上安装多路径设备映射器 (DM-Multipath)。多路径设备映射器可以实现主机节点和后端存储之间的多个 I/O 路径配置为一个单一逻辑设备，从而可以提供链路冗余和提高性能。主机通过访问包含多个 I/O 路径的逻辑设备，就可以有效提高后端存储系统的可靠性。

3.1 Multipath安装

服务器端配置双地址，其他在客户端配置，包含mutipath服务。

# ip add
.......
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:95:34:bd brd ff:ff:ff:ff:ff:ff
    inet 10.5.30.2/28 brd 10.5.30.15 scope global noprefixroute ens3
       valid_lft forever preferred_lft forever
    inet 10.5.30.3/28 brd 10.5.30.15 scope global secondary noprefixroute ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::4dad:d1ff:bac4:fc10/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

以下在客户端配置

yum install device-mapper-multipath -y

设置开机启动

systemctl enable multipathd.service

添加配置文件

需要multipath正常工作只需要如下配置即可，如果想要了解详细的配置，请参考 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-single/dm_multipath/index

vi /etc/multipath.conf
blacklist {
    devnode "^sda"
}
defaults {
    user_friendly_names yes
    path_grouping_policy multibus
    failback immediate
    no_path_retry fail
}

启动服务

systemctl start multipathd.service

3.2 Multipath挂载

卸载之前挂载的lun及iscsi链接

umount /mnt
iscsiadm -m node -T iqn.1994-05.com.redhat:c1ec1f96a0b7 -p 10.5.30.2:3260 --logout
Logging out of session [sid: 1, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260]
Logout of [sid: 1, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260] successful.

查看服务

multipath -ll
mpatha (360000000000000000e00000000010001) dm-2 IET     ,VIRTUAL-DISK    
size=4.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 4:0:0:1 sdb 8:16 active ready running
  `- 5:0:0:1 sdc 8:32 active ready running

重新建立2个iscsi链接

[root@docker1 /]# iscsiadm -m discovery -t st -p 10.5.30.2
10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7
[root@docker1 /]# iscsiadm -m discovery -t st -p 10.5.30.3
10.5.30.3:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7
[root@docker1 /]# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260] (multiple)
Logging in to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.3,3260] (multiple)
Login to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260] successful.
Login to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.3,3260] successful.

查看iscsi连接

iscsiadm -m session
tcp: [2] 10.5.30.3:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7 (non-flash)
tcp: [3] 10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7 (non-flash)

查看硬盘情况

[root@docker1 /]# lsblk 
NAME            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda               8:0    0   200G  0 disk  
├─sda1            8:1    0     1G  0 part  /boot
└─sda2            8:2    0 197.9G  0 part  
  ├─centos-root 253:0    0   190G  0 lvm   /
  └─centos-swap 253:1    0   7.9G  0 lvm   [SWAP]
sdb               8:16   0     4G  0 disk  
└─mpatha        253:2    0     4G  0 mpath 
sdc               8:32   0     4G  0 disk  
└─mpatha        253:2    0     4G  0 mpath 

挂载硬盘

mount /dev/mapper/mpatha /mnt

备注命令：

发现target,询问target名称bai
[root@RedHat5 ~]# iscsiadm -m discovery -t sendtargets -p targetIP地址：端口号du
该命令将报告出target名称zhi
建立连接
得到了target名称后，dao使用下面命令建立连接
[root@RedHat5 ~]# iscsiadm -m node -T target名称 -p targetIP地址：端口号 --login
这时应该看到映射到的逻辑磁盘名号，使用fdisk -l 可以看到有关逻辑磁盘的详细内容，可以象使用本地硬盘一样对其进行分区、创建文件系统、挂载、卸载等操作。
断开连接
如果逻辑磁盘已经挂载，断开连接前应先卸载，然后使用下面命令
[root@RedHat5 ~]# iscsiadm -m node -T target名称 -p targetIP地址：端口号 --logout
关机之前应该正确的卸载和断开逻辑磁盘，否则会在target服务器上留下僵死的进程，造成服务器资源的浪费。 

3.3 Multipath盘开机自动挂载

修改/etc/fstab，加入以下内容:

/dev/mapper/mpatha      /mnt      xfs     defaults,_netdev  0 0

修改/etc/rc.d/rc.local，加入以下内容：

rm -rf /var/lib/iscsi/nodes/*
iscsiadm -m discovery -t st -p 10.5.30.2
iscsiadm -m discovery -t st -p 10.5.30.3
iscsiadm -m node --login

修改rc.local权限

chmod +x /etc/rc.d/rc.local

3.4 iscsi建立故障

重新启动服务器，发生以下故障：

iscsiadm -m discovery -t st -p 10.5.30.2
iscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.5.30.2,3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7]
10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7

将/var/lib/iscsi/nodes/所有文件删除

rm -rf /var/lib/iscsi/nodes/*

备注：查看已经建立iscsi

iscsiadm -m node

卸载iscsi盘

iscsiadm -m node --logoutall=all

iscsiadm常用命令

增加iscsi存储

(1)发现iscsi存储:iscsiadm
-m discovery -t st -p ISCSI_IP

(2)查看iscsi发现记录:iscsiadm
-m node

(3)登录iscsi存储:iscsiadm
-m node -T LUN_NAME -p ISCSI_IP -l

(4)开机自动:
iscsiadm -m node –T LUN_NAME -p ISCSI_IP -- op

update -n node.startup - v

automatic

 

删除iscsi存储

(1)登出iscsi存储
iscsiadm -m node -T LUN_NAME -p ISCSI_IP -u

(2)对出iscsi所有登录
iscsiadm -m node --logoutall=all

(3)删除iscsi发现记录:iscsiadm
-m node -o delete -T LUN_NAME -p ISCSI_IP

 

登入需验证码的节点

(1)开启认证

iscsiadm
-m node -T LUN_NAME -o update --name node.session.auth.authmethod --value=CHAP

*.使用-o同-- op

(2)添加用户

iscsiadm
-m node -T LUN_NAME -- op

update --name node.session.auth.username --value=[用户名]

(3)添加密码

iscsiadm
–m node –T LUN_NAME – op

update –name node.session.auth.password –value=[密码] 

建立链接例子1：
iscsiadm -m discovery -t st -p 10.186.9.233
iscsiadm -m discovery -t st -p 10.186.9.236
iscsiadm -m node --login

建立链接例子2：
iscsiadm -m discovery -t st -p 192.168.96.104
iscsiadm -m node -T iqn.2016-02.com.smartx:system:iscsi1 -p 192.168.96.104 -l