怎么搭建BeeGFS开发环境

这篇文章将为大家详细讲解有关怎么搭建BeeGFS开发环境，小编觉得挺实用的，因此分享给大家做个参考，希望大家阅读完这篇文章后可以有所收获。

BeeGFS常用于高性能计算中的分布式文件存储，其对巨量小文件的支持相比于其他大多数文件系统而言，要好太多，这里介绍搭建的详细过程。一共使用3台服务器，一个磁盘分成两个区（分别用于元数据和数据存储），两个网卡（一个管理网，一个存储网）。

网络配置

网络分配

管理网络用于SSH登录等功能，都是千兆口，存储网络用于数据传输，为双万兆口Mode 6绑定。

下面的配置以sacd01为例进行说明。

DNS和主机名

sacd01 $ cat /etc/hosts172.29.201.125 sacd01172.29.201.126 sacd02172.29.201.133 sacd03sacd01 $ hostnamesacd01sacd01 $ cat /etc/hostname sacd01

网卡绑定

• 查看网卡绑定情况：

sacd01 $ eths=$(lspci | grep Ethernet); for bond in `ls -d /sys/class/net/bond[0-9]`; do nics=$(ls -d $bond/slave* | awk -F_ -v bond=$bond '$1~bond {print $2}' | tr -d ":"); for nic in $nics; do addr=$(ethtool -i $nic | awk '$1~"bus-info" {print $2}' | awk -F: '{printf $2":"$3}'); echo -e $(echo $bond | awk -F/ '{print $5}') "\t" $nic "\t" $addr "\t" $(echo -e "$eths" | grep $addr | awk -F: '{printf $3}'); done; donebond0  enp26s0f2  1a:00.2 Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)bond1  enp95s0f0  5f:00.0 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)bond1  enp95s0f1  5f:00.1 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)sacd01 $ ip a | grep bond16: enp95s0f0:  mtu 1600 qdisc mq master bond1 state UP group default qlen 10007: enp95s0f1:  mtu 1600 qdisc mq master bond1 state UP group default qlen 10008: bond1:  mtu 1600 qdisc noqueue state UP group default qlen 10009: bond1.1039@bond1:  mtu 1600 qdisc noqueue state UP group default qlen 1000    inet 172.29.39.125/24 brd 172.29.39.255 scope global noprefixroute bond1.1039sacd01 $ ifconfig bond1.1039bond1.1039: flags=4163  mtu 1600        inet 172.29.39.125  netmask 255.255.255.0  broadcast 172.29.39.255        inet6 fe80::6e92:bfff:fe67:9bdc  prefixlen 64  scopeid 0x20        ether 6c:92:bf:67:9b:dc  txqueuelen 1000  (Ethernet)        RX packets 10  bytes 528 (528.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 16  bytes 992 (992.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0sacd01 $ cat /sys/class/net/bond1/bonding/mode balance-alb 6

• 查看网卡绑定配置：

sacd01 $ cat /etc/sysconfig/network-scripts/ifcfg-enp95s0f0TYPE=EthernetNAME="enp95s0f0"DEVICE=enp95s0f0ONBOOT=yesBOOTPROTO=noneMASTER=bond1SLAVE=yessacd01 $ cat /etc/sysconfig/network-scripts/ifcfg-enp95s0f1TYPE=EthernetNAME="enp95s0f1"DEVICE=enp95s0f1ONBOOT=yesBOOTPROTO=noneMASTER=bond1SLAVE=yessacd01 $ cat /etc/sysconfig/network-scripts/ifcfg-bond1DEVICE=bond1BOOTPROTO=noneONBOOT=yesTYPE=BondNM_CONTROLED=noIPV6INIT=noUSERCTL=noIPV6INIT=noBONDING_MASTER=yesBONDING_OPTS="miimon=100 mode=6"#BONDING_OPTS="xmit_hash_policy=layer2+3 mode=4 miimon=100"MTU=1600sacd01 $ cat /etc/sysconfig/network-scripts/ifcfg-bond1.1039 DEVICE=bond1.1039ONBOOT=yesBOOTPROTO=staticIPADDR=172.29.39.125NETMASK=255.255.255.0VLAN=yes

存储配置

磁盘信息

sacd01 $ lsblkNAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTsda               8:0    0 222.6G  0 disk ├─sda1            8:1    0   200M  0 part /boot└─sda2            8:2    0 222.4G  0 part   ├─centos-root 253:0    0 122.4G  0 lvm  /  └─centos-home 253:1    0   100G  0 lvm  /homesdb               8:16   0   7.3T  0 disksacd01 $ lsblk -d | awk '$1!~"sda" && ($1~"^sd" || $1~"^nv") {print $1}' | xargs -I % sh -c 'echo -e "/dev/%\t"; smartctl -i /dev/% | grep -e "Product" -e "Device Model" -e "Model Number" -e "Rotation Rate" -e "Capacity" | awk -F\: "{print $2}"' | awk -F: 'NF==1{printf "\n"$1} NF==2{sub(/^[ ]+/,"",$2);printf $2"\t"} END {printf "\n"}'/dev/sdb        Logical Volume     8,000,999,784,448 bytes [8.00 TB]sacd01 $ smartctl -i /dev/sdbsmartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-693.el7.x86_64] (local build)Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org=== START OF INFORMATION SECTION ===Vendor:               LSIProduct:              Logical VolumeRevision:             3000Compliance:           SPC-4User Capacity:        8,000,999,784,448 bytes [8.00 TB]Logical block size:   512 bytesPhysical block size:  4096 bytesLogical Unit id:      0x600508e000000000fadfaab948205506Device type:          diskLocal Time is:        Wed May  8 09:56:05 2019 CSTSMART support is:     Unavailable - device lacks SMART capability.

磁盘分区

• 分两个2T的盘：

$ sgdisk /dev/sdb --zap-all --clear --mbrtogpt$ sgdisk /dev/sdb -n 1:0:+2T -c 1:"beegfs meta" -t 1:8300 -n 2:0:+2T -c 2:"beegfs data" -t 2:8300Setting name!partNum is 0REALLY setting name!Setting name!partNum is 1REALLY setting name!The operation has completed successfully.$ sgdisk /dev/sdb -pDisk /dev/sdb: 15626952704 sectors, 7.3 TiBLogical sector size: 512 bytesDisk identifier (GUID): A3A34A32-C0FE-4133-BDDB-05A28E2F5455Partition table holds up to 128 entriesFirst usable sector is 34, last usable sector is 15626952670Partitions will be aligned on 2048-sector boundariesTotal free space is 7037018045 sectors (3.3 TiB)Number  Start (sector)    End (sector)  Size       Code  Name   1            2048      4294969343   2.0 TiB     8300  beegfs meta   2      4294969344      8589936639   2.0 TiB     8300  beegfs data$ lsblkNAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTsda               8:0    0 222.6G  0 disk ├─sda1            8:1    0   200M  0 part /boot└─sda2            8:2    0 222.4G  0 part   ├─centos-root 253:0    0 122.4G  0 lvm  /  └─centos-home 253:1    0   100G  0 lvm  /homesdb               8:16   0   7.3T  0 disk ├─sdb1            8:17   0     2T  0 part └─sdb2            8:18   0     2T  0 part

磁盘格式化

• 元数据分区使用EXT4文件系统：

$ mkfs.ext4 -i 2048 -I 512 -J size=400 -Odir_index,filetype /dev/sdb1mke2fs 1.42.9 (28-Dec-2013)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)Stride=0 blocks, Stripe width=0 blocks1073741824 inodes, 536870912 blocks26843545 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=161061273632768 block groups16384 blocks per group, 16384 fragments per group32768 inodes per groupSuperblock backups stored on blocks:         16384, 49152, 81920, 114688, 147456, 409600, 442368, 802816, 1327104,         2048000, 3981312, 5619712, 10240000, 11943936, 35831808, 39337984,         51200000, 107495424, 256000000, 275365888, 322486272Allocating group tables: done                            Writing inode tables: done                            Creating journal (102400 blocks): doneWriting superblocks and filesystem accounting information: done

• 存储分区使用XFS文件系统：

$ mkfs.xfs /dev/sdb2meta-data=/dev/sdb2              isize=512    agcount=4, agsize=134217728 blks         =                       sectsz=4096  attr=2, projid32bit=1         =                       crc=1        finobt=0, sparse=0data     =                       bsize=4096   blocks=536870912, imaxpct=5         =                       sunit=0      swidth=0 blksnaming   =version 2              bsize=4096   ascii-ci=0 ftype=1log      =internal log           bsize=4096   blocks=262144, version=2         =                       sectsz=4096  sunit=1 blks, lazy-count=1realtime =none                   extsz=4096   blocks=0, rtextents=0

磁盘挂载

$ mkdir -pv /beegfs_meta /beegfs_data$ mount /dev/sdb1 /beegfs_meta$ mount /dev/sdb2 /beegfs_data$ echo "/dev/sdb1 /beegfs_meta   xfs     defaults    0   0" | tee -a /etc/fstab$ echo "/dev/sdb2 /beegfs_data   xfs     defaults    0   0" | tee -a /etc/fstab$ df -h | grep beegfs/dev/sdb1                2.0T   33M  2.0T   1% /beegfs_meta/dev/sdb2                2.0T   33M  2.0T   1% /beegfs_data$  mount | grep beegfs/dev/sdb1 on /beegfs_meta type xfs (rw,relatime,attr2,inode64,noquota)/dev/sdb2 on /beegfs_data type xfs (rw,relatime,attr2,inode64,noquota)

CPU配置

$ lscpuArchitecture:          x86_64CPU op-mode(s):        32-bit, 64-bitByte Order:            Little EndianCPU(s):                72On-line CPU(s) list:   0-71Thread(s) per core:    2Core(s) per socket:    18Socket(s):             2NUMA node(s):          2Vendor ID:             GenuineIntelCPU family:            6Model:                 85Model name:            Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHzStepping:              4CPU MHz:               1499.941CPU max MHz:           3700.0000CPU min MHz:           1000.0000BogoMIPS:              4600.00Virtualization:        VT-xL1d cache:             32KL1i cache:             32KL2 cache:              1024KL3 cache:              25344KNUMA node0 CPU(s):     0-17,36-53NUMA node1 CPU(s):     18-35,54-71Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req

内存配置

$ dmidecode -t memory | grep "Size: [0-9]* " -B7 -A12 | grep -e "^Handle" -e "Size:" -e "Type:" -e "Speed:" -e "Manufacturer:" -e "Part Number"Handle 0x0038, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x003B, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x003E, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x0043, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x0046, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x0049, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x004E, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x0051, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x0054, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x0059, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x005C, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK    Handle 0x005F, DMI type 17, 40 bytes        Size: 32 GB        Type: DDR4        Speed: 2666 MT/s        Manufacturer: Hynix        Part Number: HMA84GR7AFR4N-VK$ dmidecode -t memory | grep "Size: [0-9]* " | wc -l12$ free -h              total        used        free      shared  buff/cache   availableMem:           376G        6.8G        368G         10M        983M        368GSwap:            0B          0B          0B

关于"怎么搭建BeeGFS开发环境"这篇文章就分享到这里了，希望以上内容可以对大家有一定的帮助，使各位可以学到更多知识，如果觉得文章不错，请把它分享出去让更多的人看到。