分布式文件系统mogilefs的简单应用
简介
MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的DangaInteractive公司开发。Danga团队开发了包括Memcached、MogileFS、Perlbal等多个知名的开源项目。
特性
1、应用层：用户空间文件系统，无须特殊的核心组件
2、无单点：（tracker, mogstore, database(MySQL)）
3、自动文件复制：复制的最小单位不是文件，而class；
4、传输中立，无特殊协议：可以通过NFS或HTTP进行通信；
5、简单的命名空间： 每个文件对应一个key：用于domain定义名称空间
组成部分
MogileFS：
tracker：追踪元数据
database：存储元数据
storage：存储数据
tracker：
mogilefsd（守护进程）,职责：
replication：节点之间的文件复制
deletion：删除文件
queryworker：响应客户请求的文件元数据访问请求
reaper：在存储失败后将文件复制请求重新放置于队列中
monitor：检测主机和设备的健康状态
database：
存储mogilefs的元数据，一般使用mysql，建议使用冗余方案保证可用性（MMM,MHA）
mogilefs专门提供了数据结构管理工具mogdbsetup；
storage：
mogstored（进程名），一个准备停当的mogstored节点可通过mogadm命令添加至现在的集群中，
存储节点需定义"设备dev"用作存储空间；每个设备在当前集群中需通过一个唯一的DevID标识。
client：
客户端用于与mogilefs简历通信，完成数据存取；
安装包：
MogileFS-Server-2.46-2.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
perl-MogileFS-Client-1.14-1.el6.noarch.rpm
perl-Net-Netmask-1.9015-8.el6.noarch.rpm
perl-Perlbal-1.78-1.el6.noarch.rpm
架构：
1.安装mogilefs和数据库
# ls
MogileFS-Server-2.46-2.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
# yum localinstall *.rpm -y
# yum install mysql-server mysql -y
2.数据库授权远程访问，并继承授权权限
mysql> grant all on *.* to root@'192.168.95.%' identified by 'liaobin' with grant option;
mysql> flush privileges;
3.创建mofilefs的数据库用户以及授权
mysql> grant all on *.* to mguser@'192.168.95.%' identified by 'liaobin';
mysql> flush privileges;
4.用mogilefs自带工具mogdbsetup初始化mogilefs数据库
# mogdbsetup --dbhost=192.168.95.30 --dbname=mogilefs --dbrootuser=root --dbrootpass=liaobin --dbuser=mguser --dbpass=liaobin --yes
查看初始化的数据库
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mogilefs |
| mysql |
| test |
+--------------------+
4 rows in set (0.01 sec)
5.修改tracker（mogilefsd）配置文件
# vim /etc/mogilefs/mogilefsd.conf
# Enable daemon mode to work in background and use syslog
daemonize = 1
# Where to store the pid of the daemon (must be the same in the init script)
pidfile = /var/run/mogilefsd/mogilefsd.pid
# Database connection information
db_dsn = DBI:mysql:mogilefs:host=192.168.95.30 #配置database的连接信息
db_user = mguser #配置mogilefs的数据库用户密码
db_pass = liaobin
# IP:PORT to listen on for mogilefs client requests
listen = 0.0.0.0:7001 #配置监听的地址和端口
# Optional, if you don't define the port above.
conf_port = 7001
# Number of query workers to start by default.
query_jobs = 10 #生成多少个用于查询的工作进程
# Number of delete workers to start by default.
delete_jobs = 1 #生成多少个用于删除的工作进程
# Number of replicate workers to start by default.
replicate_jobs = 5 #生成多少个用于删除的工作进程
# Number of reaper workers to start by default.
# (you don't usually need to increase this)
reaper_jobs = 1
# Number of fsck workers to start by default.
# (these can cause a lot of load when fsck'ing)
#fsck_jobs = 1
# Minimum amount of space to reserve in megabytes
# default: 100
# Consider setting this to be larger than the largest file you
# would normally be uploading.
#min_free_space = 200
# Number of seconds to wait for a storage node to respond.
# default: 2
# Keep this low, so busy storage nodes are quickly ignored.
#node_timeout = 2
# Number of seconds to wait to connect to a storage node.
# default: 2
# Keep this low so overloaded nodes get skipped.
#conn_timeout = 2
# Allow replication to use the secondary node get port,
# if you have apache or similar configured for GET's
#repl_use_get_port = 1
6.启动mogilefsd服务
[root@localhost mgfs]# /etc/init.d/mogstored start
7.为每个节点新增加分区，作为存储
# fdisk /dev/sda
.........
# partx -a /dev/sda
# mkfs.ext4 /dev/sda5
# mkdir /mgfs
# vim /etc/fstab
#新增如下条目
/dev/sda5 /mgfs ext4 defaults 0 0
# mount -a
# mount
/dev/sda5 on /mgfs type ext4 (rw)
8.三台服务器分别创建存储目录，并修改权限
95.30：
# mkdir /mgfs/dev1
# chown -R /mgfs/dev1
95.34：
# mkdir /mgfs/dev2
# chown -R /mgfs/dev2
95.35：
# mkdir /mgfs/dev3
# chown -R /mgfs/dev3
9.修改store(mogstored)的配置文件
#vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
#注意此处，写到dev(1,2,3)的上级目录就好
docroot = /mgfs
10.启动mogstored服务
# /etc/init.d/mogstored start
11.用mogadm管理
创建存储主机：
# mogadm --trackers=192.168.95.30:7001 host add store1 --ip=192.168.95.30
# mogadm --trackers=192.168.95.30:7001 host add store2 --ip=192.168.95.34
# mogadm --trackers=192.168.95.30:7001 host add store3 --ip=192.168.95.35
查看创建的主机：
# mogadm --trackers=192.168.95.30:7001 host list
创建存储设备：
# mogadm --trackers=192.168.95.30:7001 device add store1 1 --status=alive
# mogadm --trackers=192.168.95.30:7001 device add store2 2 --status=alive
# mogadm --trackers=192.168.95.30:7001 device add store3 3 --status=alive
查看创建的存储设备：
# mogadm --trackers=192.168.95.30:7001 device list
创建域domain：
mogadm --trackers=192.168.95.30:7001 domain add img
mogadm --trackers=192.168.95.30:7001 domain add conf
查看创建的域：
mogadm --trackers=192.168.95.30:7001 domain list
上传文件测试：
mogupload --trackers=192.168.95.30:7001 --domain=img --key="1.jpg" --file="/root/1.jpg"
查看上传的文件：
# mogfileinfo --trackers=192.168.95.30:7001 --domain=img --key="1.jpg"
- file: 1.jpg
class: default
devcount: 2
domain: img
fid: 9
key: 1.jpg
length: 34892
- http://192.168.95.35:7500/dev3/0/000/000/0000000009.fid
- http://192.168.95.30:7500/dev1/0/000/000/0000000009.fid
用fid的url访问测试：
正常访问
nginx反向代理mogilefs
1.安装nginx和mogilefs模块
安装好开发环境：
# yum groupinstall "Development tools" "Server Platform Development" -y
# yum install pcre-devel -y
下载解压nginx包和mogilefs包：
开始编译安装：
#cd nginx-1.6.2
# ./configure \
--prefix=/usr/local/nginx \
--sbin-path=/usr/local/nginx/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/ \
--http-proxy-temp-path=/var/tmp/nginx/proxy/ \
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
--http-scgi-temp-path=/var/tmp/nginx/scgi \
--with-pcre --add-module=../nginx_mogilefs_module-1.0.4
# make && make install
给nginx创建服务脚本
# vim /etc/init.d/nginx
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /etc/nginx/nginx.conf
# config: /etc/sysconfig/nginx
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() {
# make required directories
user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
options=`$nginx -V 2>&1 | grep 'configure arguments:'`
for opt in $options; do
if [ `echo $opt | grep '.*-temp-path'` ]; then
value=`echo $opt | cut -d "=" -f 2`
if [ ! -d "$value" ]; then
# echo "creating" $value
mkdir -p $value && chown -R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
make_dirs
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
configtest || return $?
stop
sleep 1
start
}
reload() {
configtest || return $?
echo -n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac
给予执行权限：
# chmod +x /etc/init.d/nginx
创建nginx用户和必须目录：
# useradd -r nginx
# mkdir /var/tmp/nginx/client -pv
修改nginx配置文件让其支持mogilefs反向代理（红色字体为添加项）：
# vim /etc/nginx/nginx.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream stores {
server 192.168.95.30:7001;
server 192.168.95.34:7001;
server 192.168.95.35:7001;
}
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
location /img/ {
mogilefs_tracker stores;
mogilefs_domain img;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
location /upload/ {
mogilefs_tracker stores;
mogilefs_domain img;
mogilefs_methods PUT GET DELETE;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
#error_page 404 /404.html;
}
用路径访问测试nginx代理：
再上传一个文件测试：
# mogupload --trackers=192.168.95.30:7001 --domain=img --key="2.jpg" --file="/root/2.jpg"
测试删除文件：
# curl -X DELETE http://192.168.95.34/upload/1.jpg
