MogileFS是一個開源的分布式文件存儲系統，由LiveJournal旗下的Danga Interactive公司開發。Danga團隊開發了包括 Memcached、MogileFS、Perlbal 等多個知名的開源項目。目前使用MogileFS 的公司非常多，如日本排名先前的幾個互聯公司及國內的yupoo(又拍)、digg、豆瓣、1號店、大眾點評、搜狗和安居客等，分別為所在的組織或公司管理著海量的圖片。
MogileFS由3個部分組成：
(1)server：主要包括mogilefsd和mogstored兩個應用程序。mogilefsd實現的是tracker，它通過數據庫來保存元數據信息，包括站點domain、class、host等；mogstored是存儲節點(store node)，它其實是個WebDAV服務，默認監聽在7500端口，接受客戶端的文件存儲請求。在MogileFS安裝完后，要運行mogadm工具將所有的store node注冊到mogilefsd的數據庫里，mogilefsd會對這些節點進行管理和監控。
(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。
(3)客戶端API：MogileFS的客戶端API很多，例如Perl、PHP、Java、Python等，用這個模塊可以編寫客戶端程序，實現文件的備份管理功能等。
現在我們在做一個mogilefs集群分布式存儲,節點設計如下:
192.168.42.150 node1 [Nginx,Tracker, Storage,DataBase]
192.168.42.151 node2 [Tracker, Storage]
192.168.42.152 node3 [Tracker, Storage]
1.準備這三臺機器,初始化工作,時間同步,分別在這三臺機器上安裝mogilefs,我們這里用rpm安裝
cd /usr/local/src #下載RPM安裝包 wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-2.46-2.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Utils-2.19-1.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/perl-MogileFS-Client-1.14-1.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/perl-Perlbal-1.78-1.el6.noarch.rpm #安裝依賴 yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y #安裝rpm yum install *.rpm -y
2.在node1節點上部署mariadb
yum install mariadb-server -y vim /etc/my.cnf.d/server.cnf [mysqld] skip_name_resolve=1 innodb_file_per_table = 1 #啟動mysql,做以下操作 systemctl start mariadb mysql MariaDB [(none)]> GRANT ALL ON *.* TO 'root'@'192.168.42.%' IDENTIFIED BY 'mageedu'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'moguser'@'192.168.42.%' IDENTIFIED BY 'mogpass' WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> CREATE DATABASE mogilefs; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> exit Bye #設定數據庫,生成數據表 mogdbsetup --dbhost=192.168.42.150 --dbrootpass='mageedu' --dbuser='moguser' --dbpass='mogpass' --dbname=mogilefs MariaDB [(none)]> use mogilefs; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [mogilefs]> show tables; +----------------------+ | Tables_in_mogilefs | +----------------------+ | checksum | | class | | device | | domain | | file | | file_on | | file_on_corrupt | | file_to_delete | | file_to_delete2 | | file_to_delete_later | | file_to_queue | | file_to_replicate | | fsck_log | | host | | server_settings | | tempfile | | unreachable_fids | +----------------------+ 17 rows in set (0.00 sec)
3.配置mogilefsd
vim /etc/mogilefs/mogilefsd.conf db_dsn = DBI:mysql:mogilefs:host=192.168.42.150 db_user = moguser db_pass = mogpass listen = 0.0.0.0:7001 conf_port = 7001
4.啟動mogilefsd
service mogilefsd start [root@node1 my.cnf.d]# service mogilefsd start Starting mogilefsd (via systemctl): [ OK ] [root@node1 my.cnf.d]# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:22 *:* LISTEN 0 1024 *:7001 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 50 *:3306 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::*
5.配置storage(可以是所有節點)
mkdir -pv /data/mogilefs/dev1 #創建設備目錄
chown -R mogilefs.mogilefs /data/mogilefs/dev1 #權限
vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data/mogilefs/
修改腳本
lockfile下添加
pidfile=/var/run/mogilefsd/mogstored.pid
start()修改
[ $RETVAL = 0 ] && success && touch ${lockfile} && echo $(pidof mogstored) > ${pidfile} || failure
stop()修改
[ $RETVAL = 0 ] && success && rm -f ${lockfile} ${pidfile} || failure
#啟動mogstored
[root@node1 init.d]# service mogstored start
Starting mogstored (via systemctl): [ OK ]
[root@node1 init.d]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:7500 *:*
LISTEN 0 1024 *:7501 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 1024 *:7001 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 *:3306 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
6.node2,node3上分別創建設備目錄和分發配置文件,腳本文件
node2上操作: mkdir -pv /data/mogilefs/dev2 chown -R mogilefs.mogilefs /data/mogilefs/dev2 node3上操作: mkdir -pv /data/mogilefs/dev3 chown -R mogilefs.mogilefs /data/mogilefs/dev3 #分發腳本和配置文件 scp /etc/rc.d/init.d/mogstored root@192.168.42.151:/etc/rc.d/init.d/ scp /etc/mogilefs/mogstored.conf root@192.168.42.151:/etc/mogilefs/ scp /etc/rc.d/init.d/mogstored root@192.168.42.152:/etc/rc.d/init.d/ scp /etc/mogilefs/mogstored.conf root@192.168.42.152:/etc/mogilefs/ 在node2,和node3啟動mogstored service mogstored start
7.node1節點上需要添加
#查看幫助 mogadm -help [root@node1 init.d]# mogadm check Checking trackers... 127.0.0.1:7001 ... OK Checking hosts... No devices found on tracker(s). 我們還可以使用mogadm --trackers=192.168.42.150:7001 check 得到同樣的結果 #添加主機 mogadm --trackers=192.168.42.150:7001 host add 192.168.42.150 --ip=192.168.42.150 --status=alive mogadm --trackers=192.168.42.150:7001 host add 192.168.42.151 --ip=192.168.42.151 --status=alive mogadm --trackers=192.168.42.150:7001 host add 192.168.42.152 --ip=192.168.42.152 --status=alive [root@node1 init.d]# mogadm host list 192.168.42.150 [1]: alive IP: 192.168.42.150:7500 192.168.42.151 [2]: alive IP: 192.168.42.151:7500 192.168.42.152 [3]: alive IP: 192.168.42.152:7500
添加設備
(1).沒有添加設備之前查看,是沒有結果的
[root@node1 init.d]# mogadm device list
192.168.42.150 [1]: alive
used(G) free(G) total(G) weight(%)
192.168.42.151 [2]: alive
used(G) free(G) total(G) weight(%)
192.168.42.152 [3]: alive
used(G) free(G) total(G) weight(%)
(2)添加設備的操作命令
mogadm device add 192.168.42.150 1 mogadm device add 192.168.42.151 2 mogadm device add 192.168.42.152 3
(3)此時查看有結果了
[root@node1 init.d]# mogadm device list
192.168.42.150 [1]: alive
used(G) free(G) total(G) weight(%)
dev1: alive 1.688 15.298 16.986 100
192.168.42.151 [2]: alive
used(G) free(G) total(G) weight(%)
dev2: alive 1.557 15.430 16.986 100
192.168.42.152 [3]: alive
used(G) free(G) total(G) weight(%)
dev3: alive 1.557 15.430 16.986 100
(4)此時查看mogadm check
[root@node1 init.d]# mogadm check
Checking trackers...
127.0.0.1:7001 ... OK
Checking hosts...
[ 1] 192.168.42.150 ... OK
[ 2] 192.168.42.151 ... OK
[ 3] 192.168.42.152 ... OK
Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%
---- ------------ ---------- ---------- ---------- ------ ---------- -----
[ 1] dev1 16.986 1.689 15.297 9.95% writeable 0.0
[ 2] dev2 16.986 1.557 15.429 9.17% writeable 0.0
[ 3] dev3 16.986 1.557 15.429 9.17% writeable 0.0
---- ------------ ---------- ---------- ---------- ------
total: 50.959 4.803 46.156 9.43%
添加域
(1)查看沒有添加域的狀態 [root@node1 init.d]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- (2)添加域操作 mogadm domain add files mogadm domain add images (3)再次查看列出 [root@node1 init.d]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE
添加class
(1)查看一下class列表 [root@node1 init.d]# mogadm class list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE (2).添加class操作(不知為什么這一步會導致只有一個副本) 需要這樣添加 mogadm class add images jpeg --mindevcount=2 --replpolicy="MultipleHosts(2)" mogadm class add images jpeg mogadm class add images png mogadm class add images gif (3).再次查看列表 [root@node1 init.d]# mogadm class list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE images gif 2 MultipleHosts() NONE images jpeg 2 MultipleHosts() NONE images png 2 MultipleHosts() NONE
上傳圖片測試
(1)上傳圖片
mogupload --tracker=192.168.42.150:7001 --domain=images --key='1.gif' --file='/tmp/83e.gif' --class=gif
mogupload --tracker=192.168.42.150:7001 --domain=images --key='1.jpg' --file='/tmp/zhuoku194.jpg' --class=jpeg
mogupload --tracker=192.168.42.150:7001 --domain=images --key='2.jpg' --file='/tmp/1.jpg' --class=jpeg
(2)查看圖片信息
mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.gif'
[root@node1 init.d]# mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.gif'
- file: /1.gif
class: gif
devcount: 1
domain: images
fid: 2
key: /1.gif
length: 499461
- http://192.168.42.151:7500/dev2/0/000/000/0000000002.fid
mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.jpg'
[root@node1 init.d]# mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.jpg'
- file: /1.jpg
class: jpeg
devcount: 1
domain: images
fid: 3
key: /1.jpg
length: 374408
- http://192.168.42.150:7500/dev1/0/000/000/0000000003.fid
(3)在瀏覽器訪問
http://192.168.42.151:7500/dev2/0/000/000/0000000002.fid
http://192.168.42.150:7500/dev1/0/000/000/0000000003.fid
都是ok的
刪除文件:
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='1.jpg'
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='2.jpg'
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='1.gif'
刪除class:
mogadm class delete images jpeg
mogadm class delete images gitj
mogadm class delete images png
刪除域:
mogadm domain delete files
mogadm domain delete images
#配置Nginx
nginx 的mogilefs模塊的官方文檔地址：
http://www.grid.net.ru/nginx/mogilefs.en.html
wget http://www.grid.net.ru/nginx/download/nginx_mogilefs_module-1.0.4.tar.gz
tar xvf nginx_mogilefs_module-1.0.4.tar.gz
mv nginx_mogilefs_module-1.0.4 /usr/local/
cd /usr/local/src
wget http://192.168.42.26/nginx-1.11.8.tar.gz
tar xvf nginx-1.11.8.tar.gz
cd nginx-1.11.8
yum groupinstall "Development Tools" "Server Platform Developments"
yum install pcre-devel openssl-devel zlib-devel -y
groupadd -r nginx
useradd -r -g nginx nginx
我的編譯安裝方法
./configure --with-pcre --user=nginx --group=nginx --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_realip_module --add-module=/usr/local/nginx_mogilefs_module-1.0.4
make && make install
編譯會可能報錯:
cc1: all warnings being treated as errors
make[1]: *** [objs/addon/nginx_mogilefs_module-1.0.4/ngx_http_mogilefs_module.o] Error 1
make[1]: Leaving directory `/usr/local/src/nginx-1.11.8'
make: *** [build] Error 2
解決辦法:
我們這里打開
vim objs/Makefile
#去掉警告
-Werror
#編輯nginx配置文件
user nginx;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream trackers {
server 192.168.42.150:7001 weight=1;
server 192.168.42.151:7001 weight=1;
server 192.168.42.152:7001 weight=1;
check interval=1000 rise=2 fall=5 timeout=1000;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
location /images/ {
mogilefs_tracker trackers;
mogilefs_domain images;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
瀏覽器訪問:
http://192.168.42.150/images/2.jpg http://192.168.42.150/images/1.gif 都OK
此時我們的trackers其實只有一個,但我們在nginx配置文件中寫了三個,所以我們需要把另外兩個節點也要 安裝tracker
scp /etc/mogilefs/mogilefsd.conf 192.168.42.151:/etc/mogilefs/ scp /etc/mogilefs/mogilefsd.conf 192.168.42.152:/etc/mogilefs/ 啟動mogilefsd:(兩臺節點) service mogilefsd start
三個節點都能查看內容列表
[root@node1 conf]# moglistkeys --trackers=192.168.42.150:7001 --domain=images 1.gif 1.jpg 2.jpg [root@node1 conf]# moglistkeys --trackers=192.168.42.151:7001 --domain=images 1.gif 1.jpg 2.jpg [root@node1 conf]# moglistkeys --trackers=192.168.42.152:7001 --domain=images 1.gif 1.jpg 2.jpg
原創文章，作者：srayban，如若轉載，請注明出處：http://www.www58058.com/78103
不錯，很詳細，干貨。