ceph代码分析之命令处理流程
一、概述:
查看ceph集群的基本状态,ceph -s;查看ceph命令源码(/usr/bin/ceph),这个脚本是由源码里面的ceph.in文件生成而来,哪ceph -s是如何与ceph cluster进行交互的呢?下面开始从代码(py代码)开始入手分析:
从该文件起初的导入部分可以得知:
由send_command()去与ceph cluster通信:
1 def send_command(cluster, target=('mon', ''), cmd=None, inbuf='', timeout=0, 2 verbose=False): 3 """ 注释部分:使用librados的mon_command/osd_command/pg_command给daemon发送指令 4 Send a command to a daemon using librados's 5 mon_command, osd_command, or pg_command. Any bulk input data 6 comes in inbuf. 7 8 Returns (ret, outbuf, outs); ret is the return code, outbuf is 9 the outbl "bulk useful output" buffer, and outs is any status 10 or error message (intended for stderr). 11 12 If target is osd.N, send command to that osd (except for pgid cmds) 13 """ 14 cmd = cmd or [] 15 try: #target[0] 用以区分发送的目标类型,mon、osd、pg、mds的 16 if target[0] == 'osd': 17 osdid = target[1] 18 19 if verbose: 20 print >> sys.stderr, 'submit {0} to osd.{1}'.\ 21 format(cmd, osdid) 22 ret, outbuf, outs = \ 23 cluster.osd_command(osdid, cmd, inbuf, timeout) #osd中消息发送主体 24 25 elif target[0] == 'pg': 26 pgid = target[1] 27 # pgid will already be in the command for the pg <pgid> 28 # form, but for tell <pgid>, we need to put it in 29 if cmd: 30 cmddict = json.loads(cmd[0]) 31 cmddict['pgid'] = pgid 32 else: 33 cmddict = dict(pgid=pgid) 34 cmd = [json.dumps(cmddict)] 35 if verbose: 36 print >> sys.stderr, 'submit {0} for pgid {1}'.\ 37 format(cmd, pgid) 38 ret, outbuf, outs = \ 39 cluster.pg_command(pgid, cmd, inbuf, timeout) 40 41 elif target[0] == 'mon': 42 if verbose: 43 print >> sys.stderr, '{0} to {1}'.\ 44 format(cmd, target[0]) 45 if target[1] == '': 46 ret, outbuf, outs = cluster.mon_command(cmd, inbuf, timeout) 47 else: 48 ret, outbuf, outs = cluster.mon_command(cmd, inbuf, timeout, target[1]) 49 elif target[0] == 'mds': 50 mds_spec = target[1] 51 52 if verbose: 53 print >> sys.stderr, 'submit {0} to mds.{1}'.\ 54 format(cmd, mds_spec) 55 56 try: 57 from cephfs import LibCephFS 58 except ImportError: 59 raise RuntimeError("CephFS unavailable, have you installed libcephfs?") 60 61 filesystem = LibCephFS(cluster.conf_defaults, cluster.conffile) 62 filesystem.conf_parse_argv(cluster.parsed_args) 63 64 filesystem.init() 65 ret, outbuf, outs = \ 66 filesystem.mds_command(mds_spec, cmd, inbuf) 67 filesystem.shutdown() 68 else: 69 raise ArgumentValid("Bad target type '{0}'".format(target[0])) 70 71 except Exception as e: 72 if not isinstance(e, ArgumentError): 73 raise RuntimeError('"{0}": exception {1}'.format(cmd, e)) 74 else: 75 raise 76 77 return ret, outbuf, outs
此时以给mon的指令为例,继续分析:
rados_mon_command_target()调用;
client->mon_command()调用;client为librados::RadosClient;
继续start_mon_command():
之后调用MonClient::_send_command()里面继续_send_mon_message(m),然后调用底层封装的socket发送消息;
接来下来从mon端(即server端)入手,从接收到消息,解包开始:消息的tag为:MSG_MON_COMMAND,消息主体是一个结构体MMonCommand
从mon/Monitor.cc中的handle_command函数入手,查询ceph cluster状态 get_cluster_status()函数:
继续跟get_cluster_status()函数: