SDN功能实现（五）---修改OVS源码实现自定义openflow基本字段（延迟更新）
文章推荐：（提前了解背景）
Open vSwitch源码阅读笔记
SDNLAB技术分享（六）：Open vSwitch匹配处理流程和拓展性
ovs流表
OpenvSwitch 流表转换
上面文章不需要全部看懂，了解即可！！
一：功能目的和扩展字段含义
（一）功能目的：实现openflow字段的延迟更新
参考hard_timeout字段：是为了使得openflow流表项在工作一段时间（即hard_timeout）后，自动失效！！！
而我所要实现的功能就是设置一个字段，可以实现微秒级别的，用来控制一条流表项的生效时间！！！
（二）扩展字段：effect_sec和effect_usec含义
这两个字段来源于一个时间戳结构体：struct timeval; 该结构体可以精确时间到微秒级别
struct timeval { time_t tv_sec; suseconds_t tv_usec; };
#include <stdio.h> #include <sys/time.h> int main() { struct timeval tv; gettimeofday(&tv, NULL ); printf("---%d---%d-\n",tv.tv_sec,tv.tv_usec); return 0; }
（三）功能演示
1.启动mininet环境
sudo mn --topo=single,3 --switch=ovsk --controller=none
2.下发普通流表项
sh ovs-ofctl add-flow s1 in_port=1,actions=output:3,2 sh ovs-ofctl add-flow s1 in_port=2,actions=output:1
3.下发自定义字段的流表项
sh ovs-ofctl add-flow s1 in_port=3,effect_sec=1620909655,effect_usec=1811,actions=output:1
下发包含自定义基本字段的流表项之后，使用dpctl dump-flows查看流表项目：
发现我们下发的流表并没有被添加上去（因为还没有到生效时间），所以没有被添加。
此时使用ping命令，发现在没有达到生效时间时，无法ping通，当达到生效时间后，可以ping通！！！
并且查看流表项：发现我们之前添加的流表项也生效了
二：数据结构了解以及扩展数据
（一）ofputil_flow_mod结构体：整合所有版本的openflow消息结构体（不依赖某个版本）--- 解析下发的流表项字符串到ofputil_flow_mod结构体中
在文件openvswitch-2.11.4/include/openvswitch/ofp-flow.h中为ofputil_flow_mod结构体添加自定义字段
struct ofputil_flow_mod { struct ovs_list list_node; /* For queuing flow_mods. */ struct minimatch match; int priority; ovs_be64 cookie; /* Cookie bits to match. */ ovs_be64 cookie_mask; /* 1-bit in each 'cookie' bit to match. */ ovs_be64 new_cookie; /* New cookie to install or UINT64_MAX. */ bool modify_cookie; /* Set cookie of existing flow to 'new_cookie'? */ uint8_t table_id; uint16_t command; uint16_t idle_timeout; uint16_t hard_timeout; uint32_t buffer_id; ofp_port_t out_port; uint32_t out_group; enum ofputil_flow_mod_flags flags; uint16_t importance; /* Eviction precedence. */ struct ofpact *ofpacts; /* Series of "struct ofpact"s. */ size_t ofpacts_len; /* Length of ofpacts, in bytes. */ uint64_t ofpacts_tlv_bitmap; /* 1-bit for each present TLV in 'ofpacts'. */ uint64_t effect_sec; //----------------------------------修改------------- uint64_t effect_usec; //----------------------------------修改------------- };
（二）为所有需要的openflow协议体添加自定义基本字段---用于解析ofputil_flow_mod到对应的版本下的openflow协议
1.在文件openvswitch-2.11.4/include/openflow/openflow-1.1.h中为ofp11_flow_mod结构体添加自定义字段
/* Flow setup and teardown (controller -> datapath). */ struct ofp11_flow_mod { ovs_be64 cookie; /* Opaque controller-issued identifier. */ ovs_be64 cookie_mask; /* Flow actions. */ uint8_t table_id; /* ID of the table to put the flow in */ uint8_t command; /* One of OFPFC_*. */ ovs_be16 idle_timeout; /* Idle time before discarding (seconds). */ ovs_be16 hard_timeout; /* Max time before discarding (seconds). */ ovs_be16 priority; /* Priority level of flow entry. */ ovs_be32 buffer_id; ovs_be32 out_port; ovs_be32 out_group; ovs_be16 flags; /* One of OFPFF_*. */ ovs_be16 importance; ovs_be64 effect_sec; /*---------------------------------修改-------------*/ ovs_be64 effect_usec; /*----------------------------------修改-------------*/ }; OFP_ASSERT(sizeof(struct ofp11_flow_mod) == 56); /*-------必须是8的倍数-------*/
2.在文件openvswitch-2.11.4/include/openflow/openflow-1.0.h中为ofp10_flow_mod结构体添加自定义字段
/* Flow setup and teardown (controller -> datapath). */ struct ofp10_flow_mod { struct ofp10_match match; /* Fields to match */ ovs_be64 cookie; /* Opaque controller-issued identifier. */ ovs_be16 command; /* One of OFPFC_*. */ ovs_be16 idle_timeout; /* Idle time before discarding (seconds). */ ovs_be16 hard_timeout; /* Max time before discarding (seconds). */ ovs_be16 priority; /* Priority level of flow entry. */ ovs_be32 buffer_id; ovs_be16 out_port; ovs_be16 flags; /* One of OFPFF_*. */ ovs_be64 effect_sec; /*---------------------------------修改-------------*/ ovs_be64 effect_usec; /*----------------------------------修改-------------*/ }; OFP_ASSERT(sizeof(struct ofp10_flow_mod) == 80);
（三）为所有需要的openflow stats协议体添加自定义基本字段，响应OFPST_FLOW请求---用于使用dump-flows命令后显示流表项信息
1.在文件openvswitch-2.11.4/include/openflow/nicira-ext.h中为nx_flow_stats结构体添加自定义字段
struct nx_flow_stats { ovs_be16 length; /* Length of this entry. */ uint8_t table_id; /* ID of table flow came from. */ uint8_t pad; ovs_be32 duration_sec; /* Time flow has been alive in seconds. */ ovs_be32 duration_nsec; ovs_be16 priority; /* Priority of the entry. */ ovs_be16 idle_timeout; /* Number of seconds idle before expiration. */ ovs_be16 hard_timeout; /* Number of seconds before expiration. */ ovs_be16 match_len; /* Length of nx_match. */ ovs_be16 idle_age; /* Seconds since last packet, plus one. */ ovs_be16 hard_age; /* Seconds since last modification, plus one. */ ovs_be64 cookie; /* Opaque controller-issued identifier. */ ovs_be64 packet_count; /* Number of packets, UINT64_MAX if unknown. */ ovs_be64 byte_count; /* Number of bytes, UINT64_MAX if unknown. */
ovs_be64 effect_sec; /*---------------------------------修改-------------*/ ovs_be64 effect_usec; /*----------------------------------修改-------------*/ }; OFP_ASSERT(sizeof(struct nx_flow_stats) == 64);
2.在文件openvswitch-2.11.4/include/openflow/openflow-1.0.h中为ofp10_flow_stats结构体添加自定义字段
/* Body of reply to OFPST_FLOW request. */ struct ofp10_flow_stats { ovs_be16 length; /* Length of this entry. */ uint8_t table_id; /* ID of table flow came from. */ uint8_t pad; struct ofp10_match match; /* Description of fields. */ ovs_be32 duration_sec; /* Time flow has been alive in seconds. */ ovs_be32 duration_nsec; ovs_be16 priority; ovs_be16 idle_timeout; /* Number of seconds idle before expiration. */ ovs_be16 hard_timeout; /* Number of seconds before expiration. */ uint8_t pad2[6]; /* Align to 64 bits. */ ovs_32aligned_be64 cookie; /* Opaque controller-issued identifier. */ ovs_32aligned_be64 packet_count; /* Number of packets in flow. */ ovs_32aligned_be64 byte_count; /* Number of bytes in flow. */ ovs_be64 effect_sec; /*---------------------------------修改-------------*/ ovs_be64 effect_usec; /*----------------------------------修改-------------*/ }; OFP_ASSERT(sizeof(struct ofp10_flow_stats) == 104);
3.在文件openvswitch-2.11.4/include/openflow/openflow-1.1.h中为ofp11_flow_stats结构体添加自定义字段
/* Body of reply to OFPST_FLOW request. */ struct ofp11_flow_stats { ovs_be16 length; /* Length of this entry. */ uint8_t table_id; /* ID of table flow came from. */ uint8_t pad; ovs_be32 duration_sec; /* Time flow has been alive in seconds. */ ovs_be32 duration_nsec; ovs_be16 priority; ovs_be16 idle_timeout; /* Number of seconds idle before expiration. */ ovs_be16 hard_timeout; /* Number of seconds before expiration. */ ovs_be16 flags; /* OF 1.3: Set of OFPFF*. */ ovs_be16 importance; /* Eviction precedence (OF1.4+). */ uint8_t pad2[2]; /* Align to 64-bits. */ ovs_be64 cookie; /* Opaque controller-issued identifier. */ ovs_be64 packet_count; /* Number of packets in flow. */ ovs_be64 byte_count; /* Number of bytes in flow. */ ovs_be64 effect_sec; /*---------------------------------修改-------------*/ ovs_be64 effect_usec; /*----------------------------------修改-------------*/ }; OFP_ASSERT(sizeof(struct ofp11_flow_stats) == 64);
4.在文件openvswitch-2.11.4/include/openflow/ofp-flow.h中为ofputil_flow_stats结构体添加自定义字段---重点：这个结构体和前面1、2、3结构体的关系同（一）与（二）的关系
（1）通过ofctl_dump_flows--回调-->vconn_dump_flows（内部传参修改struct ofputil_flow_stats **fsesp）--调用--> ofputil_encode_flow_stats_request去解析各个版本下的openflow协议状态;最后将所有要显示的数据放入ofputil_flow_stats中
（2）再通过ofctl_dump_flows--调用-->ofputil_flow_stats_format（传参struct ofputil_flow_stats ）--调用-->ds_put_format格式化要显示的流表项字符串信息
/* Flow stats reply, independent of protocol. */ struct ofputil_flow_stats { struct match match; ovs_be64 cookie; uint8_t table_id; uint16_t priority; uint16_t idle_timeout; uint16_t hard_timeout; uint32_t duration_sec; uint32_t duration_nsec; int idle_age; /* Seconds since last packet, -1 if unknown. */ int hard_age; /* Seconds since last change, -1 if unknown. */ uint64_t packet_count; /* Packet count, UINT64_MAX if unknown. */ uint64_t byte_count; /* Byte count, UINT64_MAX if unknown. */ const struct ofpact *ofpacts; size_t ofpacts_len; enum ofputil_flow_mod_flags flags; uint16_t importance; /* Eviction precedence. */ uint64_t effect_sec; //----------------------------------修改------------- uint64_t effect_usec; //----------------------------------修改------------- };
（四）添加规则状态（未生效状态）和修改规则结构体---最后会将上面的openflow流表转换为规则插入datapath中进行缓存
在openvswitch/openvswitch-2.11.4/ofproto/ofproto-provider.h文件中
1.添加规则状态
enum OVS_PACKED_ENUM rule_state { RULE_INITIALIZED, RULE_INSERTED, RULE_REMOVED, RULE_EFFECTED, /* ----------rule effect time------------*/ };
2.修改规则结构体
struct rule { struct ofproto *const ofproto; /* The ofproto that contains this rule. */ const struct cls_rule cr; /* In owning ofproto's classifier. */ const uint8_t table_id; /* Index in ofproto's 'tables' array. */ enum rule_state state; ...... /* Timeouts. */ uint16_t hard_timeout OVS_GUARDED; /* In seconds from ->modified. */ uint16_t idle_timeout OVS_GUARDED; /* In seconds from ->used. */ /*------------------- effect time -------------------*/ uint64_t effect_sec OVS_GUARDED; uint64_t effect_usec OVS_GUARDED; const struct ofproto_flow_mod* ofm; /*----------存储部分classifier_insert需要用到的参数---------*/ ...... const struct rule_actions * const actions; ...... struct ovs_list expirable OVS_GUARDED_BY(ofproto_mutex); struct ovs_list effectable OVS_GUARDED_BY(ofproto_mutex); /*-----effect time --------*/ ...... };
3.修改struct ofproto_flow_mod结构体，因为需要保留请求中的数据
/* flow_mod with execution context. */ struct ofproto_flow_mod { /* Allocated by 'init' phase, may be freed after 'start' phase, as these * are not needed for 'revert' nor 'finish'. * * This structure owns a reference to 'temp_rule' (if it is nonnull) that * must be eventually be released with ofproto_rule_unref(). */ struct rule *temp_rule; struct rule_criteria criteria; struct cls_conjunction *conjs; size_t n_conjs; /* Replicate needed fields from ofputil_flow_mod to not need it after the * flow has been created. */ uint16_t command; bool modify_cookie; /* Fields derived from ofputil_flow_mod. */ bool modify_may_add_flow; bool modify_keep_counts; enum nx_flow_update_event event; /* These are only used during commit execution. * ofproto_flow_mod_uninit() does NOT clean these up. */ ovs_version_t version; /* Version in which changes take * effect. */ bool learn_adds_rule; /* Learn execution adds a rule. */ struct rule_collection old_rules; /* Affected rules. */ struct rule_collection new_rules; /* Replacement rules. */ /*--------------存储请求--------------------*/ struct openflow_mod_requester * omr; };
三：控制面实现，解析下发的流表项字符串到ofputil_flow_mod结构体中
流表下发一般是通过以下两种方式:
1.controller通过openflow协议下发FLOW_MOD命令给ovs的Userspace流表。
2.ovs-ofctl通过openflow协议下发FLOW_MOD给ovs的Userspace流表。
ovs-ofctl add-flow最终调用 ofctl_flow_mod(ctx->argc, ctx->argv, OFPFC_ADD);
（一）调用流程
ofctl_flow_mod--调用-->parse_ofp_flow_mod_str--调用-->parse_ofp_str--调用-->parse_ofp_str__实现解析字符串到ofputil_flow_mod结构体中
（二）修改parse_ofp_str__函数，使得解析字符串中的effect_sec和effect_usec字段
static char * OVS_WARN_UNUSED_RESULT parse_ofp_str__(struct ofputil_flow_mod *fm, int command, char *string, //----------------重点：1.字符串到ofputil_flow_mod const struct ofputil_port_map *port_map, const struct ofputil_table_map *table_map, enum ofputil_protocol *usable_protocols) { VLOG_INFO("---------------parse_ofp_str__------------start--------\n"); enum { F_OUT_PORT = 1 << 0, F_ACTIONS = 1 << 1, F_IMPORTANCE = 1 << 2, F_TIMEOUT = 1 << 3, F_PRIORITY = 1 << 4, F_FLAGS = 1 << 5, } fields; char *act_str = NULL; char *name, *value; *usable_protocols = OFPUTIL_P_ANY; if (command == -2) { size_t len; string += strspn(string, " \t\r\n"); /* Skip white space. */ len = strcspn(string, ", \t\r\n"); /* Get length of the first token. */ if (!strncmp(string, "add", len)) { command = OFPFC_ADD; } else if (!strncmp(string, "delete", len)) { command = OFPFC_DELETE; } else if (!strncmp(string, "delete_strict", len)) { command = OFPFC_DELETE_STRICT; } else if (!strncmp(string, "modify", len)) { command = OFPFC_MODIFY; } else if (!strncmp(string, "modify_strict", len)) { command = OFPFC_MODIFY_STRICT; } else { len = 0; command = OFPFC_ADD; } string += len; } switch (command) { case -1: fields = F_OUT_PORT; break; case OFPFC_ADD: fields = F_ACTIONS | F_TIMEOUT | F_PRIORITY | F_FLAGS | F_IMPORTANCE; break; case OFPFC_DELETE: fields = F_OUT_PORT; break; case OFPFC_DELETE_STRICT: fields = F_OUT_PORT | F_PRIORITY; break; case OFPFC_MODIFY: fields = F_ACTIONS | F_TIMEOUT | F_PRIORITY | F_FLAGS; break; case OFPFC_MODIFY_STRICT: fields = F_ACTIONS | F_TIMEOUT | F_PRIORITY | F_FLAGS; break; default: OVS_NOT_REACHED(); } *fm = (struct ofputil_flow_mod) { .priority = OFP_DEFAULT_PRIORITY, .table_id = 0xff, .command = command, .buffer_id = UINT32_MAX, .out_port = OFPP_ANY, .out_group = OFPG_ANY, .effect_sec = 0, .effect_usec = 0, }; VLOG_INFO("---------------parse_ofp_str__----flow_mod idle_timeout:%d--------start--------\n",fm->idle_timeout); VLOG_INFO("---------------parse_ofp_str__----flow_mod effect_time:%d %d--------start--------\n",fm->effect_sec,fm->effect_usec); /* For modify, by default, don't update the cookie. */ if (command == OFPFC_MODIFY || command == OFPFC_MODIFY_STRICT) { fm->new_cookie = OVS_BE64_MAX; } if (fields & F_ACTIONS) { act_str = ofp_extract_actions(string); if (!act_str) { return xstrdup("must specify an action"); } } struct match match = MATCH_CATCHALL_INITIALIZER; while (ofputil_parse_key_value(&string, &name, &value)) { const struct ofp_protocol *p; const struct mf_field *mf; char *error = NULL; if (ofp_parse_protocol(name, &p)) { match_set_dl_type(&match, htons(p->dl_type)); if (p->nw_proto) { match_set_nw_proto(&match, p->nw_proto); } match_set_default_packet_type(&match); } else if (!strcmp(name, "eth")) { match_set_packet_type(&match, htonl(PT_ETH)); } else if (fields & F_FLAGS && !strcmp(name, "send_flow_rem")) { fm->flags |= OFPUTIL_FF_SEND_FLOW_REM; } else if (fields & F_FLAGS && !strcmp(name, "check_overlap")) { fm->flags |= OFPUTIL_FF_CHECK_OVERLAP; } else if (fields & F_FLAGS && !strcmp(name, "reset_counts")) { fm->flags |= OFPUTIL_FF_RESET_COUNTS; *usable_protocols &= OFPUTIL_P_OF12_UP; } else if (fields & F_FLAGS && !strcmp(name, "no_packet_counts")) { fm->flags |= OFPUTIL_FF_NO_PKT_COUNTS; *usable_protocols &= OFPUTIL_P_OF13_UP; } else if (fields & F_FLAGS && !strcmp(name, "no_byte_counts")) { fm->flags |= OFPUTIL_FF_NO_BYT_COUNTS; *usable_protocols &= OFPUTIL_P_OF13_UP; } else if (!strcmp(name, "no_readonly_table") || !strcmp(name, "allow_hidden_fields")) { /* ignore these fields. */ } else if ((mf = mf_from_name(name)) != NULL) { error = ofp_parse_field(mf, value, port_map, &match, usable_protocols); } else if (strchr(name, '[')) { error = parse_subfield(name, value, &match, usable_protocols); } else { if (!*value) { return xasprintf("field %s missing value", name); } if (!strcmp(name, "table")) { if (!ofputil_table_from_string(value, table_map, &fm->table_id)) { return xasprintf("unknown table \"%s\"", value); } if (fm->table_id != 0xff) { *usable_protocols &= OFPUTIL_P_TID; } } else if (fields & F_OUT_PORT && !strcmp(name, "out_port")) { if (!ofputil_port_from_string(value, port_map, &fm->out_port)) { error = xasprintf("%s is not a valid OpenFlow port", value); } } else if (fields & F_OUT_PORT && !strcmp(name, "out_group")) { *usable_protocols &= OFPUTIL_P_OF11_UP; if (!ofputil_group_from_string(value, &fm->out_group)) { error = xasprintf("%s is not a valid OpenFlow group", value); } } else if (fields & F_PRIORITY && !strcmp(name, "priority")) { uint16_t priority = 0; error = str_to_u16(value, name, &priority); fm->priority = priority; } else if (fields & F_TIMEOUT && !strcmp(name, "idle_timeout")) { error = str_to_u16(value, name, &fm->idle_timeout); } else if (fields & F_TIMEOUT && !strcmp(name, "hard_timeout")) { error = str_to_u16(value, name, &fm->hard_timeout); } else if (fields & F_IMPORTANCE && !strcmp(name, "importance")) { error = str_to_u16(value, name, &fm->importance); } else if(!strcmp(name, "effect_sec")){ //-------------------effect_sec--------------- error = str_to_u64(value, &fm->effect_sec); } else if(!strcmp(name, "effect_usec")){ //-----------------effect_usec----------------- error = str_to_u64(value, &fm->effect_usec); } else if (!strcmp(name, "cookie")) { char *mask = strchr(value, '/'); if (mask) { /* A mask means we're searching for a cookie. */ if (command == OFPFC_ADD) { return xstrdup("flow additions cannot use " "a cookie mask"); } *mask = '\0'; error = str_to_be64(value, &fm->cookie); if (error) { return error; } error = str_to_be64(mask + 1, &fm->cookie_mask); /* Matching of the cookie is only supported through NXM or * OF1.1+. */ if (fm->cookie_mask != htonll(0)) { *usable_protocols &= OFPUTIL_P_NXM_OF11_UP; } } else { /* No mask means that the cookie is being set. */ if (command != OFPFC_ADD && command != OFPFC_MODIFY && command != OFPFC_MODIFY_STRICT) { return xstrdup("cannot set cookie"); } error = str_to_be64(value, &fm->new_cookie); fm->modify_cookie = true; } } else if (!strcmp(name, "duration") || !strcmp(name, "n_packets") || !strcmp(name, "n_bytes") || !strcmp(name, "idle_age") || !strcmp(name, "hard_age")) { /* Ignore these, so that users can feed the output of * "ovs-ofctl dump-flows" back into commands that parse * flows. */ } else { error = xasprintf("unknown keyword %s", name); } } if (error) { return error; } } VLOG_INFO("---------------parse_ofp_str__----flow_mod idle_timeout:%d--------start--------\n",fm->idle_timeout); VLOG_INFO("---------------parse_ofp_str__----flow_mod effect_sec:%d--effect_usec:%d------start--------\n",fm->effect_sec,fm->effect_usec); /* Copy ethertype to flow->dl_type for matches on packet_type * (OFPHTN_ETHERTYPE, ethertype). */ if (match.wc.masks.packet_type == OVS_BE32_MAX && pt_ns(match.flow.packet_type) == OFPHTN_ETHERTYPE) { match.flow.dl_type = pt_ns_type_be(match.flow.packet_type); } /* Check for usable protocol interdependencies between match fields. */ if (match.flow.dl_type == htons(ETH_TYPE_IPV6)) { const struct flow_wildcards *wc = &match.wc; /* Only NXM and OXM support matching L3 and L4 fields within IPv6. * * (IPv6 specific fields as well as arp_sha, arp_tha, nw_frag, and * nw_ttl are covered elsewhere so they don't need to be included in * this test too.) */ if (wc->masks.nw_proto || wc->masks.nw_tos || wc->masks.tp_src || wc->masks.tp_dst) { *usable_protocols &= OFPUTIL_P_NXM_OXM_ANY; } } if (!fm->cookie_mask && fm->new_cookie == OVS_BE64_MAX && (command == OFPFC_MODIFY || command == OFPFC_MODIFY_STRICT)) { /* On modifies without a mask, we are supposed to add a flow if * one does not exist. If a cookie wasn't been specified, use a * default of zero. */ fm->new_cookie = htonll(0); } if (fields & F_ACTIONS) { enum ofputil_protocol action_usable_protocols; struct ofpbuf ofpacts; char *error; ofpbuf_init(&ofpacts, 32); struct ofpact_parse_params pp = { .port_map = port_map, .table_map = table_map, .ofpacts = &ofpacts, .usable_protocols = &action_usable_protocols }; error = ofpacts_parse_instructions(act_str, &pp); *usable_protocols &= action_usable_protocols; if (!error) { enum ofperr err; struct ofpact_check_params cp = { .match = &match, .max_ports = OFPP_MAX, .table_id = fm->table_id, .n_tables = 255, }; err = ofpacts_check(ofpacts.data, ofpacts.size, &cp); *usable_protocols &= cp.usable_protocols; if (!err && !*usable_protocols) { err = OFPERR_OFPBAC_MATCH_INCONSISTENT; } if (err) { error = xasprintf("actions are invalid with specified match " "(%s)", ofperr_to_string(err)); } } if (error) { ofpbuf_uninit(&ofpacts); return error; } fm->ofpacts_len = ofpacts.size; fm->ofpacts = ofpbuf_steal_data(&ofpacts); } else { fm->ofpacts_len = 0; fm->ofpacts = NULL; } minimatch_init(&fm->match, &match); return NULL; }
static char * OVS_WARN_UNUSED_RESULT parse_ofp_str__(struct ofputil_flow_mod *fm, int command, char *string, //----------------重点：1.字符串到ofputil_flow_mod const struct ofputil_port_map *port_map, const struct ofputil_table_map *table_map, enum ofputil_protocol *usable_protocols) {
......*fm = (struct ofputil_flow_mod) { .priority = OFP_DEFAULT_PRIORITY, .table_id = 0xff, .command = command, .buffer_id = UINT32_MAX, .out_port = OFPP_ANY, .out_group = OFPG_ANY, .effect_sec = 0, //设置默认值 .effect_usec = 0, };
struct match match = MATCH_CATCHALL_INITIALIZER; while (ofputil_parse_key_value(&string, &name, &value)) { const struct ofp_protocol *p; const struct mf_field *mf; char *error = NULL; if (ofp_parse_protocol(name, &p)) { match_set_dl_type(&match, htons(p->dl_type)); if (p->nw_proto) { match_set_nw_proto(&match, p->nw_proto); } match_set_default_packet_type(&match); } else if (!strcmp(name, "eth")) { match_set_packet_type(&match, htonl(PT_ETH)); } else if (fields & F_FLAGS && !strcmp(name, "send_flow_rem")) { fm->flags |= OFPUTIL_FF_SEND_FLOW_REM; } else if (fields & F_FLAGS && !strcmp(name, "check_overlap")) { fm->flags |= OFPUTIL_FF_CHECK_OVERLAP; } else if (fields & F_FLAGS && !strcmp(name, "reset_counts")) { fm->flags |= OFPUTIL_FF_RESET_COUNTS; *usable_protocols &= OFPUTIL_P_OF12_UP; } else if (fields & F_FLAGS && !strcmp(name, "no_packet_counts")) { fm->flags |= OFPUTIL_FF_NO_PKT_COUNTS; *usable_protocols &= OFPUTIL_P_OF13_UP; } else if (fields & F_FLAGS && !strcmp(name, "no_byte_counts")) { fm->flags |= OFPUTIL_FF_NO_BYT_COUNTS; *usable_protocols &= OFPUTIL_P_OF13_UP; } else if (!strcmp(name, "no_readonly_table") || !strcmp(name, "allow_hidden_fields")) { /* ignore these fields. */ } else if ((mf = mf_from_name(name)) != NULL) { error = ofp_parse_field(mf, value, port_map, &match, usable_protocols); } else if (strchr(name, '[')) { error = parse_subfield(name, value, &match, usable_protocols); } else { if (!*value) { return xasprintf("field %s missing value", name); } if (!strcmp(name, "table")) { if (!ofputil_table_from_string(value, table_map, &fm->table_id)) { return xasprintf("unknown table \"%s\"", value); } if (fm->table_id != 0xff) { *usable_protocols &= OFPUTIL_P_TID; } } else if (fields & F_OUT_PORT && !strcmp(name, "out_port")) { if (!ofputil_port_from_string(value, port_map, &fm->out_port)) { error = xasprintf("%s is not a valid OpenFlow port", value); } } else if (fields & F_OUT_PORT && !strcmp(name, "out_group")) { *usable_protocols &= OFPUTIL_P_OF11_UP; if (!ofputil_group_from_string(value, &fm->out_group)) { error = xasprintf("%s is not a valid OpenFlow group", value); } } else if (fields & F_PRIORITY && !strcmp(name, "priority")) { uint16_t priority = 0; error = str_to_u16(value, name, &priority); fm->priority = priority; } else if (fields & F_TIMEOUT && !strcmp(name, "idle_timeout")) { error = str_to_u16(value, name, &fm->idle_timeout); } else if (fields & F_TIMEOUT && !strcmp(name, "hard_timeout")) { error = str_to_u16(value, name, &fm->hard_timeout); } else if (fields & F_IMPORTANCE && !strcmp(name, "importance")) { error = str_to_u16(value, name, &fm->importance); } else if(!strcmp(name, "effect_sec")){ //-------------------effect_sec--------------- error = str_to_u64(value, &fm->effect_sec); } else if(!strcmp(name, "effect_usec")){ //-----------------effect_usec----------------- error = str_to_u64(value, &fm->effect_usec); } else if (!strcmp(name, "cookie")) { ...... } else if (!strcmp(name, "duration") || !strcmp(name, "n_packets") || !strcmp(name, "n_bytes") || !strcmp(name, "idle_age") || !strcmp(name, "hard_age")) { } else { error = xasprintf("unknown keyword %s", name); } } if (error) { return error; } }
......
处理match和action
...... }
四：控制面实现，转换由三获取ofputil_flow_mod结构体数据到对应的openflow版本中去
（一）调用流程
ofctl_flow_mod--调用-->ofctl_flow_mod__--调用-->ofputil_encode_flow_mod--返回-->struct ofpbuf *msg，其中msg中封装了各个版本的openflow协议信息
msg = ofpraw_alloc(OFPRAW_OFPT11_FLOW_MOD, version, tailroom);
最后在ofctl_flow_mod__中通过下面函数转发到对应的网桥中去（控制面和数据面交互）！！！https://www.cnblogs.com/liuhongru/p/11399046.html
transact_noreply(vconn, ofputil_encode_flow_mod(fm, protocol));
ofctl_flow_mod__会打开一个指向ovs-vswitchd的socket，将ofputil_flow_mod变成openflow的协议，发出去transact_noreply
补充：vconn与OVS网桥的连接
connmgr即connect manager连接管理器，主要完成OVS网桥的连接管理。每一个网桥ofproto都有一个connmgr实体来管理连接。
（二）修改ofputil_encode_flow_mod函数，ofputil_flow_mod结构体数据到对应的openflow版本中去
struct ofpbuf * ofputil_encode_flow_mod(const struct ofputil_flow_mod *fm, //---------------重点：从ofputil_flow_mod中解析ofp11_flow_mod enum ofputil_protocol protocol) { enum ofp_version version = ofputil_protocol_to_ofp_version(protocol); ovs_be16 raw_flags = ofputil_encode_flow_mod_flags(fm->flags, version); struct ofpbuf *msg; struct match match; minimatch_expand(&fm->match, &match); VLOG_INFO("--------------ofputil_encode_flow_mod-------------start--------------"); switch (protocol) { case OFPUTIL_P_OF11_STD: case OFPUTIL_P_OF12_OXM: case OFPUTIL_P_OF13_OXM: case OFPUTIL_P_OF14_OXM: case OFPUTIL_P_OF15_OXM: case OFPUTIL_P_OF16_OXM: { struct ofp11_flow_mod *ofm; //---------------重点：从ofputil_flow_mod中解析ofp11_flow_mod int tailroom; VLOG_INFO("--------------ofputil_encode_flow_mod-------ofp11_flow_mod------start--------------"); tailroom = ofputil_match_typical_len(protocol) + fm->ofpacts_len; msg = ofpraw_alloc(OFPRAW_OFPT11_FLOW_MOD, version, tailroom); ofm = ofpbuf_put_zeros(msg, sizeof *ofm); if ((protocol == OFPUTIL_P_OF11_STD && (fm->command == OFPFC_MODIFY || fm->command == OFPFC_MODIFY_STRICT) && fm->cookie_mask == htonll(0)) || fm->command == OFPFC_ADD) { ofm->cookie = fm->new_cookie; } else { ofm->cookie = fm->cookie & fm->cookie_mask; } ofm->cookie_mask = fm->cookie_mask; if (fm->table_id != OFPTT_ALL || (protocol != OFPUTIL_P_OF11_STD && (fm->command == OFPFC_DELETE || fm->command == OFPFC_DELETE_STRICT))) { ofm->table_id = fm->table_id; } else { ofm->table_id = 0; } ofm->command = fm->command; ofm->idle_timeout = htons(fm->idle_timeout); ofm->hard_timeout = htons(fm->hard_timeout); ofm->effect_sec = htonl(fm->effect_sec); ofm->effect_usec = htonl(fm->effect_usec); ofm->priority = htons(fm->priority); ofm->buffer_id = htonl(fm->buffer_id); ofm->out_port = ofputil_port_to_ofp11(fm->out_port); ofm->out_group = htonl(fm->out_group); ofm->flags = raw_flags; if (version >= OFP14_VERSION && fm->command == OFPFC_ADD) { ofm->importance = htons(fm->importance); } else { ofm->importance = 0; } ofputil_put_ofp11_match(msg, &match, protocol); ofpacts_put_openflow_instructions(fm->ofpacts, fm->ofpacts_len, msg, version); break; } case OFPUTIL_P_OF10_STD: case OFPUTIL_P_OF10_STD_TID: { struct ofp10_flow_mod *ofm; VLOG_INFO("--------------ofputil_encode_flow_mod-------ofp10_flow_mod------start--------------"); msg = ofpraw_alloc(OFPRAW_OFPT10_FLOW_MOD, OFP10_VERSION, fm->ofpacts_len); ofm = ofpbuf_put_zeros(msg, sizeof *ofm); ofputil_match_to_ofp10_match(&match, &ofm->match); ofm->cookie = fm->new_cookie; ofm->command = ofputil_tid_command(fm, protocol); ofm->idle_timeout = htons(fm->idle_timeout); ofm->hard_timeout = htons(fm->hard_timeout); ofm->effect_sec = htonl(fm->effect_sec); ofm->effect_usec = htonl(fm->effect_usec); ofm->priority = htons(fm->priority); ofm->buffer_id = htonl(fm->buffer_id); ofm->out_port = htons(ofp_to_u16(fm->out_port)); ofm->flags = raw_flags; ofpacts_put_openflow_actions(fm->ofpacts, fm->ofpacts_len, msg, version); break; } case OFPUTIL_P_OF10_NXM: case OFPUTIL_P_OF10_NXM_TID: { struct nx_flow_mod *nfm; int match_len; VLOG_INFO("--------------ofputil_encode_flow_mod-------nx_flow_mod------start--------------"); msg = ofpraw_alloc(OFPRAW_NXT_FLOW_MOD, OFP10_VERSION, NXM_TYPICAL_LEN + fm->ofpacts_len); nfm = ofpbuf_put_zeros(msg, sizeof *nfm); nfm->command = ofputil_tid_command(fm, protocol); nfm->cookie = fm->new_cookie; match_len = nx_put_match(msg, &match, fm->cookie, fm->cookie_mask); nfm = msg->msg; nfm->idle_timeout = htons(fm->idle_timeout); nfm->hard_timeout = htons(fm->hard_timeout); nfm->priority = htons(fm->priority); nfm->buffer_id = htonl(fm->buffer_id); nfm->out_port = htons(ofp_to_u16(fm->out_port)); nfm->flags = raw_flags; nfm->match_len = htons(match_len); ofpacts_put_openflow_actions(fm->ofpacts, fm->ofpacts_len, msg, version); break; } default: OVS_NOT_REACHED(); } ofpmsg_update_length(msg); return msg; }
struct ofpbuf * ofputil_encode_flow_mod(const struct ofputil_flow_mod *fm, //---------------重点：从ofputil_flow_mod中解析ofp11_flow_mod enum ofputil_protocol protocol) { enum ofp_version version = ofputil_protocol_to_ofp_version(protocol); ovs_be16 raw_flags = ofputil_encode_flow_mod_flags(fm->flags, version); struct ofpbuf *msg; struct match match; minimatch_expand(&fm->match, &match);switch (protocol) { case OFPUTIL_P_OF11_STD: case OFPUTIL_P_OF12_OXM: case OFPUTIL_P_OF13_OXM: case OFPUTIL_P_OF14_OXM: case OFPUTIL_P_OF15_OXM: case OFPUTIL_P_OF16_OXM: { struct ofp11_flow_mod *ofm; //---------------重点：从ofputil_flow_mod中解析ofp11_flow_mod int tailroom; tailroom = ofputil_match_typical_len(protocol) + fm->ofpacts_len; msg = ofpraw_alloc(OFPRAW_OFPT11_FLOW_MOD, version, tailroom); ofm = ofpbuf_put_zeros(msg, sizeof *ofm); if ((protocol == OFPUTIL_P_OF11_STD && (fm->command == OFPFC_MODIFY || fm->command == OFPFC_MODIFY_STRICT) && fm->cookie_mask == htonll(0)) || fm->command == OFPFC_ADD) { ofm->cookie = fm->new_cookie; } else { ofm->cookie = fm->cookie & fm->cookie_mask; } ofm->cookie_mask = fm->cookie_mask; if (fm->table_id != OFPTT_ALL || (protocol != OFPUTIL_P_OF11_STD && (fm->command == OFPFC_DELETE || fm->command == OFPFC_DELETE_STRICT))) { ofm->table_id = fm->table_id; } else { ofm->table_id = 0; } ofm->command = fm->command; ofm->idle_timeout = htons(fm->idle_timeout); ofm->hard_timeout = htons(fm->hard_timeout); ofm->effect_sec = htonll(fm->effect_sec); ofm->effect_usec = htonll(fm->effect_usec); ofm->priority = htons(fm->priority); ofm->buffer_id = htonl(fm->buffer_id); ofm->out_port = ofputil_port_to_ofp11(fm->out_port); ofm->out_group = htonl(fm->out_group); ofm->flags = raw_flags; if (version >= OFP14_VERSION && fm->command == OFPFC_ADD) { ofm->importance = htons(fm->importance); } else { ofm->importance = 0; } ofputil_put_ofp11_match(msg, &match, protocol); ofpacts_put_openflow_instructions(fm->ofpacts, fm->ofpacts_len, msg, version); break; } case OFPUTIL_P_OF10_STD: case OFPUTIL_P_OF10_STD_TID: { struct ofp10_flow_mod *ofm; VLOG_INFO("--------------ofputil_encode_flow_mod-------ofp10_flow_mod------start--------------"); msg = ofpraw_alloc(OFPRAW_OFPT10_FLOW_MOD, OFP10_VERSION, fm->ofpacts_len); ofm = ofpbuf_put_zeros(msg, sizeof *ofm); ofputil_match_to_ofp10_match(&match, &ofm->match); ofm->cookie = fm->new_cookie; ofm->command = ofputil_tid_command(fm, protocol); ofm->idle_timeout = htons(fm->idle_timeout); ofm->hard_timeout = htons(fm->hard_timeout); ofm->effect_sec = htonll(fm->effect_sec); ofm->effect_usec = htonll(fm->effect_usec); ofm->priority = htons(fm->priority); ofm->buffer_id = htonl(fm->buffer_id); ofm->out_port = htons(ofp_to_u16(fm->out_port)); ofm->flags = raw_flags; ofpacts_put_openflow_actions(fm->ofpacts, fm->ofpacts_len, msg, version); break; } case OFPUTIL_P_OF10_NXM: case OFPUTIL_P_OF10_NXM_TID: { struct nx_flow_mod *nfm; int match_len; VLOG_INFO("--------------ofputil_encode_flow_mod-------nx_flow_mod------start--------------"); msg = ofpraw_alloc(OFPRAW_NXT_FLOW_MOD, OFP10_VERSION, NXM_TYPICAL_LEN + fm->ofpacts_len); nfm = ofpbuf_put_zeros(msg, sizeof *nfm); nfm->command = ofputil_tid_command(fm, protocol); nfm->cookie = fm->new_cookie; match_len = nx_put_match(msg, &match, fm->cookie, fm->cookie_mask); nfm = msg->msg; nfm->idle_timeout = htons(fm->idle_timeout); nfm->hard_timeout = htons(fm->hard_timeout); nfm->priority = htons(fm->priority); nfm->buffer_id = htonl(fm->buffer_id); nfm->out_port = htons(ofp_to_u16(fm->out_port)); nfm->flags = raw_flags; nfm->match_len = htons(match_len); ofpacts_put_openflow_actions(fm->ofpacts, fm->ofpacts_len, msg, version); break; } default: OVS_NOT_REACHED(); } ofpmsg_update_length(msg); return msg; }
五：控制面实现，dump-flows显示流表项
（一）ofctl调用流程
ofctl_dump_flows --调用-->
static void ofctl_dump_flows(struct ovs_cmdl_context *ctx) { ....struct ofputil_flow_stats_request fsr; enum ofputil_protocol protocol; struct vconn *vconn; vconn = prepare_dump_flows(ctx->argc, ctx->argv, false, &fsr, &protocol); struct ofputil_flow_stats *fses; size_t n_fses; run(vconn_dump_flows(vconn, &fsr, protocol, &fses, &n_fses), "dump flows"); if (n_criteria) { qsort(fses, n_fses, sizeof *fses, compare_flows); } struct ds s = DS_EMPTY_INITIALIZER; for (size_t i = 0; i < n_fses; i++) { ds_clear(&s); ofputil_flow_stats_format(&s, &fses[i], ports_to_show(ctx->argv[1]), tables_to_show(ctx->argv[1]), show_stats); printf(" %s\n", ds_cstr(&s)); //打印输出结果字符串 } ds_destroy(&s); for (size_t i = 0; i < n_fses; i++) { free(CONST_CAST(struct ofpact *, fses[i].ofpacts)); } free(fses); vconn_close(vconn); }
分支一：
--调用--> vconn_dump_flows --调用--> ofputil_encode_flow_stats_request获取协议版本对应的请求，然后根据请求去调用recv_flow_stats_reply --调用--> ofputil_decode_flow_stats_reply 获取统一的ofputil_flow_stats结构体，而不是某一个版本的结构体，详细参考二（三）中！！！
分支二：获取分支一之后的状态结构体之后，开始处理转换为字符串！！！
--调用--> ofputil_flow_stats_format转换ofputil_flow_stats结构体数据为字符串
（二）调用流程（二）handle_openflow，处理方法如（一）类似，甚至相同！！！
handle_openflow --调用--> handle_single_part_openflow --调用--> handle_flow_stats_request --调用-->
(1) ofputil_decode_flow_stats_request 获取协议版本对应的请求，然后根据请求去调用recv_flow_stats_reply
(2) ofputil_append_flow_stats_reply 响应请求 获取统一的ofputil_flow_stats结构体
(3) ofconn_send_replies ----> ofconn_send_reply ----> ofconn_send ----> rconn_send ----> rconn_send__ ----> copy_to_monitor ----> vconn_send ----> do_send ----> ofp_to_string会打印出所要显示的字符串 ----> ofp_to_string__
----> ofp_print_flow_stats_reply ----> ofputil_decode_flow_stats_reply
（三）修改ofputil_flow_stats_format获取要显示的字符串
void
ofputil_flow_stats_format(struct ds *string,
const struct ofputil_flow_stats *fs,
const struct ofputil_port_map *port_map,
const struct ofputil_table_map *table_map,
bool show_stats)
{
if (show_stats || fs->cookie) {
ds_put_format(string, "%scookie=%s0x%"PRIx64", ",
colors.param, colors.end, ntohll(fs->cookie));
}
if (show_stats) {
ds_put_format(string, "%sduration=%s", colors.param, colors.end);
ofp_print_duration(string, fs->duration_sec, fs->duration_nsec);
ds_put_cstr(string, ", ");
}
if (show_stats || fs->table_id
|| ofputil_table_map_get_name(table_map, fs->table_id) != NULL) {
ds_put_format(string, "%stable=%s", colors.special, colors.end);
ofputil_format_table(fs->table_id, table_map, string);
ds_put_cstr(string, ", ");
}
if (show_stats) {
print_flow_stat(string, "n_packets", fs->packet_count);
print_flow_stat(string, "n_bytes", fs->byte_count);
}
if (fs->idle_timeout != OFP_FLOW_PERMANENT) {
ds_put_format(string, "%sidle_timeout=%s%"PRIu16", ",
colors.param, colors.end, fs->idle_timeout);
}
if (fs->hard_timeout != OFP_FLOW_PERMANENT) {
ds_put_format(string, "%shard_timeout=%s%"PRIu16", ",
colors.param, colors.end, fs->hard_timeout);
}
if (fs->effect_sec != 0) {
ds_put_format(string, "%seffect_sec=%s%"PRIu64", ",
colors.param, colors.end, fs->effect_sec);
}
if (fs->effect_usec != 0) {
ds_put_format(string, "%seffect_usec=%s%"PRIu64", ",
colors.param, colors.end, fs->effect_usec);
}
if (fs->flags) {
ofputil_flow_mod_flags_format(string, fs->flags);
}
if (fs->importance != 0) {
ds_put_format(string, "%simportance=%s%"PRIu16", ",
colors.param, colors.end, fs->importance);
}
if (show_stats && fs->idle_age >= 0) {
ds_put_format(string, "%sidle_age=%s%d, ",
colors.param, colors.end, fs->idle_age);
}
if (show_stats && fs->hard_age >= 0 && fs->hard_age != fs->duration_sec) {
ds_put_format(string, "%shard_age=%s%d, ",
colors.param, colors.end, fs->hard_age);
}
/* Print the match, followed by a space (but omit the space if the match
* was an empty string). */
size_t length = string->length;
match_format(&fs->match, port_map, string, fs->priority);
if (string->length != length) {
ds_put_char(string, ' ');
}
ds_put_format(string, "%sactions=%s", colors.actions, colors.end);
struct ofpact_format_params fp = {
.port_map = port_map,
.table_map = table_map,
.s = string,
};
ofpacts_format(fs->ofpacts, fs->ofpacts_len, &fp);
}
（四）修改ofputil_decode_flow_stats_reply获取字符串
int ofputil_decode_flow_stats_reply(struct ofputil_flow_stats *fs, //对比ofputil_decode_flow_mod函数，是将ofputil_flow_mod转openflow struct ofpbuf *msg, bool flow_age_extension, struct ofpbuf *ofpacts) { const struct ofp_header *oh; size_t instructions_len; enum ofperr error; enum ofpraw raw; error = (msg->header ? ofpraw_decode(&raw, msg->header) : ofpraw_pull(&raw, msg)); if (error) { return error; } oh = msg->header; if (!msg->size) { return EOF; } else if (raw == OFPRAW_OFPST15_FLOW_REPLY) { VLOG_INFO("------------ofputil_decode_flow_stats_reply------OFPRAW_OFPST15_FLOW_REPLY------"); const struct ofp15_flow_desc *ofd; size_t length; uint16_t padded_match_len; uint16_t stat_len; uint8_t oxs_field_set; ofd = ofpbuf_try_pull(msg, sizeof *ofd); if (!ofd) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply has %" PRIu32 " leftover " "bytes at end", msg->size); return EINVAL; } length = ntohs(ofd->length); if (length < sizeof *ofd) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply claims invalid " "length %" PRIuSIZE, length); return EINVAL; } if (ofputil_pull_ofp11_match(msg, NULL, NULL, &fs->match, &padded_match_len)) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply bad match"); return EINVAL; } fs->priority = ntohs(ofd->priority); fs->table_id = ofd->table_id; fs->cookie = ofd->cookie; fs->idle_timeout = ntohs(ofd->idle_timeout); fs->hard_timeout = ntohs(ofd->hard_timeout); fs->importance = ntohs(ofd->importance); error = ofputil_decode_flow_mod_flags(ofd->flags, -1, oh->version, &fs->flags); if (error) { return error; } struct oxs_stats oxs; if (oxs_pull_stat(msg, &oxs, &stat_len, &oxs_field_set)) { VLOG_WARN_RL(&rl, "OXS OFPST_FLOW reply bad stats"); return EINVAL; } fs->duration_sec = oxs.duration_sec; fs->duration_nsec = oxs.duration_nsec; fs->packet_count = oxs.packet_count; fs->byte_count = oxs.byte_count; fs->idle_age = oxs.idle_age == UINT32_MAX ? -1 : oxs.idle_age; fs->hard_age = -1; instructions_len = length - sizeof *ofd - padded_match_len - stat_len; } else if (raw == OFPRAW_OFPST11_FLOW_REPLY || raw == OFPRAW_OFPST13_FLOW_REPLY) { VLOG_INFO("------------ofputil_decode_flow_stats_reply------ofp11_flow_stats------"); const struct ofp11_flow_stats *ofs; size_t length; uint16_t padded_match_len; ofs = ofpbuf_try_pull(msg, sizeof *ofs); if (!ofs) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply has %"PRIu32" leftover " "bytes at end", msg->size); return OFPERR_OFPBRC_BAD_LEN; } length = ntohs(ofs->length); if (length < sizeof *ofs) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply claims invalid " "length %"PRIuSIZE, length); return OFPERR_OFPBRC_BAD_LEN; } error = ofputil_pull_ofp11_match(msg, NULL, NULL, &fs->match, &padded_match_len); if (error) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply bad match"); return error; } instructions_len = length - sizeof *ofs - padded_match_len; fs->priority = ntohs(ofs->priority); fs->table_id = ofs->table_id; fs->duration_sec = ntohl(ofs->duration_sec); fs->duration_nsec = ntohl(ofs->duration_nsec); fs->idle_timeout = ntohs(ofs->idle_timeout); fs->hard_timeout = ntohs(ofs->hard_timeout); fs->effect_sec = ntohll(ofs->effect_sec); //处理------effect fs->effect_usec = ntohll(ofs->effect_usec); if (oh->version >= OFP14_VERSION) { fs->importance = ntohs(ofs->importance); } else { fs->importance = 0; } if (raw == OFPRAW_OFPST13_FLOW_REPLY) { error = ofputil_decode_flow_mod_flags(ofs->flags, -1, oh->version, &fs->flags); if (error) { return error; } } else { fs->flags = 0; } fs->idle_age = -1; fs->hard_age = -1; fs->cookie = ofs->cookie; fs->packet_count = ntohll(ofs->packet_count); fs->byte_count = ntohll(ofs->byte_count); } else if (raw == OFPRAW_OFPST10_FLOW_REPLY) { const struct ofp10_flow_stats *ofs; size_t length; VLOG_INFO("------------ofputil_decode_flow_stats_reply------ofp10_flow_stats------"); ofs = ofpbuf_try_pull(msg, sizeof *ofs); if (!ofs) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply has %"PRIu32" leftover " "bytes at end", msg->size); return OFPERR_OFPBRC_BAD_LEN; } length = ntohs(ofs->length); if (length < sizeof *ofs) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply claims invalid " "length %"PRIuSIZE, length); return OFPERR_OFPBRC_BAD_LEN; } instructions_len = length - sizeof *ofs; fs->cookie = get_32aligned_be64(&ofs->cookie); ofputil_match_from_ofp10_match(&ofs->match, &fs->match); fs->priority = ntohs(ofs->priority); fs->table_id = ofs->table_id; fs->duration_sec = ntohl(ofs->duration_sec); fs->duration_nsec = ntohl(ofs->duration_nsec); fs->idle_timeout = ntohs(ofs->idle_timeout); fs->hard_timeout = ntohs(ofs->hard_timeout); fs->effect_sec = ntohll(ofs->effect_sec); //处理------effect fs->effect_usec = ntohll(ofs->effect_usec); fs->importance = 0; fs->idle_age = -1; fs->hard_age = -1; fs->packet_count = ntohll(get_32aligned_be64(&ofs->packet_count)); fs->byte_count = ntohll(get_32aligned_be64(&ofs->byte_count)); fs->flags = 0; } else if (raw == OFPRAW_NXST_FLOW_REPLY) { const struct nx_flow_stats *nfs; size_t match_len, length; VLOG_INFO("------------ofputil_decode_flow_stats_reply------nx_flow_stats------"); nfs = ofpbuf_try_pull(msg, sizeof *nfs); if (!nfs) { VLOG_WARN_RL(&rl, "NXST_FLOW reply has %"PRIu32" leftover " "bytes at end", msg->size); return OFPERR_OFPBRC_BAD_LEN; } length = ntohs(nfs->length); match_len = ntohs(nfs->match_len); if (length < sizeof *nfs + ROUND_UP(match_len, 8)) { VLOG_WARN_RL(&rl, "NXST_FLOW reply with match_len=%"PRIuSIZE" " "claims invalid length %"PRIuSIZE, match_len, length); return OFPERR_OFPBRC_BAD_LEN; } error = nx_pull_match(msg, match_len, &fs->match, NULL, NULL, false, NULL, NULL); if (error) { return error; } instructions_len = length - sizeof *nfs - ROUND_UP(match_len, 8); fs->cookie = nfs->cookie; fs->table_id = nfs->table_id; fs->duration_sec = ntohl(nfs->duration_sec); fs->duration_nsec = ntohl(nfs->duration_nsec); fs->priority = ntohs(nfs->priority); fs->idle_timeout = ntohs(nfs->idle_timeout); fs->hard_timeout = ntohs(nfs->hard_timeout); fs->effect_sec = ntohl(nfs->effect_sec); //处理------effect fs->effect_usec = ntohl(nfs->effect_usec); VLOG_INFO("-----tt----ofputil_decode_flow_stats_reply----%d %d------------",fs->effect_sec,fs->effect_usec); fs->importance = 0; fs->idle_age = -1; fs->hard_age = -1; if (flow_age_extension) { if (nfs->idle_age) { fs->idle_age = ntohs(nfs->idle_age) - 1; } if (nfs->hard_age) { fs->hard_age = ntohs(nfs->hard_age) - 1; } } fs->packet_count = ntohll(nfs->packet_count); fs->byte_count = ntohll(nfs->byte_count); fs->flags = 0; } else { OVS_NOT_REACHED(); } error = ofpacts_pull_openflow_instructions(msg, instructions_len, oh->version, NULL, NULL, ofpacts); if (error) { VLOG_WARN_RL(&rl, "OFPST_FLOW reply bad instructions"); return error; } fs->ofpacts = ofpacts->data; fs->ofpacts_len = ofpacts->size; return 0; }
（五）修改ofputil_append_flow_stats_reply获取字符串
void ofputil_append_flow_stats_reply(const struct ofputil_flow_stats *fs, //----------对应ofputil_decode_flow_stats_reply函数，相反赋值 struct ovs_list *replies, const struct tun_table *tun_table) { struct ofputil_flow_stats *fs_ = CONST_CAST(struct ofputil_flow_stats *, fs); const struct tun_table *orig_tun_table; struct ofpbuf *reply = ofpbuf_from_list(ovs_list_back(replies)); size_t start_ofs = reply->size; enum ofp_version version = ofpmp_version(replies); enum ofpraw raw = ofpmp_decode_raw(replies); orig_tun_table = fs->match.flow.tunnel.metadata.tab; fs_->match.flow.tunnel.metadata.tab = tun_table; VLOG_INFO("------------ofputil_append_flow_stats_reply-----start-----%d-----%d----",fs->effect_sec,fs->effect_usec); if (raw == OFPRAW_OFPST15_FLOW_REPLY) { VLOG_INFO("------------ofputil_append_flow_stats_reply-----OFPRAW_OFPST15_FLOW_REPLY----"); struct ofp15_flow_desc *ofd; ofpbuf_put_uninit(reply, sizeof *ofd); oxm_put_match(reply, &fs->match, version); struct oxs_stats oxs = { .duration_sec = fs->duration_sec, .duration_nsec = fs->duration_nsec, .idle_age = fs->idle_age >= 0 ? fs->idle_age : UINT32_MAX, .packet_count = fs->packet_count, .byte_count = fs->byte_count, .flow_count = UINT32_MAX, }; oxs_put_stats(reply, &oxs); ofpacts_put_openflow_instructions(fs->ofpacts, fs->ofpacts_len, reply, version); ofd = ofpbuf_at_assert(reply, start_ofs, sizeof *ofd); ofd->length = htons(reply->size - start_ofs); ofd->table_id = fs->table_id; ofd->priority = htons(fs->priority); ofd->idle_timeout = htons(fs->idle_timeout); ofd->hard_timeout = htons(fs->hard_timeout); ofd->cookie = fs->cookie; memset(ofd->pad2, 0, sizeof ofd->pad2); ofd->pad = 0; ofd->importance = htons(fs->importance); ofd->flags = ofputil_encode_flow_mod_flags(fs->flags, version); } else if (raw == OFPRAW_OFPST11_FLOW_REPLY || raw == OFPRAW_OFPST13_FLOW_REPLY) { //处理------effect struct ofp11_flow_stats *ofs; VLOG_INFO("------------ofputil_append_flow_stats_reply-----OFPRAW_OFPST15_FLOW_REPLY----"); ofpbuf_put_uninit(reply, sizeof *ofs); oxm_put_match(reply, &fs->match, version); ofpacts_put_openflow_instructions(fs->ofpacts, fs->ofpacts_len, reply, version); ofs = ofpbuf_at_assert(reply, start_ofs, sizeof *ofs); ofs->length = htons(reply->size - start_ofs); ofs->table_id = fs->table_id; ofs->pad = 0; ofs->duration_sec = htonl(fs->duration_sec); ofs->duration_nsec = htonl(fs->duration_nsec); ofs->priority = htons(fs->priority); ofs->idle_timeout = htons(fs->idle_timeout); ofs->hard_timeout = htons(fs->hard_timeout); ofs->effect_sec = htonll(fs->effect_sec); //处理------effect ofs->effect_usec = htonll(fs->effect_usec); if (version >= OFP14_VERSION) { ofs->importance = htons(fs->importance); } else { ofs->importance = 0; } if (raw == OFPRAW_OFPST13_FLOW_REPLY) { ofs->flags = ofputil_encode_flow_mod_flags(fs->flags, version); } else { ofs->flags = 0; } memset(ofs->pad2, 0, sizeof ofs->pad2); ofs->cookie = fs->cookie; ofs->packet_count = htonll(unknown_to_zero(fs->packet_count)); ofs->byte_count = htonll(unknown_to_zero(fs->byte_count)); } else if (raw == OFPRAW_OFPST10_FLOW_REPLY) { struct ofp10_flow_stats *ofs; VLOG_INFO("------------ofputil_append_flow_stats_reply-----OFPRAW_OFPST10_FLOW_REPLY----"); ofpbuf_put_uninit(reply, sizeof *ofs); ofpacts_put_openflow_actions(fs->ofpacts, fs->ofpacts_len, reply, version); ofs = ofpbuf_at_assert(reply, start_ofs, sizeof *ofs); ofs->length = htons(reply->size - start_ofs); ofs->table_id = fs->table_id; ofs->pad = 0; ofputil_match_to_ofp10_match(&fs->match, &ofs->match); ofs->duration_sec = htonl(fs->duration_sec); ofs->duration_nsec = htonl(fs->duration_nsec); ofs->priority = htons(fs->priority); ofs->idle_timeout = htons(fs->idle_timeout); ofs->hard_timeout = htons(fs->hard_timeout); ofs->effect_sec = htonll(fs->effect_sec); ofs->effect_usec = htonll(fs->effect_usec); memset(ofs->pad2, 0, sizeof ofs->pad2); put_32aligned_be64(&ofs->cookie, fs->cookie); put_32aligned_be64(&ofs->packet_count, htonll(unknown_to_zero(fs->packet_count))); put_32aligned_be64(&ofs->byte_count, htonll(unknown_to_zero(fs->byte_count))); } else if (raw == OFPRAW_NXST_FLOW_REPLY) { struct nx_flow_stats *nfs; int match_len; VLOG_INFO("------------ofputil_append_flow_stats_reply-----OFPRAW_NXST_FLOW_REPLY----"); ofpbuf_put_uninit(reply, sizeof *nfs); match_len = nx_put_match(reply, &fs->match, 0, 0); ofpacts_put_openflow_actions(fs->ofpacts, fs->ofpacts_len, reply, version); nfs = ofpbuf_at_assert(reply, start_ofs, sizeof *nfs); nfs->length = htons(reply->size - start_ofs); nfs->table_id = fs->table_id; nfs->pad = 0; nfs->duration_sec = htonl(fs->duration_sec); nfs->duration_nsec = htonl(fs->duration_nsec); nfs->priority = htons(fs->priority); nfs->idle_timeout = htons(fs->idle_timeout); nfs->hard_timeout = htons(fs->hard_timeout); nfs->effect_sec = htonll(fs->effect_sec); nfs->effect_usec = htonll(fs->effect_usec); nfs->idle_age = htons(fs->idle_age < 0 ? 0 : fs->idle_age < UINT16_MAX ? fs->idle_age + 1 : UINT16_MAX); nfs->hard_age = htons(fs->hard_age < 0 ? 0 : fs->hard_age < UINT16_MAX ? fs->hard_age + 1 : UINT16_MAX); nfs->match_len = htons(match_len); nfs->cookie = fs->cookie; nfs->packet_count = htonll(fs->packet_count); nfs->byte_count = htonll(fs->byte_count); } else { VLOG_INFO("------------ofputil_append_flow_stats_reply-----OVS_NOT_REACHED----"); OVS_NOT_REACHED(); } ofpmp_postappend(replies, start_ofs); fs_->match.flow.tunnel.metadata.tab = orig_tun_table; }
六：ovs-vswitchd的启动分析（见链接），是七的铺垫
其中run方法后面会用到的！！！
七：数据面处理来自控制面（由四转发而来）的openflow协议（重点）
（一）调用流程
Ovs-vswitchd会监听socket，在ovs-vswitchd.c中
bridge_run --调用--> bridge_run__ --调用--> ofproto_run --调用--> connmgr_run(p->connmgr, handle_openflow); --调用--> ofconn_run(ofconn, handle_openflow) --调用--> handle_openflow(ofconn, &msgs);
当交换机有openflow调用的时候，handle_openflow会被调用去处理连接!!!!
handle_openflow--调用-->handle_flow_mod
分支（一）：
--调用--> ofputil_decode_flow_mod解析各个版本的openflow协议到ofputil_flow_mod结构体中（包括基本字段、条件字段、action）
分支（二）：调用分支一之后，所有信息都解析完成，存放在ofputil_flow_mod fm中，下面开始通过调用分支二下发到数据平面（datapath）
--调用--> handle_flow_mod__ --调用-->
(1) ofproto_flow_mod_init将规则初始化赋值给了ofm中的temp_rule --调用--> add_flow_init --调用--> ofproto_rule_create创建并初始化规则 --回调--> rule_construct初始化struct rule_dpif（内部保护规则rule）
(2) ofproto_flow_mod_start --调用--> add_flow_start 开始添加流表项 --调用--> replace_rule_start 开始替换规则
(3) ofproto_flow_mod_finish --调用--> add_flow_finish --调用--> replace_rule_finish 替换规则完成
(4) ofmonitor_flush 通知底层内核更新流表项
（二）修改ofproto_rule_create创建并初始化规则，为规则添加我们自定义的字段
1.handle_flow_mod__方法，保存后面需要的openflow_mod_requester请求数据到ofputil_flow_mod* fm中自定义字段中去
static enum ofperr handle_flow_mod__(struct ofproto *ofproto, const struct ofputil_flow_mod *fm, //协议公共数据结构，前面handle_flow_mod方法就是为了获取他 const struct openflow_mod_requester *req) OVS_EXCLUDED(ofproto_mutex) //声明锁 { struct ofproto_flow_mod ofm; //具有执行上下文的流模式 enum ofperr error; error = ofproto_flow_mod_init(ofproto, &ofm, fm, NULL); //将规则赋值给了ofm中的temp_rules if (error) { return error; } ovs_mutex_lock(&ofproto_mutex); //加锁----------加解锁中间才是重要的东西 ofm.version = ofproto->tables_version + 1; ofm.omr = req; //将规则ofm中的temp_rules中，旧的放入old_rules;新的放入new_rules;中去 error = ofproto_flow_mod_start(ofproto, &ofm); //处理规则？？？----将规则加入ovs list if (!error) { VLOG_INFO("--------------------note ovs datapath----------start----"); ofproto_bump_tables_version(ofproto); //判断版本，不用管 ofproto_flow_mod_finish(ofproto, &ofm, req); //这里处理了 //下面才是重點----------- ofmonitor_flush(ofproto->connmgr); //ofproto->connmgr是OpenFlow交换机的连接管理器 VLOG_INFO("--------------------note ovs datapath------------end--"); } ovs_mutex_unlock(&ofproto_mutex); //解锁 return error; }
1.add_flow_init方法，fm参数中已经在分支一中赋值了
static enum ofperr add_flow_init(struct ofproto *ofproto, struct ofproto_flow_mod *ofm, const struct ofputil_flow_mod *fm) //ofputil_flow_mod中还是包含了openflow协议的所有信息的 OVS_EXCLUDED(ofproto_mutex) { struct oftable *table; struct cls_rule cr; uint8_t table_id; enum ofperr error; if (!check_table_id(ofproto, fm->table_id)) { return OFPERR_OFPBRC_BAD_TABLE_ID; } /* Pick table. */ //选择表id if (fm->table_id == 0xff) { if (ofproto->ofproto_class->rule_choose_table) { error = ofproto->ofproto_class->rule_choose_table(ofproto, &fm->match, &table_id); if (error) { return error; } ovs_assert(table_id < ofproto->n_tables); } else { table_id = 0; } } else if (fm->table_id < ofproto->n_tables) { table_id = fm->table_id; } else { return OFPERR_OFPBRC_BAD_TABLE_ID; } table = &ofproto->tables[table_id]; //------------选择对应的openflow表 if (table->flags & OFTABLE_READONLY && !(fm->flags & OFPUTIL_FF_NO_READONLY)) { //只读表，不允许添加，返回错误 return OFPERR_OFPBRC_EPERM; } if (!(fm->flags & OFPUTIL_FF_HIDDEN_FIELDS) && !minimatch_has_default_hidden_fields(&fm->match)) { VLOG_WARN_RL(&rl, "%s: (add_flow) only internal flows can set " "non-default values to hidden fields", ofproto->name); return OFPERR_OFPBRC_EPERM; } if (!ofm->temp_rule) { //如果上下文中没有临时规则（新添加），则下面进行初始化 cls_rule_init_from_minimatch(&cr, &fm->match, fm->priority); //初始化对象，增加match到rule中 /* Allocate new rule. Destroys 'cr'. */ //分配新规则 //返回指示“flow”中存在哪些隧道元数据字段的位图 uint64_t map = miniflow_get_tun_metadata_present_map(fm->match.flow); error = ofproto_rule_create(ofproto, &cr, table - ofproto->tables, //-----重点，将新规则结合到ofproto中去 fm->new_cookie, fm->idle_timeout, //回调rule_construct()初始化rule_dpif的参数 fm->hard_timeout, fm->effect_sec, fm->effect_usec,fm->flags, fm->importance, fm->ofpacts, fm->ofpacts_len, map, fm->ofpacts_tlv_bitmap, &ofm->temp_rule); //所有规则数据返回到&ofm->temp_rule中！！！-------重点！！！ if (error) { return error; } get_conjunctions(fm, &ofm->conjs, &ofm->n_conjs); //重點---------------????? } return 0; }
2.ofproto_rule_create方法
static enum ofperr ofproto_rule_create(struct ofproto *ofproto, struct cls_rule *cr, uint8_t table_id, ovs_be64 new_cookie, uint16_t idle_timeout, uint16_t hard_timeout, uint64_t effect_sec,uint64_t effect_usec, enum ofputil_flow_mod_flags flags, uint16_t importance, const struct ofpact *ofpacts, size_t ofpacts_len, uint64_t match_tlv_bitmap, uint64_t ofpacts_tlv_bitmap, struct rule **new_rule) OVS_NO_THREAD_SAFETY_ANALYSIS { struct rule *rule; enum ofperr error; /* Allocate new rule. */ rule = ofproto->ofproto_class->rule_alloc(); //回调函数rule_alloc()分配rule_dpif和rule对象 if (!rule) { cls_rule_destroy(cr); VLOG_WARN_RL(&rl, "%s: failed to allocate a rule.", ofproto->name); return OFPERR_OFPFMFC_UNKNOWN; } /* Initialize base state. */ //接下来初始化rule对象中的相关参数…… *CONST_CAST(struct ofproto **, &rule->ofproto) = ofproto; cls_rule_move(CONST_CAST(struct cls_rule *, &rule->cr), cr); ovs_refcount_init(&rule->ref_count); ovs_mutex_init(&rule->mutex); ovs_mutex_lock(&rule->mutex); *CONST_CAST(ovs_be64 *, &rule->flow_cookie) = new_cookie; rule->created = rule->modified = time_msec(); rule->idle_timeout = idle_timeout; rule->hard_timeout = hard_timeout; rule->effect_sec = effect_sec; rule->effect_usec = effect_usec; *CONST_CAST(uint16_t *, &rule->importance) = importance; rule->removed_reason = OVS_OFPRR_NONE; *CONST_CAST(uint8_t *, &rule->table_id) = table_id; rule->flags = flags & OFPUTIL_FF_STATE; *CONST_CAST(const struct rule_actions **, &rule->actions) = rule_actions_create(ofpacts, ofpacts_len); //以上规则初始 //下面处理ovs_list ovs_list_init(&rule->meter_list_node); rule->eviction_group = NULL; rule->monitor_flags = 0; rule->add_seqno = 0; rule->modify_seqno = 0; ovs_list_init(&rule->expirable); ovs_list_init(&rule->effectable); //------修改effectable---------- ovs_mutex_unlock(&rule->mutex); /* Construct rule, initializing derived state. */ error = ofproto->ofproto_class->rule_construct(rule); //回调rule_construct()初始化rule_dpif的参数 if (error) { ofproto_rule_destroy__(rule); return error; } rule->state = RULE_INITIALIZED; rule->match_tlv_bitmap = match_tlv_bitmap; rule->ofpacts_tlv_bitmap = ofpacts_tlv_bitmap; mf_vl_mff_ref(&rule->ofproto->vl_mff_map, match_tlv_bitmap); mf_vl_mff_ref(&rule->ofproto->vl_mff_map, ofpacts_tlv_bitmap); *new_rule = rule; return 0; }
其中的rule->effectable链中存放的是所有还没有生效的规则，会在后面run方法中遍历和检测，使之生效！！！
（三）修改replace_rule_start，开始替换规则
1.add_flow_start
/* ofm->temp_rule is consumed only in the successful case. */ //开始使用规则了！！！ static enum ofperr add_flow_start(struct ofproto *ofproto, struct ofproto_flow_mod *ofm) OVS_REQUIRES(ofproto_mutex) { struct rule *old_rule = NULL; struct rule *new_rule = ofm->temp_rule; //------------使用 const struct rule_actions *actions = rule_get_actions(new_rule); //获取规则actions,返回一个rule_actions指针 struct oftable *table = &ofproto->tables[new_rule->table_id]; //获取对应的table enum ofperr error; /* Must check actions while holding ofproto_mutex to avoid a race. */ error = ofproto_check_ofpacts(ofproto, actions->ofpacts, //检查锁，避免竞争 actions->ofpacts_len); if (error) { return error; } /* Check for the existence of an identical rule. * This will not return rules earlier marked for removal. */ //检查是否存在相同的规则（旧的） old_rule = rule_from_cls_rule(classifier_find_rule_exactly(&table->cls, &new_rule->cr, ofm->version)); if (!old_rule) { /* Check for overlap, if requested. 检查重叠*/ if (new_rule->flags & OFPUTIL_FF_CHECK_OVERLAP && classifier_rule_overlaps(&table->cls, &new_rule->cr, ofm->version)) { return OFPERR_OFPFMFC_OVERLAP; } /* If necessary, evict an existing rule to clear out space. */ //如果流表项太多，就删除表中的部分流表项 if (table->n_flows >= table->max_flows) { if (!choose_rule_to_evict(table, &old_rule)) { return OFPERR_OFPFMFC_TABLE_FULL; } eviction_group_remove_rule(old_rule); /* Marks 'old_rule' as an evicted rule rather than replaced rule. */ old_rule->removed_reason = OFPRR_EVICTION; } } else { ofm->modify_cookie = true; } if (old_rule) { //----------这里检测出来是否有旧的流表项了 rule_collection_add(&ofm->old_rules, old_rule); }/* Take ownership of the temp_rule. */ //将去重处理后的新new_rule放入ofm->new_rules,并且清空ofm->temp_rule-------------重点！！！！ rule_collection_add(&ofm->new_rules, new_rule); ofm->temp_rule = NULL; //开始真正插入操作！！！-----重点---------------开始将规则 /* replace_rule_start主要操作就是新的流表替换旧的流表的操作，如果存在旧流表，则调用ofproto_rule_remove__删除， 然后调用ofproto_rule_insert__和classifier_insert添加流表。其中classifier_insert主要是将rule->cr添加到table->cls中 */ replace_rule_start(ofproto, ofm, old_rule, new_rule); return 0; }
2.replace_rule_start 处理包含自定义字段的流表项，放入effectable链中，不直接插入datapath。
static void replace_rule_start(struct ofproto *ofproto, struct ofproto_flow_mod *ofm, //---------插入cls struct rule *old_rule, struct rule *new_rule) { struct oftable *table = &ofproto->tables[new_rule->table_id]; //获取表 /* 'old_rule' may be either an evicted rule or replaced rule. */ //“旧规则”可以是逐出规则或替换规则。 if (old_rule) {/* Copy values from old rule for modify semantics. */ if (old_rule->removed_reason != OFPRR_EVICTION) { //非驱逐--替换（用旧的替换新的部分信息） bool change_cookie = (ofm->modify_cookie && new_rule->flow_cookie != OVS_BE64_MAX && new_rule->flow_cookie != old_rule->flow_cookie); ovs_mutex_lock(&new_rule->mutex); ovs_mutex_lock(&old_rule->mutex); if (ofm->command != OFPFC_ADD) { new_rule->idle_timeout = old_rule->idle_timeout; new_rule->hard_timeout = old_rule->hard_timeout; *CONST_CAST(uint16_t *, &new_rule->importance) = old_rule->importance; new_rule->flags = old_rule->flags; new_rule->created = old_rule->created; } if (!change_cookie) { *CONST_CAST(ovs_be64 *, &new_rule->flow_cookie) = old_rule->flow_cookie; } ovs_mutex_unlock(&old_rule->mutex); ovs_mutex_unlock(&new_rule->mutex); } /* Remove the old rule from data structures. */ if(new_rule->effect_sec || new_rule->effect_usec){//把ofm中的old_rule剔除---------- rule_collection_remove(&ofm->old_rules, old_rule); }else{/* Mark the old rule for removal in the next version. */ //标记旧规则，以便后面删除 cls_rule_make_invisible_in_version(&old_rule->cr, ofm->version); ofproto_rule_remove__(ofproto, old_rule); } } else { table->n_flows++; } /* Insert flow to ofproto data structures, so that later flow_mods may * relate to it. This is reversible, in case later errors require this to * be reverted. */ //将流插入到原始数据结构中，以便以后的流与之相关。这是可逆的，以防以后的错误需要恢复。 //先判断是否应该插入，effectable if(new_rule->effect_sec || new_rule->effect_usec){ if(new_rule->state == RULE_EFFECTED || new_rule->state == RULE_INSERTED) return; struct ofproto_flow_mod* tofm = (struct ofproto_flow_mod*)xmalloc(sizeof(*ofm)); //用来一直存放 memcpy(tofm,ofm,sizeof(*ofm)); tofm->omr = (struct openflow_mod_requester*)xmalloc(sizeof(struct openflow_mod_requester)); memcpy(tofm->omr,ofm->omr,sizeof(struct openflow_mod_requester)); tofm->conjs = (struct cls_conjunction *)xmalloc(ofm->n_conjs*sizeof(struct cls_conjunction)); memcpy(tofm->conjs,ofm->conjs,ofm->n_conjs*sizeof(struct cls_conjunction)); rule_collection_add(&tofm->old_rules, old_rule); new_rule->ofm = tofm;
ofproto_rule_effect__(ofproto,new_rule); //进行插入操作 return; } ofproto_rule_insert__(ofproto, new_rule); /* Make the new rule visible for classifier lookups only from the next * version. */ classifier_insert(&table->cls, &new_rule->cr, ofm->version, ofm->conjs, ofm->n_conjs); }
分析代码1：
//先判断是否应该插入，effectable if(new_rule->effect_sec || new_rule->effect_usec){ //如果我们设置了自定义字段的值的话！！！，开始将这条规则通过ofproto_rule_effect__方法插入effectable链中，并且改变该规则的状态 if(new_rule->state == RULE_EFFECTED || new_rule->state == RULE_INSERTED) return; struct ofproto_flow_mod* tofm = (struct ofproto_flow_mod*)xmalloc(sizeof(*ofm)); //我们需要将ofproto_flow_mod ofm存放到规则中的自定义字段中，因为规则生效时，进行插入datapath过程中需要使用到这些数据 memcpy(tofm,ofm,sizeof(*ofm)); //ofm在开始时是存放在栈中，不是堆中，生存周期没有办法达到我们需要的长度，所以我们变为堆中存放，并且把需要的数据拷贝过来 tofm->omr = (struct openflow_mod_requester*)xmalloc(sizeof(struct openflow_mod_requester)); //把请求数据一块保存，后面插入也是需要这个数据的 memcpy(tofm->omr,ofm->omr,sizeof(struct openflow_mod_requester)); //ofm->omr在handle_flow_mod__方法中被赋值了，所以我们这里可以直接获取openflow_mod_requester的数据 tofm->conjs = (struct cls_conjunction *)xmalloc(ofm->n_conjs*sizeof(struct cls_conjunction)); memcpy(tofm->conjs,ofm->conjs,ofm->n_conjs*sizeof(struct cls_conjunction)); rule_collection_add(&tofm->old_rules, old_rule); //把旧流表项添加到链中，当新的流表项生效时，再去把旧的流表项移除！！！！，注意我们后面分析的代码2 new_rule->ofm = tofm; ofproto_rule_effect__(ofproto,new_rule); //调用自定义方法ofproto_rule_effect__进行插入操作，插入自定义的effectable链中去，并且修改流表规则的状态 return; }
分析代码2：
if(new_rule->effect_sec || new_rule->effect_usec){//把ofm中的old_rule剔除----------原本old_rule中是存在这个旧的规则，但是我们因为新规则还没有生效，所以不能把旧规则删除了，所以我们把这个规则先移除来，不被删除掉 rule_collection_remove(&ofm->old_rules, old_rule); //如果effect字段被使用了，我们后面会添加到effectable链中单独处理，对于当前正在运行的旧的流表项（相互冲突的），我们在新流表项生效之前，始终运行！！！，所以我们不能去移除他 }else{ //对于没有使用自定义字段的规则，如果存在冲突的旧规则，则直接移除 //标记旧规则，以便后面删除 cls_rule_make_invisible_in_version(&old_rule->cr, ofm->version); ofproto_rule_remove__(ofproto, old_rule); }
3.自定义方法ofproto_rule_effect__，添加规则到链表中去
static void
ofproto_rule_effect__(struct ofproto *ofproto, struct rule *rule)
OVS_REQUIRES(ofproto_mutex)
{
/* A rule may not be reinserted. */
ovs_assert(rule->state != RULE_EFFECTED); //生效系列操作
if(rule->effect_sec || rule->effect_usec){
VLOG_INFO("------!!!---------ofproto_rule_effect__-------effectable----start---%d-----%d",rule->ofm->version,rule->ofm->n_conjs);
ovs_list_insert(&ofproto->effectable, &rule->effectable); //effectable入链
}
rule->state = RULE_EFFECTED; //规则已经插入
}
（四）分析ofproto_flow_mod_finish方法（未修改）
1.add_flow_finish方法
static void add_flow_finish(struct ofproto *ofproto, struct ofproto_flow_mod *ofm, const struct openflow_mod_requester *req) OVS_REQUIRES(ofproto_mutex) {struct rule *old_rule = rule_collection_n(&ofm->old_rules) ? rule_collection_rules(&ofm->old_rules)[0] : NULL;struct rule *new_rule = rule_collection_rules(&ofm->new_rules)[0];struct ovs_list dead_cookies = OVS_LIST_INITIALIZER(&dead_cookies);//下面代替才是重点-----------------！！！ replace_rule_finish(ofproto, ofm, req, old_rule, new_rule, &dead_cookies); learned_cookies_flush(ofproto, &dead_cookies);if (old_rule) { ovsrcu_postpone(remove_rule_rcu, old_rule); } else { ofmonitor_report(ofproto->connmgr, new_rule, NXFME_ADDED, 0, req ? req->ofconn : NULL, req ? req->request->xid : 0, NULL);/* Send Vacancy Events for OF1.4+. */ send_table_status(ofproto, new_rule->table_id); } }
2.replace_rule_finish方法
/* Adds the 'new_rule', replacing the 'old_rule'. */ static void replace_rule_finish(struct ofproto *ofproto, struct ofproto_flow_mod *ofm, const struct openflow_mod_requester *req, struct rule *old_rule, struct rule *new_rule, struct ovs_list *dead_cookies) OVS_REQUIRES(ofproto_mutex) { struct rule *replaced_rule; replaced_rule = (old_rule && old_rule->removed_reason != OFPRR_EVICTION) ? old_rule : NULL; ofproto->ofproto_class->rule_insert(new_rule, replaced_rule, //流表变化标识设置 ofm->modify_keep_counts); learned_cookies_inc(ofproto, rule_get_actions(new_rule));if (old_rule) {const struct rule_actions *old_actions = rule_get_actions(old_rule); const struct rule_actions *new_actions = rule_get_actions(new_rule); learned_cookies_dec(ofproto, old_actions, dead_cookies);if (replaced_rule) { enum nx_flow_update_event event = ofm->command == OFPFC_ADD ? NXFME_ADDED : NXFME_MODIFIED; bool changed_cookie = (new_rule->flow_cookie != old_rule->flow_cookie); bool changed_actions = !ofpacts_equal(new_actions->ofpacts, new_actions->ofpacts_len, old_actions->ofpacts, old_actions->ofpacts_len);if (event != NXFME_MODIFIED || changed_actions || changed_cookie) { //-------------------填充ofconn->updates用于消息发送 ofmonitor_report(ofproto->connmgr, new_rule, event, 0, req ? req->ofconn : NULL, req ? req->request->xid : 0, changed_actions ? old_actions : NULL); } } else { /* XXX: This is slight duplication with delete_flows_finish__() */ ofmonitor_report(ofproto->connmgr, old_rule, NXFME_DELETED, OFPRR_EVICTION, req ? req->ofconn : NULL, req ? req->request->xid : 0, NULL); } } }
（五）修改run方法，位于main的while循环中，循环处理effectable流表链！！！
在文件openvswitch-2.11.4/ofproto/ofproto-dpif.c中：
1.修改run方法，处理达到生效时间的流表规则，使之生效
#include <time.h>
static int run(struct ofproto *ofproto_) { struct ofproto_dpif *ofproto = ofproto_dpif_cast(ofproto_); uint64_t new_seq, new_dump_seq; struct timeval tv; //---------------获取当前时间--------------- if (mbridge_need_revalidate(ofproto->mbridge)) { ofproto->backer->need_revalidate = REV_RECONFIGURE; ovs_rwlock_wrlock(&ofproto->ml->rwlock); mac_learning_flush(ofproto->ml); ovs_rwlock_unlock(&ofproto->ml->rwlock); mcast_snooping_mdb_flush(ofproto->ms); } /* Always updates the ofproto->ams_seqno to avoid frequent wakeup during * flow restore. Even though nothing is processed during flow restore, * all queued 'ams' will be handled immediately when flow restore * completes. */ ofproto->ams_seqno = seq_read(ofproto->ams_seq); /* Do not perform any periodic activity required by 'ofproto' while * waiting for flow restore to complete. */ if (!ofproto_get_flow_restore_wait()) { struct ofproto_async_msg *am; struct ovs_list ams; guarded_list_pop_all(&ofproto->ams, &ams); LIST_FOR_EACH_POP (am, list_node, &ams) { connmgr_send_async_msg(ofproto->up.connmgr, am); ofproto_async_msg_free(am); } } if (ofproto->netflow) { netflow_run(ofproto->netflow); } if (ofproto->sflow) { dpif_sflow_run(ofproto->sflow); } if (ofproto->ipfix) { dpif_ipfix_run (ofproto->ipfix); } new_seq = seq_read(connectivity_seq_get()); if (ofproto->change_seq != new_seq) { struct ofport_dpif *ofport; HMAP_FOR_EACH (ofport, up.hmap_node, &ofproto->up.ports) { port_run(ofport); } ofproto->change_seq = new_seq; } if (ofproto->lacp_enabled || ofproto->has_bonded_bundles) { struct ofbundle *bundle; HMAP_FOR_EACH (bundle, hmap_node, &ofproto->bundles) { bundle_run(bundle); } } stp_run(ofproto); rstp_run(ofproto); ovs_rwlock_wrlock(&ofproto->ml->rwlock); if (mac_learning_run(ofproto->ml)) { ofproto->backer->need_revalidate = REV_MAC_LEARNING; } ovs_rwlock_unlock(&ofproto->ml->rwlock); if (mcast_snooping_run(ofproto->ms)) { ofproto->backer->need_revalidate = REV_MCAST_SNOOPING; } new_dump_seq = seq_read(udpif_dump_seq(ofproto->backer->udpif)); if (ofproto->dump_seq != new_dump_seq) { struct rule *rule, *next_rule; ovs_mutex_lock(&ofproto_mutex); LIST_FOR_EACH_SAFE (rule, next_rule, effectable, &ofproto->up.effectable) { //对于小于effect生效时间的，不进行处理 if(rule->effect_sec!=0 || rule->effect_usec!=0){ gettimeofday(&tv, NULL); //获取当前时间 if(!((tv.tv_sec < rule->effect_sec)||(tv.tv_sec == rule->effect_sec && tv.tv_usec < rule->effect_usec))){ //先对比秒,当前时刻小于生效时间，则return VLOG_INFO("-----------run--local_time--%d--%d->>>>>>-effect_time--%d--%d--",tv.tv_sec,tv.tv_usec,rule->effect_sec,rule->effect_usec); rule->effect_sec = 0; //归0 rule->effect_usec = 0; replace_rule_effect(&ofproto->up,rule); //进行插入 ------- 没有主动去调用handle_flow_mod函数，导致更新不及时！！！，不能进行处理 ovs_list_remove(&rule->effectable); //出链操作 } } } ovs_mutex_unlock(&ofproto_mutex); } if (ofproto->dump_seq != new_dump_seq) { //这里是用来遍历expirable链表，用来检测hard_timeout和idle_timeout是否到期的;effectable链就是模仿他实现的 struct rule *rule, *next_rule; long long now = time_msec(); /* We know stats are relatively fresh, so now is a good time to do some * periodic work. */ ofproto->dump_seq = new_dump_seq; /* Expire OpenFlow flows whose idle_timeout or hard_timeout * has passed. */ /* #define LIST_FOR_EACH_SAFE(ITER, NEXT, MEMBER, LIST) \ for (INIT_CONTAINER(ITER, (LIST)->next, MEMBER); \ (&(ITER)->MEMBER != (LIST) \ ? INIT_CONTAINER(NEXT, (ITER)->MEMBER.next, MEMBER), 1 \ : 0); \ (ITER) = (NEXT)) */ ovs_mutex_lock(&ofproto_mutex); LIST_FOR_EACH_SAFE (rule, next_rule, expirable, &ofproto->up.expirable) { rule_expire(rule_dpif_cast(rule), now); //-------这里校验是否超时 } ovs_mutex_unlock(&ofproto_mutex); /* All outstanding data in existing flows has been accounted, so it's a * good time to do bond rebalancing. */ if (ofproto->has_bonded_bundles) { struct ofbundle *bundle; HMAP_FOR_EACH (bundle, hmap_node, &ofproto->bundles) { if (bundle->bond) { bond_rebalance(bundle->bond); } } } } return 0; }
代码分析1：
if (ofproto->dump_seq != new_dump_seq) { struct rule *rule, *next_rule; ovs_mutex_lock(&ofproto_mutex); LIST_FOR_EACH_SAFE (rule, next_rule, effectable, &ofproto->up.effectable) { //遍历交换机的effectable链表 //对于小于effect生效时间的，不进行处理 if(rule->effect_sec!=0 || rule->effect_usec!=0){ //其实这里的规则的effect_sec或者effect_usec都是有值的 gettimeofday(&tv, NULL); //获取当前时间 if(!((tv.tv_sec < rule->effect_sec)||(tv.tv_sec == rule->effect_sec && tv.tv_usec < rule->effect_usec))){ //先对比秒和微秒,当前时刻大于生效时间，则当前流表项可以开始工作了，可以插入交换机进行生效处理！！！！ rule->effect_sec = 0; //归0，对于处理后的我们把值置为0,其实这里也可以不置为0,置为0之后后面不会在dump-flows中显示设置的生效时间了 rule->effect_usec = 0; replace_rule_effect(&ofproto->up,rule); //调用自定义replace_rule_effect方法进行插入，在内部主动去调用ofmonitor_flush函数，使得datapath及时更新！！！否則的话，可能更新不及时，不能进行按照新的流表项处理 ovs_list_remove(&rule->effectable); //出链操作，后面就不会再重复处理了！！！ } } } ovs_mutex_unlock(&ofproto_mutex); }
2.调用自定义方法replace_rule_effect，使得流表添加，并且被datapath及时处理（下面全部添加）
void replace_rule_effect(struct ofproto *ofproto,struct rule *new_rule){
ovs_assert(new_rule->state != RULE_INSERTED); //判断是否生效---是否是没有添加过的流表项const struct rule_actions *actions = rule_get_actions(new_rule); //获取规则actions,返回一个rule_actions指针 rule结构体中有action
struct oftable *table = &ofproto->tables[new_rule->table_id]; //获取表
struct ofproto_flow_mod* ofm = new_rule->ofm; //获取对应规则的ofproto_flow_mod数据
struct rule* old_rule = NULL;
enum ofperr error;
/* Check for the existence of an identical rule.
* This will not return rules earlier marked for removal. */
//检查是否存在相同的规则（旧的）
/* Must check actions while holding ofproto_mutex to avoid a race. */
error = ofproto_check_ofpacts(ofproto, actions->ofpacts, //检查锁，避免竞争-----------------出错！！！！
actions->ofpacts_len);
if (error) {return ;
}
old_rule = rule_from_cls_rule(classifier_find_rule_exactly(&table->cls, //去流表中查找旧的规则（于新规则有冲突的表项）
&new_rule->cr,
ofm->version));
if (!old_rule) {/* Check for overlap, if requested. 检查重叠*/
if (new_rule->flags & OFPUTIL_FF_CHECK_OVERLAP
&& classifier_rule_overlaps(&table->cls, &new_rule->cr,
ofm->version)) {return ;
}
/* If necessary, evict an existing rule to clear out space. */
//如果流表项太多，就删除表中的部分流表项
if (table->n_flows >= table->max_flows) {
if (!choose_rule_to_evict(table, &old_rule)) {return ;
}
eviction_group_remove_rule(old_rule);
/* Marks 'old_rule' as an evicted rule rather than replaced rule.
*/
old_rule->removed_reason = OFPRR_EVICTION;
}
} else {
ofm->modify_cookie = true;
}
if (old_rule) { //如果存在旧规则，就放入ofm的old_rules链中
rule_collection_add(&ofm->old_rules, old_rule);
}
/* Take ownership of the temp_rule. */
//将去重处理后的新new_rule放入ofm->new_rules,并且清空ofm->temp_rule-------------重点！！！！//开始真正插入操作！！！-----重点---------------开始将规则
//“旧规则”可以是逐出规则或替换规则。
if (old_rule) { //开始删除旧的规则/* Copy values from old rule for modify semantics. */
if (old_rule->removed_reason != OFPRR_EVICTION) { //非驱逐--替换（用旧的替换新的部分信息）
bool change_cookie = (ofm->modify_cookie
&& new_rule->flow_cookie != OVS_BE64_MAX
&& new_rule->flow_cookie != old_rule->flow_cookie);
ovs_mutex_lock(&new_rule->mutex);
ovs_mutex_lock(&old_rule->mutex);
if (ofm->command != OFPFC_ADD) {
new_rule->idle_timeout = old_rule->idle_timeout;
new_rule->hard_timeout = old_rule->hard_timeout;
*CONST_CAST(uint16_t *, &new_rule->importance) = old_rule->importance;
new_rule->flags = old_rule->flags;
new_rule->created = old_rule->created;
}
if (!change_cookie) {*CONST_CAST(ovs_be64 *, &new_rule->flow_cookie)
= old_rule->flow_cookie;
}
ovs_mutex_unlock(&old_rule->mutex);
ovs_mutex_unlock(&new_rule->mutex);
}/* Mark the old rule for removal in the next version. */
//标记旧规则，以便后面删除
cls_rule_make_invisible_in_version(&old_rule->cr, ofm->version);/* Remove the old rule from data structures. */
ofproto_rule_remove__(ofproto, old_rule); //移除旧的规则！！！！
} else {
table->n_flows++;
}
ofproto_rule_insert__(ofproto, new_rule); //插入新的规则（就是我们设置的生效时间下的流表项）
/* Make the new rule visible for classifier lookups only from the next
* version. */
classifier_insert(&table->cls, &new_rule->cr, new_rule->ofm->version, new_rule->ofm->conjs,
new_rule->ofm->n_conjs);
new_rule->state = RULE_INSERTED; //修改规则状态为已安装状态！！！//-----------更新完成之后进行通知-
ofproto_bump_tables_version(ofproto); //判断版本，不用管//ofproto_flow_mod_finish(ofproto, new_rule->ofm, new_rule->ofm->omr); //------------------這裏開始出錯！-------------------::::注釋之後可以正常開始了：：：：：：//下面才是重點
ofmonitor_flush(ofproto->connmgr); //ofproto->connmgr是OpenFlow交换机的连接管理器-------------通知内核，及时修改流表
}
目前第一个较为稳定的版本已经实现完成！！！
对于修改源码后，编译程序的方法见：
openvswitch2.11.0修改源码后重新编译（2）
