[Erlang 0104] 当Erlang遇到Solr

    Joe Armstrong的访谈中有一段关于"打开黑盒子"的阐述,给我留下很深的印象:Joe Armstrong在做XWindows开发时没有使用对应的类库,而是在了解XWindows底层实现后选择了直接和套接字通信,"把这20条消息映射到Erlang术语上,变个小魔术，然后可以向窗口直接发送消息，它们就开始执行动作了". [访谈全文] 回到今天的任务:Erlang使用Solr服务?当问题落实到数据通信协议的时候,就豁然开朗了,转换为我们熟悉的技术方案组合.先看下Solr的简介:

 

  Solr 

 

   Solr (pronounced "solar") is an open source enterprise search platform from the Apache Lucene project. Its major features include full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is highly scalable. Solr is the most popular enterprise search engine. Solr 4 adds NoSQL features.    Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Apache Tomcat or Jetty. Solr uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it usable from most popular programming languages. Solr's powerful external configuration allows it to be tailored to many types of application without Java coding, and it has a plugin architecture to support more advanced customization.

 

2014-2-17 15:03:21 补充一点Solr的东西:

1. Solr 可以做到 NEAR-Real-time Search ,  这个应该归功于softcommit
2. Solr 4 之前的版本只有提交到Lucene索引才能检索到,现在可以做到Real Time Get;换句话话说一个刚刚提交的数据,还没有Commit,现在是可以做修改的;能够做到这点是因为Transcation log,数据提交到Solr会写入transaction log.这个设计解耦了update durability和update visibility 
3. 目前Solr 4.x还是需要以文档为单位进行数据更新,重新做索引;换句话说新增字段的时候每条数据都要重新处理,重新做索引
4. Solr在任意时刻只有一个Active Searcher,Active Searcher有一份只读的Lucene索引快照.换句话说,当有新内容提交到Solr之后,这条数据对当前的Searcher是不可见的;新条目从不可见到可见:关闭当前的Searcher然后打开一个新的Searcher,新的Searcher会加载最新的索引的快照;这就是我们Commit数据到Solr的本质,关闭Old Searcher的时候会等把当前in-progress请求全部处理完;Old Searcher中缓存的数据也会被清理掉,这样是有道理的,有的数据可能已经删除了,可能有新数据了;New Searcher需要做一些预计算,所以Solr会有一个Warming Searcher的设计:等待New Searcher预热完毕再关闭之前的Searcher,这样搜索性能不会有明显的下降,对于Solr重新启动的情况,要看useColdSearcher是否开启,如果是false那就要阻塞到Searcher完成预热;Warming Searcher有两种方式:1.从old cache复制 2.执行cache warming queries,这个语句是开发者根据自己的业务定制的,不能用默认的;因为开启之后会有性能损失,默认是注释掉的,由开发者根据实际情况执行定制;一旦达到maxWarmingSearchers的数量上限,Commit就会失败,预热太多个Searcher会消耗CPU和Memeory;
5.  filtercache相当于缓存了若干数据分组结果,但是如果数据分组数量有限,命中率太低,那效果可以想象非常差;这个策略如果使用不当,就会浪费内存,比如:过滤条件过于复杂极小概率命中;solrconfig.xml中有对应的配置节,可以配置autowarmcount,自动预热是从old searcher中提取一些语句在new searcher中执行;autowarm会影响new searcher的启动速度,如果commit比较快,要求不停的创建新的searcher,就悲剧了;所以autowarmcount设置的值一般比较小;
6. queryResultWindowsSize 可以预取分页数据.queryResultMaxDocsCached 每条检索语句最多返回多少条文档;enableLazyFieldLoading可以优化取文档部分字段的情况,延迟加载的字段在用到的时候再请求;
7. segement merging是怎么回事?一份Luncene索引由若干个Segement构成,每一个查询都要在所有的Segement里面拿结果,Segment太多会影响性能;所以需要用合并机制

 

 

 

 

  搭建全文搜索服务Solr的确是一个不错的选择,分分钟就可以搭建起来Solr的环境,配置好IK什么的,那Erlang应用如何使用Solr服务呢?从上面维基百科的介绍中,我们可以捕捉到一些信息:REST-full API,XML,JSON,HTTP.看到这里已经全是我们熟悉的技术方案了,我们深入进去看下:

 

esolr

 

     2008年ppolv (Pablo Polvorin)在trapexit.org提交了一个Solr的功能模块,[地址: http://forum.trapexit.org/viewtopic.php?t=13059 ],完成了操作Solr的基本功能:

     |> Add/Update documents esolr:add/2
     |> Delete documents esolr:delete/2
     |> Search esolr:search/3

 

先看看怎么使用这些上面的接口:

 

%% 测试代码

-module(t).

-compile(export_all).


start()->
   SearchUrl="http://192.168.0.160:8080/solr/hear_me/select",
   UpdateUrl="http://192.168.0.160:8080/solr/hear_me/update",
   MltUrl="http://192.168.0.160:8080/solr/hear_me/mlt",
   {ok,Pid}=esolr:start([{select_url, SearchUrl}, {update_url, UpdateUrl}, {morelikethis_url, MltUrl}]),
   Pid.

search(SolrPid)->
  esolr:search("10",[{fields,"*,*"}],SolrPid).


add(SolrPid) ->
   esolr:add([{doc,[{id,"ai234"}, {text,<<"Look me mom!, I'm searching now">>}]}],SolrPid),
   esolr:add([{doc,[{id,"a3456"}, {text,<<"Look me mom!, I'm searching now">>}]}],SolrPid),
   esolr:commit(SolrPid).

测试结果如下:

Eshell V5.9  (abort with ^G)
1> P=t:start().
<0.34.0>
2> t:add(P).
ok
3> esolr:search("searching",[{fields,"*,*"}],P).
{ok,[{"numFound",2},{"start",0}],
    [{doc,[{"id",<<"ai234">>},
           {"_version_",1440978100186775552}]},
     {doc,[{"id",<<"a3456">>},
           {"_version_",1440978100212989952}]}],
    []}
4> t:search(P).
{ok,[{"numFound",9},{"start",0}],
    [{doc,[{"c_type",1},
           {"c_tags",
            [<<"女人">>,
             <<230,148,190,229,188,131>>,
             <<"家庭">>,
             <<229,165,179,229,143,139>>,
             <<229,165,179,229,173,169,229,173,144>>,
             <<229,176,143,229,173,169,229,173,144>>,
             <<231,166,187,229,169,154>>,
             <<229,135,186,230,137,139>>,
             <<229,133,132,229,188,159>>]},
           {"c_pub_date",<<"2013-07-12T16:29:11.593Z">>},
           {"id",<<"97">>},
           {"_version_",1440342611812417536}]},
     {doc,[{"c_type",1},
           {"c_tags",
            [<<231,189,145,231,187,156>>,
             <<229,165,179,229,143,139>>,
             <<228,187,139,231,187,141>>,
             <<233,171,152,228,184,173>>,
             <<229,144,140,229,173,166>>,
             <<230,156,139,229,143,139>>,
             <<229,140,151,228,186,172>>,
 ..... ...

　

代码实现　

  翻开代码,下面这个方法包含了大部分技术要点:

 

make_post_request(Request,PendingInfo,
State=#esolr{update_url=URL,pending=P,auto_commit=AC,dirty=Dirty},
Timeout) ->
     {ok,RequestId} = httpc:request(post,{URL,[{"connection", "close"}],"text/xml",Request},[{timeout,Timeout}],[{sync,false}]),
     Pendings = gb_trees:insert(RequestId,PendingInfo,P),
     if
          (AC == always) and Dirty -> 
                      CommitRequest = encode_commit(),
                      {ok,C_RequestId} = httpc:request(post,{URL,[{"connection", "close"}],"text/xml",CommitRequest},
                                                    [{timeout,State#esolr.commit_timeout}],[{sync,false}]),
                      Pendings2 = gb_trees:insert(C_RequestId,{auto,auto_commit},Pendings),
                      error_logger:info_report([{auto_commit,send}]),
                        {noreply,State#esolr{pending=Pendings2,dirty=false}};
         
          true -> {noreply,State#esolr{pending=Pendings}}
     end.

 

首先在init阶段开启了inets:start(),make_post_request发起HTTP请求靠的是httpc,每一次请求之后都会把RequestId和请求发起者({From,_}里面的From)对应关系存储到gb_tree.在后面的handle_info代码段,可以看到对HTTPResponse的消息的接收.

 

% @hidden
handle_info({http,{RequestId,HttpResponse}},State = #esolr{pending=P}) ->
     case gb_trees:lookup(RequestId,P) of
          {value,{Client,RequestOp}} -> handle_http_response(HttpResponse,RequestOp,Client),
                              {noreply,State#esolr{pending=gb_trees:delete(RequestId,P)}};
          none -> {noreply,State}
                    %% the requestid isn't here, probably the request was deleted after a timeout
     end;

parse_search_response(Response,Client) ->
     {value,{"response",{obj,SearchRespFields}},RestResponse} = lists:keytake("response",1, Response),
     {value,{"docs",Docs},RespFields} =  lists:keytake("docs",1,SearchRespFields),
     gen_server:reply(Client,{ok,RespFields,[{doc,DocFields} || {obj,DocFields}<-Docs],RestResponse}).

在parse_search_response方法里面gen_server:reply调用最终完成了对请求的应答.

 

XML & Json

 

既然要处理XML,当然要用到xmerl模块了,encode_*系列模块基本上都是用它完成数据的encode,比如:

 

Eshell V5.10.2  (abort with ^G)
1> xmerl:export_simple([{commit,[]}],xmerl_xml).
["<?xml version=\"1.0\"?>",[["<","commit","/>"]]]
2>

 HTTPResponse解析还会用到xmerl_scan,xmerl_xpath

 

handle_http_response({{_HttpV,200,_Reason},_Headers,Data},Op,Client) ->
     {Response,[]} = xmerl_scan:string(binary_to_list(Data)),
     [Header] = xmerl_xpath:string("/response/lst[@name='responseHeader']",Response),
     case parse_xml_response_header(Header) of
          {ok,QTime} ->  parse_xml_response(Op,Response,QTime,Client);
          {error,Error} ->  response_error(Op,Client,Error)
     end;

 除了XML之外,还要解析JSON,这里使用的是RFC4627.

 

扩展

 

这个简单的功能模块,呃,是不是太简陋了?而且你会发现代码太老了?这段代码后续被修改应用在了Zotontic项目实现搜索功能,之前盘点Erlang Web Server和Web Framework的时候提到过这个CMS系统 [地址:https://github.com/arjan/mod_search_solr] 这个项目里面把原有代码做了重构,并增加了很多实用的接口比如翻页 "MoreLikeThis"功能封装.可以在Github上获取代码试一下,Zotontic的代码略显庞大,只取必需的模块编译即可;

 

 

ok,今天就到这里.

 

 

最后小图一张 Miss Nine