Hive中视图机制的初步使用及分析

作者: 大圆那些事 | 文章可以转载，请以超链接形式标明文章原始出处和作者信息

网址: http://www.cnblogs.com/panfeng412/archive/2013/04/29/hive-view-usage-and-analysis.html

本文是对Hive中逻辑视图的介绍，通过一个简单的视图例子，说明其使用方法及执行过程。

Hive 0.6版本及以上支持视图（View，详见Hive的RELEASE_NOTES.txt），Hive View具有以下特点：

1）View是逻辑视图，暂不支持物化视图（后续将在1.0.3版本以后支持）；

2）View是只读的，不支持LOAD/INSERT/ALTER。需要改变View定义，可以是用Alter View；

3）View内可能包含ORDER BY/LIMIT语句，假如一个针对View的查询也包含这些语句， 则View中的语句优先级高；

4）支持迭代View。

CDH4中自带的Hive版本为0.10.0，支持的View是逻辑视图，因此本质上来说View只是为了使用上的方便，从执行效率上来说没有区别，甚至可能因为要多一次对MetaStore元数据的操作效率略有下降（这里只是一种理论上的推测，实际可能看不出太大区别）。

下面是简单的验证过程（感兴趣的可以看下，以下过程如有问题，可以一起交流）：

1）创建一个测试表：

hive> create table test (id int, name string);
OK
Time taken: 0.19 seconds
hive> desc test;                                                                           
OK
id      int
name    string
Time taken: 0.16 seconds

2）创建一个View之前，使用explain命令查看创建View的命令是如何被Hive解释执行的：

hive> explain create view test_view (id, name_length) as  select id, length(name) from test;
OK
ABSTRACT SYNTAX TREE:
  (TOK_CREATEVIEW (TOK_TABNAME test_view) (TOK_TABCOLNAME (TOK_TABCOL id TOK_NULL) (TOK_TABCOL name_length TOK_NULL)) (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME test))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL id)) (TOK_SELEXPR (TOK_FUNCTION length (TOK_TABLE_OR_COL name)))))))
STAGE DEPENDENCIES:
  Stage-0 is a root stage
STAGE PLANS:
  Stage: Stage-0
      Create View Operator:
        Create View
          if not exists: false
          or replace: false
          columns: id int, name_length int
          expanded text: SELECT `id` AS `id`, `_c1` AS `name_length` FROM (select `test`.`id`, length(`test`.`name`) from `default`.`test`) `test_view`
          name: test_view
          original text: select id, length(name) from test
Time taken: 0.088 seconds

可见，创建View的过程解释后并没有实际执行Map Reduce的Stage，只包含一个Create View Operator的Stage，这个阶段只是对MySQL MetaStore进行元数据操作，记录View的相关元数据而已。

3）接下来，实际创建这个View：

hive> create view test_view (id, name_length) as  select id, length(name) from test;   
OK
Time taken: 0.1 seconds

4）执行这个View之前，先explain查看实际被翻译后的执行过程：

hive> explain select name_length from test_view;
OK
ABSTRACT SYNTAX TREE:
  (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME test_view))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL name_length)))))
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage
STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        test_view:test_view:test
          TableScan
            alias: test
            Select Operator
              expressions:
                    expr: length(name)
                    type: int
              outputColumnNames: _col1
              Select Operator
                expressions:
                      expr: _col1
                      type: int
                outputColumnNames: _col1
                Select Operator
                  expressions:
                        expr: _col1
                        type: int
                  outputColumnNames: _col0
                  File Output Operator
                    compressed: false
                    GlobalTableId: 0
                    table:
                        input format: org.apache.hadoop.mapred.TextInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
  Stage: Stage-0
    Fetch Operator
      limit: -1
Time taken: 0.107 seconds

可以看出，对View进行的查找过程，实际还是对原始test表进行的查询操作（分为Stage-0和Stage-1两个阶段）。

5）最后，实际对这个View执行一次查询，显示Stage-1阶段对原始表test进行了MapReduce过程：

hive> select name_length from test_view;       
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201303092253_0057, Tracking URL = http://jobtracker.host:50030/jobdetails.jsp?jobid=job_201303092253_0057
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_201303092253_0057
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2013-03-13 22:43:39,044 Stage-1 map = 0%,  reduce = 0%
2013-03-13 22:43:42,074 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.73 sec
2013-03-13 22:43:43,086 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.73 sec
2013-03-13 22:43:44,098 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.73 sec
2013-03-13 22:43:45,113 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 0.73 sec
MapReduce Total cumulative CPU time: 730 msec
Ended Job = job_201303092253_0057
MapReduce Jobs Launched:
Job 0: Map: 1   Cumulative CPU: 0.73 sec   HDFS Read: 250 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 730 msec
OK
Time taken: 15.793 seconds