1.hive介绍及安装配置

1.Hive介绍

  1. 数据库OLTP 在线事务处理
  2. 数据仓库OLAP 在线分析处理 延迟高
  3. 类sql方式(HQL)
  4. 使用sql方式,用来读写,管理位于分布式存储系统上的大型数据集的数据仓库技术
  5. hive是基于Hadoop的一个数据仓库工具,可以将结构化的数据文件映射为一张数据库表,并提供完整的sql查询功能,可以将sql语句转换为MapReduce任务进行运行。其优点是学习成本低,可以通过类SQL语句快速实现简单的MapReduce统计,不必开发专门的MapReduce应用,十分适合数据仓库的统计分析。
  6. hive使用的是hdfs做为存储
  7. 使用maprecude做为计算模型
  8. 用于海量数据计算分析

2.安装

  1. 基于hadoop完全分布式环境(搭建过程略)

  2. 下载hive包

    #wget https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz
    
  3. 解压到指定路径和创建软链接

    #tar xf apache-hive-2.1.1-bin.tar.gz -C /soft/
    #ln -s /soft/apache-hive-2.1.1-bin /soft/hive
    
  4. 配置环境变量

    #vim  /etc/profile
    添加一下内容
    HIVE_HOME=/soft/hive
    PATH=$PATH:$HIVE_HOME/bin
    #source /etc/profile
    
  5. 测试

    #hive --version
    

3.配置mysql相关

  1. 安装mysql(过程省略)

  2. 创建相关数据库并授权

    mysql>create database dbhive;
    mysql>use hive;
    mysql>grant all on dbhive.* to "hive"@"%" identified by "123456"; 
    

5. 配置hive

  1. 修改hive-site.xml(修改mysql相关配置)

    #cd /soft/hive/conf
    #vim hive-site.xml
    
javax.jdo.option.ConnectionPassword 123456 password to use against metastore database javax.jdo.option.ConnectionUserName hive Username to use against metastore database javax.jdo.option.ConnectionURL jdbc:mysql://192.168.10.103:3306/dbhive ```
  1. 复制mysql驱动程序

    #cp /root/mysql-connector-java-5.1.38-bin.jar /soft/hive/lib/
    
  2. 在mysql中初始化hive的schema

    #cd /soft/hive/bin/
    #./schematool -dbType mysql -initSchema
    which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/soft/jdk/bin:/soft/hadoop/bin:/soft/hadoop/sbin:/soft/hive/bin:/root/bin)
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    Metastore connection URL:	 jdbc:mysql://192.168.10.103:3306/dbhive
    Metastore Connection Driver :	 com.mysql.jdbc.Driver
    Metastore connection User:	 hive
    Starting metastore schema initialization to 2.1.0
    Initialization script hive-schema-2.1.0.mysql.sql
    Initialization script completed
    schemaTool completed
    
  3. 查看mysql

    mysql> use dbhive;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A
    
    Database changed
    mysql> show tables;
    +---------------------------+
    | Tables_in_dbhive          |
    +---------------------------+
    | AUX_TABLE                 |
    | BUCKETING_COLS            |
    | CDS                       |
    | COLUMNS_V2                |
    | COMPACTION_QUEUE          |
    | COMPLETED_COMPACTIONS     |
    | COMPLETED_TXN_COMPONENTS  |
    | DATABASE_PARAMS           |
    | DBS                       |
    | DB_PRIVS                  |
    | DELEGATION_TOKENS         |
    | FUNCS                     |
    | FUNC_RU                   |
    | GLOBAL_PRIVS              |
    | HIVE_LOCKS                |
    | IDXS                      |
    | INDEX_PARAMS              |
    | KEY_CONSTRAINTS           |
    | MASTER_KEYS               |
    | NEXT_COMPACTION_QUEUE_ID  |
    | NEXT_LOCK_ID              |
    | NEXT_TXN_ID               |
    | NOTIFICATION_LOG          |
    | NOTIFICATION_SEQUENCE     |
    | NUCLEUS_TABLES            |
    | PARTITIONS                |
    | PARTITION_EVENTS          |
    | PARTITION_KEYS            |
    | PARTITION_KEY_VALS        |
    | PARTITION_PARAMS          |
    | PART_COL_PRIVS            |
    | PART_COL_STATS            |
    | PART_PRIVS                |
    | ROLES                     |
    | ROLE_MAP                  |
    | SDS                       |
    | SD_PARAMS                 |
    | SEQUENCE_TABLE            |
    | SERDES                    |
    | SERDE_PARAMS              |
    | SKEWED_COL_NAMES          |
    | SKEWED_COL_VALUE_LOC_MAP  |
    | SKEWED_STRING_LIST        |
    | SKEWED_STRING_LIST_VALUES |
    | SKEWED_VALUES             |
    | SORT_COLS                 |
    | TABLE_PARAMS              |
    | TAB_COL_STATS             |
    | TBLS                      |
    | TBL_COL_PRIVS             |
    | TBL_PRIVS                 |
    | TXNS                      |
    | TXN_COMPONENTS            |
    | TYPES                     |
    | TYPE_FIELDS               |
    | VERSION                   |
    | WRITE_SET                 |
    +---------------------------+
    57 rows in set (0.00 sec)
    
  4. 登录hive

    #hive
    
posted @ 2017-07-15 21:17  刘耀  阅读(507)  评论(0编辑  收藏  举报