1.搭建Hadoop伪分布式平台

Linux配置

配置hosts

vim /etc/hosts

ssh免密登录

ssh-keygen #生成秘钥
ssh-copy-id hostname #复制ssh秘钥

Hadoop2.6.0

解压，配置环境变量

tar -xzvf hadoop.tar -C /
vim /etc/profile 
	export HADOOP_HOME=
	export PATH=
source /etc/profile #使变量生效

修改配置文件

• hadoop-env.sh

export JAVA_HOME=/usr/src/jdk

• core-site.xml

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/src/hadoop/tmp</value>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:9000</value>
        </property>
</configuration>

• hdfs-site.xml

<configuration>
        <property>
                <name>dfs.namenode.dir</name>
                <value>/usr/software/hadoop/data/name</value>
        </property>
        <property>
                <name>dfs.datanode.dir</name>
                <value>/usr/software/hadoop/data/data</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
</configuration>

• mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>

• yran-site.xml

<configuration>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

格式化hdfs

hadoop namenode -format

web界面

http://master:50070

Hive

在mysql中创建hive数据库

create database hive;
grant all on hive.* to 'hive' @  'localhost' identified by 'hive';

修改配置文件

vim hive/conf/hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <!-- 插入一下代码 -->
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <!--用户名（这4是新添加的，记住删除配置文件原有的哦！）-->
    <value>hive</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <!--密码-->
    <value>hive</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>mysql
    <value>jdbc:mysql://localhost:3306/hive</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name><!--mysql驱动程序-->
    <value>com.mysql.jdbc.Driver</value>
  </property>
   <!-- 打印表头信息，不指定字段时，显示"表名.字段" -->
  <property>
      <name>hive.cli.print.header</name>
      <value>true</value>
  </property>
  <!-- 打印数据库信息-->
  <property>
      <name>hive.cli.print.current.db</name>
      <value>true</value>
  </property>
          <!-- 到此结束代码 -->
  </configuration>

复制mysql驱动到hive/lib

cp mysql-connector-java-2.1.26-bin.jar hive/lib

初始化hive元数据

bin/schematool -dbType mysql -initSchema

常见错误

java.net.ConnectException: Call From master/188.188.14.231 to slave1:9000 failed on connection exception: java.net.ConnectException: 拒绝连接;

解决：hadoop未启动，启动即可

Found class jline.Terminal, but interface was expected

yarn下的jline和hadoop中的版本不同，复制hive/lib目录下的jline到Hadoop/yarn

Failed to get schema version.

检查数据库授权和配置文件是否有误