hadoop安装
Hadoop安装
安装前准备
- 关闭防火墙
tu@tu-virtual-machine:~$ sudo ufw disable
[sudo] tu 的密码：
防火墙在系统启动时自动禁用
tu@tu-virtual-machine:~$ sudo ufw status
状态：不活动
2. ssh安装
tu@tu-virtual-machine:~$ sudo dpkg -l | grep ssh
ii libssh-4:amd64 0.6.3-4.3ubuntu0.6 amd64 tiny C SSH library (OpenSSL flavor)
ii openssh-client 1:7.2p2-4ubuntu2.10 amd64 secure shell (SSH) client, for secure access to remote machines
tu@tu-virtual-machine:~$ sudo apt-get install openssh-server
在安装ssh的时候可能会又依赖包的问题
在设置>软件与更新>更换为华为的yum源即可解决
安装远程连接软件 xshell CRT ...
安装JDK
1. 下载并上传到系统上
tu@tu-virtual-machine:~$ ls
examples.desktop jdk1.8 jdk-8u171-windows-x64.zip 模板 图片 下载 桌面
hadoop-2.7.3.tar.gz 公共的 视频 文档 音乐
2. 解压
tu@tu-virtual-machine:~$ tar -zxvf jdk8u171-linux-x64.tar.gz
3. 建立JDK链接
tu@tu-virtual-machine:~$ ln -s jdk1.8.0_171/ jdk
4. 设置JDK的环境变量
vim ~/.bashrc
export JAVA_HOME=~/jdk
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:.
export PATH=${JAVA_HOME}/bin:$PATH
5. 使设置生效
source ~/.bashrc
6. 检验是否安装成功
tu@tu-virtual-machine:~$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
安装hadoop
1. 解压
tu@tu-virtual-machine:~$ tar -zxvf hadoop-2.7.3.tar.gz
2. 建立软链接
tu@tu-virtual-machine:~$ ln -s hadoop-2.7.3 hadoop
3. 编写环境变量
export HADOOP_HOME=~/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
4. 使设置生效
source ~/.bashrc
5. 验证
tu@tu-virtual-machine:~$ whereis hdfs
hdfs: /home/tu/hadoop-2.7.3/bin/hdfs /home/tu/hadoop-2.7.3/bin/hdfs.cmd
tu@tu-virtual-machine:~$ whereis start-all.sh
start-all: /home/tu/hadoop-2.7.3/sbin/start-all.sh /home/tu/hadoop-2.7.3/sbin/start-all.cmd
hadoop安装 ( 单机安装 )
1. 编辑配置文件
# The java implementation to use.
export JAVA_HOME=/home/tu/jdk
2. 测试hadoop
tu@tu-virtual-machine:~/hadoop/etc/hadoop$ mkdir ~/input
tu@tu-virtual-machine:~/hadoop/etc/hadoop$ cd ~/input
tu@tu-virtual-machine:~/input$ vim data.txt
hello world
hello hadoop
...
tu@tu-virtual-machine:~/hadoop/share/hadoop/mapreduce$ hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount~/input/data.txt ~/output
tu@tu-virtual-machine:~/hadoop/share/hadoop/mapreduce$ cd ~/output
tu@tu-virtual-machine:~/output$ ls
part-r-00000 _SUCCESS
tu@tu-virtual-machine:~/output$ cat part-r-00000
hadoop 1
hello 2
world 1
hadoop安装 ( 伪分布式 )
1. 修改主机名
tu@tu-virtual-machine:~$ sudo vim /etc/hostname
nodel
重启主机生效
2. 修改/etc/hosts文件
tu@nodel:~$ sudo vim /etc/hosts
[sudo] tu 的密码：
127.0.0.1 localhost
127.0.1.1 tu-virtual-machine
192.168.200.140 nodel
3. 免密登录设置
生成密钥对
tu@nodel:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tu/.ssh/id_rsa):
Created directory '/home/tu/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tu/.ssh/id_rsa.
Your public key has been saved in /home/tu/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:p2phzZY9D/cNxhFKo7Rhj3N0tfhvnw6yVR90y5jeic4 tu@nodel
The key's randomart image is:
+---[RSA 2048]----+
| ..|
| + + + .|
| o O = +.|
| = + B o|
| oSo.o + B |
| o =o+ o B =|
| . o. * B =+|
| .. B o.+|
| .. . E.o.|
+----[SHA256]-----+
tu@nodel:~$ ls ~/.ssh
id_rsa id_rsa.pub
追加公钥
tu@nodel:~$ ssh-copy-id -i ~/.ssh/id_rsa.pub nodel
tu@nodel:~$ ls ~/.ssh
authorized_keys id_rsa id_rsa.pub known_hosts
验证
tu@nodel:~$ ssh nodel
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-112-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
124 个可升级软件包。
98 个安全更新。
Last login: Sun Nov 15 15:28:18 2020 from 192.168.200.1
tu@nodel:~$
4. 编辑hadoop的配置文件
[core-site.xml]
tu@nodel:~/hadoop/etc/hadoop$ vim core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://nodel:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/tu/hadoop/tmp</value>
</property>
</configuration>
[hdfs-site.xml]
tu@nodel:~/hadoop/etc/hadoop$ vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>`
</configuration>
[mapred-site.xml]
tu@nodel:~/hadoop/etc/hadoop$ cp mapred-site.xml.template mapred-site.xml
tu@nodel:~/hadoop/etc/hadoop$ vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
[yarn-site.xml]
tu@nodel:~/hadoop/etc/hadoop$ vim yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>nodel</value>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
5.格式化HDFS
tu@nodel:~/hadoop/etc/hadoop$ hdfs namenode -format
6. 验证操作
tu@nodel:~/hadoop/etc/hadoop$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [nodel]
nodel: namenode running as process 9181. Stop it first.
localhost: datanode running as process 9330. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 9517. Stop it first.
starting yarn daemons
starting resourcemanager, logging to /home/tu/hadoop-2.7.3/logs/yarn-tu-resourcemanager-nodel.out
localhost: starting nodemanager, logging to /home/tu/hadoop-2.7.3/logs/yarn-tu-nodemanager-nodel.out
tu@nodel:~/hadoop/etc/hadoop$ jps
10512 Jps
10401 NodeManager
9330 DataNode
10101 ResourceManager
9181 NameNode
9517 SecondaryNameNode
7. 通过Web访问hadoop
在浏览器中输入网址:
192.168.200.140:50070
192.168.200.140:50090
192.168.200.140:8088
Hbase的安装
本地模式
1. 解压
tu@nodel:~$ tar -zxvf hbase-1.3.1-bin.tar.gz
2. 建立软连接
tu@nodel:~$ ln -s hbase-1.3.1 hbase
3. 编辑环境变量
export HBASE_HOME=~/hbase
export PATH=$HBASE_HOME/bin:$PATH
tu@nodel:~$ source ~/.bashrc
4. 在hbase下创建data目录
tu@nodel:~$ cd ~/hbase
tu@nodel:~/hbase$ mkdir data
5. 修改配置文件
tu@nodel:~/hbase/conf$ vim hbase-env.sh
...
export JAVA_HOME=~/jdk
tu@nodel:~/hbase/conf$ vim hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///~hbase/data</value>
</property>
</configuration>
6. 启动Hbase
tu@nodel:~/hbase/conf$ start-hbase.sh
7. 检查进程
tu@nodel:~/hbase/conf$ jps
10401 NodeManager
11506 HMaster
9330 DataNode
10101 ResourceManager
9181 NameNode
11741 Jps
9517 SecondaryNameNode
Hive的安装
本地模式
1. 解压
tu@nodel:~$ tar -zxvf apache-hive-2.3.3-bin.tar.gz
2. 建立软链接
tu@nodel:~$ ln -s apache-hive-2.3.3-bin hive
3. 编辑环境变量
export HIVE_HOME=~/hive
export PATH=$HIVE_HOME/bin:$PATH
tu@nodel:~$ source ~/.bashrc
4. 新建配置文件
tu@nodel:~/hive/conf$ vim hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jbo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?useSSL=false</value>
</property>
<property>
<name>javax.jbo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jbo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jbo.option.ConnectionPassword</name>
<value>000000</value>
</property>
</configuration>
5. 启动
tu@nodel:~$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/tu/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/tu/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/home/tu/apache-hive-2.3.3-bin/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
安装pig
1. 解压
tu@nodel:~$ tar -zxvf pig-0.17.0.tar.gz
2. 建立软链接
tu@nodel:~$ ln -s pig-0.17.0 pig
3.编辑环境变量文件
export PIG_HOME=~pig
export PATH=$PIG_HOME/bin:$PATH
tu@nodel:~$ source ~/.bashrc
4. 本地模式启动
tu@nodel:~$ pig -x local
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/tu/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/tu/hbase-1.3.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/11/15 16:56:19 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
20/11/15 16:56:19 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2020-11-15 16:56:19,247 [main] INFO org.apache.pig.Main - Apache Pig version 0.17.0 (r1797386) compiled Jun 02 2017, 15:41:58
2020-11-15 16:56:19,247 [main] INFO org.apache.pig.Main - Logging error messages to: /home/tu/pig_1605430579246.log
2020-11-15 16:56:19,271 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/tu/.pigbootup not found
2020-11-15 16:56:19,410 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2020-11-15 16:56:19,413 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2020-11-15 16:56:19,549 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2020-11-15 16:56:19,561 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-default-568a54da-b2e3-4c35-80d8-a1b118bd92e3
2020-11-15 16:56:19,561 [main] WARN org.apache.pig.PigServer - ATS is disabled since yarn.timeline-service.enabled set to false
grunt>
5. mapreduce模式启动 ( 需要hadoop,yarn 启动 )
tu@nodel:~$ pig
20/11/15 16:57:23 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
20/11/15 16:57:23 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
20/11/15 16:57:23 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2020-11-15 16:57:23,677 [main] INFO org.apache.pig.Main - Apache Pig version 0.17.0 (r1797386) compiled Jun 02 2017, 15:41:58
2020-11-15 16:57:23,678 [main] INFO org.apache.pig.Main - Logging error messages to: /home/tu/pig_1605430643676.log
2020-11-15 16:57:23,692 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/tu/.pigbootup not found
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/tu/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/tu/hbase-1.3.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2020-11-15 16:57:24,035 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2020-11-15 16:57:24,035 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://nodel:8020
2020-11-15 16:57:24,566 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-default-9a9eef35-4182-42af-93fb-56580f183568
2020-11-15 16:57:24,566 [main] WARN org.apache.pig.PigServer - ATS is disabled since yarn.timeline-service.enabled set to false
grunt>
浙公网安备 33010602011771号