hdfs 命令学习

appendToFile

Usage: hdfs dfs -appendToFile <localsrc> ... <dst>

从本地追加一个或者多个文件到hdfs中，数据源是本地，目的地是hdfs中

hdfs dfs -appendToFile localfile /user/hadoop/hadoopfile

hdfs dfs -appendToFile localfile1 localfile2 /user/hadoop/hadoopfile

hdfs dfs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile

cat

Usage: hdfs dfs -cat URI [URI ...]

查看内容.

hdfs dfs -cat /user/hadoop.txt

 

Chgrp【change group】

Usage: hdfs dfs -chgrp [-R] GROUP URI [URI ...]

修改所属组.

Options

The -R option will make the change recursively through the directory structure

 

chmod

Usage: hdfs dfs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]

修改权限.

Options

The -R option will make the change recursively through the directory structure.

 

chown

Usage: hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]

修改所有者.

Options

The -R option will make the change recursively through the directory structure.

 

copyFromLocal

Usage: hdfs dfs -copyFromLocal <localsrc> URI

把本地的文件复制到hdfs上

hdfs dfs -copyFromLocal  test.txt /test/hadoop

option   -f  把原来已经存在的hdfs上的相同文件覆盖

 

copyToLocal

Usage: hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI <localdst>

Similar to get command, except that the destination is restricted to a local file reference.

 

count

Usage: hdfs dfs -count [-q] [-h] <paths>

列出文件夹数量、文件数量、内容大小. The output columns with -count are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE FILE_NAME

Example:

hdfs dfs -count /t_phone_contacts_edate

文件夹数量   文件数量  内容大小     

35                  1656        68588622049   /t_phone_contacts_edate

 

cp

Usage: hdfs dfs -cp [-f] [-p | -p[topax]] URI [URI ...] <dest>

复制文件(夹)，可以覆盖，可以保留原有权限信息

Options:

The -f option will overwrite the destination if it already exists.

The -p option will preserve file attributes [topx] (timestamps, ownership, permission, ACL, XAttr). If -p is specified with no arg, then preserves timestamps, ownership, permission. If -pa is specified, then preserves permission also because ACL is a super-set of permission. Determination of whether raw namespace extended attributes are preserved is independent of the -p flag.

Example:

hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2

hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir

 

du

Usage: hdfs dfs -du [-s] [-h] URI [URI ...]

显示文件(夹)大小.

hdfs dfs -du -h /

191.6 G  /hbase
5.5 K    /hbaseImportTsv1
22.1 K   /hive
10.2 G   /hive_t_phone_contacts_top_tmp
28.7 K   /hivetemp
5.6 K    /home
1.2 T    /signalling_info
0        /signalling_info_tmp
195.5 M  /spark
40.2 M   /spark-eventlogs
3.5 G    /t_info_quhao
26.2 G   /t_phone_contacts_calling
67.5 G   /t_phone_contacts_edate
0        /t_phone_contacts_mouth
18.0 G   /t_phone_contacts_top
477.3 M  /tmp
19.9 M   /user

dus

Usage: hdfs dfs -dus <args>

Displays a summary of file lengths.

Note: This command is deprecated. Instead use hdfs dfs -du -s.

expunge

Usage: hdfs dfs -expunge

清空回收站.

 

get

Usage: hdfs dfs -get [-ignorecrc] [-crc] <src> <localdst>

从hdfs中下载到本地目录中

Example:hdfs dfs -get  /t_phone_contacts_edate/edate=10020160701/000056_0.snappy /home/hadoop

 

getfacl

Usage: hdfs dfs -getfacl [-R] <path>

显示权限信息.

Options:

-R: List the ACLs of all files and directories recursively.

path: File or directory to list.

Examples:

hdfs dfs -getfacl /t_phone_contacts_edate/edate=10020160701/000056_0.snappy
# file: /t_phone_contacts_edate/edate=10020160701/000056_0.snappy
# owner: hadoop
# group: supergroup
user::rwx
group::rwx
other::rwx

 

ls

Usage: hdfs dfs -ls [-R] <args>

Options:

The -R option will return stat recursively through the directory structure.

For a file returns stat on the file with the following format:

permissions number_of_replicas userid groupid filesize modification_date modification_time filename

For a directory it returns list of its direct children as in Unix. A directory is listed as:

permissions userid groupid modification_date modification_time dirname

Example:

hdfs dfs -ls /
Found 17 items
drwxrwxrwx   - hadoop supergroup          0 2016-09-18 11:47 /hbase
drwxrwxrwx   - hadoop supergroup          0 2016-08-24 23:14 /hbaseImportTsv1
drwxrwxrwx   - hadoop supergroup          0 2016-03-31 18:45 /hive
drwxrwxrwx   - hadoop supergroup          0 2016-09-02 12:53 /hive_t_phone_contacts_top_tmp
drwxrwxrwx   - hadoop supergroup          0 2016-04-08 11:54 /hivetemp
drwxrwxrwx   - hadoop supergroup          0 2016-05-10 14:50 /home
drwxrwxrwx   - hadoop supergroup          0 2016-09-22 00:40 /signalling_info
drwxrwxrwx   - hadoop supergroup          0 2016-05-10 14:16 /signalling_info_tmp
drwxrwxrwx   - hadoop supergroup          0 2016-04-13 10:34 /spark
drwxrwxrwx   - hadoop supergroup          0 2016-09-22 21:02 /spark-eventlogs
drwxrwxrwx   - hadoop supergroup          0 2016-09-22 20:53 /t_info_quhao
drwxrwxrwx   - hadoop supergroup          0 2016-09-20 23:53 /t_phone_contacts_calling
drwxrwxrwx   - hadoop supergroup          0 2016-09-22 21:01 /t_phone_contacts_edate
drwxrwxrwx   - hadoop supergroup          0 2016-09-22 12:29 /t_phone_contacts_mouth
drwxrwxrwx   - hadoop supergroup          0 2016-09-02 10:12 /t_phone_contacts_top
drwxrwxrwx   - hadoop supergroup          0 2016-05-20 14:24 /tmp
drwxrwxrwx   - hadoop supergroup          0 2016-04-07 18:27 /user

 

mkdir

Usage: hdfs dfs -mkdir [-p] <paths>

Takes path uri's as argument and creates directories.

Options:

The -p option behavior is much like Unix mkdir -p, creating parent directories along the path.

Example:

hdfs dfs -mkdir /hadoop

 

moveFromLocal

Usage: hdfs dfs -moveFromLocal <localsrc> <dst>

Similar to put command, except that the source localsrc is deleted after it's copied.

将本地的文件上传到hdfs上，并删除本地的文件

 

mv

Usage: hdfs dfs -mv URI [URI ...] <dest>

Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across file systems is not permitted.

从文件夹A移到文件夹B中

Example:

hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2

 

put

Usage: hdfs dfs -put <localsrc> ... <dst>

Example:

hdfs dfs -put /hadoop.txt /test

rm

Usage: hdfs dfs -rm [-f] [-r|-R] [-skipTrash] URI [URI ...]

Delete files specified as args.

Options:

The -f option will not display a diagnostic message or modify the exit status to reflect an error if the file does not exist.

删除文件

The -R option deletes the directory and any content under it recursively.

递归删除文件夹的内容

The -r option is equivalent to -R.

递归删除文件夹的内容

The -skipTrash option will bypass trash, if enabled, and delete the specified file(s) immediately. This can be useful when it is necessary to delete files from an over-quota directory.

 

rmr

Usage: hdfs dfs -rmr [-skipTrash] URI [URI ...]

Recursive version of delete.

Note: This command is deprecated. Instead use hdfs dfs -rm -r

 

test

Usage: hdfs dfs -test -[ezd] URI

Options:

The -e option will check to see if the file exists, returning 0 if true.

是否存在文件，一般用于自己管控HDFS

The -z option will check to see if the file is zero length, returning 0 if true.

The -d option will check to see if the path is directory, returning 0 if true.

Example:

$HADOOP_HOME/bin/hadoop fs -test -e /t_phone_contacts_top
if [ $? -eq 0 ] ;then
        ###$HADOOP_HOME/bin/hadoop fs -rmr /t_phone_contacts_top
        echo "/t_phone_contacts_top is exist,delete.... "
else
        echo "Error! /t_phone_contacts_top is not exist"
fi

HDFS dfsadmin命令
hdfs dfsadmin -setQuota 10 /lisi    //限制文件中文件的数量

hdfs dfsadmin -clrQuota  /lisi  //清除文件数量的限制

hdfs dfsadmin -setSpaceQuota 4k /lisi    //限制文件中文件的大小

hdfs dfsadmin -clrSpaceQuota  /lisi  //清除文件大小的限制

hdfs dfsadmin -safemode  设置hadoop集群为安全模式，在安全模式下只能读不能写，在hadoop集群启动的时候，有30秒是安全模式的