hadoop基准測试

_写測试hadoop jarhadoop-0.20.2-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000

----- TestDFSIO ----- : write

           Date & time: Fri Jul 24 14:24:36 CST 2015

       Number of files: 10

 Total MBytes processed: 10000

     Throughput mb/sec: 24.302163378583963

 Average IO rate mb/sec: 24.46268653869629

  IOrate std deviation: 2.0279575539782315

    Test exec time sec: 72.853

_读測试hadoop jarhadoop-0.20.2-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000

 

 ----- TestDFSIO ----- : read

           Date & time: Fri Jul 24 14:27:11 CST 2015

       Number of files: 10

 Total MBytes processed: 10000

     Throughput mb/sec: 80.73566336457803

 Average IO rate mb/sec: 106.7965087890625

  IOrate std deviation: 70.18198197030486

    Test exec time sec: 46.772

 

清除数据

[root@ZhejiangYiwuF1210 hadoop]# hadoop jarhadoop-0.20.2-test.jar TestDFSIO -clean 

 

_排序測试

1.在各节点上分别执行2个map任务。每一个节点产生1GB大小的随机二进制数据，并输出到文件夹/examples/terasort-input

hadoop jar hadoop-0.20.2-examples.jar teragen 10000000/examples/terasort-input

   Launched map tasks=2

 FileSystemCounters

   HDFS_BYTES_WRITTEN=1000000000

 Map-Reduce Framework

   Map input records=10000000

   Spilled Records=0

   Map input bytes=10000000

   Map output records=10000000

MapReduce（map:2个,reduce:0个）

2.排序。并将结果输出到文件夹/examples/terasort-output

#hadoop jar hadoop-0.20.2-examples.jar terasort /examples/terasort-input/examples/terasort-output

MapReduce（map:16个,reduce:1个）

 

3.检查是否已经排好序

# hadoop jar hadoop-0.20.2-examples.jarteravalidate /examples/terasort-output /examples/terasort-validate

MapReduce（map:1个,reduce:1个）

 

 

_{Namenode的负载測试}

# hadoop jar hadoop-0.20.2-test.jarnnbench -operation create_write -maps 12 -reduces 6 -blockSize 1 -bytesToWrite0 -numberOfFiles 1000 -replicationFactorPerFile 3 -readFileAfterOpen true-baseDir /benchmarks/NNBench-`hostname -s`

-------------- NNBench -------------- :

                          

Version:NameNode Benchmark 0.4

Date& time: 2015-07-24 14:52:04,989

 

TestOperation: create_write

Starttime: 2015-07-24 14:48:46,912

Maps torun: 12

Reducesto run: 6

BlockSize (bytes): 1

Bytes towrite: 0

Bytesper checksum: 1

Numberof files: 1000

Replicationfactor: 3

Successfulfile operations: 0

MapReduce（map:12个,reduce:6个）

_{MapReduce连续性測试}

mrbench会多次反复执行一个小作业，用于检查在机群上小作业的执行是否可反复以及执行是否高效

执行一个小作业10次

hadoop jar hadoop-0.20.2-test.jar mrbench -numRuns 10

DataLines       Maps   Reduces AvgTime (milliseconds)

1               2       1      29637

以上结果表示平均作业完毕时间是29637(milliseconds)

 

_{文件系统一致性的分布式检查}

hadoop jarhadoop-0.20.2-test.jar DistributedFSCheck

 

----- DistributedFSCheck ----- :

               Date & time: Fri Jul 2415:04:18 CST 2015

   Total number of blocks: 58

   Total number of  files: 3171

Number of corrupted blocks: 0

 Number of corrupted files: 0

MapReduce（map:2个,reduce:1个）