随笔分类 -  Nutch & Hadoop

As of the official Nutch 1.3 release the source code architecture has been greatly simplified to allow us to run Nutch in one of two modes; namely local and deploy. By default, Nutch no longer comes with a Hadoop distribution, however when run in local mode e.g. running Nutch in a single process on one machine, then we use Hadoop as a dependency. This may suit you fine if you have a small site to crawl and index, but most people choose Nutch because of its capability to run on in deploy mode, within a Hadoop cluster. This gives you the benefit of a distributed file system (HDFS) and MapReduce processing style.
about hadoop-eclipse-plugin used by IDE
摘要：Apache Hadoop Development Tools (HDT) is still in development phase. So, no official distribution of Hadoop 2.2.0 Eclipse Plugin is available now. But... 阅读全文
posted @ 2014-11-03 14:27 小开风 阅读(320) 评论(0) 推荐(0)
File System Shell
摘要：OverviewappendToFilecatchgrpchmodchowncopyFromLocalcopyToLocalcountcpdudusexpungegetgetfaclgetmergelslsrmkdirmoveFromLocalmoveToLocalmvputrmrmrsetfacl... 阅读全文
posted @ 2014-08-06 08:04 小开风 阅读(249) 评论(0) 推荐(0)
User Commands
摘要：archiveCreates a hadoop archive【v.存档; n.档案文件;档案室; 】. More information can be found at Hadoop Archives.distcpCopy file or directories recursively. More... 阅读全文
posted @ 2014-08-06 07:55 小开风 阅读(132) 评论(0) 推荐(0)
building hadoop2.4.1 on centos7[在centos7上面构建hadoop2.4.1]
摘要：本文介绍在centos7上面通过hadoop2.4.1源码构建hadoop distribution 版本，即hadoop的运行版本。 为何要自己building，而不用Apache的distribution 【bin】版本，因为hadoop涉及到Linux系统的底层实现，如： hado... 阅读全文
posted @ 2014-07-30 22:43 小开风 阅读(442) 评论(0) 推荐(0)
Hadoop MapReduce Next Generation - Setting up a Single Node Cluster
摘要：Hadoop MapReduce Next Generation - Setting up a Single Node Cluster.PurposeThis document describes how to set up and configure a single-node Hadoop in... 阅读全文
posted @ 2014-07-30 22:04 小开风 阅读(383) 评论(0) 推荐(0)
hadoop namenode多次格式化后，导致datanode启动不了
摘要：jpshadoop namenode -formatdfs directory : /home/hadoop/dfs --data --current/VERSION#Wed Jul 30 20:41:03 CST 2014storageID=DS-ab96ad90-7352-4cd5-a0de... 阅读全文
posted @ 2014-07-30 20:53 小开风 阅读(642) 评论(1) 推荐(0)
about Q&A in installing linux[centos6,7]
摘要：keywords:grub1,grub2,gnome,kde,question describe:install centos7 by U disk，出现问题，解决办法：install centos7test this media &install centos7troubles shooting当... 阅读全文
posted @ 2014-07-25 11:02 小开风 阅读(654) 评论(0) 推荐(0)
prepare—Article【准备篇】之SSH_tool#PuTTY
摘要：第一：下载PuTTY： url ： http://www.openssh.com/下载界面：安装后：详解以上命令①②PuTTYgen is a key generator. It generates pairs of public and private keys to be used with ... 阅读全文
posted @ 2014-06-24 14:07 小开风 阅读(241) 评论(0) 推荐(0)