multi-node cluster (linux)

echo $PATH　　source /etc/environment 　　export JAVA_HOME=${JAVA_HOME}　　rmp -Uvh　　tar -xvf

ssh　　ssh-keygen　　hostname　　id　　whoami　　df -h .　　ln -s　　gvfs-tree　　/etc/init.d/ssh restart　　　

cut -d ":" -f 1 /etc/passwd　　compgen -u/-g　

some important files for system configuration or else : 

/usr /opt /boot /net /etc /root /sbin /bin /initrd

 

problem shooting

  • index of jdk rpm
  • open a connection to your authentication agent　　eval `ssh-agent -s`　ssh-add
  • ssh localhost problem(vm to localhost): 　　1the username(master to slave) and hostname   2hosts

ssh & sshd:　　

1- ssh : The command we use to connect to remote machines - the client. 
2- sshd : The daemon that is running on the server and allows clients to connect to the server.
ssh is pre-enabled on Linux, but in order to start sshd daemon, we need to install ssh first. 

The ECDSA host key for localhost has changed, and the key for the corresponding IP address 192.168..is unchanged. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time.

/usr/sbin/sshd -t 

echo $?

ssh-add -D　　delete all keys

mac: ssh localhost/***@localhost password :*　　root@localhost permission denied　　cannot ssh slave 

ubuntu: ssh ***@master *　　root@master permission denied　　hduser@slave denied　　

So,for master side can ssh by *** not root.(*** have root privilege in slave while a common user in master.)

 After ssh-keygen -R hostname, for slave side can ssh hostname by hduser. https://www.digitalocean.com/community/questions/warning-remote-host-identification-has-changed

ssh-copy-id -i $HOME/.ssh/id_rsa.pub hduser@slave

And my sshd_config :

PermitRootLogin without-password

StrictModes yes

RSAAuthentication yes

PubkeyAuthentication yes

then ,i set PermitRootLogin yes. Now, i can ssh root@slave.(ssh can connect by any user except s-to-m by root but with passphare)

  • allow root to max os　　By default, the root user is not enabled [in Mountain Lion].

https://superuser.com/questions/555810/how-do-i-ssh-login-into-my-mac-as-root

  •  ubuntu  login loop

https://askubuntu.com/questions/146137/login-screen-loops-unless-you-login-as-guest

(.profile .bashrc .xauthority .xsession-error  xorg ) i have tried to modify all these files following the instructions but failed

solution1:some answers didn't suit me. I have to reinstall xubuntu-desktop .

solution2: disable the guest session by modifying .profile ,get the result"fail to start session"

　　　　　/var/log/lightdm/lightdm.log file to get the debug info https://ubuntuforums.org/showthread.php?t=2226247

　　　　  change the .dmrc and set the session = ubuntu ,then reboot

  • change  autologin-user　

nano /etc/lightdm/lightdm.conf

/var/lib/AccountsService/users　　nano

/var/cache/lightdm/dmrc　　rm

After rebooting it seems nothing changed. just switch account.

permanently : /etc/sysctl.conf

Ubuntu is that using 0.0.0.0 for the various networking-related Hadoop configuration options will result in Hadoop binding to the IPv6 addresses of my Ubuntu box. 

  • export : not a valid identifier　　cannot have any blank space between = and the left of =
  • ssh without passphare

http://www.thecloudavenue.com/2012/01/how-to-setup-password-less-ssh-to.html

after ssh-copy-id, ssh slave still have to input password because ssh by root of slave.

so, i remove root privilege of ***.(id,w,who　　nano /etc/passwd)

 

  • multi-node cluster

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ 

1　　conf/masters(m only)　　defines on which machines Hadoop will start secondary NameNodes 

The primary NameNode and the JobTracker will always be the machines on which you run the bin/start-dfs.sh and bin/start-mapred.sh scripts

2　　bin/hadoop-daemon.sh start [namenode | secondarynamenode | datanode | jobtracker | tasktracker]

which will not take the “conf/masters“ and “conf/slaves“ files into account. and be deprecated

3　　 the machine on which bin/start-dfs.sh is run will become the primaryNameNode.

4　　secondary NameNode :

merges the fsimage and the edits log files periodically and keeps edits log size within a limit.

5　　conf/slaves(m only)　　helps you to make “full” cluster restarts easier

The conf/slaves file lists the hosts, one per line, where the Hadoop slave daemons (DataNodes and TaskTrackers) will be run. We want both the master box and the slave box to act as Hadoop slaves because we want both of them to store and process data.

used only by the scripts like bin/start-dfs.sh or bin/stop-dfs.sh

6　　dfs.replication

It defines how many machines a single file should be replicated to before it becomes available. If you set this to a value higher than the number of available slave nodes (more precisely, the number of DataNodes), you will start seeing a lot of “(Zero targets found, forbidden1.size=1)” type errors in the log files.

7　　“mapred.reduce.tasks“

As a rule of thumb, use num_tasktrackers * num_reduce_slots_per_tasktracker * 0.99.

If num_tasktrackers is small (as in the case of this tutorial), use(num_tasktrackers - 1) * num_reduce_slots_per_tasktracker.

8　　 Do not format a running cluster because this will erase all existing data in the HDFS filesytem!

9　　

 

  •  difference of `hdfs dfs` and `hadoop dfs` and `hadoop fs`

https://stackoverflow.com/questions/18142960/whats-the-difference-between-hadoop-fs-shell-commands-and-hdfs-dfs-shell-co

  • wget wrapper

https://superuser.com/questions/691612/does-wget-have-a-download-history

  • ubuntu用户的配置文件及密码（解释/etc/passwd /etc/shadow /etc/group 格式）

http://blog.sina.com.cn/s/blog_5edae1a101017gfn.html 

 

！！： take up each problem one at a time

 

posted on 2017-09-21 02:08  satyrs  阅读(318)  评论(0)    收藏  举报

导航