第一章：大数据 の Linux 基础 [更新中]
本课主题
- Linux 休系结构图
- Linux 系统启动的顺序
- Linux 查看内存和 CPU 指令
- 环境变量加载顺序
- Linux 内存结构
Linux 休系结构图
Linux 大致分为三个层次，第一层是就是 用户空间层，就是离我们最近的层，它一般有Shell和应用程序，大数据基乎所有的应用软件都在用户空间层，另外就是核心层，它是 Linux 的内核，它负责与硬件进行交互，为系统的用户空间提供服务。为了不让用户空间直接访问内核的地址空间，它做了限制，加了一层 System Call，防止系统当机。可以通过查看。
cat /usr/include/bits/syscall.h
Linux 系统的启动顺序
- 加载BIOS，执行BIOS内置程序
- 读取 Master Boot Record (MBR) 中 Boot Loader 中的引导信息
- 加载内核 Kernel boot 到内存中
- 内核开始执行 /sbin/init，并加载 /etc/inittab，然后执行 rc.sysinit 进行初始化，它分别7个启动级别，从 0 到 6，分别表示操作系统的7个状况：0-关机；1-单用户模式；2-没有网络的多用户模式；3-全功能的多用户模式；4-没有用；5-X11；6-表示当前启动状态是重启；比如说你想关机，你可以输入 init 0
/etc/inittab# Default runlevel. The runlevels used are: # 0 - halt (Do NOT set initdefault to this) # 1 - Single user mode # 2 - Multiuser, without NFS (The same as 3, if you do not have networking) # 3 - Full multiuser mode # 4 - unused # 5 - X11 # 6 - reboot (Do NOT set initdefault to this) # id:5:initdefault:
- 启动核心的外挂模块 /etc/modules.conf
- 按照启动级别 (服务器默认是 5) 执行 /etc/rc5.d/ 下的脚本
- 执行 /bin/login 程序
- 可以使用 chkconfig --list 来查看当前的整个程序启动的一个列表
chkconfig --list[root@localhost rc5.d]# chkconfig --list NetworkManager 0:off 1:off 2:on 3:on 4:on 5:on 6:off auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off blk-availability 0:off 1:on 2:on 3:on 4:on 5:on 6:off crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off dnsmasq 0:off 1:off 2:off 3:off 4:off 5:off 6:off firstboot 0:off 1:off 2:off 3:off 4:off 5:off 6:off haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off network 0:off 1:off 2:on 3:on 4:on 5:on 6:off ntpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off ntpdate 0:off 1:off 2:off 3:off 4:off 5:off 6:off postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off pppoe-server 0:off 1:off 2:off 3:off 4:off 5:off 6:off rdisc 0:off 1:off 2:off 3:off 4:off 5:off 6:off restorecond 0:off 1:off 2:off 3:off 4:off 5:off 6:off rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off saslauthd 0:off 1:off 2:off 3:off 4:off 5:off 6:off spice-vdagentd 0:off 1:off 2:off 3:off 4:off 5:on 6:off sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off svnserve 0:off 1:off 2:off 3:off 4:off 5:off 6:off udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off wdaemon 0:off 1:off 2:off 3:off 4:off 5:off 6:off winbind 0:off 1:off 2:off 3:off 4:off 5:off 6:off wpa_supplicant 0:off 1:off 2:off 3:off 4:off 5:off 6:off
chkconfig sshd --level 23 off[root@localhost rc5.d]# chkconfig --list | grep sshd sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@localhost rc5.d]# chkconfig sshd --level 23 off [root@localhost rc5.d]# chkconfig --list | grep sshd sshd 0:off 1:off 2:off 3:off 4:on 5:on 6:off
Linux 查看内存和 CPU 的基本操作指令
登录到操作系统之后查看当前 Linux 的版本
[root@localhost enmoedu]# cat /etc/issue CentOS release 6.8 (Final) Kernel \r on an \m [root@localhost enmoedu]# uname -a Linux localhost.localdomain 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux [root@localhost enmoedu]# cat /proc/version Lin
然后查看系统的 CPU、内存和磁盘
- df -lh，它把整个磁盘分为两个空间，一个是 sda1
磁盘空间[root@localhost enmoedu]# df -lh Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_deepalley-lv_root 23G 2.3G 19G 11% / tmpfs 491M 80K 491M 1% /dev/shm /dev/sda1 477M 28M 424M 7% /boot /dev/sr0 71M 71M 0 100% /media/VMware Tools
-
free -m，显示基本的内存使用情况，swap 是虚拟内存的使用情况
内存空间[root@localhost enmoedu]# free -m total used free shared buffers cached Mem: 980 465 515 2 11 176 -/+ buffers/cache: 276 703 Swap: 1983 0 1983
- 也可以到 /proc/meminfo 中查看操作系纪的运行信息
/proc/meminfo[root@localhost enmoedu]# cat /proc/meminfo | grep page Hugepagesize: 2048 kB
-
在 /proc/cpuinfo 中查看CPU的使用情况，有多少的虚拟核
CPU 情况[root@localhost enmoedu]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 61 model name : Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz stepping : 4 microcode : 33 cpu MHz : 2700.126 cache size : 3072 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb xsaveopt pln pts dtherm fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx bogomips : 5400.25 clflush size : 64 cache_alignment : 64 address sizes : 42 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 61 model name : Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz stepping : 4 microcode : 33 cpu MHz : 2700.126 cache size : 3072 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb xsaveopt pln pts dtherm fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx bogomips : 5400.25 clflush size : 64 cache_alignment : 64 address sizes : 42 bits physical, 48 bits virtual power management:
环境变量加载顺序
在 Linux 系统下会有以下的环境变量，分别是
/etc/profile #全区环境变量，每个用户第一次登录时设置 ~/.bash_profile #用户级环境变量，每个用户第一次登录时设置 ~/.bash_login ~/.profile ~/.bashrc #用户级环境变量，用户登录，打开新时设置，每登录一次就会加载一次环境变量 /etc/bashrc ~/.bash_logout #用户级环境变量，退出时执行
进一步测试环境变量的顺序，我们可以在不同的环境变量文件中输入以下语句：
echo "/etc/profile" >> ~/enmoedu/start_seq.txt; date >> ~/enmoedu/start_seq.txt; chown -R enmoedu:enmoedu ~/enmoedu/start_seq.txt; echo ".bash_profile" >> ~/enmoedu/start_seq.txt; date >> ~/enmoedu/start_seq.txt; chown -R enmoedu:enmoedu ~/enmoedu/start_seq.txt; echo ".bashrc" >> ~/enmoedu/start_seq.txt; date >> ~/enmoedu/start_seq.txt; chown -R enmoedu:enmoedu ~/enmoedu/start_seq.txt;
[更新中...]
总结一下，当启动主机的服务器的时候，优先会加载全区级别的 /etc/profile 文件，其次加载用户级别的 ~/.bash_profile 文件，然后在每一次打开终端窗口时都会加载用户级别的 ~/.bashrc 文件。
NTP 网络时间协议
[admin@elephant Asia]$ ls -l /etc/localtime
lrwxrwxrwx. 1 root root 36 Feb 17 00:27 /etc/localtime -> ../usr/share/zoneinfo/Asia/Hong_Kong
在大数据的集群中有很多台不同的主机，要确保集群中的每台主机的时间是一直的，我们可以从集群中挑出一台，这一台主机可以是集群中的一个节点，或者是外部的节点，所有其他的节点都基于同一台主机来确保时间是一直的，把这个节点当做时间服务器！那台机器如果能上网的，可以基于网络上的原子钟，如果不能上网的话可以基于本地时间。这里我们用到了一个叫 NTP 协议。
查看当前主机是不是安装了 NTP 这个包：rpm -qa | grep ntp
[admin@elephant ~]$ rpm -qa | grep ntp fontpackages-filesystem-1.44-8.el7.noarch ntpdate-4.2.6p5-25.el7.centos.x86_64 ntp-4.2.6p5-25.el7.centos.x86_64 python-ntplib-0.3.2-1.el7.noarch
再查看 NTP 的服务有没有启动：service ntp status
[admin@elephant ~]$ service ntpd status Redirecting to /bin/systemctl status ntpd.service ● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled) Active: inactive (dead) [admin@elephant ~]$ service ntpd start Redirecting to /bin/systemctl start ntpd.service ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Authenticating as: admin Password: ==== AUTHENTICATION COMPLETE === [admin@elephant ~]$ service ntpd status Redirecting to /bin/systemctl status ntpd.service ● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2017-04-24 15:16:40 HKT; 3s ago Process: 3998 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 3999 (ntpd) CGroup: /system.slice/ntpd.service └─3999 /usr/sbin/ntpd -u ntp:ntp -g Apr 24 15:16:40 elephant ntpd[3999]: Listen normally on 2 lo 127.0.0.1 UDP 123 Apr 24 15:16:40 elephant ntpd[3999]: Listen normally on 3 ens33 192.168.80.142 UDP 123 Apr 24 15:16:40 elephant ntpd[3999]: Listen normally on 4 virbr0 192.168.122.1 UDP 123 Apr 24 15:16:40 elephant ntpd[3999]: Listen normally on 5 lo ::1 UDP 123 Apr 24 15:16:40 elephant ntpd[3999]: Listen normally on 6 ens33 fe80::e79e:e14c:4194:cfd6 UDP 123 Apr 24 15:16:40 elephant ntpd[3999]: Listening on routing socket on fd #23 for interface updates Apr 24 15:16:40 elephant ntpd[3999]: 0.0.0.0 c016 06 restart Apr 24 15:16:40 elephant ntpd[3999]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM Apr 24 15:16:40 elephant ntpd[3999]: 0.0.0.0 c011 01 freq_not_set Apr 24 15:16:41 elephant ntpd[3999]: 0.0.0.0 c614 04 freq_mode
然后改 /etc/ntp.conf 文件把主机时间 e.g. elephant server 指向本地物理时钟，其他的服务器 monkey server指向那台时间服务器 elephant server
driftfile /var/lib/ntp/drift # Permit time synchronization with our time source, but do not # permit the source to query or modify the service on this system. restrict default nomodify notrap nopeer noquery # Permit all access over the loopback interface. This could # be tightened as well, but to do so would effect some of # the administrative functions. restrict 127.0.0.1 restrict ::1 # Hosts on local network are less restricted. restrict 192.168.80.142 mask 255.255.255.0 nomodify notrap # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). server 0.centos.pool.ntp.org iburst server 1.centos.pool.ntp.org iburst server 2.centos.pool.ntp.org iburst server 3.centos.pool.ntp.org iburst #broadcast 192.168.1.255 autokey # broadcast server #broadcastclient # broadcast client #broadcast 224.0.1.1 autokey # multicast server #multicastclient 224.0.1.1 # multicast client #manycastserver 239.255.254.254 # manycast server #manycastclient 239.255.254.254 autokey # manycast client # Enable public key cryptography. #crypto includefile /etc/ntp/crypto/pw # Key file containing the keys and key identifiers used when operating # with symmetric key cryptography. keys /etc/ntp/keys server 127.127.1.0 # Specify the key identifiers which are trusted. #trustedkey 4 8 42 # Specify the key identifier to use with the ntpdc utility. #requestkey 8 # Specify the key identifier to use with the ntpq utility. #controlkey 8 # Enable writing of statistics records. #statistics clockstats cryptostats loopstats peerstats # Disable the monitoring facility to prevent amplification attacks using ntpdc # monlist command when default restrict does not include the noquery flag. See # CVE-2013-5211 for more details. # Note: Monitoring will not be disabled with the limited restriction flag. disable monitor
[admin@elephant ~]$ ntpdate -d elephant 24 Apr 15:36:24 ntpdate[41312]: ntpdate 4.2.6p5@1.2349-o Mon Nov 14 18:25:09 UTC 2016 (1) Looking for host elephant and service ntp host found : elephant transmit(fe80::e79e:e14c:4194:cfd6) receive(fe80::e79e:e14c:4194:cfd6) transmit(192.168.80.142) receive(192.168.80.142) transmit(192.168.122.1) receive(192.168.122.1) transmit(fe80::e79e:e14c:4194:cfd6) receive(fe80::e79e:e14c:4194:cfd6) transmit(192.168.80.142) receive(192.168.80.142) transmit(192.168.122.1) receive(192.168.122.1) transmit(fe80::e79e:e14c:4194:cfd6) receive(fe80::e79e:e14c:4194:cfd6) transmit(192.168.80.142) receive(192.168.80.142) transmit(192.168.122.1) receive(192.168.122.1) transmit(fe80::e79e:e14c:4194:cfd6) receive(fe80::e79e:e14c:4194:cfd6) transmit(192.168.80.142) receive(192.168.80.142) transmit(192.168.122.1) receive(192.168.122.1) server fe80::e79e:e14c:4194:cfd6, port 123 stratum 3, precision -24, leap 00, trust 000 refid [fe80::e79e:e14c:4194:cfd6], delay 0.02571, dispersion 0.00000 transmitted 4, in filter 4 reference time: dca8277e.946a679d Mon, Apr 24 2017 15:27:58.579 originate timestamp: dca82984.05c29784 Mon, Apr 24 2017 15:36:36.022 transmit timestamp: dca82984.05bb0060 Mon, Apr 24 2017 15:36:36.022 filter delay: 0.02576 0.02577 0.02582 0.02571 0.00000 0.00000 0.00000 0.00000 filter offset: -0.00001 -0.00002 -0.00004 0.000007 0.000000 0.000000 0.000000 0.000000 delay 0.02571, dispersion 0.00000 offset 0.000007 server 192.168.80.142, port 123 stratum 3, precision -24, leap 00, trust 000 refid [192.168.80.142], delay 0.02567, dispersion 0.00002 transmitted 4, in filter 4 reference time: dca8277e.946a679d Mon, Apr 24 2017 15:27:58.579 originate timestamp: dca82984.38f0e040 Mon, Apr 24 2017 15:36:36.222 transmit timestamp: dca82984.38e959d3 Mon, Apr 24 2017 15:36:36.222 filter delay: 0.02580 0.02567 0.02579 0.02577 0.00000 0.00000 0.00000 0.00000 filter offset: -0.00001 0.000000 -0.00004 -0.00003 0.000000 0.000000 0.000000 0.000000 delay 0.02567, dispersion 0.00002 offset 0.000000 server 192.168.122.1, port 123 stratum 3, precision -24, leap 00, trust 000 refid [192.168.122.1], delay 0.02574, dispersion 0.00000 transmitted 4, in filter 4 reference time: dca8277e.946a679d Mon, Apr 24 2017 15:27:58.579 originate timestamp: dca82984.6c02ed61 Mon, Apr 24 2017 15:36:36.421 transmit timestamp: dca82984.6bfba740 Mon, Apr 24 2017 15:36:36.421 filter delay: 0.02579 0.02574 0.02576 0.02577 0.00000 0.00000 0.00000 0.00000 filter offset: -0.00003 -0.00003 -0.00003 -0.00003 0.000000 0.000000 0.000000 0.000000 delay 0.02574, dispersion 0.00000 offset -0.000031 24 Apr 15:36:36 ntpdate[41312]: adjust time server 192.168.80.142 offset 0.000000 sec
如果 NetworkManager 是启动的话，必须要通过图形化去修改。
[admin@elephant ~]$ service NetworkManager status Redirecting to /bin/systemctl status NetworkManager.service ● NetworkManager.service - Network Manager Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2017-04-24 14:51:35 HKT; 59min ago Docs: man:NetworkManager(8) Main PID: 745 (NetworkManager) CGroup: /system.slice/NetworkManager.service ├─ 745 /usr/sbin/NetworkManager --no-daemon └─3526 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-ens33.pid -lf /var/lib/NetworkManager/dhclie... Apr 24 15:38:41 elephant dhclient[3526]: DHCPACK from 192.168.80.254 (xid=0x20960047) Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5503] dhcp4 (ens33): address 192.168.80.142 Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5507] dhcp4 (ens33): plen 24 (255.255.255.0) Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5507] dhcp4 (ens33): gateway 192.168.80.2 Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5507] dhcp4 (ens33): server identifier 192.168.80.254 Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5508] dhcp4 (ens33): lease time 1800 Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5508] dhcp4 (ens33): nameserver '192.168.80.2' Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5508] dhcp4 (ens33): domain name 'localdomain' Apr 24 15:38:41 elephant NetworkManager[745]: <info> [1493019521.5508] dhcp4 (ens33): state changed bound -> bound Apr 24 15:38:41 elephant dhclient[3526]: bound to 192.168.80.142 -- renewal in 821 seconds.
搭建 Web 服务器
如果没有安装 httpd 服务，请先安装 sudo yum -y install httpd，然后再次查看 httpd 服务
[admin@elephant ~]$ rpm -qa | grep httpd httpd-tools-2.4.6-45.el7.centos.4.x86_64 httpd-2.4.6-45.el7.centos.4.x86_64
[admin@elephant ~]$ service httpd status Redirecting to /bin/systemctl status httpd.service ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled) Active: inactive (dead) Docs: man:httpd(8) man:apachectl(8) [admin@elephant ~]$ service httpd start Redirecting to /bin/systemctl start httpd.service [admin@elephant ~]$ service httpd status Redirecting to /bin/systemctl status httpd.service ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2017-04-24 16:19:51 HKT; 4s ago Docs: man:httpd(8) man:apachectl(8) Main PID: 42969 (httpd) Status: "Processing requests..." CGroup: /system.slice/httpd.service ├─42969 /usr/sbin/httpd -DFOREGROUND ├─42974 /usr/sbin/httpd -DFOREGROUND ├─42975 /usr/sbin/httpd -DFOREGROUND ├─42976 /usr/sbin/httpd -DFOREGROUND ├─42979 /usr/sbin/httpd -DFOREGROUND └─42980 /usr/sbin/httpd -DFOREGROUND Apr 24 16:19:51 elephant systemd[1]: Starting The Apache HTTP Server... Apr 24 16:19:51 elephant httpd[42969]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, usi... message Apr 24 16:19:51 elephant systemd[1]: Started The Apache HTTP Server. Hint: Some lines were ellipsized, use -l to show in full.
把 cloudera-cdh5.9 移动到 /var/www/html/ 文件夹里，再用网页打开 http://elephant/cloudera-cdh5.9/ 试试看能否可以查看 Web 服务！
hostname # 查看当前的主机名称 sudo -s # 换到超级用户 id # 用 id 去查看 /etc/sysconfig/network # 这个文件是更改 HOSTNAME，改完之后重新启动 Network 服务 hostname elephant # 改完之后重新启动 hostname 服务 # 修改 ip 地址 ifconfig -a # IP 地址 netstat -rn # Geteway 网关 cat /etc/resolv.conf # DNS 地址的信息
文件系统
文件系统是一个管理磁盘的软件，它作为一个块设备，什么是块设备呢，在每一个文件前面都有一个标指符，管理磁盘有一个自己最小的 IO 单位，文件系统把磁盘有序的格式化。
- 查看硬件设备的最小 IO 单位
查看一个 blockdev 的用量[root@localhost enmoedu]# blockdev --getss /dev/sda [root@localhost enmoedu]# fdisk -l Disk /dev/sda: 26.8 GB, 26843545600 bytes heads, 63 sectors/track, 3263 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00056b3f Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 3264 25701376 8e Linux LVM Disk /dev/mapper/vg_deepalley-lv_root: 24.2 GB, 24234688512 bytes heads, 63 sectors/track, 2946 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_deepalley-lv_swap: 2080 MB, 2080374784 bytes heads, 63 sectors/track, 252 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
-
查看文件系统的最小 IO 单位，就是Block，所谓最小 IO 单位就是一次查询时取数据的大小，必需先有磁盘然后再从磁盘上格式化成文件系统，tune2fs 是文件系统的概念然后 /dev/sda1 是分区的概念。
tune2fs -l /dev/sda1[root@localhost enmoedu]# tune2fs -l /dev/sda1 tune2fs 1.41.12 (17-May-2010) Filesystem volume name: <none> Last mounted on: /boot Filesystem UUID: 1c293ff2-4299-4aa7-9bff-165c389d4967 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 128016 Block count: 512000 Reserved block count: 25600 Free blocks: 459455 Free inodes: 127978 First block: 1 Block size: 1024 Fragment size: 1024 Reserved GDT blocks: 256 Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 2032 Inode blocks per group: 254 Flex block group size: 16 Filesystem created: Tue Dec 27 00:07:15 2016 Last mount time: Tue Apr 25 03:00:13 2017 Last write time: Tue Apr 25 03:00:13 2017 Mount count: 9 Maximum mount count: -1 Last checked: Tue Dec 27 00:07:15 2016 Check interval: 0 (<none>) Lifetime writes: 67 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 3eeda192-f1f4-4da8-a332-32e1d6ee09fc Journal backup: inode blocks
- 重点查看 BlockSize 的大小，一个基本的 Block 大小是 1024K
Block size 的大小[root@localhost enmoedu]# tune2fs -l /dev/sda1 | grep Block Block count: 512000 Block size: 1024 Blocks per group: 8192
什么是 Inode 节点
整个 /dev/sda1 的文件系统会分成不同的块组，然后格式化成 EXT2 文件格式，这个块组的第一个块叫 superblock 超级块占用了 1K 的空间，在中间有一个叫 Inode 的节点叫 Inode Table，这个节点是存储元数据信息的。Inode Table 分为 15 个指针，前12个是直接指针，后3个是间接指针，分别是1级间接指针、2级间接指针和3级间接指针，所以说是它的文件系统的大小是通过 Inode 节点的数据结构来。1个指针指向1个数据块，1个地址指向 1个 Block，有12个直接指针，所以有 12K，1级间接指针有 256K 个块，2 级间接指针有 256K 存储指针就是 256K x 256K = 6636K，1个文件对应1个 Inode Table，1个 Inode Table 有很多指针总共有 16G。
Linux 的操作系统是由文件系统句柄限制的，这种限制有时候是指整个操作系统能够打开的文件总数，有时候也指单个进程打开文件的数量，这里的 nofile 指的就是单个进程能够打开的文件句柄数，什么是文件句柄数，它是整行的一串数，这串数字代表的是打开一个文件对于进程来说是一个唯一的标示，Linux 操作系统默认的限制不是特别高，它有两个限制来，一个是软限制、一个是硬限制，软限制是指一些警告。
[root@localhost dev]# ls -la total 4 drwxr-xr-x. 21 root root 3840 Apr 25 03:00 . dr-xr-xr-x. 22 root root 4096 Apr 25 03:00 .. drwxr-xr-x. 2 root root 40 Apr 25 03:00 .mdadm drwxr-xr-x. 6 root root 140 Apr 25 03:00 .udev lrwxrwxrwx. 1 root root 13 Apr 25 03:00 MAKEDEV -> /sbin/MAKEDEV crw-rw----. 1 root video 10, 175 Apr 25 03:00 agpgart drwxr-xr-x. 2 root root 640 Apr 25 03:00 block drwxr-xr-x. 2 root root 80 Apr 25 03:00 bsg crw-------. 1 root root 10, 234 Apr 25 03:00 btrfs-control drwxr-xr-x. 3 root root 60 Apr 25 03:00 bus lrwxrwxrwx. 1 root root 3 Apr 25 03:00 cdrom -> sr0 lrwxrwxrwx. 1 root root 3 Apr 25 03:00 cdrw -> sr0 drwxr-xr-x. 2 root root 3040 Apr 25 03:00 char crw-------. 1 root root 5, 1 Apr 25 03:00 console lrwxrwxrwx. 1 root root 11 Apr 25 03:00 core -> /proc/kcore drwxr-xr-x. 4 root root 80 Apr 25 03:00 cpu crw-rw----. 1 root root 10, 61 Apr 25 03:00 cpu_dma_latency crw-rw----. 1 root root 10, 62 Apr 25 03:00 crash drwxr-xr-x. 6 root root 120 Apr 25 03:00 disk brw-rw----. 1 root disk 253, 0 Apr 25 03:00 dm-0 brw-rw----. 1 root disk 253, 1 Apr 25 03:00 dm-1 crw-rw----+ 1 root audio 14, 9 Apr 25 03:00 dmmidi drwxr-xr-x. 2 root root 100 Apr 25 03:00 dri lrwxrwxrwx. 1 root root 3 Apr 25 03:00 dvd -> sr0 lrwxrwxrwx. 1 root root 3 Apr 25 03:00 dvdrw -> sr0 lrwxrwxrwx. 1 root root 3 Apr 25 03:00 fb -> fb0 crw-rw----. 1 root video 29, 0 Apr 25 03:00 fb0 lrwxrwxrwx. 1 root root 13 Apr 25 03:00 fd -> /proc/self/fd crw-rw-rw-. 1 root root 1, 7 Apr 25 03:00 full crw-rw-rw-. 1 root root 10, 229 Apr 25 03:00 fuse crw-rw----. 1 root root 248, 0 Apr 25 03:00 hidraw0 crw-rw----. 1 root root 10, 228 Apr 25 03:00 hpet drwxr-xr-x. 2 root root 40 Apr 25 03:00 hugepages crw-------. 1 root root 229, 0 Apr 25 03:00 hvc0 drwxr-xr-x. 4 root root 260 Apr 25 03:00 input crw-rw----. 1 root root 1, 11 Apr 25 03:00 kmsg srw-rw-rw-. 1 root root 0 Apr 25 03:00 log brw-rw----. 1 root disk 7, 0 Apr 25 03:00 loop0 brw-rw----. 1 root disk 7, 1 Apr 25 03:00 loop1 brw-rw----. 1 root disk 7, 2 Apr 25 03:00 loop2 brw-rw----. 1 root disk 7, 3 Apr 25 03:00 loop3 brw-rw----. 1 root disk 7, 4 Apr 25 03:00 loop4 brw-rw----. 1 root disk 7, 5 Apr 25 03:00 loop5 brw-rw----. 1 root disk 7, 6 Apr 25 03:00 loop6 brw-rw----. 1 root disk 7, 7 Apr 25 03:00 loop7 crw-rw----. 1 root lp 6, 0 Apr 25 03:00 lp0 crw-rw----. 1 root lp 6, 1 Apr 25 03:00 lp1 crw-rw----. 1 root lp 6, 2 Apr 25 03:00 lp2 crw-rw----. 1 root lp 6, 3 Apr 25 03:00 lp3 drwxr-xr-x. 2 root root 100 Apr 25 03:00 mapper crw-rw----. 1 root root 10, 227 Apr 25 03:00 mcelog crw-r-----. 1 root kmem 1, 1 Apr 25 03:00 mem crw-rw----+ 1 root audio 14, 2 Apr 25 03:00 midi drwxr-xr-x. 2 root root 60 Apr 25 03:00 net crw-rw----. 1 root root 10, 60 Apr 25 03:00 network_latency crw-rw----. 1 root root 10, 59 Apr 25 03:00 network_throughput crw-rw-rw-. 1 root root 1, 3 Apr 25 03:00 null crw-r-----. 1 root kmem 10, 144 Apr 25 03:00 nvram crw-rw----. 1 root root 1, 12 Apr 25 03:00 oldmem crw-r-----. 1 root kmem 1, 4 Apr 25 03:00 port crw-------. 1 root root 108, 0 Apr 25 03:00 ppp crw-rw-rw-. 1 root tty 5, 2 Apr 25 03:51 ptmx drwxr-xr-x. 2 root root 0 Apr 25 03:00 pts brw-rw----. 1 root disk 1, 0 Apr 25 03:00 ram0 brw-rw----. 1 root disk 1, 1 Apr 25 03:00 ram1 brw-rw----. 1 root disk 1, 10 Apr 25 03:00 ram10 brw-rw----. 1 root disk 1, 11 Apr 25 03:00 ram11 brw-rw----. 1 root disk 1, 12 Apr 25 03:00 ram12 brw-rw----. 1 root disk 1, 13 Apr 25 03:00 ram13 brw-rw----. 1 root disk 1, 14 Apr 25 03:00 ram14 brw-rw----. 1 root disk 1, 15 Apr 25 03:00 ram15 brw-rw----. 1 root disk 1, 2 Apr 25 03:00 ram2 brw-rw----. 1 root disk 1, 3 Apr 25 03:00 ram3 brw-rw----. 1 root disk 1, 4 Apr 25 03:00 ram4 brw-rw----. 1 root disk 1, 5 Apr 25 03:00 ram5 brw-rw----. 1 root disk 1, 6 Apr 25 03:00 ram6 brw-rw----. 1 root disk 1, 7 Apr 25 03:00 ram7 brw-rw----. 1 root disk 1, 8 Apr 25 03:00 ram8 brw-rw----. 1 root disk 1, 9 Apr 25 03:00 ram9 crw-rw-rw-. 1 root root 1, 8 Apr 25 03:00 random drwxr-xr-x. 2 root root 60 Apr 25 03:00 raw lrwxrwxrwx. 1 root root 4 Apr 25 03:00 root -> dm-0 lrwxrwxrwx. 1 root root 4 Apr 25 03:00 rtc -> rtc0 crw-rw----. 1 root root 253, 0 Apr 25 03:00 rtc0 lrwxrwxrwx. 1 root root 3 Apr 25 03:00 scd0 -> sr0 brw-rw----. 1 root disk 8, 0 Apr 25 03:00 sda brw-rw----. 1 root disk 8, 1 Apr 25 03:00 sda1 brw-rw----. 1 root disk 8, 2 Apr 25 03:00 sda2 crw-rw----. 1 root cdrom 21, 0 Apr 25 03:00 sg0 crw-rw----. 1 root disk 21, 1 Apr 25 03:00 sg1 drwxrwxrwt. 2 root root 100 Apr 25 03:00 shm crw-rw----. 1 root root 10, 231 Apr 25 03:00 snapshot drwxr-xr-x. 3 root root 200 Apr 25 03:00 snd brw-rw----+ 1 root cdrom 11, 0 Apr 25 03:00 sr0 lrwxrwxrwx. 1 root root 15 Apr 25 03:00 stderr -> /proc/self/fd/2 lrwxrwxrwx. 1 root root 15 Apr 25 03:00 stdin -> /proc/self/fd/0 lrwxrwxrwx. 1 root root 15 Apr 25 03:00 stdout -> /proc/self/fd/1 lrwxrwxrwx. 1 root root 4 Apr 25 03:00 systty -> tty0 crw-rw-rw-. 1 root tty 5, 0 Apr 25 03:00 tty crw--w----. 1 root tty 4, 0 Apr 25 03:00 tty0 crw--w----. 1 root tty 4, 1 Apr 25 03:00 tty1 crw--w----. 1 root tty 4, 10 Apr 25 03:00 tty10 crw--w----. 1 root tty 4, 11 Apr 25 03:00 tty11 crw--w----. 1 root tty 4, 12 Apr 25 03:00 tty12 crw--w----. 1 root tty 4, 13 Apr 25 03:00 tty13 crw--w----. 1 root tty 4, 14 Apr 25 03:00 tty14 crw--w----. 1 root tty 4, 15 Apr 25 03:00 tty15 crw--w----. 1 root tty 4, 16 Apr 25 03:00 tty16 crw--w----. 1 root tty 4, 17 Apr 25 03:00 tty17 crw--w----. 1 root tty 4, 18 Apr 25 03:00 tty18 crw--w----. 1 root tty 4, 19 Apr 25 03:00 tty19 crw-------. 1 root root 4, 2 Apr 25 03:00 tty2 crw--w----. 1 root tty 4, 20 Apr 25 03:00 tty20 crw--w----. 1 root tty 4, 21 Apr 25 03:00 tty21 crw--w----. 1 root tty 4, 22 Apr 25 03:00 tty22 crw--w----. 1 root tty 4, 23 Apr 25 03:00 tty23 crw--w----. 1 root tty 4, 24 Apr 25 03:00 tty24 crw--w----. 1 root tty 4, 25 Apr 25 03:00 tty25 crw--w----. 1 root tty 4, 26 Apr 25 03:00 tty26 crw--w----. 1 root tty 4, 27 Apr 25 03:00 tty27 crw--w----. 1 root tty 4, 28 Apr 25 03:00 tty28 crw--w----. 1 root tty 4, 29 Apr 25 03:00 tty29 crw-------. 1 root root 4, 3 Apr 25 03:00 tty3 crw--w----. 1 root tty 4, 30 Apr 25 03:00 tty30 crw--w----. 1 root tty 4, 31 Apr 25 03:00 tty31 crw--w----. 1 root tty 4, 32 Apr 25 03:00 tty32 crw--w----. 1 root tty 4, 33 Apr 25 03:00 tty33 crw--w----. 1 root tty 4, 34 Apr 25 03:00 tty34 crw--w----. 1 root tty 4, 35 Apr 25 03:00 tty35 crw--w----. 1 root tty 4, 36 Apr 25 03:00 tty36 crw--w----. 1 root tty 4, 37 Apr 25 03:00 tty37 crw--w----. 1 root tty 4, 38 Apr 25 03:00 tty38 crw--w----. 1 root tty 4, 39 Apr 25 03:00 tty39 crw-------. 1 root root 4, 4 Apr 25 03:00 tty4 crw--w----. 1 root tty 4, 40 Apr 25 03:00 tty40 crw--w----. 1 root tty 4, 41 Apr 25 03:00 tty41 crw--w----. 1 root tty 4, 42 Apr 25 03:00 tty42 crw--w----. 1 root tty 4, 43 Apr 25 03:00 tty43 crw--w----. 1 root tty 4, 44 Apr 25 03:00 tty44 crw--w----. 1 root tty 4, 45 Apr 25 03:00 tty45 crw--w----. 1 root tty 4, 46 Apr 25 03:00 tty46 crw--w----. 1 root tty 4, 47 Apr 25 03:00 tty47 crw--w----. 1 root tty 4, 48 Apr 25 03:00 tty48 crw--w----. 1 root tty 4, 49 Apr 25 03:00 tty49 crw-------. 1 root root 4, 5 Apr 25 03:00 tty5 crw--w----. 1 root tty 4, 50 Apr 25 03:00 tty50 crw--w----. 1 root tty 4, 51 Apr 25 03:00 tty51 crw--w----. 1 root tty 4, 52 Apr 25 03:00 tty52 crw--w----. 1 root tty 4, 53 Apr 25 03:00 tty53 crw--w----. 1 root tty 4, 54 Apr 25 03:00 tty54 crw--w----. 1 root tty 4, 55 Apr 25 03:00 tty55 crw--w----. 1 root tty 4, 56 Apr 25 03:00 tty56 crw--w----. 1 root tty 4, 57 Apr 25 03:00 tty57 crw--w----. 1 root tty 4, 58 Apr 25 03:00 tty58 crw--w----. 1 root tty 4, 59 Apr 25 03:00 tty59 crw-------. 1 root root 4, 6 Apr 25 03:00 tty6 crw--w----. 1 root tty 4, 60 Apr 25 03:00 tty60 crw--w----. 1 root tty 4, 61 Apr 25 03:00 tty61 crw--w----. 1 root tty 4, 62 Apr 25 03:00 tty62 crw--w----. 1 root tty 4, 63 Apr 25 03:00 tty63 crw--w----. 1 root tty 4, 7 Apr 25 03:00 tty7 crw--w----. 1 root tty 4, 8 Apr 25 03:00 tty8 crw--w----. 1 root tty 4, 9 Apr 25 03:00 tty9 crw-rw----. 1 root dialout 4, 64 Apr 25 03:00 ttyS0 crw-rw----. 1 root dialout 4, 65 Apr 25 03:00 ttyS1 crw-rw----. 1 root dialout 4, 66 Apr 25 03:00 ttyS2 crw-rw----. 1 root dialout 4, 67 Apr 25 03:00 ttyS3 crw-rw-rw-. 1 root root 1, 9 Apr 25 03:00 urandom crw-rw----. 1 root root 249, 0 Apr 25 03:00 usbmon0 crw-rw----. 1 root root 249, 1 Apr 25 03:00 usbmon1 crw-rw----. 1 root root 249, 2 Apr 25 03:00 usbmon2 drwxr-xr-x. 4 root root 80 Apr 25 03:00 v4l crw-rw----. 1 vcsa tty 7, 0 Apr 25 03:00 vcs crw-rw----. 1 vcsa tty 7, 1 Apr 25 03:00 vcs1 crw-rw----. 1 vcsa tty 7, 2 Apr 25 03:00 vcs2 crw-rw----. 1 vcsa tty 7, 3 Apr 25 03:00 vcs3 crw-rw----. 1 vcsa tty 7, 4 Apr 25 03:00 vcs4 crw-rw----. 1 vcsa tty 7, 5 Apr 25 03:00 vcs5 crw-rw----. 1 vcsa tty 7, 6 Apr 25 03:00 vcs6 crw-rw----. 1 vcsa tty 7, 128 Apr 25 03:00 vcsa crw-rw----. 1 vcsa tty 7, 129 Apr 25 03:00 vcsa1 crw-rw----. 1 vcsa tty 7, 130 Apr 25 03:00 vcsa2 crw-rw----. 1 vcsa tty 7, 131 Apr 25 03:00 vcsa3 crw-rw----. 1 vcsa tty 7, 132 Apr 25 03:00 vcsa4 crw-rw----. 1 vcsa tty 7, 133 Apr 25 03:00 vcsa5 crw-rw----. 1 vcsa tty 7, 134 Apr 25 03:00 vcsa6 drwxr-xr-x. 2 root root 80 Apr 25 03:00 vg_deepalley crw-rw----. 1 root root 10, 63 Apr 25 03:00 vga_arbiter crw-rw----+ 1 root video 81, 0 Apr 25 03:00 video0 crw-------. 1 root root 10, 57 Apr 25 03:00 vmci crw-rw-rw-. 1 root root 10, 56 Apr 25 03:00 vsock crw-rw-rw-. 1 root root 1, 5 Apr 25 03:00 zero
文件系统中有以下类型：
- 常文件 regular file
- 目录文件 directory file
- 字符设备 character
- 块设备 block
查看当前系统已经打开了的文件数量，有那几种：
[root@localhost ~]# lsof | more COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME init 1 root cwd DIR 253,0 4096 2 / init 1 root rtd DIR 253,0 4096 2 / init 1 root txt REG 253,0 150352 41 /sbin/init init 1 root mem REG 253,0 65960 393344 /lib64/libnss_files-2.12.so init 1 root mem REG 253,0 1923352 393328 /lib64/libc-2.12.so init 1 root mem REG 253,0 90880 392450 /lib64/libgcc_s-4.4.7-20120601.so.1 init 1 root mem REG 253,0 43944 393356 /lib64/librt-2.12.so init 1 root mem REG 253,0 142688 393352 /lib64/libpthread-2.12.so init 1 root mem REG 253,0 265728 393457 /lib64/libdbus-1.so.3.4.0 init 1 root mem REG 253,0 39896 393659 /lib64/libnih-dbus.so.1.0.0 init 1 root mem REG 253,0 101920 393661 /lib64/libnih.so.1.0.0 init 1 root mem REG 253,0 154664 393321 /lib64/ld-2.12.so init 1 root 0u CHR 1,3 0t0 4601 /dev/null init 1 root 1u CHR 1,3 0t0 4601 /dev/null init 1 root 2u CHR 1,3 0t0 4601 /dev/null init 1 root 3r FIFO 0,8 0t0 9742 pipe init 1 root 4w FIFO 0,8 0t0 9742 pipe init 1 root 5r DIR 0,10 0 1 inotify init 1 root 6r DIR 0,10 0 1 inotify init 1 root 7u unix 0xffff880039c23b40 0t0 9743 @/com/ubuntu/upstart init 1 root 9u unix 0xffff88003d4f9840 0t0 13862 socket
[root@localhost ~]# lsof | awk 'NR>1{print $5}' | sort | uniq -c | sort 1 pack 2 IPv6 6 IPv4 32 sock 74 unknown 232 CHR 239 FIFO 332 DIR 477 unix 3109 REG [root@localhost ~]#
修改 nofile 文件
查看文件数量的限制
[root@localhost ~]# unlimit -a bash: unlimit: command not found [root@localhost ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 3836 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 3836 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
文件限制最大能调到 1048566
[root@localhost ~]# ulimit -n 1048576 [root@localhost ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 3836 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 3836 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@localhost ~]# ulimit -n 1048577 bash: ulimit: open files: cannot modify limit: Operation not permitted
也可以查看那一个命令打开了一个文件
[root@localhost ~]# tail -f /etc/security/limits.conf #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 # End of file ... [root@localhost enmoedu]# lsof /etc/security/limits.conf COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tail 2768 root 3r REG 253,0 1835 1046659 /etc/security/limits.conf [root@localhost enmoedu]#
以上修改只针对当前登录，在下一次重启时 ulimit 会变会默认值，可以在 /etc/security/limits.conf 文件修改 ulimit 的限制让它永久生效，它是针对用户来进行限制的。
# - "soft" for enforcing the soft limits # - "hard" for enforcing hard limits # #<item> can be one of the following: # - core - limits the core file size (KB) # - data - max data size (KB) # - fsize - maximum filesize (KB) # - memlock - max locked-in-memory address space (KB) # - nofile - max number of open file descriptors # - rss - max resident set size (KB) # - stack - max stack size (KB) # - cpu - max CPU time (MIN) # - nproc - max number of processes # - as - address space limit (KB) # - maxlogins - max number of logins for this user # - maxsyslogins - max number of logins on the system # - priority - the priority to run user process with # - locks - max number of file locks the user can hold # - sigpending - max number of pending signals # - msgqueue - max memory used by POSIX message queues (bytes) # - nice - max nice priority allowed to raise to values: [-20, 19] # - rtprio - max realtime priority # #<domain> <type> <item> <value> # #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 # End of file
/etc/sysctl.conf 是针对整个统系的 ulimit 限制
cat << EOF > output.sh echo "hello" echo "world" EOF
[root@monkey enmoedu]# service iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] [root@monkey enmoedu]# chkconfig iptables off [root@monkey enmoedu]# vim /etc/selinux/config [root@monkey enmoedu]# setenforce 0
Linux 内存结构
每个内存单元都有一个地址，内存地址是从 0 开始的，CPU 通过地址找到内存单元
操作系统默认是 60 swap 值，swap=0 的意思是尽量不要做 swap，swap=100的意思是尽量去做 swap，在大数据的世界中，操作系统层面的配置要求这个是 0，PAGEIN PAGEOUT 也就是 swap 这个信息，代表的是你内存页面又交换到磁盘空间，然后把当前应用程序常用的数据从磁盘加载到内存里。应用程序在关机的时候是存储在磁盘上。
[root@elephant enmoedu]# sysctl -a | grep -i swap vm.swappiness = 60 [root@elephant enmoedu]# sysctl -w vm.swappiness=0 vm.swappiness = 0 [root@elephant enmoedu]# sysctl -a | grep -i swap vm.swappiness = 0 [root@elephant enmoedu]# cat /proc/sys/vm/swappiness 0
[root@elehpant bin]# readelf -h bash ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: EXEC (Executable file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x41b070 Start of program headers: 64 (bytes into file) Start of section headers: 904456 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 8 Size of section headers: 64 (bytes) Number of section headers: 28 Section header string table index: 27
[root@elehpant bin]# cat /proc/3229/maps 00400000-004d5000 r-xp 00000000 fd:00 1177348 /bin/bash 006d4000-006dd000 rw-p 000d4000 fd:00 1177348 /bin/bash 006dd000-006e3000 rw-p 00000000 00:00 0 01cb5000-01cf7000 rw-p 00000000 00:00 0 [heap] 7fbeb951c000-7fbeb9529000 r-xp 00000000 fd:00 393344 /lib64/libnss_files-2.12.so 7fbeb9529000-7fbeb9728000 ---p 0000d000 fd:00 393344 /lib64/libnss_files-2.12.so 7fbeb9728000-7fbeb9729000 r--p 0000c000 fd:00 393344 /lib64/libnss_files-2.12.so 7fbeb9729000-7fbeb972a000 rw-p 0000d000 fd:00 393344 /lib64/libnss_files-2.12.so 7fbeb972a000-7fbebf5bd000 r--p 00000000 fd:00 132261 /usr/lib/locale/locale-archive 7fbebf5bd000-7fbebf747000 r-xp 00000000 fd:00 393328 /lib64/libc-2.12.so 7fbebf747000-7fbebf947000 ---p 0018a000 fd:00 393328 /lib64/libc-2.12.so 7fbebf947000-7fbebf94b000 r--p 0018a000 fd:00 393328 /lib64/libc-2.12.so 7fbebf94b000-7fbebf94d000 rw-p 0018e000 fd:00 393328 /lib64/libc-2.12.so 7fbebf94d000-7fbebf951000 rw-p 00000000 00:00 0 7fbebf951000-7fbebf953000 r-xp 00000000 fd:00 393334 /lib64/libdl-2.12.so 7fbebf953000-7fbebfb53000 ---p 00002000 fd:00 393334 /lib64/libdl-2.12.so 7fbebfb53000-7fbebfb54000 r--p 00002000 fd:00 393334 /lib64/libdl-2.12.so 7fbebfb54000-7fbebfb55000 rw-p 00003000 fd:00 393334 /lib64/libdl-2.12.so 7fbebfb55000-7fbebfb72000 r-xp 00000000 fd:00 393370 /lib64/libtinfo.so.5.7 7fbebfb72000-7fbebfd71000 ---p 0001d000 fd:00 393370 /lib64/libtinfo.so.5.7 7fbebfd71000-7fbebfd75000 rw-p 0001c000 fd:00 393370 /lib64/libtinfo.so.5.7 7fbebfd75000-7fbebfd76000 rw-p 00000000 00:00 0 7fbebfd76000-7fbebfd96000 r-xp 00000000 fd:00 393321 /lib64/ld-2.12.so 7fbebff85000-7fbebff88000 rw-p 00000000 00:00 0 7fbebff8b000-7fbebff8d000 rw-p 00000000 00:00 0 7fbebff8d000-7fbebff94000 r--s 00000000 fd:00 262762 /usr/lib64/gconv/gconv-modules.cache 7fbebff94000-7fbebff95000 rw-p 00000000 00:00 0 7fbebff95000-7fbebff96000 r--p 0001f000 fd:00 393321 /lib64/ld-2.12.so 7fbebff96000-7fbebff97000 rw-p 00020000 fd:00 393321 /lib64/ld-2.12.so 7fbebff97000-7fbebff98000 rw-p 00000000 00:00 0 7ffe9e60b000-7ffe9e620000 rw-p 00000000 00:00 0 [stack] 7ffe9e698000-7ffe9e699000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
so 是共享库，共享库在物理内存是只存一份，然后共享到不同内存的虚拟内存空间。
- 虚拟内存管理可以控制物理内存的访问权限，物理内存是硬件，它不能控制访问权限；
- 虚拟内存管理容许每个进层有独立的地址空间，所谓独立就是指你不同进层中同一个虚拟内存地址的 MMU Insert 到不同的物理内存地址；
- 虚拟内存对放内存带来很大的方便；
- 借用磁盘空间来当做虚拟内存空间，让它成为虚拟内存空间的一部份。
JVM 内存
public class MyJava extends Thread { public void run() { for (int i = 0; i < 100; i++) { if ((i) % 10 == 0) { System.out.println("-------" + i + "--------"); } System.out.print(i); try { Thread.sleep(1000); System.out.println(" Sleep "+ i + " Seconds\n"); } catch (InterruptedException e) { e.printStackTrace(); } } } public static void main(String[] args) { new MyJava().start(); } }
[enmoedu@elephant ~]$ jstat -gc 3208 1000 10 S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000 512.0 512.0 0.0 0.0 4224.0 592.0 10496.0 0.0 21248.0 2395.4 0 0.000 0 0.000 0.000
[root@elehpant ~]# ipcs -a ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 0 enmoedu 600 393216 2 dest 0x00000000 32769 enmoedu 600 393216 2 dest 0x00000000 65538 enmoedu 600 393216 2 dest 0x00000000 98307 enmoedu 600 393216 2 dest 0x00000000 131076 enmoedu 600 393216 2 dest 0x00000000 163845 enmoedu 600 393216 2 dest 0x00000000 196614 enmoedu 600 393216 2 dest 0x00000000 229383 enmoedu 600 393216 2 dest 0x00000000 262152 enmoedu 600 393216 2 dest 0x00000000 294921 enmoedu 600 393216 2 dest 0x00000000 327690 enmoedu 600 393216 2 dest 0x00000000 360459 enmoedu 600 393216 2 dest ------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 0 root 600 1 0x00000000 65537 root 600 1 ------ Message Queues -------- key msqid owner perms used-bytes messages
MySQL
ACID，是指数据库管理系统（DBMS）在写入或更新资料的过程中，为保证事务（transaction）是正确可靠的，所必须具备的四个特性：原子性（atomicity，或称不可分割性） 、一致性（consistency）、隔离性（isolation，又称独立性）、持久性（durability）
缓冲区，数据写到缓冲区的目的是防止数据一下子写入数据库，这会对数据库的开销比较大。
[来至百度] 缓冲区的作用是为了解决速度不匹配的问题，高速的cpu与内存，内存与硬盘，cpu与io等速度不匹配的问题，而引入缓冲区，比如我们从磁盘里读取信息，我们先把读出的数据放在缓冲区，计算机再直接从缓冲区中读取数据，等缓冲区的数据读取完后再去磁盘中读取，这样就可以减少磁盘的读写次数，再加上计算机对缓冲区的操作大大快于对磁盘的操作，故应用缓冲区可大大提高计算机的运行速度。缓冲区就是一块内存区，它用在输入输出设备和CPU之间，用来缓存数据。它使得低速的输入输出设备和高速的CPU能够协调工作，避免低速的输入输出设备占用CPU。解放出CPU，使其能够高效率工作。
问题：缓冲和缓存有什么区别？
mysqld 有三层：连接层、SQL层和存储层，存储层分别也有关于内存、磁盘和网络的存储。在连接层，它只关心的是通信协议、线程和用户名验证的问题；在 SQL 层，它关心的是解析器，你的SQL语法对不对、还会有优化器内部优化SQL，最后是一列系的查询动作、查询缓存和日志。在存储层，它关心的是 InnoDB、MyISAM、Memory 和 NBD。
大数据文件系统是数据和元数据是分开管理的。
总结
[更新中...]
参考资料
[1] 鸟哥的 Linux － 第七章、Linux 磁碟与档案系统管理
[2] 大数据微职位第一周
[3] 有事务处理的NoSQL数据库
[4] 怎么打造一个分布式数据库
[5] 百度百科：缓冲
[6] 百度百科：缓存
[8] 手动搭建 Cloudera 环境
[9] 修改时钟和时区
[10] 鸟哥的 Linux － 第十五章、時間伺服器： NTP 伺服器
