RabbitMQ in Action (2): Running and administering Rabbit
Server management
the Erlang node and the Erlang application
Starting nodes
multiple Erlang applications can run inside the same node
an application on node asparagus can call functions in applications running on node artichoke as though those functions were local.
Also, if an application crashes for any reason (say, RabbitMQ crashing) the Erlang node will automatically attempt to restart the application
when we talk about RabbitMQ nodes, we’re referring to the RabbitMQ application and the Erlang node it runs on.
root@OpenstackIcehouse2:~# ps aux | grep rabbit
rabbitmq 2021 0.0 0.0 7488 312 ? S 19:18 0:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 2377 0.0 0.0 4444 648 ? S 19:18 0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 2393 1.1 2.0 1161984 81856 ? Sl 19:18 1:40 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@OpenstackIcehouse2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2"
rabbitmq 2740 0.0 0.0 7456 420 ? Ss 19:19 0:00 inet_gethost 4
rabbitmq 2741 0.0 0.0 9552 624 ? S 19:19 0:00 inet_gethost 4
epmd:
- The Erlang endpoint mapper daemon
- Allows distributed Erlang applications to find one another
- Becomes important if we want to do things like RabbitMQ clustering
rabbitmq-server
- RabbitMQ start-up script
beam.smp
- The SMP Erlang VM
- Started by the erl command in rabbitmq-server
- erl is essentially a front-end to beam
- This is the thing that’s actually running RabbitMQ
inet_gethost
- Started by beam to do name service lookups
service rabbitmq-server start
Stopping nodes
./sbin/rabbitmqctl stop
root@OpenstackIcehouse2:/var/log/rabbitmq# rabbitmqctl status
Status of node rabbit@OpenstackIcehouse2 ...
[{pid,6398},
{running_applications,[{rabbit,"RabbitMQ","3.2.4"},
{os_mon,"CPO CXC 138 46","2.2.14"},
{mnesia,"MNESIA CXC 138 12","4.11"},
{xmerl,"XML parser","1.3.5"},
{sasl,"SASL CXC 138 11","2.3.4"},
{stdlib,"ERTS CXC 138 10","1.19.4"},
{kernel,"ERTS CXC 138 10","2.16.4"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
{memory,[{total,97816240},
{connection_procs,1142376},
{queue_procs,492920},
{plugins,0},
{other_proc,13534496},
{mnesia,134496},
{mgmt_db,0},
{msg_index,46936},
{other_ets,807528},
{binary,60441128},
{code,16522377},
{atom,594537},
{other_system,4099446}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,1658186956},
{disk_free_limit,50000000},
{disk_free,194296217600},
{file_descriptors,[{total_limit,924},
{total_used,26},
{sockets_limit,829},
{sockets_used,24}]},
{processes,[{limit,1048576},{used,362}]},
{run_queue,0},
{uptime,359}]
...done.
Stopping and restarting the application: what’s the difference?
sometimes you just want to restart the RabbitMQ application and keep the Erlang node running.
For clustering, it’s required.
Since rabbitmq-server starts both the node and the application, it preconfigures the RabbitMQ application for standalone mode.
To add the node to an existing cluster, what you need to do is stop the application and reset the node to a pristine state so it can be prepared for clustering.
If you were to use ./rabbitmqctl stop you’d shutdown both the application and the node, forcing you to start both in standalone mode again via ./rabbitmq-server.
./rabbitmqctl stop_app
Rabbit configuration files
/etc/rabbitmq/rabbitmq.config
1) [
2) {mnesia, [{dump_log_write_threshold, 1000}]},
3) {rabbit, [{vm_memory_high_watermark, 0.4}]}
4) ].
each Erlang application gets its own hashtable for its configuration options
mnesia specifies configuration options for the Mnesia database (Mnesia is what RabbitMQ uses for storing exchange and queue metadata).
rabbit specifies RabbitMQ-specific configuration options.
{[option_name], [option_value]}
Mnesia configuration options
dump_log_write_threshold:
- Integer
- 100
- How often to flush/dump the contents of the append-only log into the actual database files. It’s specified in terms of how many entries must be in the log before a dump operation occurs. Setting this to a higher number can reduce I/O load and increase performance for persistent messages.
Rabbit configuration options:
tcp_listeners:
- Array of {"ip_address", port_number}
- [{"0.0.0.0", 5672},]
- Defines which IP addresses and ports RabbitMQ should listen on for non-SSL-encrypted traffic.
ssl_listeners:
- Array of {"ip_address", port_number}
- Defines which IP addresses and ports RabbitMQ should listen on for SSL-encrypted traffic.
ssl_options:
- Array of {"key", value}
- Specifies SSL-related options. Valid options are cacertfile (CA certificate file), certfile (server certificate file), keyfile (server key file) and fail_if_no_peer_cert (require client to have a valid certificate: True/False).
vm_memory_high_watermark:
- Decimal percentage
- 0.4
- Controls how much memory RAM RabbitMQ is allowed to consume. It’s specified in terms of a decimal number representing the percentage of installed memory Rabbit is allowed to use (0.4 = 40%).
msg_store_file_size_limit:
- Integer (bytes)
- 16777216
- The maximum size of the message store DB before RabbitMQ garbage collects the contents of the store.
queue_index_max_journal_entries:
- Integer
- 262144
- The maximum number of entries in the message store journal before they’re flushed into the message store DB and committed.
Asking permission
access control lists
a single user can be granted permissions across multiple vhosts.
Managing users
$ ./rabbitmqctl add_user cashing-tier cashMe1
Creating user "cashing-tier" ...
...done.
$ ./rabbitmqctl delete_user cashing-tier
Deleting user "cashing-tier" ...
...done.
when you delete a user, any access control entries referencing that user will be deleted automatically from the Rabbit permissions database.
$ ./rabbitmqctl list_users
Listing users ...
cashing-tier
guest
...done.
$ ./rabbitmqctl change_password cashing-tier compl3xPassword
Changing password for user "cashing-tier" ...
...done.
Rabbit’s permissions system
- Read—Any operation related to consuming messages, including “purging” an entire queue (also required for binding operations to succeed)
- Write—Publishing messages (also required for binding operations to succeed)
- Configure—Creation and deletion of queues and exchanges
An access control entry consists of four parts:
- The user being granted access.
- The vhost on which the permissions apply.
- The combination of read/write/configure permissions to grant.
- The permission scope—whether the permissions apply only to client-named queues/exchanges, server-named queues/exchanges, or both.
$ ./rabbitmqctl set_permissions -p sycamore \
cashing-tier ".*" ".*" ".*"
Setting permissions for user "cashing-tier" in vhost "sycamore" ...
...done.
- -p sycamore—This tells set_permissions which vhost the entry should apply to.
- cashing-tier—The user being granted the permissions.
- ".*" ".*" ".*"—These are the granted permissions. The values map to configure, write, and read respectively.
".*" means match any queue or exchange name
- you want to grant cashing-tier access to the oak vhost.
- You want to allow the user to be able to execute a read command against any queue or exchange, while restricting writes only to queues or exchanges whose names starts with checks-.
- Also, you want to prevent configure operations entirely.
$ ./rabbitmqctl set_permissions -p oak \
-s all cashing-tier "" "checks-.*" ".*"
Setting permissions for user "cashing-tier" in vhost "oak" ...
...done.
$ ./rabbitmqctl clear_permissions -p oak cashing-tier
Clearing permissions for user "cashing-tier" in vhost "oak" ...
...done.
Checking up
$ ./rabbitmqctl list_queues -p sycamore
$ ./rabbitmqctl list_queues name messages consumers memory
Listing queues ...
msg-inbox-logs 0 2 34632
msg-inbox-errors 0 1 34632
all-logs 3 0 43664
...done.
$ ./rabbitmqctl list_exchanges
Listing exchanges ...
logs-exchange topic
amq.rabbitmq.log topic
amq.match headers
amq.headers headers
amq.topic topic
amq.direct direct
amq.fanout fanout
direct
...done.
$ ./rabbitmqctl list_bindings
Listing bindings ...
all-logs all-logs []
msg-inbox-errors msg-inbox-errors []
msg-inbox-logs msg-inbox-logs []
logs-exchange all-logs # []
logs-exchange msg-inbox-logs *.msg-inbox []
logs-exchange msg-inbox-errors error.msg-inbox []
...done.
Understanding RabbitMQ’s logs
READING THE LOGS ON THE FILESYSTEM
LOG_BASE=/var/log/rabbitmq，Inside that folder RabbitMQ will create two log files: RABBITMQ_NODENAME-sasl.log and RABBITMQ_NODENAME.log, where RABBITMQ_NODENAME will be something like _rabbit@localhost_ or just rabbit depending on how you configured your system.
SASL (System Application Support Libraries) is a set of libraries that are part of the Erlang-OTP distribution.
They help the developer to have a set of standards when developing their Erlang apps. One of those is the logging format. So, when RabbitMQ logs Erlang-related information, it’ll go to the rabbit-sasl.log files.
ROTATING THE LOGS
$ ./rabbitmqctl rotate_logs suffix
ACCESSING THE LOGS IN REAL TIME VIA AMQP
when you were listing exchanges using rabbitmqctl you spotted an exchange called amq.rabbitmq.log whose type is topic. RabbitMQ will publish its logs to that exchange using the severity level as a routing key—you’ll get error, warning, and info.
Fixing a bad Rabbit: troubleshooting
ERLANG COOKIES
One common error that you can get when you start working with RabbitMQ is badrpc,nodedown. It usually happens when you try to use the rabbitmqctl command
rabbitmqctl will start up an Erlang node, and from there it’ll try to communicate with the RabbitMQ node using the Erlang distribution system. For this to work, you need two things: the proper Erlang cookie and the proper node name.
An Erlang cookie acts as a secret token that two Erlang nodes exchange in order to authenticate themselves.
rabbitmq使用下面路径下的cookie
openstack@OpenstackIcehouse2:/var/lib/rabbitmq$ ls -a
. .. .erlang.cookie mnesia
In order for rabbitmqctl to communicate with the RabbitMQ node, it needs to share the same cookie.
ERLANG NODES
When you start an Erlang node, you can give it two mutually exclusive options, name and sname, which will specify the node name.
That node name can be long or short, hence the s in sname.
If you start your node with a long name, it will get something like rabbit@hostname.network.tld. If you use short names, you’ll see something similar to this: rabbit@hostname. The latter is the default way of starting RabbitMQ.
MNESIA AND HOST NAMES
RabbitMQ uses Mnesia to store information about queues, exchanges, bindings, and so on. One of the things that RabbitMQ does at startup time
is launch the Mnesia database.
Mnesia creates a database schema based on the machine hostname. If you list the contents of the MNESIA_BASE folder, you’ll see a folder called rabbit@hostname. If hostname changes due to some network reconfiguration, then Mnesia won’t be able to load the old schema.
openstack@OpenstackIcehouse2:~$ ps aux | grep rabbit
rabbitmq 1883 0.0 0.0 7488 312 ? S May28 0:07 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
opensta+ 3775 0.0 0.0 11744 916 pts/9 R+ 22:58 0:00 grep --color=auto rabbit
rabbitmq 18927 0.0 0.0 4444 652 ? S 13:42 0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 18935 1.2 1.9 1183552 80416 ? Sl 13:42 7:13 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@OpenstackIcehouse2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2"
rabbitmq 19061 0.0 0.0 7456 420 ? Ss 13:42 0:00 inet_gethost 4
rabbitmq 19062 0.0 0.0 9552 636 ? S 13:42 0:00 inet_gethost 4
openstack@OpenstackIcehouse2:/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2$ ls
cluster_nodes.config rabbit_durable_route.DCD
DECISION_TAB.LOG rabbit_runtime_parameters.DCD
LATEST.LOG rabbit_serial
msg_store_persistent rabbit_user.DCD
msg_store_transient rabbit_user_permission.DCD
nodes_running_at_shutdown rabbit_vhost.DCD
queues schema.DAT
rabbit_durable_exchange.DCD schema_version
rabbit_durable_queue.DCD
ERLANG TROUBLESHOOTING TIPS
start your Erlang node using test as node name.
openstack@OpenstackIcehouse2:~$ erl -sname test
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]
Eshell V5.10.4 (abort with ^G)
(test@OpenstackIcehouse2)1>
Erlang Read Eval Print Loop
find out your node name (输入node命令的时候，前有空格后有点)：
(test@OpenstackIcehouse2)1> node().
test@OpenstackIcehouse2
let’s check which other nodes are running on your machine:
(test@OpenstackIcehouse2)2> net_adm:names().
{ok,[{"rabbit",59939},{"test",40695}]}
There you called the names function from the net_adm module. As you can see, RabbitMQ is running on your machine, using rabbit as the node name and 59106 as the port.
that’s not the port you use to connect to RabbitMQ from an AMQP client.
the Erlang Port Mapper Daemon (epmd). Whenever you start a distributed Erlang node, it’ll register with the epmd process, giving it the address and port assigned to it by the OS kernel. Then when another Erlang node comes to life, it’ll do the same. Finally if the latter wants to connect to your first node, it’ll go through epmd to obtain the other node address.
root@OpenstackIcehouse2:~# netstat -nap | grep beam
tcp 0 0 0.0.0.0:40695 0.0.0.0:* LISTEN 3887/beam.smp
tcp 0 0 0.0.0.0:59939 0.0.0.0:* LISTEN 18935/beam.smp
tcp6 0 0 :::5672 :::* LISTEN 18935/beam.smp
前两个是epmd的port，后一个才是rabbitmq的port
Now that you know that you can see the RabbitMQ node, let’s try to establish a connection to it:
(test@OpenstackIcehouse2)2> net_adm:ping('rabbit@OpenstackIcehouse2').
pong
If the answer is pong, then you succeeded; if you get pang as reply, that means you couldn’t connect to the other node. Keep in mind that for all this to work you have to share the same Erlang cookie.
Let’s confirm that you’re connected to the rabbit node:
(test@OpenstackIcehouse2)3> nodes().
[rabbit@OpenstackIcehouse2]
Finally, let’s execute a function on the remote rabbit node:
(test@OpenstackIcehouse2)4> rpc:call('rabbit@OpenstackIcehouse2', erlang, system_info, [process_count]).
443
(test@OpenstackIcehouse2)7> rpc:call('rabbit@OpenstackIcehouse2', mnesia, info, []).
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
rabbit_user_permission: with 1 records occupying 331 words of mem
rabbit_topic_trie_edge: with 22 records occupying 1372 words of mem
rabbit_queue : with 40 records occupying 1519 words of mem
rabbit_semi_durable_route: with 0 records occupying 89 words of mem
schema : with 20 records occupying 2707 words of mem
rabbit_exchange_serial: with 0 records occupying 299 words of mem
rabbit_route : with 80 records occupying 3726 words of mem
rabbit_exchange: with 29 records occupying 1190 words of mem
rabbit_vhost : with 1 records occupying 310 words of mem
mirrored_sup_childspec: with 0 records occupying 89 words of mem
rabbit_listener: with 1 records occupying 355 words of mem
gm_group : with 0 records occupying 299 words of mem
rabbit_topic_trie_binding: with 21 records occupying 998 words of mem
rabbit_reverse_route: with 80 records occupying 3726 words of mem
rabbit_durable_route: with 0 records occupying 299 words of mem
rabbit_user : with 1 records occupying 318 words of mem
rabbit_topic_trie_node: with 25 records occupying 838 words of mem
rabbit_durable_queue: with 0 records occupying 299 words of mem
rabbit_durable_exchange: with 8 records occupying 538 words of mem
rabbit_runtime_parameters: with 0 records occupying 299 words of mem
===> System info in version "4.11", debug level = none <===
opt_disc. Directory "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2" is used.
use fallback at restart = false
running db nodes = [rabbit@OpenstackIcehouse2]
stopped db nodes = []
master node tables = []
remote = []
ram_copies = [gm_group,mirrored_sup_childspec,rabbit_exchange,
rabbit_exchange_serial,rabbit_listener,rabbit_queue,
rabbit_reverse_route,rabbit_route,
rabbit_semi_durable_route,rabbit_topic_trie_binding,
rabbit_topic_trie_edge,rabbit_topic_trie_node]
disc_copies = [rabbit_durable_exchange,rabbit_durable_queue,
rabbit_durable_route,rabbit_runtime_parameters,
rabbit_user,rabbit_user_permission,rabbit_vhost,schema]
disc_only_copies = []
[{rabbit@OpenstackIcehouse2,disc_copies}] = [rabbit_runtime_parameters,
rabbit_durable_exchange,
rabbit_durable_queue,rabbit_user,
rabbit_durable_route,
rabbit_vhost,schema,
rabbit_user_permission]
[{rabbit@OpenstackIcehouse2,ram_copies}] = [rabbit_topic_trie_node,
rabbit_reverse_route,
rabbit_topic_trie_binding,
gm_group,rabbit_listener,
mirrored_sup_childspec,
rabbit_exchange,rabbit_route,
rabbit_exchange_serial,
rabbit_semi_durable_route,
rabbit_queue,
rabbit_topic_trie_edge]
331 transactions committed, 0 aborted, 2 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
ok
(test@OpenstackIcehouse2)8> q().
ok
