Pacemaker inherits most of its logging setting from either CMAN or Corosync - depending on what its running on top of.
In order to avoid spamming syslog, Pacemaker only logs a summary of its actions (NOTICE and above) to syslog.
If the level of detail in syslog is insufficient, you should enable a cluster log file. Normally one is configured by default and it contains everything except debug and trace messages.
To find the location of this file, either examine your CMAN (
cluster.conf) or Corosync (
corosync.conf) configuration file or look for syslog entries such as:
pacemakerd: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
If you do not see a line like this, either update the cluster configuration or set
crm_reportalso knows how to find all the Pacemaker related logs and blackbox files
If the level of detail in the cluster log file is still insufficient, or you simply wish to go blind, you can turn on debugging in Corosync/CMAN, or set
A minor advantage of setting
PCMK_debug is that the value can be a comma-separated list of processes which should produce debug logging instead of a global yes/no.
When an ERROR occurs
Pacemaker includes support for a blackbox.
When enabled, the blackbox contains a rolling buffer of all logs (not just those sent to syslog or a file) and is written to disk after a crash or assertion failure.
The blackbox recorder can be enabled by setting
/etc/sysconfig/pacemaker or at runtime by sending SIGUSR1. Eg.
killall -USR1 crmd
When enabled you’ll see a log such as:
crmd: notice: crm_enable_blackbox: Initiated blackbox recorder: /var/lib/pacemaker/blackbox/crmd-1811
If a crash occurs, the blackbox will be available at that location. To extract the contents, pass it to
Which produces output like:
Dumping the contents of /var/lib/pacemaker/blackbox/crmd-1811 [debug] shm size:5242880; real_size:5242880; rb->word_size:1310720 [debug] read total of: 5242892 Ringbuffer: ->NORMAL ->write_pt  ->read_pt  ->size [1310720 words] =>free [5220524 bytes] =>used [22352 bytes] ... trace May 19 23:20:55 gio_read_socket(368):0: 0x11ab920.5 1 (ref=1) trace May 19 23:20:55 pcmk_ipc_accept(458):0: Connection 0x11aee00 info May 19 23:20:55 crm_client_new(302):0: Connecting 0x11aee00 for uid=0 gid=0 pid=24425 id=0e943a2a-dd64-49bc-b9d5-10fa6c6cb1bd debug May 19 23:20:55 handle_new_connection(465):2147483648: IPC credentials authenticated (24414-24425-14) ... [debug] Free'ing ringbuffer: /dev/shm/qb-create_from_file-header
When an ERROR occurs you’ll also see the function and line number that produced it such as:
crmd: Problem detected at child_death_dispatch:872 (mainloop.c), please see /var/lib/pacemaker/blackbox/crmd-1811.1 for additional details crmd: Problem detected at main:94 (crmd.c), please see /var/lib/pacemaker/blackbox/crmd-1811.2 for additional details
Again, simply pass the files to
qb-blackbox to extract and query the contents.
Note the a counter is added to the end so as to avoid name collisions.
Diving into files and functions
In case you have not already guessed, all logs include the name of the function that generated them. So:
crmd: notice: crm_update_peer_state: cman_event_callback: Node corosync-host-1 - state is now lost (was member)
came from the function
To obtain more detail from that or any other function, you can set
/etc/sysconfig/pacemaker to a comma separated list of function names. Eg.
For a bigger stick, you may also activate trace logging for all the functions in a particular source file or files by setting
PCMK_trace_files as well.
These additional logs are sent to the cluster log file. Note that enabling tracing options also alters the output format.
crmd: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
the output includes file and line information:
crmd: ( cluster.c:215 ) notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
But wait there’s still more
Still need more detail? You’re in luck! The blackbox can be dumped at any time, not just when an error occurs.
First, make sure the blackbox is active (we’ll assume its the
crmd that needs to be debugged):
killall -USR1 crmd
Next, discard any previous contents by dumping them to disk
killall -TRAP crmd
now cause whatever condition you’re trying to debug, and send
-TRAP when you’re ready to see the result.
killall -TRAP crmd
You can now look for the result in syslog:
grep -e crm_write_blackbox: /var/log/messages
This will include a filename containing the trace logging:
crmd: notice: crm_write_blackbox: Blackbox dump requested, please see /var/lib/pacemaker/blackbox/crmd-1811.1 for contents crmd: notice: crm_write_blackbox: Blackbox dump requested, please see /var/lib/pacemaker/blackbox/crmd-1811.2 for contents
To extract the trace loging for our test, pass the most recent file to
At this point you’ll probably want to use grep :)