| <?xml version='1.0'?> |
| <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" |
| "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> |
| <!-- SPDX-License-Identifier: LGPL-2.1-or-later --> |
| |
| <refentry id="systemd-coredump" conditional='ENABLE_COREDUMP' |
| xmlns:xi="http://www.w3.org/2001/XInclude"> |
| |
| <refentryinfo> |
| <title>systemd-coredump</title> |
| <productname>systemd</productname> |
| </refentryinfo> |
| |
| <refmeta> |
| <refentrytitle>systemd-coredump</refentrytitle> |
| <manvolnum>8</manvolnum> |
| </refmeta> |
| |
| <refnamediv> |
| <refname>systemd-coredump</refname> |
| <refname>systemd-coredump.socket</refname> |
| <refname>systemd-coredump@.service</refname> |
| <refpurpose>Acquire, save and process core dumps</refpurpose> |
| </refnamediv> |
| |
| <refsynopsisdiv> |
| <para><filename>/usr/lib/systemd/systemd-coredump</filename></para> |
| <para><filename>/usr/lib/systemd/systemd-coredump</filename> <option>--backtrace</option></para> |
| <para><filename>systemd-coredump@.service</filename></para> |
| <para><filename>systemd-coredump.socket</filename></para> |
| </refsynopsisdiv> |
| |
| <refsect1> |
| <title>Description</title> |
| <para><filename>systemd-coredump@.service</filename> is a system service to process core dumps. It will |
| log a summary of the event to |
| <citerefentry><refentrytitle>systemd-journald.service</refentrytitle><manvolnum>8</manvolnum></citerefentry>, |
| including information about the process identifier, owner, the signal that killed the process, and the |
| stack trace if possible. It may also save the core dump for later processing. See the "Information about |
| the crashed process" section below.</para> |
| |
| <para>The behavior of a specific program upon reception of a signal is governed by a few |
| factors which are described in detail in |
| <citerefentry project='man-pages'><refentrytitle>core</refentrytitle><manvolnum>5</manvolnum></citerefentry>. |
| In particular, the core dump will only be processed when the related resource limits are sufficient. |
| </para> |
| |
| <para>Core dumps can be written to the journal or saved as a file. In both cases, they can be retrieved |
| for further processing, for example in |
| <citerefentry project='man-pages'><refentrytitle>gdb</refentrytitle><manvolnum>1</manvolnum></citerefentry>. |
| See <citerefentry><refentrytitle>coredumpctl</refentrytitle><manvolnum>1</manvolnum></citerefentry>, |
| in particular the <command>list</command> and <command>debug</command> verbs.</para> |
| |
| <para>By default, <command>systemd-coredump</command> will log the core dump to the journal, including a |
| backtrace if possible, and store the core dump (an image of the memory contents of the process) itself in |
| an external file in <filename>/var/lib/systemd/coredump</filename>. These core dumps are deleted after a |
| few days by default; see <filename>/usr/lib/tmpfiles.d/systemd.conf</filename> for details. Note that the |
| removal of core files from the file system and the purging of journal entries are independent, and the |
| core file may be present without the journal entry, and journal entries may point to since-removed core |
| files. Some metadata is attached to core files in the form of extended attributes, so the core files are |
| useful for some purposes even without the full metadata available in the journal entry.</para> |
| |
| <refsect2> |
| <title>Invocation of <command>systemd-coredump</command></title> |
| |
| <para>The <command>systemd-coredump</command> executable does the actual work. It is invoked twice: |
| once as the handler by the kernel, and the second time in the |
| <filename>systemd-coredump@.service</filename> to actually write the data to the journal and process |
| and save the core file.</para> |
| |
| <para>When the kernel invokes <command>systemd-coredump</command> to handle a core dump, it runs in |
| privileged mode, and will connect to the socket created by the |
| <filename>systemd-coredump.socket</filename> unit, which in turn will spawn an unprivileged |
| <filename>systemd-coredump@.service</filename> instance to process the core dump. Hence |
| <filename>systemd-coredump.socket</filename> and <filename>systemd-coredump@.service</filename> are |
| helper units which do the actual processing of core dumps and are subject to normal service |
| management.</para> |
| |
| <para>It is also possible to invoke <command>systemd-coredump</command> with |
| <option>--backtrace</option> option. In this case, <command>systemd-coredump</command> expects a |
| journal entry in the journal |
| <ulink url="https://systemd.io/JOURNAL_EXPORT_FORMATS#journal-export-format">Journal Export Format</ulink> |
| on standard input. The entry should contain a <varname>MESSAGE=</varname> field and any additional |
| metadata fields the caller deems reasonable. <command>systemd-coredump</command> will append additional |
| metadata fields in the same way it does for core dumps received from the kernel. In this mode, no core |
| dump is stored in the journal.</para> |
| </refsect2> |
| </refsect1> |
| |
| <refsect1> |
| <title>Configuration</title> |
| <para>For programs started by <command>systemd</command>, process resource limits can be set by directive |
| <varname>LimitCORE=</varname>, see |
| <citerefentry><refentrytitle>systemd.exec</refentrytitle><manvolnum>5</manvolnum></citerefentry>. |
| </para> |
| |
| <para>In order to be used by the kernel to handle core dumps, |
| <command>systemd-coredump</command> must be configured in |
| <citerefentry project='man-pages'><refentrytitle>sysctl</refentrytitle><manvolnum>8</manvolnum></citerefentry> |
| parameter <varname>kernel.core_pattern</varname>. The syntax of this parameter is explained in |
| <citerefentry project='man-pages'><refentrytitle>core</refentrytitle><manvolnum>5</manvolnum></citerefentry>. |
| systemd installs the file <filename>/usr/lib/sysctl.d/50-coredump.conf</filename> which configures |
| <varname>kernel.core_pattern</varname> accordingly. This file may be masked or overridden to use a different |
| setting following normal |
| <citerefentry><refentrytitle>sysctl.d</refentrytitle><manvolnum>5</manvolnum></citerefentry> |
| rules. If the sysctl configuration is modified, it must be updated in the kernel before it |
| takes effect, see |
| <citerefentry project='man-pages'><refentrytitle>sysctl</refentrytitle><manvolnum>8</manvolnum></citerefentry> |
| and |
| <citerefentry><refentrytitle>systemd-sysctl</refentrytitle><manvolnum>8</manvolnum></citerefentry>. |
| </para> |
| |
| <para>In order to be used in the <option>--backtrace</option> mode, an appropriate backtrace |
| handler must be installed on the sender side. For example, in case of |
| <citerefentry project='die-net'><refentrytitle>python</refentrytitle><manvolnum>1</manvolnum></citerefentry>, this |
| means a <varname>sys.excepthook</varname> must be installed, see |
| <ulink url="https://github.com/systemd/systemd-coredump-python">systemd-coredump-python</ulink>. |
| </para> |
| |
| <para>The behavior of <command>systemd-coredump</command> itself is configured through the configuration file |
| <filename>/etc/systemd/coredump.conf</filename> and corresponding snippets |
| <filename>/etc/systemd/coredump.conf.d/*.conf</filename>, see |
| <citerefentry><refentrytitle>coredump.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>. A new |
| instance of <command>systemd-coredump</command> is invoked upon receiving every core dump. Therefore, changes |
| in these files will take effect the next time a core dump is received.</para> |
| |
| <para>Resources used by core dump files are restricted in two ways. Parameters like maximum size of acquired |
| core dumps and files can be set in files <filename>/etc/systemd/coredump.conf</filename> and snippets mentioned |
| above. In addition the storage time of core dump files is restricted by <command>systemd-tmpfiles</command>, |
| corresponding settings are by default in <filename>/usr/lib/tmpfiles.d/systemd.conf</filename>. The default is |
| to delete core dumps after a few days; see the above file for details.</para> |
| |
| <refsect2> |
| <title>Disabling coredump processing</title> |
| |
| <para>To disable potentially resource-intensive processing by <command>systemd-coredump</command>, set |
| <programlisting>Storage=none |
| ProcessSizeMax=0</programlisting> in |
| <citerefentry><refentrytitle>coredump.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>. |
| </para> |
| </refsect2> |
| </refsect1> |
| |
| <refsect1> |
| <title>Information about the crashed process</title> |
| |
| <para><citerefentry><refentrytitle>coredumpctl</refentrytitle><manvolnum>1</manvolnum></citerefentry> can |
| be used to retrieve saved core dumps independently of their location, to display information, and to |
| process them e.g. by passing to the GNU debugger (gdb).</para> |
| |
| <para>Data stored in the journal can be also viewed with |
| <citerefentry><refentrytitle>journalctl</refentrytitle><manvolnum>1</manvolnum></citerefentry> as usual |
| (or from any other process, using the |
| <citerefentry><refentrytitle>sd-journal</refentrytitle><manvolnum>3</manvolnum></citerefentry> API). |
| The relevant messages have <constant>MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1</constant>:</para> |
| <programlisting>$ journalctl MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 -o verbose |
| … |
| MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 |
| COREDUMP_PID=552351 |
| COREDUMP_UID=1000 |
| COREDUMP_GID=1000 |
| COREDUMP_SIGNAL_NAME=SIGSEGV |
| COREDUMP_SIGNAL=11 |
| COREDUMP_TIMESTAMP=1614342930000000 |
| COREDUMP_COMM=Web Content |
| COREDUMP_EXE=/usr/lib64/firefox/firefox |
| COREDUMP_USER_UNIT=app-gnome-firefox-552136.scope |
| COREDUMP_CMDLINE=/usr/lib64/firefox/firefox -contentproc -childID 5 -isForBrowser … |
| COREDUMP_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/app-….scope |
| COREDUMP_FILENAME=/var/lib/systemd/coredump/core.Web….552351.….zst |
| … |
| </programlisting> |
| |
| <para>The following fields are saved (if known) with the journal entry</para> |
| |
| <variablelist class='journal-directives'> |
| <varlistentry> |
| <term><varname>COREDUMP_UID=</varname></term> |
| <term><varname>COREDUMP_PID=</varname></term> |
| <term><varname>COREDUMP_GID=</varname></term> |
| <listitem><para>The process number (PID), owner user number (UID), and group number (GID) of the |
| crashed process.</para> |
| |
| <para>When the crashed process was part of a container (or in a process or user namespace in |
| general), those are the values as seen <emphasis>outside</emphasis>, in the namespace where |
| <filename>systemd-coredump</filename> is running.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_TIMESTAMP=</varname></term> |
| <listitem><para>The time of the crash as reported by the kernel (in µs since the epoch).</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_RLIMIT=</varname></term> |
| <listitem><para>The core file size soft resource limit, see |
| <citerefentry project='man-pages'><refentrytitle>getrlimit</refentrytitle><manvolnum>2</manvolnum></citerefentry>. |
| </para></listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_UNIT=</varname></term> |
| <term><varname>COREDUMP_SLICE=</varname></term> |
| <listitem><para>The system unit and slice names.</para> |
| |
| <para>When the crashed process was in container, those are the units names |
| <emphasis>outside</emphasis>, in the main system manager.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_CGROUP=</varname></term> |
| <listitem><para>Control group information in the format used in |
| <filename>/proc/self/cgroup</filename>. On systems with the unified cgroup hierarchy, this is a |
| single path prefixed with <literal>0::</literal>, and multiple paths prefixed with controller numbers |
| on legacy systems.</para> |
| |
| <para>When the crashed process was in a container, this is the full path, as seen outside of the |
| container.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_OWNER_UID=</varname></term> |
| <term><varname>COREDUMP_USER_UNIT=</varname></term> |
| <listitem><para>The numerical UID of the user owning the login session or systemd user unit of the |
| crashed process, and the user manager unit. Both fields are only present for user processes. |
| </para> |
| |
| <para>When the crashed process was in container, those are the values <emphasis>outside</emphasis>, |
| in the main system.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_SIGNAL_NAME=</varname></term> |
| <term><varname>COREDUMP_SIGNAL=</varname></term> |
| |
| <listitem><para>The terminating signal name (with the <literal>SIG</literal> prefix |
| <footnote><para><citerefentry |
| project='man-pages'><refentrytitle>kill</refentrytitle><manvolnum>1</manvolnum></citerefentry> |
| expects signal names <emphasis>without</emphasis> the prefix; <citerefentry |
| project='man-pages'><refentrytitle>kill</refentrytitle><manvolnum>2</manvolnum></citerefentry> uses |
| the prefix; all systemd tools accept signal names both with and without the prefix. |
| </para></footnote>) and numerical value. (Both are included because signal numbers vary by |
| architecture.)</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_CWD=</varname></term> |
| <term><varname>COREDUMP_ROOT=</varname></term> |
| |
| <listitem><para>The current working directory and root directory of the crashed process.</para> |
| |
| <para>When the crashed process is in a container, those paths are relative to the root of the |
| container's mount namespace.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_OPEN_FDS=</varname></term> |
| |
| <listitem><para>Information about open file descriptors, in the following format:</para> |
| <programlisting><replaceable>fd</replaceable>:<replaceable>/path/to/file</replaceable> |
| pos: ... |
| flags: ... |
| ... |
| |
| <replaceable>fd</replaceable>:<replaceable>/path/to/file</replaceable> |
| pos: ... |
| flags: ... |
| ... |
| </programlisting> |
| |
| <para>The first line contains the file descriptor number <replaceable>fd</replaceable> and the path, |
| while subsequent lines show the contents of |
| <filename>/proc/<replaceable>pid</replaceable>/fdinfo/<replaceable>fd</replaceable></filename>. |
| </para></listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_EXE=</varname></term> |
| |
| <listitem><para>The destination of the <filename>/proc/<replaceable>pid</replaceable>/exe</filename> |
| symlink.</para> |
| |
| <para>When the crashed process is in a container, that path is relative to the root of the |
| container's mount namespace.</para></listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_COMM=</varname></term> |
| <term><varname>COREDUMP_PROC_STATUS=</varname></term> |
| <term><varname>COREDUMP_PROC_MAPS=</varname></term> |
| <term><varname>COREDUMP_PROC_LIMITS=</varname></term> |
| <term><varname>COREDUMP_PROC_MOUNTINFO=</varname></term> |
| <term><varname>COREDUMP_ENVIRON=</varname></term> |
| |
| <listitem><para>Fields that map the per-process entries in the <filename>/proc/</filename> |
| filesystem: <filename>/proc/<replaceable>pid</replaceable>/comm</filename> (the command name |
| associated with the process), <filename>/proc/<replaceable>pid</replaceable>/exe</filename> (the |
| filename of the executed command), <filename>/proc/<replaceable>pid</replaceable>/status</filename> |
| (various metadata about the process), <filename>/proc/<replaceable>pid</replaceable>/maps</filename> |
| (memory regions visible to the process and their access permissions), |
| <filename>/proc/<replaceable>pid</replaceable>/limits</filename> (the soft and hard resource limits), |
| <filename>/proc/<replaceable>pid</replaceable>/mountinfo</filename> (mount points in the process's |
| mount namespace), <filename>/proc/<replaceable>pid</replaceable>/environ</filename> |
| (the environment block of the crashed process).</para> |
| |
| <para>See |
| <citerefentry project='man-pages'><refentrytitle>proc</refentrytitle><manvolnum>5</manvolnum></citerefentry> |
| for more information.</para></listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_HOSTNAME=</varname></term> |
| |
| <listitem><para>The system hostname.</para> |
| |
| <para>When the crashed process was in container, this is the container hostname.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_CONTAINER_CMDLINE=</varname></term> |
| |
| <listitem><para>For processes running in a container, the commandline of the process spawning the |
| container (the first parent process with a different mount namespace).</para></listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP=</varname></term> |
| |
| <listitem><para>When the core is stored in the journal, the core image itself.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_FILENAME=</varname></term> |
| |
| <listitem><para>When the core is stored externally, the path to the core file.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_TRUNCATED=</varname></term> |
| |
| <listitem><para>Set to <literal>1</literal> when the saved coredump was truncated. (A partial core |
| image may still be processed by some tools, though obviously not all information is available.) |
| </para></listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>COREDUMP_PACKAGE_NAME=</varname></term> |
| <term><varname>COREDUMP_PACKAGE_VERSION=</varname></term> |
| <term><varname>COREDUMP_PACKAGE_JSON=</varname></term> |
| |
| <listitem><para>If the executable contained .package metadata ELF notes, they will be |
| parsed and attached. The <varname>package</varname> and <varname>packageVersion</varname> |
| of the 'main' ELF module (ie: the executable) will be appended individually. The |
| JSON-formatted content of all modules will be appended as a single JSON object, each with |
| the module name as the key. For more information about this metadata format and content, see |
| <ulink url="https://systemd.io/COREDUMP_PACKAGE_METADATA/">the coredump metadata spec</ulink>.</para> |
| </listitem> |
| </varlistentry> |
| |
| <varlistentry> |
| <term><varname>MESSAGE=</varname></term> |
| |
| <listitem><para>The message generated by <command>systemd-coredump</command> that includes the |
| backtrace if it was successfully generated. When <command>systemd-coredump</command> is invoked with |
| <option>--backtrace</option>, this field is provided by the caller.</para></listitem> |
| </varlistentry> |
| </variablelist> |
| |
| <para>Various other fields exist in the journal entry, but pertain to the logging process, |
| i.e. <command>systemd-coredump</command>, not the crashed process. See |
| <citerefentry><refentrytitle>systemd.journal-fields</refentrytitle><manvolnum>7</manvolnum></citerefentry>. |
| </para> |
| |
| <para>The following fields are saved (if known) with the external file listed in |
| <varname>COREDUMP_FILENAME=</varname> as extended attributes:</para> |
| |
| <variablelist> |
| <varlistentry> |
| <term><varname>user.coredump.pid</varname></term> |
| <term><varname>user.coredump.uid</varname></term> |
| <term><varname>user.coredump.gid</varname></term> |
| <term><varname>user.coredump.signal</varname></term> |
| <term><varname>user.coredump.timestamp</varname></term> |
| <term><varname>user.coredump.rlimit</varname></term> |
| <term><varname>user.coredump.hostname</varname></term> |
| <term><varname>user.coredump.comm</varname></term> |
| <term><varname>user.coredump.exe</varname></term> |
| |
| <listitem><para>Those are the same as <varname>COREDUMP_PID=</varname>, |
| <varname>COREDUMP_UID=</varname>, <varname>COREDUMP_GID=</varname>, |
| <varname>COREDUMP_SIGNAL=</varname>, <varname>COREDUMP_TIMESTAMP=</varname>, |
| <varname>COREDUMP_RLIMIT=</varname>, <varname>COREDUMP_HOSTNAME=</varname>, |
| <varname>COREDUMP_COMM=</varname>, and <varname>COREDUMP_EXE=</varname>, described above. |
| </para></listitem> |
| </varlistentry> |
| </variablelist> |
| |
| <para>Those can be viewed using |
| <citerefentry project='man-pages'><refentrytitle>getfattr</refentrytitle><manvolnum>1</manvolnum></citerefentry>. |
| For the core file described in the journal entry shown above: |
| <programlisting>$ getfattr --absolute-names -d /var/lib/systemd/coredump/core.Web….552351.….zst |
| # file: /var/lib/systemd/coredump/core.Web….552351.….zst |
| user.coredump.pid="552351" |
| user.coredump.uid="1000" |
| user.coredump.gid="1000" |
| user.coredump.signal="11" |
| user.coredump.timestamp="1614342930000000" |
| user.coredump.comm="Web Content" |
| user.coredump.exe="/usr/lib64/firefox/firefox" |
| … |
| </programlisting> |
| </para> |
| </refsect1> |
| |
| <refsect1> |
| <title>See Also</title> |
| <para> |
| <citerefentry><refentrytitle>coredump.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>, |
| <citerefentry><refentrytitle>coredumpctl</refentrytitle><manvolnum>1</manvolnum></citerefentry>, |
| <citerefentry><refentrytitle>systemd-journald.service</refentrytitle><manvolnum>8</manvolnum></citerefentry>, |
| <citerefentry><refentrytitle>systemd-tmpfiles</refentrytitle><manvolnum>8</manvolnum></citerefentry>, |
| <citerefentry project='man-pages'><refentrytitle>core</refentrytitle><manvolnum>5</manvolnum></citerefentry>, |
| <citerefentry><refentrytitle>sysctl.d</refentrytitle><manvolnum>5</manvolnum></citerefentry>, |
| <citerefentry><refentrytitle>systemd-sysctl.service</refentrytitle><manvolnum>8</manvolnum></citerefentry>. |
| </para> |
| </refsect1> |
| </refentry> |
