doc/html/hdf5_profile_user_guide.shtml - SchedMD/slurm - Git at Google


            
            <!--#include virtual="header.txt"-->

            
            <!--  Copyright (C) 2013 Bull S. A. S.

            
                  Bull, Rue Jean Jaures, B.P.68, 78340, Les Clayes-sous-Bois. -->


            <h1>Profiling Using HDF5 User Guide</h1>


            <h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>

            
            <ul>

            
            <li><a href="#Overview">Overview</a></li>

            
            <li><a href="#Administration">Administration</a></li>

            
            <li><a href="#Profiling">Profiling Jobs</a></li>

            
            <li><a href="#HDF5">HDF5</a></li>

            
            <li><a href="#DataSeries">Data Structure</a></li>

            
            </ul>


            <h2 id="Overview">Overview<a class="slurm_link" href="#Overview"></a></h2>

            
            <p>The acct_gather_profile/hdf5 plugin allows Slurm to coordinate collecting

            
            data on jobs it runs on a cluster that is more detailed than is practical to

            
            include in its database. The data comes from periodically sampling various

            
            performance data either collected by Slurm, the operating system, or

            
            component software. The plugin will record the data from each source

            
            as a <b>Time Series</b>. Totals for each statistic can be computed from the

            
            time series with <a href="sh5util.html">sh5util</a>.</p>


            <p>Time Series are energy data collected by an acct_gather_energy plugin,

            
            I/O data from a network interface collected by an acct_gather_interconnect

            
            plugin, I/O data from parallel file systems such as Lustre collected by an

            
            acct_gather_filesystem plugin, and task performance data such as local disk I/O,

            
            cpu consumption, and memory use from a jobacct_gather plugin.

            
            Data from other sources may be added in the future.</p>


            <p>The data is collected into a file on a shared file system for each step on

            
            each allocated node of a job and then merged into an HDF5 file.

            
            This shared file system architecture was chosen because it is possible

            
            that the data is voluminous. Solutions that pass data to the Slurm control

            
            daemon via RPC may not scale to very large clusters or jobs with

            
            many allocated nodes.</p>


            <h2 id="Administration">Administration

            
            <a class="slurm_link" href="#Administration"></a>

            
            </h2>


            <h3 id="shared">Shared File System<a class="slurm_link" href="#shared"></a></h3>

            
            <div style="margin-left: 20px;">

            
            <p>The HDF5 Profile Plugin requires a common shared file system on all

            
            the compute nodes. While a job is running, the plugin writes a

            
            file into this file system for each step of the job on each node. When

            
            the job ends, the merge process is launched and the node-step files

            
            are combined into one HDF5 file for the job.</p>


            <p>The root of the directory structure is declared in the <b>ProfileHDF5Dir</b>

            
            option in the acct_gather.conf file. The directory will be created by

            
            Slurm if it doesn't exist.  Each user will have

            
            their own directory created in the ProfileHDF5Dir which contains

            
            the HDF5 files.  All the directories and files are created by the

            
            SlurmdUser which is usually root. The user specific directories, as well

            
            as the files inside, are chowned to the user running the job so they

            
            can access the files.  Since user root is usually creating these

            
            files/directories a root squashed file system will not work for

            
            the ProfileHDF5Dir.</p>


            <p>Each user that creates a profile will have a subdirectory in the profile

            
            directory that has read/write permission only for the user.</p>

            
            </div>

            
            <br>

            
            <h3 id="config">Configuration parameters

            
            <a class="slurm_link" href="#config"></a>

            
            </h3>


            <div style="margin-left: 20px;">

            
            <p>The profile plugin is enabled in the

            
            <a href="slurm.conf.html">slurm.conf</a> file and it is internally

            
            configured in the

            
            <a href="acct_gather.conf.html">acct_gather.conf</a> file.</p>

            
            </div>

            
            <div style="margin-left: 20px;">

            
            <br>

            
            <h3 id="slurm_conf">slurm.conf parameters

            
            <a class="slurm_link" href="#slurm_conf"></a>

            
            </h3>

            
            <div style="margin-left: 20px;">

            
            <dl>

            
            <dt><b>AcctGatherProfileType</b>=acct_gather_profile/hdf5</dt>

            
            <dd>Enables the HDF5 plugin.</dd>

            
            <dt><b>JobAcctGatherFrequency</b>=&lt;freq&gt;</dt>

            
            <dd>Sets the sampling frequency for data types, e.g.

            
            <code>task=sec,energy=sec,filesystem=sec,network=sec</code>.</dd>

            
            </dl>

            
            </div>

            
            </div>

            
            <div style="margin-left: 20px;">

            
            <br>

            
            <h3 id="acct_gather">acct_gather.conf parameters

            
            <a class="slurm_link" href="#acct_gather"></a>

            
            </h3>

            
            <div style="margin-left: 20px;">

            
            <p>These parameters are directly used by the HDF5 Profile Plugin.</p>

            
            <dl>

            
            <dt><b>ProfileHDF5Dir</b>=&lt;path&gt;</dt>

            
            <dd>This parameter is the path to the shared folder into which the

            
            acct_gather_profile plugin will write detailed data as an HDF5 file.

            
            The directory is assumed to be on a file system shared by the controller and

            
            all compute nodes. This is a required parameter.</dd>

            
            <dt><b>ProfileHDF5Default</b>=[options]</dt>

            
            <dd>A comma-delimited list of data types to be collected for each job

            
            submission. Use this option with caution. A node-step file will be created on

            
            every node for every step of every job. They will not automatically be merged

            
            into job files. (Even job files for large numbers of small jobs would fill the

            
            file system.) This option is intended for test environments where you

            
            might want to profile a series of jobs but do not want to have to

            
            add the --profile option to the launch scripts.

            
            The options are described below and in the man pages for acct_gather.conf,

            
            srun, salloc and sbatch commands.</dd>

            
            </dl>

            
            </div>

            
            </div>


            <div style="margin-left: 20px;">

            
            <br>

            
            <h3 id="time">Time Series Control Parameters

            
            <a class="slurm_link" href="#time"></a>

            
            </h3>

            
            <div style="margin-left: 20px;">

            
            <p>Other plugins add time series data to the HDF5 collection. The HDF5

            
            profile plugin stores samples produced by the corresponding accounting gather

            
            plugin configured in slurm.conf: AcctGatherEnergyType,

            
            AcctGatherFilesystemType, AcctGatherInterconnectType, or JobAcctGatherType.

            
            They typically have a default polling frequency specified in slurm.conf in the

            
            JobAcctGatherFrequency parameter. The polling frequency can be overridden

            
            using the --acctg-freq <a href="srun.html">srun</a> parameter.

            
            Both parameters use the form task=sec,energy=sec,filesystem=sec,network=sec.</p>


            <p>The acct_gather_energy/ipmi plugin also needs the EnergyIPMIFrequency value

            
            set in the acct_gather.conf file. This sets the rate at which that plugin

            
            samples the external sensors. This value should be the same as the energy=sec in

            
            either JobAcctGatherFrequency or --acctg-freq.</p>


            <p>Note that the IPMI and profile sampling are not synchronous.

            
            The profile sample simply takes the last available IPMI sample value.

            
            If the profile energy sample is more frequent than the IPMI sample rate,

            
            the IPMI value will be repeated. If the profile energy sample is greater

            
            than the IPMI rate, IPMI values will be lost.</p>


            <p>Also note that smallest effective IPMI (EnergyIPMIFrequency) sample rate

            
            for 2013 era Intel processors is 3 seconds.</p>


            </div>

            
            </div>

            
            <h2 id="Profiling">Profiling Jobs

            
            <a class="slurm_link" href="#Profiling"></a>

            
            </h2>

            
            <h3 id="collection">Data Collection

            
            <a class="slurm_link" href="#collection"></a>

            
            </h3>

            
            <p>The --profile option on salloc|sbatch|srun controls whether data is

            
            collected and what type of data is collected. If --profile is not specified

            
            no data is collected unless the <B>ProfileHDF5Default</B>

            
            option is used in acct_gather.conf. --profile on the command line overrides

            
            any value specified in the configuration file.</p>


            <div style="margin-left: 20px;">

            
            <dl>

            
            <dt><b>--profile</b>=&lt;all|none|[energy[,|task[,|filesystem[,|network]]]]&gt;

            
            </dt>

            
            <dd>Enables detailed data collection by the acct_gather_profile plugin.

            
            Detailed data is typically a time series stored in an HDF5 file for the job.

            
            </dd>

            
            <dd>

            
            <DL>

            
            <dt><B>All</B></dt>

            
            <DD>All data types are collected. (Cannot be combined with other values.)

            
            </DD>

            
            <dt><B>None</B></dt>

            
            <DD>No data types are collected. This is the default. (Cannot be

            
            combined with other values.)

            
            </DD>


            <dt><B>Energy</B></dt>

            
            <DD>Energy data is collected.</DD>


            <dt><B>Filesystem</B></dt>

            
            <DD>Filesystem data is collected. Currently only

            
            Lustre filesystem is supported.</DD>


            <dt><B>Network</B></dt>

            
            <DD>Network data is collected.</DD>


            <dt><B>Task</B></dt>

            
            <DD>Task (I/O, Memory, ...) data is collected.</DD>

            
            </DL>

            
            </dd>

            
            </dl>

            
            </div>

            
            <br>


            <h3 id="consolidation">Data Consolidation

            
            <a class="slurm_link" href="#consolidation"></a>

            
            </h3>

            
            <p>The node-step files are merged into one HDF5 file for the job using the

            
            <a href="sh5util.html">sh5util</a>.</p>


            <p>If the job is started with sbatch, the command line may be added to the

            
            normal launch script. For example:</p>

            
            <pre>

            
            sbatch -n1 -d$SLURM_JOB_ID --wrap="sh5util -j $SLURM_JOB_ID"

            
            </pre>


            <h3 id="extraction">Data Extraction

            
            <a class="slurm_link" href="#extraction"></a>

            
            </h3>

            
            <p>The <a href="sh5util.html">sh5util</a> program can also be used to extract

            
            specific data from the HDF5 file and write it in <i>comma separated value (csv)</i>

            
            form for importation into other analysis tools such as spreadsheets.</p>


            <h2 id="HDF5">HDF5<a class="slurm_link" href="#HDF5"></a></h2>

            
            <p>HDF5 is a well known structured data set that allows heterogeneous but

            
            related data to be stored in one file

            
            (e.g. sections for energy statistics, network I/O, task data, etc.).

            
            Its internal structure resembles a

            
            file system with <b>groups</b> being similar to <i>directories</i> and

            
            <b>data sets</b> being similar to <i>files</i>. It also allows <b>attributes</b>

            
            to be attached to groups to store application defined properties.</p>


            <p>There are commodity programs, notably

            
            <a href="https://www.hdfgroup.org/download-hdfview/">

            
            HDFView</a>, for viewing and manipulating these files.</p>


            <h2 id="DataSeries">Data Structure

            
            <a class="slurm_link" href="#DataSeries"></a>

            
            </h2>


            <div style="margin-left: 5px;">

            
            <p>In the merged job file, the top-level <b>Steps</b> group contains a group

            
            for each <b>step</b> of the job, named by step ID (<b>batch</b> for the batch

            
            step). Within each step, there will be a <b>Nodes</b> group. Each node

            
            group contains the node-level time series datasets for the configured

            
            accounting gather plugins and a <b>Tasks</b> group for per-task data.

            
            For example, step 0 of a job with energy, filesystem, network, and task

            
            profiling enabled and two tasks on node1 contains:</p>

            
            <pre>

            
            /Steps/0/Nodes/node1/Energy

            
            /Steps/0/Nodes/node1/Filesystem

            
            /Steps/0/Nodes/node1/Network

            
            /Steps/0/Nodes/node1/Tasks/0

            
            /Steps/0/Nodes/node1/Tasks/1

            
            </pre>

            
            </div>

            
            <ul>

            
            <li>

            
            The <b>Nodes</b> group will have a group for each node in the step allocation.

            
            For each node group, node-level series such as Energy, Filesystem, and

            
            Network are stored as HDF5 datasets.

            
            <ul>

            
            <li>

            
            Each dataset contains an <b>ElapsedTime</b> field for seconds since the step

            
            started and an <b>EpochTime</b> field for the sample's UNIX timestamp, followed

            
            by the fields defined by the accounting gather plugin that produced the

            
            samples.

            
            </li>

            
            <li>

            
            The <a href="sh5util.html">sh5util</a> command can aggregate time series data

            
            when extracting totals.

            
            </li>

            
            <li>

            
            The <b>Tasks</b> group contains one dataset for each task that ran on the node,

            
            named by task ID.

            
            </li>

            
            </ul>

            
            </li>

            
            </ul>


            <br>

            
            <h3 id="energy">Energy Data<a class="slurm_link" href="#energy"></a></h3>

            
            <p>An <b>AcctGatherEnergyType</b> plugin is required in slurm.conf

            
            to collect energy data.

            
            Appropriately set energy=freq in either JobAcctGatherFrequency in slurm.conf

            
            or in --acctg-freq on the command line.

            
            Also set EnergyIPMIFrequency in acct_gather.conf when using

            
            <b>acct_gather_energy/ipmi</b>.</p>


            <p>Each Energy sample contains <b>ElapsedTime</b> and <b>EpochTime</b>.

            
            Additional data items are defined by the configured energy plugin and may vary

            
            by hardware. The <a href="sh5util.html">sh5util</a> command can list the

            
            fields present in a job file.</p>


            <DL>

            
            <dt><B>acct_gather_energy/rapl</B>, <B>acct_gather_energy/ibmaem</B>,

            
            <B>acct_gather_energy/pm_counters</B></dt>

            
            <DD><B>Power</B></DD>

            
            <dt><B>acct_gather_energy/ipmi</B></dt>

            
            <DD><B>&lt;label&gt;Power</B>, where label comes from

            
            EnergyIPMIPowerSensors; the default label is <B>Node</B>.</DD>

            
            <dt><B>acct_gather_energy/gpu</B></dt>

            
            <DD><B>GPU0Power</B>, <B>GPU1Power</B>, ...</DD>

            
            <dt><B>acct_gather_energy/xcc</B></dt>

            
            <DD><B>Energy</B>, <B>CurrPower</B></DD>

            
            </DL>


            <h3 id="filesystem">Filesystem Data

            
            <a class="slurm_link" href="#filesystem"></a>

            
            </h3>

            
            <p>An <b>AcctGatherFilesystemType</b> plugin is required in slurm.conf

            
            to collect filesystem data.

            
            Appropriately set filesystem=freq in either JobAcctGatherFrequency in slurm.conf

            
            or in --acctg-freq on the command line.</p>


            <p>Each Filesystem sample contains <b>ElapsedTime</b> and <b>EpochTime</b>.

            
            The Lustre plugin adds the following data items.</p>

            
            <DL>

            
            <dt><B>Reads</B></dt>

            
            <DD>Number of read operations.</DD>

            
            <dt><B>ReadMB</B></dt>

            
            <DD>Number of megabytes read.</DD>

            
            <dt><B>Writes</B></dt>

            
            <DD>Number of write operations.</DD>

            
            <dt><B>WriteMB</B></dt>

            
            <DD>Number of megabytes written.</DD>

            
            </DL>


            <h3 id="network">Network Data

            
            <a class="slurm_link" href="#network"></a>

            
            </h3>

            
            <p>An <b>AcctGatherInterconnectType</b> plugin is required in slurm.conf

            
            to collect interconnect data.

            
            Appropriately set network=freq in either JobAcctGatherFrequency in slurm.conf

            
            or in --acctg-freq on the command line.</p>

            
            <p>Each Network sample contains <b>ElapsedTime</b> and <b>EpochTime</b>

            
            in addition to the following data items.</p>

            
            <DL>

            
            <dt><B>PacketsIn</B></dt>

            
            <DD>Number of packets coming in.</DD>

            
            <dt><B>PacketsOut</B></dt>

            
            <DD>Number of packets going out.</DD>

            
            <dt><B>InMB</B></dt>

            
            <DD>Number of megabytes coming in through the interface.</DD>

            
            <dt><B>OutMB</B></dt>

            
            <DD>Number of megabytes going out through the interface.</DD>

            
            </DL>


            <h3 id="task">Task Data<a class="slurm_link" href="#task"></a></h3>

            
            <p>A <b>JobAcctGatherType</b> plugin is required in slurm.conf

            
            to collect task data.

            
            Appropriately set task=freq in either JobAcctGatherFrequency in slurm.conf

            
            or in --acctg-freq on the command line.</p>

            
            <p>Each Task sample contains <b>ElapsedTime</b> and <b>EpochTime</b>.

            
            The job accounting gather plugins add the following data items.</p>

            
            <DL>

            
            <dt><B>CPUFrequency</B></dt>

            
            <DD>CPU Frequency at time of sample.</DD>

            
            <dt><B>CPUTime</B></dt>

            
            <DD>Seconds of CPU time used during the sample.</DD>

            
            <dt><B>CPUUtilization</B></dt>

            
            <DD>CPU Utilization during the interval.</DD>

            
            <dt><B>GPUMemMB</B></dt>

            
            <DD>GPU memory used, in megabytes.</DD>

            
            <dt><B>GPUUtilization</B></dt>

            
            <DD>GPU utilization during the interval.</DD>

            
            <dt><B>RSS</B></dt>

            
            <DD>Value of RSS at time of sample.</DD>

            
            <dt><B>VMSize</B></dt>

            
            <DD>Value of VM Size at time of sample.</DD>

            
            <dt><B>Pages</B></dt>

            
            <DD>Pages used in sample.</DD>

            
            <dt><B>ReadMB</B></dt>

            
            <DD>Number of megabytes read from local disk.</DD>

            
            <dt><B>WriteMB</B></dt>

            
            <DD>Number of megabytes written to local disk.</DD>

            
            </DL>


            <!--#include virtual="footer.txt"-->
	<!--#include virtual="header.txt"-->
	<!-- Copyright (C) 2013 Bull S. A. S.
	Bull, Rue Jean Jaures, B.P.68, 78340, Les Clayes-sous-Bois. -->

	<h1>Profiling Using HDF5 User Guide</h1>

	<h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>
	<ul>
	<li><a href="#Overview">Overview</a></li>
	<li><a href="#Administration">Administration</a></li>
	<li><a href="#Profiling">Profiling Jobs</a></li>
	<li><a href="#HDF5">HDF5</a></li>
	<li><a href="#DataSeries">Data Structure</a></li>
	</ul>


	<h2 id="Overview">Overview<a class="slurm_link" href="#Overview"></a></h2>
	<p>The acct_gather_profile/hdf5 plugin allows Slurm to coordinate collecting
	data on jobs it runs on a cluster that is more detailed than is practical to
	include in its database. The data comes from periodically sampling various
	performance data either collected by Slurm, the operating system, or
	component software. The plugin will record the data from each source
	as a <b>Time Series</b>. Totals for each statistic can be computed from the
	time series with <a href="sh5util.html">sh5util</a>.</p>

	<p>Time Series are energy data collected by an acct_gather_energy plugin,
	I/O data from a network interface collected by an acct_gather_interconnect
	plugin, I/O data from parallel file systems such as Lustre collected by an
	acct_gather_filesystem plugin, and task performance data such as local disk I/O,
	cpu consumption, and memory use from a jobacct_gather plugin.
	Data from other sources may be added in the future.</p>

	<p>The data is collected into a file on a shared file system for each step on
	each allocated node of a job and then merged into an HDF5 file.
	This shared file system architecture was chosen because it is possible
	that the data is voluminous. Solutions that pass data to the Slurm control
	daemon via RPC may not scale to very large clusters or jobs with
	many allocated nodes.</p>

	<h2 id="Administration">Administration
	<a class="slurm_link" href="#Administration"></a>
	</h2>

	<h3 id="shared">Shared File System<a class="slurm_link" href="#shared"></a></h3>
	<div style="margin-left: 20px;">
	<p>The HDF5 Profile Plugin requires a common shared file system on all
	the compute nodes. While a job is running, the plugin writes a
	file into this file system for each step of the job on each node. When
	the job ends, the merge process is launched and the node-step files
	are combined into one HDF5 file for the job.</p>

	<p>The root of the directory structure is declared in the <b>ProfileHDF5Dir</b>
	option in the acct_gather.conf file. The directory will be created by
	Slurm if it doesn't exist. Each user will have
	their own directory created in the ProfileHDF5Dir which contains
	the HDF5 files. All the directories and files are created by the
	SlurmdUser which is usually root. The user specific directories, as well
	as the files inside, are chowned to the user running the job so they
	can access the files. Since user root is usually creating these
	files/directories a root squashed file system will not work for
	the ProfileHDF5Dir.</p>

	<p>Each user that creates a profile will have a subdirectory in the profile
	directory that has read/write permission only for the user.</p>
	</div>
	<br>
	<h3 id="config">Configuration parameters
	<a class="slurm_link" href="#config"></a>
	</h3>

	<div style="margin-left: 20px;">
	<p>The profile plugin is enabled in the
	<a href="slurm.conf.html">slurm.conf</a> file and it is internally
	configured in the
	<a href="acct_gather.conf.html">acct_gather.conf</a> file.</p>
	</div>
	<div style="margin-left: 20px;">
	<br>
	<h3 id="slurm_conf">slurm.conf parameters
	<a class="slurm_link" href="#slurm_conf"></a>
	</h3>
	<div style="margin-left: 20px;">
	<dl>
	<dt><b>AcctGatherProfileType</b>=acct_gather_profile/hdf5</dt>
	<dd>Enables the HDF5 plugin.</dd>
	<dt><b>JobAcctGatherFrequency</b>=<freq></dt>
	<dd>Sets the sampling frequency for data types, e.g.
	<code>task=sec,energy=sec,filesystem=sec,network=sec</code>.</dd>
	</dl>
	</div>
	</div>
	<div style="margin-left: 20px;">
	<br>
	<h3 id="acct_gather">acct_gather.conf parameters
	<a class="slurm_link" href="#acct_gather"></a>
	</h3>
	<div style="margin-left: 20px;">
	<p>These parameters are directly used by the HDF5 Profile Plugin.</p>
	<dl>
	<dt><b>ProfileHDF5Dir</b>=<path></dt>
	<dd>This parameter is the path to the shared folder into which the
	acct_gather_profile plugin will write detailed data as an HDF5 file.
	The directory is assumed to be on a file system shared by the controller and
	all compute nodes. This is a required parameter.</dd>
	<dt><b>ProfileHDF5Default</b>=[options]</dt>
	<dd>A comma-delimited list of data types to be collected for each job
	submission. Use this option with caution. A node-step file will be created on
	every node for every step of every job. They will not automatically be merged
	into job files. (Even job files for large numbers of small jobs would fill the
	file system.) This option is intended for test environments where you
	might want to profile a series of jobs but do not want to have to
	add the --profile option to the launch scripts.
	The options are described below and in the man pages for acct_gather.conf,
	srun, salloc and sbatch commands.</dd>
	</dl>
	</div>
	</div>


	<div style="margin-left: 20px;">
	<br>
	<h3 id="time">Time Series Control Parameters
	<a class="slurm_link" href="#time"></a>
	</h3>
	<div style="margin-left: 20px;">
	<p>Other plugins add time series data to the HDF5 collection. The HDF5
	profile plugin stores samples produced by the corresponding accounting gather
	plugin configured in slurm.conf: AcctGatherEnergyType,
	AcctGatherFilesystemType, AcctGatherInterconnectType, or JobAcctGatherType.
	They typically have a default polling frequency specified in slurm.conf in the
	JobAcctGatherFrequency parameter. The polling frequency can be overridden
	using the --acctg-freq <a href="srun.html">srun</a> parameter.
	Both parameters use the form task=sec,energy=sec,filesystem=sec,network=sec.</p>

	<p>The acct_gather_energy/ipmi plugin also needs the EnergyIPMIFrequency value
	set in the acct_gather.conf file. This sets the rate at which that plugin
	samples the external sensors. This value should be the same as the energy=sec in
	either JobAcctGatherFrequency or --acctg-freq.</p>

	<p>Note that the IPMI and profile sampling are not synchronous.
	The profile sample simply takes the last available IPMI sample value.
	If the profile energy sample is more frequent than the IPMI sample rate,
	the IPMI value will be repeated. If the profile energy sample is greater
	than the IPMI rate, IPMI values will be lost.</p>

	<p>Also note that smallest effective IPMI (EnergyIPMIFrequency) sample rate
	for 2013 era Intel processors is 3 seconds.</p>

	</div>
	</div>
	<h2 id="Profiling">Profiling Jobs
	<a class="slurm_link" href="#Profiling"></a>
	</h2>
	<h3 id="collection">Data Collection
	<a class="slurm_link" href="#collection"></a>
	</h3>
	<p>The --profile option on salloc\|sbatch\|srun controls whether data is
	collected and what type of data is collected. If --profile is not specified
	no data is collected unless the <B>ProfileHDF5Default</B>
	option is used in acct_gather.conf. --profile on the command line overrides
	any value specified in the configuration file.</p>

	<div style="margin-left: 20px;">
	<dl>
	<dt><b>--profile</b>=<all\|none\|[energy[,\|task[,\|filesystem[,\|network]]]]>
	</dt>
	<dd>Enables detailed data collection by the acct_gather_profile plugin.
	Detailed data is typically a time series stored in an HDF5 file for the job.
	</dd>
	<dd>
	<DL>
	<dt><B>All</B></dt>
	<DD>All data types are collected. (Cannot be combined with other values.)
	</DD>
	<dt><B>None</B></dt>
	<DD>No data types are collected. This is the default. (Cannot be
	combined with other values.)
	</DD>

	<dt><B>Energy</B></dt>
	<DD>Energy data is collected.</DD>

	<dt><B>Filesystem</B></dt>
	<DD>Filesystem data is collected. Currently only
	Lustre filesystem is supported.</DD>

	<dt><B>Network</B></dt>
	<DD>Network data is collected.</DD>

	<dt><B>Task</B></dt>
	<DD>Task (I/O, Memory, ...) data is collected.</DD>
	</DL>
	</dd>
	</dl>
	</div>
	<br>

	<h3 id="consolidation">Data Consolidation
	<a class="slurm_link" href="#consolidation"></a>
	</h3>
	<p>The node-step files are merged into one HDF5 file for the job using the
	<a href="sh5util.html">sh5util</a>.</p>

	<p>If the job is started with sbatch, the command line may be added to the
	normal launch script. For example:</p>
	<pre>
	sbatch -n1 -d$SLURM_JOB_ID --wrap="sh5util -j $SLURM_JOB_ID"
	</pre>

	<h3 id="extraction">Data Extraction
	<a class="slurm_link" href="#extraction"></a>
	</h3>
	<p>The <a href="sh5util.html">sh5util</a> program can also be used to extract
	specific data from the HDF5 file and write it in <i>comma separated value (csv)</i>
	form for importation into other analysis tools such as spreadsheets.</p>

	<h2 id="HDF5">HDF5<a class="slurm_link" href="#HDF5"></a></h2>
	<p>HDF5 is a well known structured data set that allows heterogeneous but
	related data to be stored in one file
	(e.g. sections for energy statistics, network I/O, task data, etc.).
	Its internal structure resembles a
	file system with <b>groups</b> being similar to <i>directories</i> and
	<b>data sets</b> being similar to <i>files</i>. It also allows <b>attributes</b>
	to be attached to groups to store application defined properties.</p>

	<p>There are commodity programs, notably
	<a href="https://www.hdfgroup.org/download-hdfview/">
	HDFView</a>, for viewing and manipulating these files.</p>


	<h2 id="DataSeries">Data Structure
	<a class="slurm_link" href="#DataSeries"></a>
	</h2>

	<div style="margin-left: 5px;">
	<p>In the merged job file, the top-level <b>Steps</b> group contains a group
	for each <b>step</b> of the job, named by step ID (<b>batch</b> for the batch
	step). Within each step, there will be a <b>Nodes</b> group. Each node
	group contains the node-level time series datasets for the configured
	accounting gather plugins and a <b>Tasks</b> group for per-task data.
	For example, step 0 of a job with energy, filesystem, network, and task
	profiling enabled and two tasks on node1 contains:</p>
	<pre>
	/Steps/0/Nodes/node1/Energy
	/Steps/0/Nodes/node1/Filesystem
	/Steps/0/Nodes/node1/Network
	/Steps/0/Nodes/node1/Tasks/0
	/Steps/0/Nodes/node1/Tasks/1
	</pre>
	</div>
	<ul>
	<li>
	The <b>Nodes</b> group will have a group for each node in the step allocation.
	For each node group, node-level series such as Energy, Filesystem, and
	Network are stored as HDF5 datasets.
	<ul>
	<li>
	Each dataset contains an <b>ElapsedTime</b> field for seconds since the step
	started and an <b>EpochTime</b> field for the sample's UNIX timestamp, followed
	by the fields defined by the accounting gather plugin that produced the
	samples.
	</li>
	<li>
	The <a href="sh5util.html">sh5util</a> command can aggregate time series data
	when extracting totals.
	</li>
	<li>
	The <b>Tasks</b> group contains one dataset for each task that ran on the node,
	named by task ID.
	</li>
	</ul>
	</li>
	</ul>

	<br>
	<h3 id="energy">Energy Data<a class="slurm_link" href="#energy"></a></h3>
	<p>An <b>AcctGatherEnergyType</b> plugin is required in slurm.conf
	to collect energy data.
	Appropriately set energy=freq in either JobAcctGatherFrequency in slurm.conf
	or in --acctg-freq on the command line.
	Also set EnergyIPMIFrequency in acct_gather.conf when using
	<b>acct_gather_energy/ipmi</b>.</p>

	<p>Each Energy sample contains <b>ElapsedTime</b> and <b>EpochTime</b>.
	Additional data items are defined by the configured energy plugin and may vary
	by hardware. The <a href="sh5util.html">sh5util</a> command can list the
	fields present in a job file.</p>

	<DL>
	<dt><B>acct_gather_energy/rapl</B>, <B>acct_gather_energy/ibmaem</B>,
	<B>acct_gather_energy/pm_counters</B></dt>
	<DD><B>Power</B></DD>
	<dt><B>acct_gather_energy/ipmi</B></dt>
	<DD><B><label>Power</B>, where label comes from
	EnergyIPMIPowerSensors; the default label is <B>Node</B>.</DD>
	<dt><B>acct_gather_energy/gpu</B></dt>
	<DD><B>GPU0Power</B>, <B>GPU1Power</B>, ...</DD>
	<dt><B>acct_gather_energy/xcc</B></dt>
	<DD><B>Energy</B>, <B>CurrPower</B></DD>
	</DL>

	<h3 id="filesystem">Filesystem Data
	<a class="slurm_link" href="#filesystem"></a>
	</h3>
	<p>An <b>AcctGatherFilesystemType</b> plugin is required in slurm.conf
	to collect filesystem data.
	Appropriately set filesystem=freq in either JobAcctGatherFrequency in slurm.conf
	or in --acctg-freq on the command line.</p>

	<p>Each Filesystem sample contains <b>ElapsedTime</b> and <b>EpochTime</b>.
	The Lustre plugin adds the following data items.</p>
	<DL>
	<dt><B>Reads</B></dt>
	<DD>Number of read operations.</DD>
	<dt><B>ReadMB</B></dt>
	<DD>Number of megabytes read.</DD>
	<dt><B>Writes</B></dt>
	<DD>Number of write operations.</DD>
	<dt><B>WriteMB</B></dt>
	<DD>Number of megabytes written.</DD>
	</DL>

	<h3 id="network">Network Data
	<a class="slurm_link" href="#network"></a>
	</h3>
	<p>An <b>AcctGatherInterconnectType</b> plugin is required in slurm.conf
	to collect interconnect data.
	Appropriately set network=freq in either JobAcctGatherFrequency in slurm.conf
	or in --acctg-freq on the command line.</p>
	<p>Each Network sample contains <b>ElapsedTime</b> and <b>EpochTime</b>
	in addition to the following data items.</p>
	<DL>
	<dt><B>PacketsIn</B></dt>
	<DD>Number of packets coming in.</DD>
	<dt><B>PacketsOut</B></dt>
	<DD>Number of packets going out.</DD>
	<dt><B>InMB</B></dt>
	<DD>Number of megabytes coming in through the interface.</DD>
	<dt><B>OutMB</B></dt>
	<DD>Number of megabytes going out through the interface.</DD>
	</DL>

	<h3 id="task">Task Data<a class="slurm_link" href="#task"></a></h3>
	<p>A <b>JobAcctGatherType</b> plugin is required in slurm.conf
	to collect task data.
	Appropriately set task=freq in either JobAcctGatherFrequency in slurm.conf
	or in --acctg-freq on the command line.</p>
	<p>Each Task sample contains <b>ElapsedTime</b> and <b>EpochTime</b>.
	The job accounting gather plugins add the following data items.</p>
	<DL>
	<dt><B>CPUFrequency</B></dt>
	<DD>CPU Frequency at time of sample.</DD>
	<dt><B>CPUTime</B></dt>
	<DD>Seconds of CPU time used during the sample.</DD>
	<dt><B>CPUUtilization</B></dt>
	<DD>CPU Utilization during the interval.</DD>
	<dt><B>GPUMemMB</B></dt>
	<DD>GPU memory used, in megabytes.</DD>
	<dt><B>GPUUtilization</B></dt>
	<DD>GPU utilization during the interval.</DD>
	<dt><B>RSS</B></dt>
	<DD>Value of RSS at time of sample.</DD>
	<dt><B>VMSize</B></dt>
	<DD>Value of VM Size at time of sample.</DD>
	<dt><B>Pages</B></dt>
	<DD>Pages used in sample.</DD>
	<dt><B>ReadMB</B></dt>
	<DD>Number of megabytes read from local disk.</DD>
	<dt><B>WriteMB</B></dt>
	<DD>Number of megabytes written to local disk.</DD>
	</DL>

	<!--#include virtual="footer.txt"-->