sphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget /translations/zh_CN/trace/eventsmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget /translations/zh_TW/trace/eventsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget /translations/it_IT/trace/eventsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget /translations/ja_JP/trace/eventsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget /translations/ko_KR/trace/eventsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget /translations/sp_SP/trace/eventsmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhsection)}(hhh](htitle)}(h Event Tracingh]h Event Tracing}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhh:/var/lib/git/docbuild/linux/Documentation/trace/events.rsthKubh field_list)}(hhh](hfield)}(hhh](h field_name)}(hAuthorh]hAuthor}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhKubh field_body)}(h Theodore Ts'oh]h paragraph)}(hhh]hTheodore Ts’o}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhubah}(h]h ]h"]h$]h&]uh1hhhubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hhh](h)}(hUpdatedh]hUpdated}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhKubh)}(hLi Zefan and Tom Zanussi h]h)}(hLi Zefan and Tom Zanussih]hLi Zefan and Tom Zanussi}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhhubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubeh}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h1. Introductionh]h1. Introduction}(hj*hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj'hhhhhK ubh)}(hTracepoints (see Documentation/trace/tracepoints.rst) can be used without creating custom kernel modules to register probe functions using the event tracing infrastructure.h]hTracepoints (see Documentation/trace/tracepoints.rst) can be used without creating custom kernel modules to register probe functions using the event tracing infrastructure.}(hj8hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hj'hhubh)}(hNot all tracepoints can be traced using the event tracing system; the kernel developer must provide code snippets which define how the tracing information is saved into the tracing buffer, and how the tracing information should be printed.h]hNot all tracepoints can be traced using the event tracing system; the kernel developer must provide code snippets which define how the tracing information is saved into the tracing buffer, and how the tracing information should be printed.}(hjFhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj'hhubeh}(h] introductionah ]h"]1. introductionah$]h&]uh1hhhhhhhhK ubh)}(hhh](h)}(h2. Using Event Tracingh]h2. Using Event Tracing}(hj_hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj\hhhhhKubh)}(hhh](h)}(h!2.1 Via the 'set_event' interfaceh]h%2.1 Via the ‘set_event’ interface}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhjmhhhhhKubh)}(hiThe events which are available for tracing can be found in the file /sys/kernel/tracing/available_events.h]hiThe events which are available for tracing can be found in the file /sys/kernel/tracing/available_events.}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjmhhubh)}(htTo enable a particular event, such as 'sched_wakeup', simply echo it to /sys/kernel/tracing/set_event. For example::h]hwTo enable a particular event, such as ‘sched_wakeup’, simply echo it to /sys/kernel/tracing/set_event. For example:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjmhhubh literal_block)}(h4# echo sched_wakeup >> /sys/kernel/tracing/set_eventh]h4# echo sched_wakeup >> /sys/kernel/tracing/set_event}hjsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1jhhhK hjmhhubhnote)}(hD'>>' is necessary, otherwise it will firstly disable all the events.h]h)}(hjh]hH‘>>’ is necessary, otherwise it will firstly disable all the events.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK"hjubah}(h]h ]h"]h$]h&]uh1jhjmhhhhhNubh)}(hcTo disable an event, echo the event name to the set_event file prefixed with an exclamation point::h]hbTo disable an event, echo the event name to the set_event file prefixed with an exclamation point:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK$hjmhhubj)}(h7# echo '!sched_wakeup' >> /sys/kernel/tracing/set_eventh]h7# echo '!sched_wakeup' >> /sys/kernel/tracing/set_event}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhK'hjmhhubh)}(hATo disable all events, echo an empty line to the set_event file::h]h@To disable all events, echo an empty line to the set_event file:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK)hjmhhubj)}(h&# echo > /sys/kernel/tracing/set_eventh]h&# echo > /sys/kernel/tracing/set_event}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhK+hjmhhubh)}(hDTo enable all events, echo ``*:*`` or ``*:`` to the set_event file::h](hTo enable all events, echo }(hjhhhNhNubhliteral)}(h``*:*``h]h*:*}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh or }(hjhhhNhNubj)}(h``*:``h]h*:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh to the set_event file:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK-hjmhhubj)}(h*# echo *:* > /sys/kernel/tracing/set_eventh]h*# echo *:* > /sys/kernel/tracing/set_event}hj1sbah}(h]h ]h"]h$]h&]jjuh1jhhhK/hjmhhubh)}(hXnThe events are organized into subsystems, such as ext4, irq, sched, etc., and a full event name looks like this: :. The subsystem name is optional, but it is displayed in the available_events file. All of the events in a subsystem can be specified via the syntax ``:*``; for example, to enable all irq events, you can use the command::h](hXThe events are organized into subsystems, such as ext4, irq, sched, etc., and a full event name looks like this: :. The subsystem name is optional, but it is displayed in the available_events file. All of the events in a subsystem can be specified via the syntax }(hj?hhhNhNubj)}(h``:*``h]h :*}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1jhj?ubhA; for example, to enable all irq events, you can use the command:}(hj?hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK1hjmhhubj)}(h.# echo 'irq:*' > /sys/kernel/tracing/set_eventh]h.# echo 'irq:*' > /sys/kernel/tracing/set_event}hj_sbah}(h]h ]h"]h$]h&]jjuh1jhhhK8hjmhhubh)}(h[The set_event file may also be used to enable events associated to only a specific module::h]hZThe set_event file may also be used to enable events associated to only a specific module:}(hjmhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK:hjmhhubj)}(h6# echo ':mod:' > /sys/kernel/tracing/set_eventh]h6# echo ':mod:' > /sys/kernel/tracing/set_event}hj{sbah}(h]h ]h"]h$]h&]jjuh1jhhhK=hjmhhubh)}(hWill enable all events in the module ````. If the module is not yet loaded, the string will be saved and when a module is that matches ```` is loaded, then it will apply the enabling of events then.h](h%Will enable all events in the module }(hjhhhNhNubj)}(h ````h]h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh_. If the module is not yet loaded, the string will be saved and when a module is that matches }(hjhhhNhNubj)}(h ````h]h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh; is loaded, then it will apply the enabling of events then.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK?hjmhhubh)}(h]The text before ``:mod:`` will be parsed to specify specific events that the module creates::h](hThe text before }(hjhhhNhNubj)}(h ``:mod:``h]h:mod:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhC will be parsed to specify specific events that the module creates:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKChjmhhubj)}(h=# echo ':mod:' > /sys/kernel/tracing/set_eventh]h=# echo ':mod:' > /sys/kernel/tracing/set_event}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKFhjmhhubh)}(h|The above will enable any system or event that ```` matches. If ```` is ``"*"`` then it will match all events.h](h/The above will enable any system or event that }(hjhhhNhNubj)}(h ````h]h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh matches. If }(hjhhhNhNubj)}(h ````h]h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh is }(hjhhhNhNubj)}(h``"*"``h]h"*"}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh then it will match all events.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKHhjmhhubh)}(h1To enable only a specific event within a system::h]h0To enable only a specific event within a system:}(hj-hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKKhjmhhubj)}(hF# echo '::mod:' > /sys/kernel/tracing/set_eventh]hF# echo '::mod:' > /sys/kernel/tracing/set_event}hj;sbah}(h]h ]h"]h$]h&]jjuh1jhhhKMhjmhhubh)}(h]If ```` is ``"*"`` then it will match all events within the system for a given module.h](hIf }(hjIhhhNhNubj)}(h ````h]h}(hjQhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjIubh is }(hjIhhhNhNubj)}(h``"*"``h]h"*"}(hjchhhNhNubah}(h]h ]h"]h$]h&]uh1jhjIubhD then it will match all events within the system for a given module.}(hjIhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKOhjmhhubeh}(h]via-the-set-event-interfaceah ]h"]!2.1 via the 'set_event' interfaceah$]h&]uh1hhj\hhhhhKubh)}(hhh](h)}(h2.2 Via the 'enable' toggleh]h2.2 Via the ‘enable’ toggle}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKSubh)}(h]The events available are also listed in /sys/kernel/tracing/events/ hierarchy of directories.h]h]The events available are also listed in /sys/kernel/tracing/events/ hierarchy of directories.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKUhjhhubh)}(h To enable event 'sched_wakeup'::h]h#To enable event ‘sched_wakeup’:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKXhjhhubj)}(h?# echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enableh]h?# echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKZhjhhubh)}(hTo disable it::h]hTo disable it:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK\hjhhubj)}(h?# echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enableh]h?# echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhK^hjhhubh)}(h)To enable all events in sched subsystem::h]h(To enable all events in sched subsystem:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK`hjhhubj)}(h2# echo 1 > /sys/kernel/tracing/events/sched/enableh]h2# echo 1 > /sys/kernel/tracing/events/sched/enable}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKbhjhhubh)}(hTo enable all events::h]hTo enable all events:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKdhjhhubj)}(h,# echo 1 > /sys/kernel/tracing/events/enableh]h,# echo 1 > /sys/kernel/tracing/events/enable}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKfhjhhubh)}(h?When reading one of these enable files, there are four results:h]h?When reading one of these enable files, there are four results:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhjhhubh block_quote)}(h- 0 - all events this file affects are disabled - 1 - all events this file affects are enabled - X - there is a mixture of events enabled and disabled - ? - this file does not affect any event h]h bullet_list)}(hhh](h list_item)}(h-0 - all events this file affects are disabledh]h)}(hj/h]h-0 - all events this file affects are disabled}(hj1hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKjhj-ubah}(h]h ]h"]h$]h&]uh1j+hj(ubj,)}(h,1 - all events this file affects are enabledh]h)}(hjFh]h,1 - all events this file affects are enabled}(hjHhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKkhjDubah}(h]h ]h"]h$]h&]uh1j+hj(ubj,)}(h5X - there is a mixture of events enabled and disabledh]h)}(hj]h]h5X - there is a mixture of events enabled and disabled}(hj_hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKlhj[ubah}(h]h ]h"]h$]h&]uh1j+hj(ubj,)}(h(? - this file does not affect any event h]h)}(h'? - this file does not affect any eventh]h'? - this file does not affect any event}(hjvhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKmhjrubah}(h]h ]h"]h$]h&]uh1j+hj(ubeh}(h]h ]h"]h$]h&]bullet-uh1j&hhhKjhj"ubah}(h]h ]h"]h$]h&]uh1j hhhKjhjhhubeh}(h]via-the-enable-toggleah ]h"]2.2 via the 'enable' toggleah$]h&]uh1hhj\hhhhhKSubh)}(hhh](h)}(h2.3 Boot optionh]h2.3 Boot option}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKpubh)}(h>In order to facilitate early boot debugging, use boot option::h]h=In order to facilitate early boot debugging, use boot option:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKrhjhhubj)}(htrace_event=[event-list]h]htrace_event=[event-list]}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKthjhhubh)}(hQevent-list is a comma separated list of events. See section 2.1 for event format.h]hQevent-list is a comma separated list of events. See section 2.1 for event format.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKvhjhhubeh}(h] boot-optionah ]h"]2.3 boot optionah$]h&]uh1hhj\hhhhhKpubeh}(h]using-event-tracingah ]h"]2. using event tracingah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h'3. Defining an event-enabled tracepointh]h'3. Defining an event-enabled tracepoint}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKzubh)}(h0See The example provided in samples/trace_eventsh]h0See The example provided in samples/trace_events}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK|hjhhubeh}(h]$defining-an-event-enabled-tracepointah ]h"]'3. defining an event-enabled tracepointah$]h&]uh1hhhhhhhhKzubh)}(hhh](h)}(h4. Event formatsh]h4. Event formats}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXEach trace event has a 'format' file associated with it that contains a description of each field in a logged event. This information can be used to parse the binary trace stream, and is also the place to find the field names that can be used in event filters (see section 5).h]hXEach trace event has a ‘format’ file associated with it that contains a description of each field in a logged event. This information can be used to parse the binary trace stream, and is also the place to find the field names that can be used in event filters (see section 5).}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hIt also displays the format string that will be used to print the event in text mode, along with the event name and ID used for profiling.h]hIt also displays the format string that will be used to print the event in text mode, along with the event name and ID used for profiling.}(hj1hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hEvery event has a set of ``common`` fields associated with it; these are the fields prefixed with ``common_``. The other fields vary between events and correspond to the fields defined in the TRACE_EVENT definition for that event.h](hEvery event has a set of }(hj?hhhNhNubj)}(h ``common``h]hcommon}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1jhj?ubh? fields associated with it; these are the fields prefixed with }(hj?hhhNhNubj)}(h ``common_``h]hcommon_}(hjYhhhNhNubah}(h]h ]h"]h$]h&]uh1jhj?ubhz. The other fields vary between events and correspond to the fields defined in the TRACE_EVENT definition for that event.}(hj?hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h'Each field in the format has the form::h]h&Each field in the format has the form:}(hjqhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(h.field:field-type field-name; offset:N; size:N;h]h.field:field-type field-name; offset:N; size:N;}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(hlwhere offset is the offset of the field in the trace record and size is the size of the data item, in bytes.h]hlwhere offset is the offset of the field in the trace record and size is the size of the data item, in bytes.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hLFor example, here's the information displayed for the 'sched_wakeup' event::h]hQFor example, here’s the information displayed for the ‘sched_wakeup’ event:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hX2# cat /sys/kernel/tracing/events/sched/sched_wakeup/format name: sched_wakeup ID: 60 format: field:unsigned short common_type; offset:0; size:2; field:unsigned char common_flags; offset:2; size:1; field:unsigned char common_preempt_count; offset:3; size:1; field:int common_pid; offset:4; size:4; field:int common_tgid; offset:8; size:4; field:char comm[TASK_COMM_LEN]; offset:12; size:16; field:pid_t pid; offset:28; size:4; field:int prio; offset:32; size:4; field:int success; offset:36; size:4; field:int cpu; offset:40; size:4; print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, REC->prio, REC->success, REC->cpuh]hX2# cat /sys/kernel/tracing/events/sched/sched_wakeup/format name: sched_wakeup ID: 60 format: field:unsigned short common_type; offset:0; size:2; field:unsigned char common_flags; offset:2; size:1; field:unsigned char common_preempt_count; offset:3; size:1; field:int common_pid; offset:4; size:4; field:int common_tgid; offset:8; size:4; field:char comm[TASK_COMM_LEN]; offset:12; size:16; field:pid_t pid; offset:28; size:4; field:int prio; offset:32; size:4; field:int success; offset:36; size:4; field:int cpu; offset:40; size:4; print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, REC->prio, REC->success, REC->cpu}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(hThis event contains 10 fields, the first 5 common and the remaining 5 event-specific. All the fields for this event are numeric, except for 'comm' which is a string, a distinction important for event filtering.h]hThis event contains 10 fields, the first 5 common and the remaining 5 event-specific. All the fields for this event are numeric, except for ‘comm’ which is a string, a distinction important for event filtering.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h] event-formatsah ]h"]4. event formatsah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h5. Event filteringh]h5. Event filtering}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXTrace events can be filtered in the kernel by associating boolean 'filter expressions' with them. As soon as an event is logged into the trace buffer, its fields are checked against the filter expression associated with that event type. An event with field values that 'match' the filter will appear in the trace output, and an event whose values don't match will be discarded. An event with no filter associated with it matches everything, and is the default when no filter has been set for an event.h]hXTrace events can be filtered in the kernel by associating boolean ‘filter expressions’ with them. As soon as an event is logged into the trace buffer, its fields are checked against the filter expression associated with that event type. An event with field values that ‘match’ the filter will appear in the trace output, and an event whose values don’t match will be discarded. An event with no filter associated with it matches everything, and is the default when no filter has been set for an event.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(h5.1 Expression syntaxh]h5.1 Expression syntax}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXRA filter expression consists of one or more 'predicates' that can be combined using the logical operators '&&' and '||'. A predicate is simply a clause that compares the value of a field contained within a logged event with a constant value and returns either 0 or 1 depending on whether the field value matched (1) or didn't match (0)::h]hX_A filter expression consists of one or more ‘predicates’ that can be combined using the logical operators ‘&&’ and ‘||’. A predicate is simply a clause that compares the value of a field contained within a logged event with a constant value and returns either 0 or 1 depending on whether the field value matched (1) or didn’t match (0):}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(h$field-name relational-operator valueh]h$field-name relational-operator value}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(hParentheses can be used to provide arbitrary logical groupings and double-quotes can be used to prevent the shell from interpreting operators as shell metacharacters.h]hParentheses can be used to provide arbitrary logical groupings and double-quotes can be used to prevent the shell from interpreting operators as shell metacharacters.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hqThe field-names available for use in filters can be found in the 'format' files for trace events (see section 4).h]huThe field-names available for use in filters can be found in the ‘format’ files for trace events (see section 4).}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hFThe relational-operators depend on the type of the field being tested:h]hFThe relational-operators depend on the type of the field being tested:}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h/The operators available for numeric fields are:h]h/The operators available for numeric fields are:}(hjChhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h==, !=, <, <=, >, >=, &h]h==, !=, <, <=, >, >=, &}(hjQhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hAnd for string fields they are:h]hAnd for string fields they are:}(hj_hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h ==, !=, ~h]h ==, !=, ~}(hjmhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hZThe glob (~) accepts a wild card character (\*,?) and character classes ([). For example::h]hYThe glob (~) accepts a wild card character (*,?) and character classes ([). For example:}(hj{hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hJprev_comm ~ "*sh" prev_comm ~ "sh*" prev_comm ~ "*sh*" prev_comm ~ "ba*sh"h]hJprev_comm ~ "*sh" prev_comm ~ "sh*" prev_comm ~ "*sh*" prev_comm ~ "ba*sh"}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(hIf the field is a pointer that points into user space (for example "filename" from sys_enter_openat), then you have to append ".ustring" to the field name::h]hIf the field is a pointer that points into user space (for example “filename” from sys_enter_openat), then you have to append “.ustring” to the field name:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hfilename.ustring ~ "password"h]hfilename.ustring ~ "password"}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(hbAs the kernel will have to know how to retrieve the memory that the pointer is at from user space.h]hbAs the kernel will have to know how to retrieve the memory that the pointer is at from user space.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hQYou can convert any long type to a function address and search by function name::h]hPYou can convert any long type to a function address and search by function name:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(h,call_site.function == security_prepare_credsh]h,call_site.function == security_prepare_creds}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(hX5The above will filter when the field "call_site" falls on the address within "security_prepare_creds". That is, it will compare the value of "call_site" and the filter will return true if it is greater than or equal to the start of the function "security_prepare_creds" and less than the end of that function.h]hXEThe above will filter when the field “call_site” falls on the address within “security_prepare_creds”. That is, it will compare the value of “call_site” and the filter will return true if it is greater than or equal to the start of the function “security_prepare_creds” and less than the end of that function.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hpThe ".function" postfix can only be attached to values of size long, and can only be compared with "==" or "!=".h]h|The “.function” postfix can only be attached to values of size long, and can only be compared with “==” or “!=”.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hCpumask fields or scalar fields that encode a CPU number can be filtered using a user-provided cpumask in cpulist format. The format is as follows::h]hCpumask fields or scalar fields that encode a CPU number can be filtered using a user-provided cpumask in cpulist format. The format is as follows:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hCPUS{$cpulist}h]hCPUS{$cpulist}}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubh)}(h-Operators available to cpumask filtering are:h]h-Operators available to cpumask filtering are:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h& (intersection), ==, !=h]h& (intersection), ==, !=}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(heFor example, this will filter events that have their .target_cpu field present in the given cpumask::h]hdFor example, this will filter events that have their .target_cpu field present in the given cpumask:}(hj1hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(htarget_cpu & CPUS{17-42}h]htarget_cpu & CPUS{17-42}}hj?sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubeh}(h]expression-syntaxah ]h"]5.1 expression syntaxah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(h5.2 Setting filtersh]h5.2 Setting filters}(hjXhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjUhhhhhMubh)}(hpA filter for an individual event is set by writing a filter expression to the 'filter' file for the given event.h]htA filter for an individual event is set by writing a filter expression to the ‘filter’ file for the given event.}(hjfhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjUhhubh)}(h For example::h]h For example:}(hjthhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjUhhubj)}(h]# cd /sys/kernel/tracing/events/sched/sched_wakeup # echo "common_preempt_count > 4" > filterh]h]# cd /sys/kernel/tracing/events/sched/sched_wakeup # echo "common_preempt_count > 4" > filter}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhM hjUhhubh)}(h"A slightly more involved example::h]h!A slightly more involved example:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjUhhubj)}(h# cd /sys/kernel/tracing/events/signal/signal_generate # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filterh]h# cd /sys/kernel/tracing/events/signal/signal_generate # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjUhhubh)}(hIf there is an error in the expression, you'll get an 'Invalid argument' error when setting it, and the erroneous string along with an error message can be seen by looking at the filter e.g.::h]hIf there is an error in the expression, you’ll get an ‘Invalid argument’ error when setting it, and the erroneous string along with an error message can be seen by looking at the filter e.g.:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjUhhubj)}(hX# cd /sys/kernel/tracing/events/signal/signal_generate # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter -bash: echo: write error: Invalid argument # cat filter ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash ^ parse_error: Field not foundh]hX# cd /sys/kernel/tracing/events/signal/signal_generate # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter -bash: echo: write error: Invalid argument # cat filter ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash ^ parse_error: Field not found}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjUhhubh)}(hCurrently the caret ('^') for an error always appears at the beginning of the filter string; the error message should still be useful though even without more accurate position info.h]hCurrently the caret (‘^’) for an error always appears at the beginning of the filter string; the error message should still be useful though even without more accurate position info.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjUhhubeh}(h]setting-filtersah ]h"]5.2 setting filtersah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h5.2.1 Filter limitationsh]h5.2.1 Filter limitations}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM%ubh)}(hXIf a filter is placed on a string pointer ``(char *)`` that does not point to a string on the ring buffer, but instead points to kernel or user space memory, then, for safety reasons, at most 1024 bytes of the content is copied onto a temporary buffer to do the compare. If the copy of the memory faults (the pointer points to memory that should not be accessed), then the string compare will be treated as not matching.h](h*If a filter is placed on a string pointer }(hjhhhNhNubj)}(h ``(char *)``h]h(char *)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhXn that does not point to a string on the ring buffer, but instead points to kernel or user space memory, then, for safety reasons, at most 1024 bytes of the content is copied onto a temporary buffer to do the compare. If the copy of the memory faults (the pointer points to memory that should not be accessed), then the string compare will be treated as not matching.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM'hjhhubeh}(h]filter-limitationsah ]h"]5.2.1 filter limitationsah$]h&]uh1hhjhhhhhM%ubh)}(hhh](h)}(h5.3 Clearing filtersh]h5.3 Clearing filters}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM/ubh)}(hITo clear the filter for an event, write a '0' to the event's filter file.h]hOTo clear the filter for an event, write a ‘0’ to the event’s filter file.}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM1hjhhubh)}(h_To clear the filters for all events in a subsystem, write a '0' to the subsystem's filter file.h]heTo clear the filters for all events in a subsystem, write a ‘0’ to the subsystem’s filter file.}(hj6hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM4hjhhubeh}(h]clearing-filtersah ]h"]5.3 clearing filtersah$]h&]uh1hhjhhhhhM/ubh)}(hhh](h)}(h5.4 Subsystem filtersh]h5.4 Subsystem filters}(hjOhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjLhhhhhM8ubh)}(hXFor convenience, filters for every event in a subsystem can be set or cleared as a group by writing a filter expression into the filter file at the root of the subsystem. Note however, that if a filter for any event within the subsystem lacks a field specified in the subsystem filter, or if the filter can't be applied for any other reason, the filter for that event will retain its previous setting. This can result in an unintended mixture of filters which could lead to confusing (to the user who might think different filters are in effect) trace output. Only filters that reference just the common fields can be guaranteed to propagate successfully to all events.h]hXFor convenience, filters for every event in a subsystem can be set or cleared as a group by writing a filter expression into the filter file at the root of the subsystem. Note however, that if a filter for any event within the subsystem lacks a field specified in the subsystem filter, or if the filter can’t be applied for any other reason, the filter for that event will retain its previous setting. This can result in an unintended mixture of filters which could lead to confusing (to the user who might think different filters are in effect) trace output. Only filters that reference just the common fields can be guaranteed to propagate successfully to all events.}(hj]hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM:hjLhhubh)}(hOHere are a few subsystem filter examples that also illustrate the above points:h]hOHere are a few subsystem filter examples that also illustrate the above points:}(hjkhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMEhjLhhubh)}(h8Clear the filters on all events in the sched subsystem::h]h7Clear the filters on all events in the sched subsystem:}(hjyhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMHhjLhhubj)}(hu# cd /sys/kernel/tracing/events/sched # echo 0 > filter # cat sched_switch/filter none # cat sched_wakeup/filter noneh]hu# cd /sys/kernel/tracing/events/sched # echo 0 > filter # cat sched_switch/filter none # cat sched_wakeup/filter none}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMJhjLhhubh)}(hvSet a filter using only common fields for all events in the sched subsystem (all events end up with the same filter)::h]huSet a filter using only common fields for all events in the sched subsystem (all events end up with the same filter):}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMQhjLhhubj)}(h# cd /sys/kernel/tracing/events/sched # echo common_pid == 0 > filter # cat sched_switch/filter common_pid == 0 # cat sched_wakeup/filter common_pid == 0h]h# cd /sys/kernel/tracing/events/sched # echo common_pid == 0 > filter # cat sched_switch/filter common_pid == 0 # cat sched_wakeup/filter common_pid == 0}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMThjLhhubh)}(hAttempt to set a filter using a non-common field for all events in the sched subsystem (all events but those that have a prev_pid field retain their old filters)::h]hAttempt to set a filter using a non-common field for all events in the sched subsystem (all events but those that have a prev_pid field retain their old filters):}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM[hjLhhubj)}(h# cd /sys/kernel/tracing/events/sched # echo prev_pid == 0 > filter # cat sched_switch/filter prev_pid == 0 # cat sched_wakeup/filter common_pid == 0h]h# cd /sys/kernel/tracing/events/sched # echo prev_pid == 0 > filter # cat sched_switch/filter prev_pid == 0 # cat sched_wakeup/filter common_pid == 0}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhM_hjLhhubeh}(h]subsystem-filtersah ]h"]5.4 subsystem filtersah$]h&]uh1hhjhhhhhM8ubh)}(hhh](h)}(h5.5 PID filteringh]h5.5 PID filtering}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMgubh)}(hThe set_event_pid file in the same directory as the top events directory exists, will filter all events from tracing any task that does not have the PID listed in the set_event_pid file. ::h]hThe set_event_pid file in the same directory as the top events directory exists, will filter all events from tracing any task that does not have the PID listed in the set_event_pid file.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMihjhhubj)}(hK# cd /sys/kernel/tracing # echo $$ > set_event_pid # echo 1 > events/enableh]hK# cd /sys/kernel/tracing # echo $$ > set_event_pid # echo 1 > events/enable}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMnhjhhubh)}(h,Will only trace events for the current task.h]h,Will only trace events for the current task.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMrhjhhubh)}(hGTo add more PIDs without losing the PIDs already included, use '>>'. ::h]hHTo add more PIDs without losing the PIDs already included, use ‘>>’.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMthjhhubj)}(h!# echo 123 244 1 >> set_event_pidh]h!# echo 123 244 1 >> set_event_pid}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMwhjhhubeh}(h] pid-filteringah ]h"]5.5 pid filteringah$]h&]uh1hhjhhhhhMgubeh}(h]event-filteringah ]h"]5. event filteringah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h6. Event triggersh]h6. Event triggers}(hj? hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj< hhhhhM{ubh)}(hXTrace events can be made to conditionally invoke trigger 'commands' which can take various forms and are described in detail below; examples would be enabling or disabling other trace events or invoking a stack trace whenever the trace event is hit. Whenever a trace event with attached triggers is invoked, the set of trigger commands associated with that event is invoked. Any given trigger can additionally have an event filter of the same form as described in section 5 (Event filtering) associated with it - the command will only be invoked if the event being invoked passes the associated filter. If no filter is associated with the trigger, it always passes.h]hXTrace events can be made to conditionally invoke trigger ‘commands’ which can take various forms and are described in detail below; examples would be enabling or disabling other trace events or invoking a stack trace whenever the trace event is hit. Whenever a trace event with attached triggers is invoked, the set of trigger commands associated with that event is invoked. Any given trigger can additionally have an event filter of the same form as described in section 5 (Event filtering) associated with it - the command will only be invoked if the event being invoked passes the associated filter. If no filter is associated with the trigger, it always passes.}(hjM hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM}hj< hhubh)}(hTriggers are added to and removed from a particular event by writing trigger expressions to the 'trigger' file for the given event.h]hTriggers are added to and removed from a particular event by writing trigger expressions to the ‘trigger’ file for the given event.}(hj[ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj< hhubh)}(hA given event can have any number of triggers associated with it, subject to any restrictions that individual commands may have in that regard.h]hA given event can have any number of triggers associated with it, subject to any restrictions that individual commands may have in that regard.}(hji hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj< hhubh)}(hX Event triggers are implemented on top of "soft" mode, which means that whenever a trace event has one or more triggers associated with it, the event is activated even if it isn't actually enabled, but is disabled in a "soft" mode. That is, the tracepoint will be called, but just will not be traced, unless of course it's actually enabled. This scheme allows triggers to be invoked even for events that aren't enabled, and also allows the current event filter implementation to be used for conditionally invoking triggers.h]hXEvent triggers are implemented on top of “soft” mode, which means that whenever a trace event has one or more triggers associated with it, the event is activated even if it isn’t actually enabled, but is disabled in a “soft” mode. That is, the tracepoint will be called, but just will not be traced, unless of course it’s actually enabled. This scheme allows triggers to be invoked even for events that aren’t enabled, and also allows the current event filter implementation to be used for conditionally invoking triggers.}(hjw hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj< hhubh)}(hXHThe syntax for event triggers is roughly based on the syntax for set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands' section of Documentation/trace/ftrace.rst), but there are major differences and the implementation isn't currently tied to it in any way, so beware about making generalizations between the two.h]hXRThe syntax for event triggers is roughly based on the syntax for set_ftrace_filter ‘ftrace filter commands’ (see the ‘Filter commands’ section of Documentation/trace/ftrace.rst), but there are major differences and the implementation isn’t currently tied to it in any way, so beware about making generalizations between the two.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj< hhubj)}(hWriting into trace_marker (See Documentation/trace/ftrace.rst) can also enable triggers that are written into /sys/kernel/tracing/events/ftrace/print/triggerh]h)}(hWriting into trace_marker (See Documentation/trace/ftrace.rst) can also enable triggers that are written into /sys/kernel/tracing/events/ftrace/print/triggerh]hWriting into trace_marker (See Documentation/trace/ftrace.rst) can also enable triggers that are written into /sys/kernel/tracing/events/ftrace/print/trigger}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jhj< hhhhhNubh)}(hhh](h)}(h6.1 Expression syntaxh]h6.1 Expression syntax}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hATriggers are added by echoing the command to the 'trigger' file::h]hDTriggers are added by echoing the command to the ‘trigger’ file:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(h.# echo 'command[:count] [if filter]' > triggerh]h.# echo 'command[:count] [if filter]' > trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(h^Triggers are removed by echoing the same command but starting with '!' to the 'trigger' file::h]heTriggers are removed by echoing the same command but starting with ‘!’ to the ‘trigger’ file:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(h/# echo '!command[:count] [if filter]' > triggerh]h/# echo '!command[:count] [if filter]' > trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(hThe [if filter] part isn't used in matching commands when removing, so leaving that off in a '!' command will accomplish the same thing as having it in.h]hThe [if filter] part isn’t used in matching commands when removing, so leaving that off in a ‘!’ command will accomplish the same thing as having it in.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hWThe filter syntax is the same as that described in the 'Event filtering' section above.h]h[The filter syntax is the same as that described in the ‘Event filtering’ section above.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hXFor ease of use, writing to the trigger file using '>' currently just adds or removes a single trigger and there's no explicit '>>' support ('>' actually behaves like '>>') or truncation support to remove all triggers (you have to use '!' for each one added.)h]hXFor ease of use, writing to the trigger file using ‘>’ currently just adds or removes a single trigger and there’s no explicit ‘>>’ support (‘>’ actually behaves like ‘>>’) or truncation support to remove all triggers (you have to use ‘!’ for each one added.)}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]id1ah ]h"]6.1 expression syntaxah$]h&]uh1hhj< hhhhhMubh)}(hhh](h)}(h6.2 Supported trigger commandsh]h6.2 Supported trigger commands}(hj) hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj& hhhhhMubh)}(h%The following commands are supported:h]h%The following commands are supported:}(hj7 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj& hhubj')}(hhh](j,)}(hXenable_event/disable_event These commands can enable or disable another trace event whenever the triggering event is hit. When these commands are registered, the other trace event is activated, but disabled in a "soft" mode. That is, the tracepoint will be called, but just will not be traced. The event tracepoint stays in this mode as long as there's a trigger in effect that can trigger it. For example, the following trigger causes kmalloc events to be traced when a read system call is entered, and the :1 at the end specifies that this enablement happens only once:: # echo 'enable_event:kmem:kmalloc:1' > \ /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger The following trigger causes kmalloc events to stop being traced when a read system call exits. This disablement happens on every read system call exit:: # echo 'disable_event:kmem:kmalloc' > \ /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger The format is:: enable_event::[:count] disable_event::[:count] To remove the above commands:: # echo '!enable_event:kmem:kmalloc:1' > \ /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger # echo '!disable_event:kmem:kmalloc' > \ /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger Note that there can be any number of enable/disable_event triggers per triggering event, but there can only be one trigger per triggered event. e.g. sys_enter_read can have triggers enabling both kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they could be combined into a single filter on kmem:kmalloc though). h](h)}(henable_event/disable_eventh]henable_event/disable_event}(hjL hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubh)}(hXoThese commands can enable or disable another trace event whenever the triggering event is hit. When these commands are registered, the other trace event is activated, but disabled in a "soft" mode. That is, the tracepoint will be called, but just will not be traced. The event tracepoint stays in this mode as long as there's a trigger in effect that can trigger it.h]hXuThese commands can enable or disable another trace event whenever the triggering event is hit. When these commands are registered, the other trace event is activated, but disabled in a “soft” mode. That is, the tracepoint will be called, but just will not be traced. The event tracepoint stays in this mode as long as there’s a trigger in effect that can trigger it.}(hjZ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubh)}(hFor example, the following trigger causes kmalloc events to be traced when a read system call is entered, and the :1 at the end specifies that this enablement happens only once::h]hFor example, the following trigger causes kmalloc events to be traced when a read system call is entered, and the :1 at the end specifies that this enablement happens only once:}(hjh hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubj)}(hg# echo 'enable_event:kmem:kmalloc:1' > \ /sys/kernel/tracing/events/syscalls/sys_enter_read/triggerh]hg# echo 'enable_event:kmem:kmalloc:1' > \ /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger}hjv sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjH ubh)}(hThe following trigger causes kmalloc events to stop being traced when a read system call exits. This disablement happens on every read system call exit::h]hThe following trigger causes kmalloc events to stop being traced when a read system call exits. This disablement happens on every read system call exit:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubj)}(he# echo 'disable_event:kmem:kmalloc' > \ /sys/kernel/tracing/events/syscalls/sys_exit_read/triggerh]he# echo 'disable_event:kmem:kmalloc' > \ /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjH ubh)}(hThe format is::h]hThe format is:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubj)}(hLenable_event::[:count] disable_event::[:count]h]hLenable_event::[:count] disable_event::[:count]}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjH ubh)}(hTo remove the above commands::h]hTo remove the above commands:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubj)}(h# echo '!enable_event:kmem:kmalloc:1' > \ /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger # echo '!disable_event:kmem:kmalloc' > \ /sys/kernel/tracing/events/syscalls/sys_exit_read/triggerh]h# echo '!enable_event:kmem:kmalloc:1' > \ /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger # echo '!disable_event:kmem:kmalloc' > \ /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjH ubh)}(hXNote that there can be any number of enable/disable_event triggers per triggering event, but there can only be one trigger per triggered event. e.g. sys_enter_read can have triggers enabling both kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they could be combined into a single filter on kmem:kmalloc though).h]hXNote that there can be any number of enable/disable_event triggers per triggering event, but there can only be one trigger per triggered event. e.g. sys_enter_read can have triggers enabling both kmem:kmalloc and sched:sched_switch, but can’t have two kmem:kmalloc versions such as kmem:kmalloc and kmem:kmalloc:1 or ‘kmem:kmalloc if bytes_req == 256’ and ‘kmem:kmalloc if bytes_alloc == 256’ (they could be combined into a single filter on kmem:kmalloc though).}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjH ubeh}(h]h ]h"]h$]h&]uh1j+hjE hhhhhNubj,)}(hX*stacktrace This command dumps a stacktrace in the trace buffer whenever the triggering event occurs. For example, the following trigger dumps a stacktrace every time the kmalloc tracepoint is hit:: # echo 'stacktrace' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger The following trigger dumps a stacktrace the first 5 times a kmalloc request happens with a size >= 64K:: # echo 'stacktrace:5 if bytes_req >= 65536' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger The format is:: stacktrace[:count] To remove the above commands:: # echo '!stacktrace' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger # echo '!stacktrace:5 if bytes_req >= 65536' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger The latter can also be removed more simply by the following (without the filter):: # echo '!stacktrace:5' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger Note that there can be only one stacktrace trigger per triggering event. h](h)}(h stacktraceh]h stacktrace}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubh)}(hYThis command dumps a stacktrace in the trace buffer whenever the triggering event occurs.h]hYThis command dumps a stacktrace in the trace buffer whenever the triggering event occurs.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubh)}(h`For example, the following trigger dumps a stacktrace every time the kmalloc tracepoint is hit::h]h_For example, the following trigger dumps a stacktrace every time the kmalloc tracepoint is hit:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubj)}(hM# echo 'stacktrace' > \ /sys/kernel/tracing/events/kmem/kmalloc/triggerh]hM# echo 'stacktrace' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj ubh)}(hiThe following trigger dumps a stacktrace the first 5 times a kmalloc request happens with a size >= 64K::h]hhThe following trigger dumps a stacktrace the first 5 times a kmalloc request happens with a size >= 64K:}(hj( hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubj)}(he# echo 'stacktrace:5 if bytes_req >= 65536' > \ /sys/kernel/tracing/events/kmem/kmalloc/triggerh]he# echo 'stacktrace:5 if bytes_req >= 65536' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger}hj6 sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj ubh)}(hThe format is::h]hThe format is:}(hjD hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubj)}(hstacktrace[:count]h]hstacktrace[:count]}hjR sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj ubh)}(hTo remove the above commands::h]hTo remove the above commands:}(hj` hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubj)}(h# echo '!stacktrace' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger # echo '!stacktrace:5 if bytes_req >= 65536' > \ /sys/kernel/tracing/events/kmem/kmalloc/triggerh]h# echo '!stacktrace' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger # echo '!stacktrace:5 if bytes_req >= 65536' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger}hjn sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj ubh)}(hRThe latter can also be removed more simply by the following (without the filter)::h]hQThe latter can also be removed more simply by the following (without the filter):}(hj| hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hj ubj)}(hP# echo '!stacktrace:5' > \ /sys/kernel/tracing/events/kmem/kmalloc/triggerh]hP# echo '!stacktrace:5' > \ /sys/kernel/tracing/events/kmem/kmalloc/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhM hj ubh)}(hHNote that there can be only one stacktrace trigger per triggering event.h]hHNote that there can be only one stacktrace trigger per triggering event.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubeh}(h]h ]h"]h$]h&]uh1j+hjE hhhhhNubj,)}(hXsnapshot This command causes a snapshot to be triggered whenever the triggering event occurs. The following command creates a snapshot every time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, the snapshot trace buffer would capture those events when the trigger event occurred:: # echo 'snapshot if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger To only snapshot once:: # echo 'snapshot:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger To remove the above commands:: # echo '!snapshot if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!snapshot:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger Note that there can be only one snapshot trigger per triggering event. h](h)}(hsnapshoth]hsnapshot}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubh)}(hTThis command causes a snapshot to be triggered whenever the triggering event occurs.h]hTThis command causes a snapshot to be triggered whenever the triggering event occurs.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubh)}(hThe following command creates a snapshot every time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, the snapshot trace buffer would capture those events when the trigger event occurred::h]hThe following command creates a snapshot every time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, the snapshot trace buffer would capture those events when the trigger event occurred:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubj)}(h^# echo 'snapshot if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/triggerh]h^# echo 'snapshot if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj ubh)}(hTo only snapshot once::h]hTo only snapshot once:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubj)}(h`# echo 'snapshot:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/triggerh]h`# echo 'snapshot:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhM!hj ubh)}(hTo remove the above commands::h]hTo remove the above commands:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM$hj ubj)}(h# echo '!snapshot if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!snapshot:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/triggerh]h# echo '!snapshot if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!snapshot:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhM&hj ubh)}(hFNote that there can be only one snapshot trigger per triggering event.h]hFNote that there can be only one snapshot trigger per triggering event.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM,hj ubeh}(h]h ]h"]h$]h&]uh1j+hjE hhhhhNubj,)}(hXKtraceon/traceoff These commands turn tracing on and off when the specified events are hit. The parameter determines how many times the tracing system is turned on and off. If unspecified, there is no limit. The following command turns tracing off the first time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, you could then examine the trace buffer to see the sequence of events that led up to the trigger event:: # echo 'traceoff:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger To always disable tracing when nr_rq > 1:: # echo 'traceoff if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger To remove the above commands:: # echo '!traceoff:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!traceoff if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger Note that there can be only one traceon or traceoff trigger per triggering event. h](h)}(htraceon/traceoffh]htraceon/traceoff}(hj8 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM/hj4 ubh)}(hThese commands turn tracing on and off when the specified events are hit. The parameter determines how many times the tracing system is turned on and off. If unspecified, there is no limit.xh]hThese commands turn tracing on and off when the specified events are hit. The parameter determines how many times the tracing system is turned on and off. If unspecified, there is no limit.}(hjF hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM1hj4 ubh)}(hXThe following command turns tracing off the first time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, you could then examine the trace buffer to see the sequence of events that led up to the trigger event::h]hXThe following command turns tracing off the first time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, you could then examine the trace buffer to see the sequence of events that led up to the trigger event:}(hjT hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM5hj4 ubj)}(h`# echo 'traceoff:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/triggerh]h`# echo 'traceoff:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger}hjb sbah}(h]h ]h"]h$]h&]jjuh1jhhhM;hj4 ubh)}(h+To always disable tracing when nr_rq > 1::h]h*To always disable tracing when nr_rq > 1:}(hjp hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM>hj4 ubj)}(h^# echo 'traceoff if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/triggerh]h^# echo 'traceoff if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger}hj~ sbah}(h]h ]h"]h$]h&]jjuh1jhhhM@hj4 ubh)}(hTo remove the above commands::h]hTo remove the above commands:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMChj4 ubj)}(h# echo '!traceoff:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!traceoff if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/triggerh]h# echo '!traceoff:1 if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!traceoff if nr_rq > 1' > \ /sys/kernel/tracing/events/block/block_unplug/trigger}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMEhj4 ubh)}(hQNote that there can be only one traceon or traceoff trigger per triggering event.h]hQNote that there can be only one traceon or traceoff trigger per triggering event.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMKhj4 ubeh}(h]h ]h"]h$]h&]uh1j+hjE hhhhhNubj,)}(hX*hist This command aggregates event hits into a hash table keyed on one or more trace event format fields (or stacktrace) and a set of running totals derived from one or more trace event format fields and/or event counts (hitcount). See Documentation/trace/histogram.rst for details and examples. h](h)}(hhisth]hhist}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMNhj ubh)}(hThis command aggregates event hits into a hash table keyed on one or more trace event format fields (or stacktrace) and a set of running totals derived from one or more trace event format fields and/or event counts (hitcount).h]hThis command aggregates event hits into a hash table keyed on one or more trace event format fields (or stacktrace) and a set of running totals derived from one or more trace event format fields and/or event counts (hitcount).}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMPhj ubh)}(h?See Documentation/trace/histogram.rst for details and examples.h]h?See Documentation/trace/histogram.rst for details and examples.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMUhj ubeh}(h]h ]h"]h$]h&]uh1j+hjE hhhhhNubeh}(h]h ]h"]h$]h&]jjuh1j&hhhMhj& hhubeh}(h]supported-trigger-commandsah ]h"]6.2 supported trigger commandsah$]h&]uh1hhj< hhhhhMubeh}(h]event-triggersah ]h"]6. event triggersah$]h&]uh1hhhhhhhhM{ubh)}(hhh](h)}(h7. In-kernel trace event APIh]h7. In-kernel trace event API}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMXubh)}(hX In most cases, the command-line interface to trace events is more than sufficient. Sometimes, however, applications might find the need for more complex relationships than can be expressed through a simple series of linked command-line expressions, or putting together sets of commands may be simply too cumbersome. An example might be an application that needs to 'listen' to the trace stream in order to maintain an in-kernel state machine detecting, for instance, when an illegal kernel state occurs in the scheduler.h]hXIn most cases, the command-line interface to trace events is more than sufficient. Sometimes, however, applications might find the need for more complex relationships than can be expressed through a simple series of linked command-line expressions, or putting together sets of commands may be simply too cumbersome. An example might be an application that needs to ‘listen’ to the trace stream in order to maintain an in-kernel state machine detecting, for instance, when an illegal kernel state occurs in the scheduler.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMZhj hhubh)}(hX The trace event subsystem provides an in-kernel API allowing modules or other kernel code to generate user-defined 'synthetic' events at will, which can be used to either augment the existing trace stream and/or signal that a particular important state has occurred.h]hXThe trace event subsystem provides an in-kernel API allowing modules or other kernel code to generate user-defined ‘synthetic’ events at will, which can be used to either augment the existing trace stream and/or signal that a particular important state has occurred.}(hj% hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMchj hhubh)}(hSA similar in-kernel API is also available for creating kprobe and kretprobe events.h]hSA similar in-kernel API is also available for creating kprobe and kretprobe events.}(hj3 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhhj hhubh)}(hBoth the synthetic event and k/ret/probe event APIs are built on top of a lower-level "dynevent_cmd" event command API, which is also available for more specialized applications, or as the basis of other higher-level trace event APIs.h]hBoth the synthetic event and k/ret/probe event APIs are built on top of a lower-level “dynevent_cmd” event command API, which is also available for more specialized applications, or as the basis of other higher-level trace event APIs.}(hjA hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMkhj hhubh)}(hOThe API provided for these purposes is describe below and allows the following:h]hOThe API provided for these purposes is describe below and allows the following:}(hjO hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMphj hhubj!)}(h- dynamically creating synthetic event definitions - dynamically creating kprobe and kretprobe event definitions - tracing synthetic events from in-kernel code - the low-level "dynevent_cmd" API h]j')}(hhh](j,)}(h0dynamically creating synthetic event definitionsh]h)}(hjf h]h0dynamically creating synthetic event definitions}(hjh hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMshjd ubah}(h]h ]h"]h$]h&]uh1j+hja ubj,)}(h;dynamically creating kprobe and kretprobe event definitionsh]h)}(hj} h]h;dynamically creating kprobe and kretprobe event definitions}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMthj{ ubah}(h]h ]h"]h$]h&]uh1j+hja ubj,)}(h,tracing synthetic events from in-kernel codeh]h)}(hj h]h,tracing synthetic events from in-kernel code}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMuhj ubah}(h]h ]h"]h$]h&]uh1j+hja ubj,)}(h!the low-level "dynevent_cmd" API h]h)}(h the low-level "dynevent_cmd" APIh]h$the low-level “dynevent_cmd” API}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMvhj ubah}(h]h ]h"]h$]h&]uh1j+hja ubeh}(h]h ]h"]h$]h&]jjuh1j&hhhMshj] ubah}(h]h ]h"]h$]h&]uh1j hhhMshj hhubh)}(hhh](h)}(h37.1 Dyamically creating synthetic event definitionsh]h37.1 Dyamically creating synthetic event definitions}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMyubh)}(hbThere are a couple ways to create a new synthetic event from a kernel module or other kernel code.h]hbThere are a couple ways to create a new synthetic event from a kernel module or other kernel code.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM{hj hhubh)}(hXRThe first creates the event in one step, using synth_event_create(). In this method, the name of the event to create and an array defining the fields is supplied to synth_event_create(). If successful, a synthetic event with that name and fields will exist following that call. For example, to create a new "schedtest" synthetic event::h]hXUThe first creates the event in one step, using synth_event_create(). In this method, the name of the event to create and an array defining the fields is supplied to synth_event_create(). If successful, a synthetic event with that name and fields will exist following that call. For example, to create a new “schedtest” synthetic event:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM~hj hhubj)}(htret = synth_event_create("schedtest", sched_fields, ARRAY_SIZE(sched_fields), THIS_MODULE);h]htret = synth_event_create("schedtest", sched_fields, ARRAY_SIZE(sched_fields), THIS_MODULE);}hj sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(hThe sched_fields param in this example points to an array of struct synth_field_desc, each of which describes an event field by type and name::h]hThe sched_fields param in this example points to an array of struct synth_field_desc, each of which describes an event field by type and name:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(hXstatic struct synth_field_desc sched_fields[] = { { .type = "pid_t", .name = "next_pid_field" }, { .type = "char[16]", .name = "next_comm_field" }, { .type = "u64", .name = "ts_ns" }, { .type = "u64", .name = "ts_ms" }, { .type = "unsigned int", .name = "cpu" }, { .type = "char[64]", .name = "my_string_field" }, { .type = "int", .name = "my_int_field" }, };h]hXstatic struct synth_field_desc sched_fields[] = { { .type = "pid_t", .name = "next_pid_field" }, { .type = "char[16]", .name = "next_comm_field" }, { .type = "u64", .name = "ts_ns" }, { .type = "u64", .name = "ts_ms" }, { .type = "unsigned int", .name = "cpu" }, { .type = "char[64]", .name = "my_string_field" }, { .type = "int", .name = "my_int_field" }, };}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(h+See synth_field_size() for available types.h]h+See synth_field_size() for available types.}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hIIf field_name contains [n], the field is considered to be a static array.h]hIIf field_name contains [n], the field is considered to be a static array.}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hIf field_names contains[] (no subscript), the field is considered to be a dynamic array, which will only take as much space in the event as is required to hold the array.h]hIf field_names contains[] (no subscript), the field is considered to be a dynamic array, which will only take as much space in the event as is required to hold the array.}(hj@hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hXBecause space for an event is reserved before assigning field values to the event, using dynamic arrays implies that the piecewise in-kernel API described below can't be used with dynamic arrays. The other non-piecewise in-kernel APIs can, however, be used with dynamic arrays.h]hXBecause space for an event is reserved before assigning field values to the event, using dynamic arrays implies that the piecewise in-kernel API described below can’t be used with dynamic arrays. The other non-piecewise in-kernel APIs can, however, be used with dynamic arrays.}(hjNhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hIf the event is created from within a module, a pointer to the module must be passed to synth_event_create(). This will ensure that the trace buffer won't contain unreadable events when the module is removed.h]hIf the event is created from within a module, a pointer to the module must be passed to synth_event_create(). This will ensure that the trace buffer won’t contain unreadable events when the module is removed.}(hj\hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hNAt this point, the event object is ready to be used for generating new events.h]hNAt this point, the event object is ready to be used for generating new events.}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hIn the second method, the event is created in several steps. This allows events to be created dynamically and without the need to create and populate an array of fields beforehand.h]hIn the second method, the event is created in several steps. This allows events to be created dynamically and without the need to create and populate an array of fields beforehand.}(hjxhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hX|To use this method, an empty or partially empty synthetic event should first be created using synth_event_gen_cmd_start() or synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(), the name of the event along with one or more pairs of args each pair representing a 'type field_name;' field specification should be supplied. For synth_event_gen_cmd_array_start(), the name of the event along with an array of struct synth_field_desc should be supplied. Before calling synth_event_gen_cmd_start() or synth_event_gen_cmd_array_start(), the user should create and initialize a dynevent_cmd object using synth_event_cmd_init().h]hXTo use this method, an empty or partially empty synthetic event should first be created using synth_event_gen_cmd_start() or synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(), the name of the event along with one or more pairs of args each pair representing a ‘type field_name;’ field specification should be supplied. For synth_event_gen_cmd_array_start(), the name of the event along with an array of struct synth_field_desc should be supplied. Before calling synth_event_gen_cmd_start() or synth_event_gen_cmd_array_start(), the user should create and initialize a dynevent_cmd object using synth_event_cmd_init().}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hJFor example, to create a new "schedtest" synthetic event with two fields::h]hMFor example, to create a new “schedtest” synthetic event with two fields:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(hXstruct dynevent_cmd cmd; char *buf; /* Create a buffer to hold the generated command */ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); /* Before generating the command, initialize the cmd object */ synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE, "pid_t", "next_pid_field", "u64", "ts_ns");h]hXstruct dynevent_cmd cmd; char *buf; /* Create a buffer to hold the generated command */ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); /* Before generating the command, initialize the cmd object */ synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE, "pid_t", "next_pid_field", "u64", "ts_ns");}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(haAlternatively, using an array of struct synth_field_desc fields containing the same information::h]h`Alternatively, using an array of struct synth_field_desc fields containing the same information:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(h~ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE, fields, n_fields);h]h~ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE, fields, n_fields);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(hX6Once the synthetic event object has been created, it can then be populated with more fields. Fields are added one by one using synth_event_add_field(), supplying the dynevent_cmd object, a field type, and a field name. For example, to add a new int field named "intfield", the following call should be made::h]hX9Once the synthetic event object has been created, it can then be populated with more fields. Fields are added one by one using synth_event_add_field(), supplying the dynevent_cmd object, a field type, and a field name. For example, to add a new int field named “intfield”, the following call should be made:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(h5ret = synth_event_add_field(&cmd, "int", "intfield");h]h5ret = synth_event_add_field(&cmd, "int", "intfield");}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(hnSee synth_field_size() for available types. If field_name contains [n] the field is considered to be an array.h]hnSee synth_field_size() for available types. If field_name contains [n] the field is considered to be an array.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hA group of fields can also be added all at once using an array of synth_field_desc with add_synth_fields(). For example, this would add just the first four sched_fields::h]hA group of fields can also be added all at once using an array of synth_field_desc with add_synth_fields(). For example, this would add just the first four sched_fields:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(h4ret = synth_event_add_fields(&cmd, sched_fields, 4);h]h4ret = synth_event_add_fields(&cmd, sched_fields, 4);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(hIf you already have a string of the form 'type field_name', synth_event_add_field_str() can be used to add it as-is; it will also automatically append a ';' to the string.h]hIf you already have a string of the form ‘type field_name’, synth_event_add_field_str() can be used to add it as-is; it will also automatically append a ‘;’ to the string.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hOnce all the fields have been added, the event should be finalized and registered by calling the synth_event_gen_cmd_end() function::h]hOnce all the fields have been added, the event should be finalized and registered by calling the synth_event_gen_cmd_end() function:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj)}(h$ret = synth_event_gen_cmd_end(&cmd);h]h$ret = synth_event_gen_cmd_end(&cmd);}hj.sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhj hhubh)}(hKAt this point, the event object is ready to be used for tracing new events.h]hKAt this point, the event object is ready to be used for tracing new events.}(hj<hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]/dyamically-creating-synthetic-event-definitionsah ]h"]37.1 dyamically creating synthetic event definitionsah$]h&]uh1hhj hhhhhMyubh)}(hhh](h)}(h07.2 Tracing synthetic events from in-kernel codeh]h07.2 Tracing synthetic events from in-kernel code}(hjUhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjRhhhhhMubh)}(hXTo trace a synthetic event, there are several options. The first option is to trace the event in one call, using synth_event_trace() with a variable number of values, or synth_event_trace_array() with an array of values to be set. A second option can be used to avoid the need for a pre-formed array of values or list of arguments, via synth_event_trace_start() and synth_event_trace_end() along with synth_event_add_next_val() or synth_event_add_val() to add the values piecewise.h]hXTo trace a synthetic event, there are several options. The first option is to trace the event in one call, using synth_event_trace() with a variable number of values, or synth_event_trace_array() with an array of values to be set. A second option can be used to avoid the need for a pre-formed array of values or list of arguments, via synth_event_trace_start() and synth_event_trace_end() along with synth_event_add_next_val() or synth_event_add_val() to add the values piecewise.}(hjchhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjRhhubeh}(h],tracing-synthetic-events-from-in-kernel-codeah ]h"]07.2 tracing synthetic events from in-kernel codeah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h+7.2.1 Tracing a synthetic event all at onceh]h+7.2.1 Tracing a synthetic event all at once}(hj|hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjyhhhhhMubh)}(hsTo trace a synthetic event all at once, the synth_event_trace() or synth_event_trace_array() functions can be used.h]hsTo trace a synthetic event all at once, the synth_event_trace() or synth_event_trace_array() functions can be used.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjyhhubh)}(hXThe synth_event_trace() function is passed the trace_event_file representing the synthetic event (which can be retrieved using trace_get_event_file() using the synthetic event name, "synthetic" as the system name, and the trace instance name (NULL if using the global trace array)), along with an variable number of u64 args, one for each synthetic event field, and the number of values being passed.h]hXThe synth_event_trace() function is passed the trace_event_file representing the synthetic event (which can be retrieved using trace_get_event_file() using the synthetic event name, “synthetic” as the system name, and the trace instance name (NULL if using the global trace array)), along with an variable number of u64 args, one for each synthetic event field, and the number of values being passed.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjyhhubh)}(htSo, to trace an event corresponding to the synthetic event definition above, code like the following could be used::h]hsSo, to trace an event corresponding to the synthetic event definition above, code like the following could be used:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjyhhubj)}(hXret = synth_event_trace(create_synth_test, 7, /* number of values */ 444, /* next_pid_field */ (u64)"clackers", /* next_comm_field */ 1000000, /* ts_ns */ 1000, /* ts_ms */ smp_processor_id(),/* cpu */ (u64)"Thneed", /* my_string_field */ 999); /* my_int_field */h]hXret = synth_event_trace(create_synth_test, 7, /* number of values */ 444, /* next_pid_field */ (u64)"clackers", /* next_comm_field */ 1000000, /* ts_ns */ 1000, /* ts_ms */ smp_processor_id(),/* cpu */ (u64)"Thneed", /* my_string_field */ 999); /* my_int_field */}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhM hjyhhubh)}(hAll vals should be cast to u64, and string vals are just pointers to strings, cast to u64. Strings will be copied into space reserved in the event for the string, using these pointers.h]hAll vals should be cast to u64, and string vals are just pointers to strings, cast to u64. Strings will be copied into space reserved in the event for the string, using these pointers.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjyhhubh)}(hXAlternatively, the synth_event_trace_array() function can be used to accomplish the same thing. It is passed the trace_event_file representing the synthetic event (which can be retrieved using trace_get_event_file() using the synthetic event name, "synthetic" as the system name, and the trace instance name (NULL if using the global trace array)), along with an array of u64, one for each synthetic event field.h]hXAlternatively, the synth_event_trace_array() function can be used to accomplish the same thing. It is passed the trace_event_file representing the synthetic event (which can be retrieved using trace_get_event_file() using the synthetic event name, “synthetic” as the system name, and the trace instance name (NULL if using the global trace array)), along with an array of u64, one for each synthetic event field.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjyhhubh)}(hpTo trace an event corresponding to the synthetic event definition above, code like the following could be used::h]hoTo trace an event corresponding to the synthetic event definition above, code like the following could be used:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjyhhubj)}(hXcu64 vals[7]; vals[0] = 777; /* next_pid_field */ vals[1] = (u64)"tiddlywinks"; /* next_comm_field */ vals[2] = 1000000; /* ts_ns */ vals[3] = 1000; /* ts_ms */ vals[4] = smp_processor_id(); /* cpu */ vals[5] = (u64)"thneed"; /* my_string_field */ vals[6] = 398; /* my_int_field */h]hXcu64 vals[7]; vals[0] = 777; /* next_pid_field */ vals[1] = (u64)"tiddlywinks"; /* next_comm_field */ vals[2] = 1000000; /* ts_ns */ vals[3] = 1000; /* ts_ms */ vals[4] = smp_processor_id(); /* cpu */ vals[5] = (u64)"thneed"; /* my_string_field */ vals[6] = 398; /* my_int_field */}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhM!hjyhhubh)}(hThe 'vals' array is just an array of u64, the number of which must match the number of field in the synthetic event, and which must be in the same order as the synthetic event fields.h]hThe ‘vals’ array is just an array of u64, the number of which must match the number of field in the synthetic event, and which must be in the same order as the synthetic event fields.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM+hjyhhubh)}(hAll vals should be cast to u64, and string vals are just pointers to strings, cast to u64. Strings will be copied into space reserved in the event for the string, using these pointers.h]hAll vals should be cast to u64, and string vals are just pointers to strings, cast to u64. Strings will be copied into space reserved in the event for the string, using these pointers.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM/hjyhhubh)}(hXKIn order to trace a synthetic event, a pointer to the trace event file is needed. The trace_get_event_file() function can be used to get it - it will find the file in the given trace instance (in this case NULL since the top trace array is being used) while at the same time preventing the instance containing it from going away::h]hXJIn order to trace a synthetic event, a pointer to the trace event file is needed. The trace_get_event_file() function can be used to get it - it will find the file in the given trace instance (in this case NULL since the top trace array is being used) while at the same time preventing the instance containing it from going away:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM3hjyhhubj)}(hxschedtest_event_file = trace_get_event_file(NULL, "synthetic", "schedtest");h]hxschedtest_event_file = trace_get_event_file(NULL, "synthetic", "schedtest");}hj$sbah}(h]h ]h"]h$]h&]jjuh1jhhhM9hjyhhubh)}(hBefore tracing the event, it should be enabled in some way, otherwise the synthetic event won't actually show up in the trace buffer.h]hBefore tracing the event, it should be enabled in some way, otherwise the synthetic event won’t actually show up in the trace buffer.}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM<hjyhhubh)}(hTo enable a synthetic event from the kernel, trace_array_set_clr_event() can be used (which is not specific to synthetic events, so does need the "synthetic" system name to be specified explicitly).h]hTo enable a synthetic event from the kernel, trace_array_set_clr_event() can be used (which is not specific to synthetic events, so does need the “synthetic” system name to be specified explicitly).}(hj@hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM?hjyhhubh)}(h(To enable the event, pass 'true' to it::h]h+To enable the event, pass ‘true’ to it:}(hjNhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMChjyhhubj)}(hntrace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", true);h]hntrace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", true);}hj\sbah}(h]h ]h"]h$]h&]jjuh1jhhhMEhjyhhubh)}(hTo disable it pass false::h]hTo disable it pass false:}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMHhjyhhubj)}(hotrace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", false);h]hotrace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", false);}hjxsbah}(h]h ]h"]h$]h&]jjuh1jhhhMJhjyhhubh)}(hFinally, synth_event_trace_array() can be used to actually trace the event, which should be visible in the trace buffer afterwards::h]hFinally, synth_event_trace_array() can be used to actually trace the event, which should be visible in the trace buffer afterwards:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMMhjyhhubj)}(hjret = synth_event_trace_array(schedtest_event_file, vals, ARRAY_SIZE(vals));h]hjret = synth_event_trace_array(schedtest_event_file, vals, ARRAY_SIZE(vals));}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMPhjyhhubh)}(hTo remove the synthetic event, the event should be disabled, and the trace instance should be 'put' back using trace_put_event_file()::h]hTo remove the synthetic event, the event should be disabled, and the trace instance should be ‘put’ back using trace_put_event_file():}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMShjyhhubj)}(htrace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", false); trace_put_event_file(schedtest_event_file);h]htrace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", false); trace_put_event_file(schedtest_event_file);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMVhjyhhubh)}(hWIf those have been successful, synth_event_delete() can be called to remove the event::h]hVIf those have been successful, synth_event_delete() can be called to remove the event:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMZhjyhhubj)}(h&ret = synth_event_delete("schedtest");h]h&ret = synth_event_delete("schedtest");}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhM]hjyhhubeh}(h]%tracing-a-synthetic-event-all-at-onceah ]h"]+7.2.1 tracing a synthetic event all at onceah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h)7.2.2 Tracing a synthetic event piecewiseh]h)7.2.2 Tracing a synthetic event piecewise}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM`ubh)}(hTo trace a synthetic using the piecewise method described above, the synth_event_trace_start() function is used to 'open' the synthetic event trace::h]hTo trace a synthetic using the piecewise method described above, the synth_event_trace_start() function is used to ‘open’ the synthetic event trace:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMbhjhhubj)}(hostruct synth_event_trace_state trace_state; ret = synth_event_trace_start(schedtest_event_file, &trace_state);h]hostruct synth_event_trace_state trace_state; ret = synth_event_trace_start(schedtest_event_file, &trace_state);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMfhjhhubh)}(hX It's passed the trace_event_file representing the synthetic event using the same methods as described above, along with a pointer to a struct synth_event_trace_state object, which will be zeroed before use and used to maintain state between this and following calls.h]hX It’s passed the trace_event_file representing the synthetic event using the same methods as described above, along with a pointer to a struct synth_event_trace_state object, which will be zeroed before use and used to maintain state between this and following calls.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMjhjhhubh)}(hXcOnce the event has been opened, which means space for it has been reserved in the trace buffer, the individual fields can be set. There are two ways to do that, either one after another for each field in the event, which requires no lookups, or by name, which does. The tradeoff is flexibility in doing the assignments vs the cost of a lookup per field.h]hXcOnce the event has been opened, which means space for it has been reserved in the trace buffer, the individual fields can be set. There are two ways to do that, either one after another for each field in the event, which requires no lookups, or by name, which does. The tradeoff is flexibility in doing the assignments vs the cost of a lookup per field.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMohjhhubh)}(hXTo assign the values one after the other without lookups, synth_event_add_next_val() should be used. Each call is passed the same synth_event_trace_state object used in the synth_event_trace_start(), along with the value to set the next field in the event. After each field is set, the 'cursor' points to the next field, which will be set by the subsequent call, continuing until all the fields have been set in order. The same sequence of calls as in the above examples using this method would be (without error-handling code)::h]hXTo assign the values one after the other without lookups, synth_event_add_next_val() should be used. Each call is passed the same synth_event_trace_state object used in the synth_event_trace_start(), along with the value to set the next field in the event. After each field is set, the ‘cursor’ points to the next field, which will be set by the subsequent call, continuing until all the fields have been set in order. The same sequence of calls as in the above examples using this method would be (without error-handling code):}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMvhjhhubj)}(hX /* next_pid_field */ ret = synth_event_add_next_val(777, &trace_state); /* next_comm_field */ ret = synth_event_add_next_val((u64)"slinky", &trace_state); /* ts_ns */ ret = synth_event_add_next_val(1000000, &trace_state); /* ts_ms */ ret = synth_event_add_next_val(1000, &trace_state); /* cpu */ ret = synth_event_add_next_val(smp_processor_id(), &trace_state); /* my_string_field */ ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state); /* my_int_field */ ret = synth_event_add_next_val(395, &trace_state);h]hX /* next_pid_field */ ret = synth_event_add_next_val(777, &trace_state); /* next_comm_field */ ret = synth_event_add_next_val((u64)"slinky", &trace_state); /* ts_ns */ ret = synth_event_add_next_val(1000000, &trace_state); /* ts_ms */ ret = synth_event_add_next_val(1000, &trace_state); /* cpu */ ret = synth_event_add_next_val(smp_processor_id(), &trace_state); /* my_string_field */ ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state); /* my_int_field */ ret = synth_event_add_next_val(395, &trace_state);}hj9sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hXfTo assign the values in any order, synth_event_add_val() should be used. Each call is passed the same synth_event_trace_state object used in the synth_event_trace_start(), along with the field name of the field to set and the value to set it to. The same sequence of calls as in the above examples using this method would be (without error-handling code)::h]hXeTo assign the values in any order, synth_event_add_val() should be used. Each call is passed the same synth_event_trace_state object used in the synth_event_trace_start(), along with the field name of the field to set and the value to set it to. The same sequence of calls as in the above examples using this method would be (without error-handling code):}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(hXret = synth_event_add_val("next_pid_field", 777, &trace_state); ret = synth_event_add_val("next_comm_field", (u64)"silly putty", &trace_state); ret = synth_event_add_val("ts_ns", 1000000, &trace_state); ret = synth_event_add_val("ts_ms", 1000, &trace_state); ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state); ret = synth_event_add_val("my_string_field", (u64)"thneed_9", &trace_state); ret = synth_event_add_val("my_int_field", 3999, &trace_state);h]hXret = synth_event_add_val("next_pid_field", 777, &trace_state); ret = synth_event_add_val("next_comm_field", (u64)"silly putty", &trace_state); ret = synth_event_add_val("ts_ns", 1000000, &trace_state); ret = synth_event_add_val("ts_ms", 1000, &trace_state); ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state); ret = synth_event_add_val("my_string_field", (u64)"thneed_9", &trace_state); ret = synth_event_add_val("my_int_field", 3999, &trace_state);}hjUsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hNote that synth_event_add_next_val() and synth_event_add_val() are incompatible if used within the same trace of an event - either one can be used but not both at the same time.h]hNote that synth_event_add_next_val() and synth_event_add_val() are incompatible if used within the same trace of an event - either one can be used but not both at the same time.}(hjchhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hFinally, the event won't be actually traced until it's 'closed', which is done using synth_event_trace_end(), which takes only the struct synth_event_trace_state object used in the previous calls::h]hFinally, the event won’t be actually traced until it’s ‘closed’, which is done using synth_event_trace_end(), which takes only the struct synth_event_trace_state object used in the previous calls:}(hjqhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(h*ret = synth_event_trace_end(&trace_state);h]h*ret = synth_event_trace_end(&trace_state);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hNote that synth_event_trace_end() must be called at the end regardless of whether any of the add calls failed (say due to a bad field name being passed in).h]hNote that synth_event_trace_end() must be called at the end regardless of whether any of the add calls failed (say due to a bad field name being passed in).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]#tracing-a-synthetic-event-piecewiseah ]h"])7.2.2 tracing a synthetic event piecewiseah$]h&]uh1hhj hhhhhM`ubh)}(hhh](h)}(h>7.3 Dyamically creating kprobe and kretprobe event definitionsh]h>7.3 Dyamically creating kprobe and kretprobe event definitions}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hTo create a kprobe or kretprobe trace event from kernel code, the kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start() functions can be used.h]hTo create a kprobe or kretprobe trace event from kernel code, the kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start() functions can be used.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hXTo create a kprobe event, an empty or partially empty kprobe event should first be created using kprobe_event_gen_cmd_start(). The name of the event and the probe location should be specified along with one or args each representing a probe field should be supplied to this function. Before calling kprobe_event_gen_cmd_start(), the user should create and initialize a dynevent_cmd object using kprobe_event_cmd_init().h]hXTo create a kprobe event, an empty or partially empty kprobe event should first be created using kprobe_event_gen_cmd_start(). The name of the event and the probe location should be specified along with one or args each representing a probe field should be supplied to this function. Before calling kprobe_event_gen_cmd_start(), the user should create and initialize a dynevent_cmd object using kprobe_event_cmd_init().}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hGFor example, to create a new "schedtest" kprobe event with two fields::h]hJFor example, to create a new “schedtest” kprobe event with two fields:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(hXstruct dynevent_cmd cmd; char *buf; /* Create a buffer to hold the generated command */ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); /* Before generating the command, initialize the cmd object */ kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); /* * Define the gen_kprobe_test event with the first 2 kprobe * fields. */ ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open", "dfd=%ax", "filename=%dx");h]hXstruct dynevent_cmd cmd; char *buf; /* Create a buffer to hold the generated command */ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); /* Before generating the command, initialize the cmd object */ kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); /* * Define the gen_kprobe_test event with the first 2 kprobe * fields. */ ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open", "dfd=%ax", "filename=%dx");}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hX5Once the kprobe event object has been created, it can then be populated with more fields. Fields can be added using kprobe_event_add_fields(), supplying the dynevent_cmd object along with a variable arg list of probe fields. For example, to add a couple additional fields, the following call could be made::h]hX4Once the kprobe event object has been created, it can then be populated with more fields. Fields can be added using kprobe_event_add_fields(), supplying the dynevent_cmd object along with a variable arg list of probe fields. For example, to add a couple additional fields, the following call could be made:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(hDret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");h]hDret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hOnce all the fields have been added, the event should be finalized and registered by calling the kprobe_event_gen_cmd_end() or kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe or kretprobe command was started::h]hOnce all the fields have been added, the event should be finalized and registered by calling the kprobe_event_gen_cmd_end() or kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe or kretprobe command was started:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(h%ret = kprobe_event_gen_cmd_end(&cmd);h]h%ret = kprobe_event_gen_cmd_end(&cmd);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hor::h]hor:}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(h(ret = kretprobe_event_gen_cmd_end(&cmd);h]h(ret = kretprobe_event_gen_cmd_end(&cmd);}hj2sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hKAt this point, the event object is ready to be used for tracing new events.h]hKAt this point, the event object is ready to be used for tracing new events.}(hj@hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hSimilarly, a kretprobe event can be created using kretprobe_event_gen_cmd_start() with a probe name and location and additional params such as $retval::h]hSimilarly, a kretprobe event can be created using kretprobe_event_gen_cmd_start() with a probe name and location and additional params such as $retval:}(hjNhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(h~ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test", "do_sys_open", "$retval");h]h~ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test", "do_sys_open", "$retval");}hj\sbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hsSimilar to the synthetic event case, code like the following can be used to enable the newly created kprobe event::h]hrSimilar to the synthetic event case, code like the following can be used to enable the newly created kprobe event:}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(hgen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test"); ret = trace_array_set_clr_event(gen_kprobe_test->tr, "kprobes", "gen_kprobe_test", true);h]hgen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test"); ret = trace_array_set_clr_event(gen_kprobe_test->tr, "kprobes", "gen_kprobe_test", true);}hjxsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hFinally, also similar to synthetic events, the following code can be used to give the kprobe event file back and delete the event::h]hFinally, also similar to synthetic events, the following code can be used to give the kprobe event file back and delete the event:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(hUtrace_put_event_file(gen_kprobe_test); ret = kprobe_event_delete("gen_kprobe_test");h]hUtrace_put_event_file(gen_kprobe_test); ret = kprobe_event_delete("gen_kprobe_test");}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubeh}(h]:dyamically-creating-kprobe-and-kretprobe-event-definitionsah ]h"]>7.3 dyamically creating kprobe and kretprobe event definitionsah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h$7.4 The "dynevent_cmd" low-level APIh]h(7.4 The “dynevent_cmd” low-level API}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hX Both the in-kernel synthetic event and kprobe interfaces are built on top of a lower-level "dynevent_cmd" interface. This interface is meant to provide the basis for higher-level interfaces such as the synthetic and kprobe interfaces, which can be used as examples.h]hXBoth the in-kernel synthetic event and kprobe interfaces are built on top of a lower-level “dynevent_cmd” interface. This interface is meant to provide the basis for higher-level interfaces such as the synthetic and kprobe interfaces, which can be used as examples.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hX;The basic idea is simple and amounts to providing a general-purpose layer that can be used to generate trace event commands. The generated command strings can then be passed to the command-parsing and event creation code that already exists in the trace event subsystem for creating the corresponding trace events.h]hX;The basic idea is simple and amounts to providing a general-purpose layer that can be used to generate trace event commands. The generated command strings can then be passed to the command-parsing and event creation code that already exists in the trace event subsystem for creating the corresponding trace events.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hXaIn a nutshell, the way it works is that the higher-level interface code creates a struct dynevent_cmd object, then uses a couple functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up a command string, which finally causes the command to be executed using the dynevent_create() function. The details of the interface are described below.h]hXaIn a nutshell, the way it works is that the higher-level interface code creates a struct dynevent_cmd object, then uses a couple functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up a command string, which finally causes the command to be executed using the dynevent_create() function. The details of the interface are described below.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hThe first step in building a new command string is to create and initialize an instance of a dynevent_cmd. Here, for instance, we create a dynevent_cmd on the stack and initialize it::h]hThe first step in building a new command string is to create and initialize an instance of a dynevent_cmd. Here, for instance, we create a dynevent_cmd on the stack and initialize it:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj)}(hstruct dynevent_cmd cmd; char *buf; int ret; buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO, foo_event_run_command);h]hstruct dynevent_cmd cmd; char *buf; int ret; buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO, foo_event_run_command);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhMhjhhubh)}(hXThe dynevent_cmd initialization needs to be given a user-specified buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used for this purpose - at 2k it's generally too big to be comfortably put on the stack, so is dynamically allocated), a dynevent type id, which is meant to be used to check that further API calls are for the correct command type, and a pointer to an event-specific run_command() callback that will be called to actually execute the event-specific command function.h]hXThe dynevent_cmd initialization needs to be given a user-specified buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used for this purpose - at 2k it’s generally too big to be comfortably put on the stack, so is dynamically allocated), a dynevent type id, which is meant to be used to check that further API calls are for the correct command type, and a pointer to an event-specific run_command() callback that will be called to actually execute the event-specific command function.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM"hjhhubh)}(hfOnce that's done, the command string can by built up by successive calls to argument-adding functions.h]hhOnce that’s done, the command string can by built up by successive calls to argument-adding functions.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM+hjhhubh)}(hXTo add a single argument, define and initialize a struct dynevent_arg or struct dynevent_arg_pair object. Here's an example of the simplest possible arg addition, which is simply to append the given string as a whitespace-separated argument to the command::h]hXTo add a single argument, define and initialize a struct dynevent_arg or struct dynevent_arg_pair object. Here’s an example of the simplest possible arg addition, which is simply to append the given string as a whitespace-separated argument to the command:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM.hjhhubj)}(hpstruct dynevent_arg arg; dynevent_arg_init(&arg, NULL, 0); arg.str = name; ret = dynevent_arg_add(cmd, &arg);h]hpstruct dynevent_arg arg; dynevent_arg_init(&arg, NULL, 0); arg.str = name; ret = dynevent_arg_add(cmd, &arg);}hj+sbah}(h]h ]h"]h$]h&]jjuh1jhhhM3hjhhubh)}(hThe arg object is first initialized using dynevent_arg_init() and in this case the parameters are NULL or 0, which means there's no optional sanity-checking function or separator appended to the end of the arg.h]hThe arg object is first initialized using dynevent_arg_init() and in this case the parameters are NULL or 0, which means there’s no optional sanity-checking function or separator appended to the end of the arg.}(hj9hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM;hjhhubh)}(hHere's another more complicated example using an 'arg pair', which is used to create an argument that consists of a couple components added together as a unit, for example, a 'type field_name;' arg or a simple expression arg e.g. 'flags=%cx'::h]hXHere’s another more complicated example using an ‘arg pair’, which is used to create an argument that consists of a couple components added together as a unit, for example, a ‘type field_name;’ arg or a simple expression arg e.g. ‘flags=%cx’:}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM@hjhhubj)}(hstruct dynevent_arg_pair arg_pair; dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';'); arg_pair.lhs = type; arg_pair.rhs = name; ret = dynevent_arg_pair_add(cmd, &arg_pair);h]hstruct dynevent_arg_pair arg_pair; dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';'); arg_pair.lhs = type; arg_pair.rhs = name; ret = dynevent_arg_pair_add(cmd, &arg_pair);}hjUsbah}(h]h ]h"]h$]h&]jjuh1jhhhMEhjhhubh)}(hXDAgain, the arg_pair is first initialized, in this case with a callback function used to check the sanity of the args (for example, that neither part of the pair is NULL), along with a character to be used to add an operator between the pair (here none) and a separator to be appended onto the end of the arg pair (here ';').h]hXHAgain, the arg_pair is first initialized, in this case with a callback function used to check the sanity of the args (for example, that neither part of the pair is NULL), along with a character to be used to add an operator between the pair (here none) and a separator to be appended onto the end of the arg pair (here ‘;’).}(hjchhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMNhjhhubh)}(hThere's also a dynevent_str_add() function that can be used to simply add a string as-is, with no spaces, delimiters, or arg check.h]hThere’s also a dynevent_str_add() function that can be used to simply add a string as-is, with no spaces, delimiters, or arg check.}(hjqhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMThjhhubh)}(hXAny number of dynevent_*_add() calls can be made to build up the string (until its length surpasses cmd->maxlen). When all the arguments have been added and the command string is complete, the only thing left to do is run the command, which happens by simply calling dynevent_create()::h]hXAny number of dynevent_*_add() calls can be made to build up the string (until its length surpasses cmd->maxlen). When all the arguments have been added and the command string is complete, the only thing left to do is run the command, which happens by simply calling dynevent_create():}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMWhjhhubj)}(hret = dynevent_create(&cmd);h]hret = dynevent_create(&cmd);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhM]hjhhubh)}(h`At that point, if the return value is 0, the dynamic event has been created and is ready to use.h]h`At that point, if the return value is 0, the dynamic event has been created and is ready to use.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM_hjhhubh)}(hPSee the dynevent_cmd function definitions themselves for the details of the API.h]hPSee the dynevent_cmd function definitions themselves for the details of the API.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMbhjhhubeh}(h]the-dynevent-cmd-low-level-apiah ]h"]$7.4 the "dynevent_cmd" low-level apiah$]h&]uh1hhj hhhhhMubeh}(h]in-kernel-trace-event-apiah ]h"]7. in-kernel trace event apiah$]h&]uh1hhhhhhhhMXubeh}(h] event-tracingah ]h"] event tracingah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerjerror_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}nameids}(jjjYjVjjjj}jjjjjj jjj9 j6 jRjOjjjjjIjFjjj1 j. j j j# j j j jjjOjLjvjsjjjjjjjju nametypes}(jjYjjjjjjj9 jRjjjIjj1 j j# j jjOjvjjjjuh}(jhjVj'jj\j}jmjjjjj jjjj6 jjOjjjUjjjFjjjLj. jj j< j j j j& jj jLj jsjRjjyjjjjjju footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}jKsRparse_messages]hsystem_message)}(hhh]h)}(hfPossible title underline, too short for the title. Treating it as ordinary text because it's so short.h]hhPossible title underline, too short for the title. Treating it as ordinary text because it’s so short.}(hjYhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjVubah}(h]h ]h"]h$]h&]levelKtypeINFOlineMusourcehuh1jThjhhhhhMuubatransform_messages] transformerN include_log] decorationNhhub.