nmsphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget&/translations/zh_CN/core-api/local_opsmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget&/translations/zh_TW/core-api/local_opsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget&/translations/it_IT/core-api/local_opsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget&/translations/ja_JP/core-api/local_opsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget&/translations/ko_KR/core-api/local_opsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget&/translations/sp_SP/core-api/local_opsmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhtarget)}(h.. _local_ops:h]h}(h]h ]h"]h$]h&]refid local-opsuh1hhKhhhhh@/var/lib/git/docbuild/linux/Documentation/core-api/local_ops.rstubhsection)}(hhh](htitle)}(h1Semantics and Behavior of Local Atomic Operationsh]h1Semantics and Behavior of Local Atomic Operations}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh field_list)}(hhh]hfield)}(hhh](h field_name)}(hAuthorh]hAuthor}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhKubh field_body)}(hMathieu Desnoyers h]h paragraph)}(hMathieu Desnoyersh]hMathieu Desnoyers}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhubah}(h]h ]h"]h$]h&]uh1hhhubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hX+This document explains the purpose of the local atomic operations, how to implement them for any given architecture and shows how they can be used properly. It also stresses on the precautions that must be taken when reading those local variables across CPUs when the order of memory writes matters.h]hX+This document explains the purpose of the local atomic operations, how to implement them for any given architecture and shows how they can be used properly. It also stresses on the precautions that must be taken when reading those local variables across CPUs when the order of memory writes matters.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhhhubhnote)}(hXNote that ``local_t`` based operations are not recommended for general kernel use. Please use the ``this_cpu`` operations instead unless there is really a special purpose. Most uses of ``local_t`` in the kernel have been replaced by ``this_cpu`` operations. ``this_cpu`` operations combine the relocation with the ``local_t`` like semantics in a single instruction and yield more compact and faster executing code.h]h)}(hXNote that ``local_t`` based operations are not recommended for general kernel use. Please use the ``this_cpu`` operations instead unless there is really a special purpose. Most uses of ``local_t`` in the kernel have been replaced by ``this_cpu`` operations. ``this_cpu`` operations combine the relocation with the ``local_t`` like semantics in a single instruction and yield more compact and faster executing code.h](h Note that }(hjhhhNhNubhliteral)}(h ``local_t``h]hlocal_t}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubhM based operations are not recommended for general kernel use. Please use the }(hjhhhNhNubj$)}(h ``this_cpu``h]hthis_cpu}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubhK operations instead unless there is really a special purpose. Most uses of }(hjhhhNhNubj$)}(h ``local_t``h]hlocal_t}(hjIhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh% in the kernel have been replaced by }(hjhhhNhNubj$)}(h ``this_cpu``h]hthis_cpu}(hj[hhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh operations. }(hjhhhNhNubj$)}(h ``this_cpu``h]hthis_cpu}(hjmhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh, operations combine the relocation with the }(hjhhhNhNubj$)}(h ``local_t``h]hlocal_t}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubhY like semantics in a single instruction and yield more compact and faster executing code.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhhhhhhhNubh)}(hhh](h)}(h"Purpose of local atomic operationsh]h"Purpose of local atomic operations}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hLocal atomic operations are meant to provide fast and highly reentrant per CPU counters. They minimize the performance cost of standard atomic operations by removing the LOCK prefix and memory barriers normally required to synchronize across CPUs.h]hLocal atomic operations are meant to provide fast and highly reentrant per CPU counters. They minimize the performance cost of standard atomic operations by removing the LOCK prefix and memory barriers normally required to synchronize across CPUs.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXHaving fast per CPU atomic counters is interesting in many cases: it does not require disabling interrupts to protect from interrupt handlers and it permits coherent counters in NMI handlers. It is especially useful for tracing purposes and for various performance monitoring counters.h]hXHaving fast per CPU atomic counters is interesting in many cases: it does not require disabling interrupts to protect from interrupt handlers and it permits coherent counters in NMI handlers. It is especially useful for tracing purposes and for various performance monitoring counters.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK"hjhhubh)}(hXLocal atomic operations only guarantee variable modification atomicity wrt the CPU which owns the data. Therefore, care must taken to make sure that only one CPU writes to the ``local_t`` data. This is done by using per cpu data and making sure that we modify it from within a preemption safe context. It is however permitted to read ``local_t`` data from any CPU: it will then appear to be written out of order wrt other memory writes by the owner CPU.h](hLocal atomic operations only guarantee variable modification atomicity wrt the CPU which owns the data. Therefore, care must taken to make sure that only one CPU writes to the }(hjhhhNhNubj$)}(h ``local_t``h]hlocal_t}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh data. This is done by using per cpu data and making sure that we modify it from within a preemption safe context. It is however permitted to read }(hjhhhNhNubj$)}(h ``local_t``h]hlocal_t}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubhl data from any CPU: it will then appear to be written out of order wrt other memory writes by the owner CPU.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK'hjhhubeh}(h]"purpose-of-local-atomic-operationsah ]h"]"purpose of local atomic operationsah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h'Implementation for a given architectureh]h'Implementation for a given architecture}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK0ubh)}(hXiIt can be done by slightly modifying the standard atomic operations: only their UP variant must be kept. It typically means removing LOCK prefix (on i386 and x86_64) and any SMP synchronization barrier. If the architecture does not have a different behavior between SMP and UP, including ``asm-generic/local.h`` in your architecture's ``local.h`` is sufficient.h](hX It can be done by slightly modifying the standard atomic operations: only their UP variant must be kept. It typically means removing LOCK prefix (on i386 and x86_64) and any SMP synchronization barrier. If the architecture does not have a different behavior between SMP and UP, including }(hjhhhNhNubj$)}(h``asm-generic/local.h``h]hasm-generic/local.h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh in your architecture’s }(hjhhhNhNubj$)}(h ``local.h``h]hlocal.h}(hj/hhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh is sufficient.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK2hjhhubh)}(hThe ``local_t`` type is defined as an opaque ``signed long`` by embedding an ``atomic_long_t`` inside a structure. This is made so a cast from this type to a ``long`` fails. The definition looks like::h](hThe }(hjGhhhNhNubj$)}(h ``local_t``h]hlocal_t}(hjOhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjGubh type is defined as an opaque }(hjGhhhNhNubj$)}(h``signed long``h]h signed long}(hjahhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjGubh by embedding an }(hjGhhhNhNubj$)}(h``atomic_long_t``h]h atomic_long_t}(hjshhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjGubh@ inside a structure. This is made so a cast from this type to a }(hjGhhhNhNubj$)}(h``long``h]hlong}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjGubh" fails. The definition looks like:}(hjGhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK8hjhhubh literal_block)}(h,typedef struct { atomic_long_t a; } local_t;h]h,typedef struct { atomic_long_t a; } local_t;}hjsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1jhhhK #include static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0);h]hk#include #include static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0);}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhK\hjhhubeh}(h]"how-to-use-local-atomic-operationsah ]h"]"how to use local atomic operationsah$]h&]uh1hhhhhhhhKXubh)}(hhh](h)}(hCountingh]hCounting}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKcubh)}(h2Counting is done on all the bits of a signed long.h]h2Counting is done on all the bits of a signed long.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKehjhhubh)}(hIn preemptible context, use ``get_cpu_var()`` and ``put_cpu_var()`` around local atomic operations: it makes sure that preemption is disabled around write access to the per cpu variable. For instance::h](hIn preemptible context, use }(hjhhhNhNubj$)}(h``get_cpu_var()``h]h get_cpu_var()}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh and }(hjhhhNhNubj$)}(h``put_cpu_var()``h]h put_cpu_var()}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh around local atomic operations: it makes sure that preemption is disabled around write access to the per cpu variable. For instance:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKghjhhubj)}(h9local_inc(&get_cpu_var(counters)); put_cpu_var(counters);h]h9local_inc(&get_cpu_var(counters)); put_cpu_var(counters);}hjCsbah}(h]h ]h"]h$]h&]jjuh1jhhhKkhjhhubh)}(hYIf you are already in a preemption-safe context, you can use ``this_cpu_ptr()`` instead::h](h=If you are already in a preemption-safe context, you can use }(hjQhhhNhNubj$)}(h``this_cpu_ptr()``h]hthis_cpu_ptr()}(hjYhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjQubh instead:}(hjQhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKnhjhhubj)}(h#local_inc(this_cpu_ptr(&counters));h]h#local_inc(this_cpu_ptr(&counters));}hjqsbah}(h]h ]h"]h$]h&]jjuh1jhhhKqhjhhubeh}(h]countingah ]h"]countingah$]h&]uh1hhhhhhhhKcubh)}(hhh](h)}(hReading the countersh]hReading the counters}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKvubh)}(hThose local counters can be read from foreign CPUs to sum the count. Note that the data seen by local_read across CPUs must be considered to be out of order relatively to other memory writes happening on the CPU that owns the data::h]hThose local counters can be read from foreign CPUs to sum the count. Note that the data seen by local_read across CPUs must be considered to be out of order relatively to other memory writes happening on the CPU that owns the data:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKxhjhhubj)}(hZlong sum = 0; for_each_online_cpu(cpu) sum += local_read(&per_cpu(counters, cpu));h]hZlong sum = 0; for_each_online_cpu(cpu) sum += local_read(&per_cpu(counters, cpu));}hjsbah}(h]h ]h"]h$]h&]jjuh1jhhhK|hjhhubh)}(hXIf you want to use a remote local_read to synchronize access to a resource between CPUs, explicit ``smp_wmb()`` and ``smp_rmb()`` memory barriers must be used respectively on the writer and the reader CPUs. It would be the case if you use the ``local_t`` variable as a counter of bytes written in a buffer: there should be a ``smp_wmb()`` between the buffer write and the counter increment and also a ``smp_rmb()`` between the counter read and the buffer read.h](hbIf you want to use a remote local_read to synchronize access to a resource between CPUs, explicit }(hjhhhNhNubj$)}(h ``smp_wmb()``h]h smp_wmb()}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh and }(hjhhhNhNubj$)}(h ``smp_rmb()``h]h smp_rmb()}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubhr memory barriers must be used respectively on the writer and the reader CPUs. It would be the case if you use the }(hjhhhNhNubj$)}(h ``local_t``h]hlocal_t}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubhG variable as a counter of bytes written in a buffer: there should be a }(hjhhhNhNubj$)}(h ``smp_wmb()``h]h smp_wmb()}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh? between the buffer write and the counter increment and also a }(hjhhhNhNubj$)}(h ``smp_rmb()``h]h smp_rmb()}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh. between the counter read and the buffer read.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hTHere is a sample module which implements a basic per cpu counter using ``local.h``::h](hGHere is a sample module which implements a basic per cpu counter using }(hjhhhNhNubj$)}(h ``local.h``h]hlocal.h}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1j#hjubh:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hXY/* test-local.c * * Sample module for local.h usage. */ #include #include #include static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0); static struct timer_list test_timer; /* IPI called on each CPU. */ static void test_each(void *info) { /* Increment the counter from a non preemptible context */ printk("Increment on cpu %d\n", smp_processor_id()); local_inc(this_cpu_ptr(&counters)); /* This is what incrementing the variable would look like within a * preemptible context (it disables preemption) : * * local_inc(&get_cpu_var(counters)); * put_cpu_var(counters); */ } static void do_test_timer(unsigned long data) { int cpu; /* Increment the counters */ on_each_cpu(test_each, NULL, 1); /* Read all the counters */ printk("Counters read from CPU %d\n", smp_processor_id()); for_each_online_cpu(cpu) { printk("Read : CPU %d, count %ld\n", cpu, local_read(&per_cpu(counters, cpu))); } mod_timer(&test_timer, jiffies + 1000); } static int __init test_init(void) { /* initialize the timer that will increment the counter */ timer_setup(&test_timer, do_test_timer, 0); mod_timer(&test_timer, jiffies + 1); return 0; } static void __exit test_exit(void) { timer_shutdown_sync(&test_timer); } module_init(test_init); module_exit(test_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Mathieu Desnoyers"); MODULE_DESCRIPTION("Local Atomic Ops");h]hXY/* test-local.c * * Sample module for local.h usage. */ #include #include #include static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0); static struct timer_list test_timer; /* IPI called on each CPU. */ static void test_each(void *info) { /* Increment the counter from a non preemptible context */ printk("Increment on cpu %d\n", smp_processor_id()); local_inc(this_cpu_ptr(&counters)); /* This is what incrementing the variable would look like within a * preemptible context (it disables preemption) : * * local_inc(&get_cpu_var(counters)); * put_cpu_var(counters); */ } static void do_test_timer(unsigned long data) { int cpu; /* Increment the counters */ on_each_cpu(test_each, NULL, 1); /* Read all the counters */ printk("Counters read from CPU %d\n", smp_processor_id()); for_each_online_cpu(cpu) { printk("Read : CPU %d, count %ld\n", cpu, local_read(&per_cpu(counters, cpu))); } mod_timer(&test_timer, jiffies + 1000); } static int __init test_init(void) { /* initialize the timer that will increment the counter */ timer_setup(&test_timer, do_test_timer, 0); mod_timer(&test_timer, jiffies + 1); return 0; } static void __exit test_exit(void) { timer_shutdown_sync(&test_timer); } module_init(test_init); module_exit(test_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Mathieu Desnoyers"); MODULE_DESCRIPTION("Local Atomic Ops");}hj<sbah}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubeh}(h]reading-the-countersah ]h"]reading the countersah$]h&]uh1hhhhhhhhKvubeh}(h](1semantics-and-behavior-of-local-atomic-operationsheh ]h"](1semantics and behavior of local atomic operations local_opseh$]h&]uh1hhhhhhhhKexpect_referenced_by_name}jXhsexpect_referenced_by_id}hhsubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerjerror_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}h]hasnameids}(jXhjWjTjjjjjjjjjjjOjLu nametypes}(jXjWjjjjjjOuh}(hhjThjjjjjjjjjjjLju footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}Rparse_messages]transform_messages]hsystem_message)}(hhh]h)}(hhh]h/Hyperlink target "local-ops" is not referenced.}hjsbah}(h]h ]h"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]levelKtypeINFOsourcehlineKuh1juba transformerN include_log] decorationNhhub.