€•àNŒsphinx.addnodes”Œdocument”“”)”}”(Œ rawsource”Œ”Œchildren”]”(Œ translations”Œ LanguagesNode”“”)”}”(hhh]”(hŒ pending_xref”“”)”}”(hhh]”Œdocutils.nodes”ŒText”“”ŒChinese (Simplified)”…””}”(hhŒparent”hubaŒ attributes”}”(Œids”]”Œclasses”]”Œnames”]”Œdupnames”]”Œbackrefs”]”Œ refdomain”Œstd”Œreftype”Œdoc”Œ reftarget”Œ/translations/zh_CN/staging/xz”Œmodname”NŒ classname”NŒ refexplicit”ˆuŒtagname”hhh ubh)”}”(hhh]”hŒChinese (Traditional)”…””}”(hhhh2ubah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ/translations/zh_TW/staging/xz”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒItalian”…””}”(hhhhFubah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ/translations/it_IT/staging/xz”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒJapanese”…””}”(hhhhZubah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ/translations/ja_JP/staging/xz”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒKorean”…””}”(hhhhnubah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ/translations/ko_KR/staging/xz”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒSpanish”…””}”(hhhh‚ubah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ/translations/sp_SP/staging/xz”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubeh}”(h]”h ]”h"]”h$]”h&]”Œcurrent_language”ŒEnglish”uh1h hhŒ _document”hŒsource”NŒline”NubhŒsection”“”)”}”(hhh]”(hŒtitle”“”)”}”(hŒXZ data compression in Linux”h]”hŒXZ data compression in Linux”…””}”(hhªhh¨hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hh£hžhhŸŒ8/var/lib/git/docbuild/linux/Documentation/staging/xz.rst”h Kubh¢)”}”(hhh]”(h§)”}”(hŒ Introduction”h]”hŒ Introduction”…””}”(hh¼hhºhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hh·hžhhŸh¶h KubhŒ paragraph”“”)”}”(hX?XZ is a general purpose data compression format with high compression ratio and relatively fast decompression. The primary compression algorithm (filter) is LZMA2. Additional filters can be used to improve compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters improve compression ratio of executable data.”h]”hX?XZ is a general purpose data compression format with high compression ratio and relatively fast decompression. The primary compression algorithm (filter) is LZMA2. Additional filters can be used to improve compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters improve compression ratio of executable data.”…””}”(hhÌhhÊhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h Khh·hžhubhÉ)”}”(hXQThe XZ decompressor in Linux is called XZ Embedded. It supports the LZMA2 filter and optionally also BCJ filters. CRC32 is supported for integrity checking. The home page of XZ Embedded is at , where you can find the latest version and also information about using the code outside the Linux kernel.”h]”(hŒÁThe XZ decompressor in Linux is called XZ Embedded. It supports the LZMA2 filter and optionally also BCJ filters. CRC32 is supported for integrity checking. The home page of XZ Embedded is at <”…””}”(hŒÁThe XZ decompressor in Linux is called XZ Embedded. It supports the LZMA2 filter and optionally also BCJ filters. CRC32 is supported for integrity checking. The home page of XZ Embedded is at <”hhØhžhhŸNh NubhŒ reference”“”)”}”(hŒ$https://tukaani.org/xz/embedded.html”h]”hŒ$https://tukaani.org/xz/embedded.html”…””}”(hhhhãhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”Œrefuri”håuh1háhhØubhŒl>, where you can find the latest version and also information about using the code outside the Linux kernel.”…””}”(hŒl>, where you can find the latest version and also information about using the code outside the Linux kernel.”hhØhžhhŸNh Nubeh}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h Khh·hžhubhÉ)”}”(hŒFor userspace, XZ Utils provide a zlib-like compression library and a gzip-like command line tool. XZ Utils can be downloaded from .”h]”(hŒ„For userspace, XZ Utils provide a zlib-like compression library and a gzip-like command line tool. XZ Utils can be downloaded from <”…””}”(hŒ„For userspace, XZ Utils provide a zlib-like compression library and a gzip-like command line tool. XZ Utils can be downloaded from <”hhýhžhhŸNh Nubhâ)”}”(hŒhttps://tukaani.org/xz/”h]”hŒhttps://tukaani.org/xz/”…””}”(hhhjhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”Œrefuri”juh1háhhýubhŒ>.”…””}”(hŒ>.”hhýhžhhŸNh Nubeh}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h Khh·hžhubeh}”(h]”Œ introduction”ah ]”h"]”Œ introduction”ah$]”h&]”uh1h¡hh£hžhhŸh¶h Kubh¢)”}”(hhh]”(h§)”}”(hŒ#XZ related components in the kernel”h]”hŒ#XZ related components in the kernel”…””}”(hj-hj+hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hj(hžhhŸh¶h KubhÉ)”}”(hŒ²The xz_dec module provides XZ decompressor with single-call (buffer to buffer) and multi-call (stateful) APIs. The usage of the xz_dec module is documented in include/linux/xz.h.”h]”hŒ²The xz_dec module provides XZ decompressor with single-call (buffer to buffer) and multi-call (stateful) APIs. The usage of the xz_dec module is documented in include/linux/xz.h.”…””}”(hj;hj9hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h Khj(hžhubhÉ)”}”(hXsThe xz_dec_test module is for testing xz_dec. xz_dec_test is not useful unless you are hacking the XZ decompressor. xz_dec_test allocates a char device major dynamically to which one can write .xz files from userspace. The decompressed output is thrown away. Keep an eye on dmesg to see diagnostics printed by xz_dec_test. See the xz_dec_test source code for the details.”h]”hXsThe xz_dec_test module is for testing xz_dec. xz_dec_test is not useful unless you are hacking the XZ decompressor. xz_dec_test allocates a char device major dynamically to which one can write .xz files from userspace. The decompressed output is thrown away. Keep an eye on dmesg to see diagnostics printed by xz_dec_test. See the xz_dec_test source code for the details.”…””}”(hjIhjGhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K hj(hžhubhÉ)”}”(hŒÞFor decompressing the kernel image, initramfs, and initrd, there is a wrapper function in lib/decompress_unxz.c. Its API is the same as in other decompress_*.c files, which is defined in include/linux/decompress/generic.h.”h]”hŒÞFor decompressing the kernel image, initramfs, and initrd, there is a wrapper function in lib/decompress_unxz.c. Its API is the same as in other decompress_*.c files, which is defined in include/linux/decompress/generic.h.”…””}”(hjWhjUhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K'hj(hžhubhÉ)”}”(hŒ«scripts/xz_wrap.sh is a wrapper for the xz command line tool found from XZ Utils. The wrapper sets compression options to values suitable for compressing the kernel image.”h]”hŒ«scripts/xz_wrap.sh is a wrapper for the xz command line tool found from XZ Utils. The wrapper sets compression options to values suitable for compressing the kernel image.”…””}”(hjehjchžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K,hj(hžhubhÉ)”}”(hX¼For kernel makefiles, two commands are provided for use with $(call if_needed). The kernel image should be compressed with $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 dictionary. It will also append a four-byte trailer containing the uncompressed size of the file, which is needed by the boot code. Other things should be compressed with $(call if_needed,xzmisc) which will use no BCJ filter and 1 MiB LZMA2 dictionary.”h]”hX¼For kernel makefiles, two commands are provided for use with $(call if_needed). The kernel image should be compressed with $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 dictionary. It will also append a four-byte trailer containing the uncompressed size of the file, which is needed by the boot code. Other things should be compressed with $(call if_needed,xzmisc) which will use no BCJ filter and 1 MiB LZMA2 dictionary.”…””}”(hjshjqhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K0hj(hžhubeh}”(h]”Œ#xz-related-components-in-the-kernel”ah ]”h"]”Œ#xz related components in the kernel”ah$]”h&]”uh1h¡hh£hžhhŸh¶h Kubh¢)”}”(hhh]”(h§)”}”(hŒNotes on compression options”h]”hŒNotes on compression options”…””}”(hjŒhjŠhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hj‡hžhhŸh¶h K9ubhÉ)”}”(hXgSince the XZ Embedded supports only streams with no integrity check or CRC32, make sure that you don't use some other integrity check type when encoding files that are supposed to be decoded by the kernel. With liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 when encoding. With the xz command line tool, use --check=none or --check=crc32.”h]”hXiSince the XZ Embedded supports only streams with no integrity check or CRC32, make sure that you don’t use some other integrity check type when encoding files that are supposed to be decoded by the kernel. With liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 when encoding. With the xz command line tool, use --check=none or --check=crc32.”…””}”(hjšhj˜hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K;hj‡hžhubhÉ)”}”(hX€Using CRC32 is strongly recommended unless there is some other layer which will verify the integrity of the uncompressed data anyway. Double checking the integrity would probably be waste of CPU cycles. Note that the headers will always have a CRC32 which will be validated by the decoder; you can only change the integrity check type (or disable it) for the actual uncompressed data.”h]”hX€Using CRC32 is strongly recommended unless there is some other layer which will verify the integrity of the uncompressed data anyway. Double checking the integrity would probably be waste of CPU cycles. Note that the headers will always have a CRC32 which will be validated by the decoder; you can only change the integrity check type (or disable it) for the actual uncompressed data.”…””}”(hj¨hj¦hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h KBhj‡hžhubhÉ)”}”(hXÐIn userspace, LZMA2 is typically used with dictionary sizes of several megabytes. The decoder needs to have the dictionary in RAM, thus big dictionaries cannot be used for files that are intended to be decoded by the kernel. 1 MiB is probably the maximum reasonable dictionary size for in-kernel use (maybe more is OK for initramfs). The presets in XZ Utils may not be optimal when creating files for the kernel, so don't hesitate to use custom settings. Example::”h]”hXÑIn userspace, LZMA2 is typically used with dictionary sizes of several megabytes. The decoder needs to have the dictionary in RAM, thus big dictionaries cannot be used for files that are intended to be decoded by the kernel. 1 MiB is probably the maximum reasonable dictionary size for in-kernel use (maybe more is OK for initramfs). The presets in XZ Utils may not be optimal when creating files for the kernel, so don’t hesitate to use custom settings. Example:”…””}”(hXÏIn userspace, LZMA2 is typically used with dictionary sizes of several megabytes. The decoder needs to have the dictionary in RAM, thus big dictionaries cannot be used for files that are intended to be decoded by the kernel. 1 MiB is probably the maximum reasonable dictionary size for in-kernel use (maybe more is OK for initramfs). The presets in XZ Utils may not be optimal when creating files for the kernel, so don't hesitate to use custom settings. Example:”hj´hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h KIhj‡hžhubhŒ literal_block”“”)”}”(hŒ.xz --check=crc32 --lzma2=dict=512KiB inputfile”h]”hŒ.xz --check=crc32 --lzma2=dict=512KiB inputfile”…””}”(hhhjÅubah}”(h]”h ]”h"]”h$]”h&]”Œ xml:space”Œpreserve”uh1jÃhŸh¶h KQhj‡hžhubhÉ)”}”(hX„An exception to above dictionary size limitation is when the decoder is used in single-call mode. Decompressing the kernel itself is an example of this situation. In single-call mode, the memory usage doesn't depend on the dictionary size, and it is perfectly fine to use a big dictionary: for maximum compression, the dictionary should be at least as big as the uncompressed data itself.”h]”hX†An exception to above dictionary size limitation is when the decoder is used in single-call mode. Decompressing the kernel itself is an example of this situation. In single-call mode, the memory usage doesn’t depend on the dictionary size, and it is perfectly fine to use a big dictionary: for maximum compression, the dictionary should be at least as big as the uncompressed data itself.”…””}”(hj×hjÕhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h KShj‡hžhubeh}”(h]”Œnotes-on-compression-options”ah ]”h"]”Œnotes on compression options”ah$]”h&]”uh1h¡hh£hžhhŸh¶h K9ubh¢)”}”(hhh]”(h§)”}”(hŒ Future plans”h]”hŒ Future plans”…””}”(hjðhjîhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hjëhžhhŸh¶h K[ubhÉ)”}”(hŒÞCreating a limited XZ encoder may be considered if people think it is useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at the fastest settings, so it isn't clear if LZMA2 encoder is wanted into the kernel.”h]”hŒàCreating a limited XZ encoder may be considered if people think it is useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at the fastest settings, so it isn’t clear if LZMA2 encoder is wanted into the kernel.”…””}”(hjþhjühžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K]hjëhžhubhÉ)”}”(hŒÜSupport for limited random-access reading is planned for the decompression code. I don't know if it could have any use in the kernel, but I know that it would be useful in some embedded projects outside the Linux kernel.”h]”hŒÞSupport for limited random-access reading is planned for the decompression code. I don’t know if it could have any use in the kernel, but I know that it would be useful in some embedded projects outside the Linux kernel.”…””}”(hj hj hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h Kbhjëhžhubeh}”(h]”Œ future-plans”ah ]”h"]”Œ future plans”ah$]”h&]”uh1h¡hh£hžhhŸh¶h K[ubh¢)”}”(hhh]”(h§)”}”(hŒ0Conformance to the .xz file format specification”h]”hŒ0Conformance to the .xz file format specification”…””}”(hj%hj#hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hj hžhhŸh¶h KhubhÉ)”}”(hX"There are a couple of corner cases where things have been simplified at expense of detecting errors as early as possible. These should not matter in practice all, since they don't cause security issues. But it is good to know this if testing the code e.g. with the test files from XZ Utils.”h]”hX$There are a couple of corner cases where things have been simplified at expense of detecting errors as early as possible. These should not matter in practice all, since they don’t cause security issues. But it is good to know this if testing the code e.g. with the test files from XZ Utils.”…””}”(hj3hj1hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h Kjhj hžhubeh}”(h]”Œ/conformance-to-the-xz-file-format-specification”ah ]”h"]”Œ0conformance to the .xz file format specification”ah$]”h&]”uh1h¡hh£hžhhŸh¶h Khubh¢)”}”(hhh]”(h§)”}”(hŒReporting bugs”h]”hŒReporting bugs”…””}”(hjLhjJhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hjGhžhhŸh¶h KqubhÉ)”}”(hŒBefore reporting a bug, please check that it's not fixed already at upstream. See to get the latest code.”h]”(hŒUBefore reporting a bug, please check that it’s not fixed already at upstream. See <”…””}”(hŒSBefore reporting a bug, please check that it's not fixed already at upstream. See <”hjXhžhhŸNh Nubhâ)”}”(hŒ$https://tukaani.org/xz/embedded.html”h]”hŒ$https://tukaani.org/xz/embedded.html”…””}”(hhhjahžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”Œrefuri”jcuh1háhjXubhŒ> to get the latest code.”…””}”(hŒ> to get the latest code.”hjXhžhhŸNh Nubeh}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h KshjGhžhubhÉ)”}”(hŒòReport bugs to or visit #tukaani on Freenode and talk to Larhzu. I don't actively read LKML or other kernel-related mailing lists, so if there's something I should know, you should email to me personally or use IRC.”h]”(hŒReport bugs to <”…””}”(hŒReport bugs to <”hj{hžhhŸNh Nubhâ)”}”(hŒlasse.collin@tukaani.org”h]”hŒlasse.collin@tukaani.org”…””}”(hhhj„hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”Œrefuri”Œmailto:lasse.collin@tukaani.org”uh1háhj{ubhŒÎ> or visit #tukaani on Freenode and talk to Larhzu. I don’t actively read LKML or other kernel-related mailing lists, so if there’s something I should know, you should email to me personally or use IRC.”…””}”(hŒÊ> or visit #tukaani on Freenode and talk to Larhzu. I don't actively read LKML or other kernel-related mailing lists, so if there's something I should know, you should email to me personally or use IRC.”hj{hžhhŸNh Nubeh}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h KwhjGhžhubhÉ)”}”(hXDon't bother Igor Pavlov with questions about the XZ implementation in the kernel or about XZ Utils. While these two implementations include essential code that is directly based on Igor Pavlov's code, these implementations aren't maintained nor supported by him.”h]”hX Don’t bother Igor Pavlov with questions about the XZ implementation in the kernel or about XZ Utils. While these two implementations include essential code that is directly based on Igor Pavlov’s code, these implementations aren’t maintained nor supported by him.”…””}”(hj¡hjŸhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÈhŸh¶h K|hjGhžhubeh}”(h]”Œreporting-bugs”ah ]”h"]”Œreporting bugs”ah$]”h&]”uh1h¡hh£hžhhŸh¶h Kqubeh}”(h]”Œxz-data-compression-in-linux”ah ]”h"]”Œxz data compression in linux”ah$]”h&]”uh1h¡hhhžhhŸh¶h Kubeh}”(h]”h ]”h"]”h$]”h&]”Œsource”h¶uh1hŒcurrent_source”NŒ current_line”NŒsettings”Œdocutils.frontend”ŒValues”“”)”}”(h¦NŒ generator”NŒ datestamp”NŒ source_link”NŒ source_url”NŒ toc_backlinks”Œentry”Œfootnote_backlinks”KŒ sectnum_xform”KŒstrip_comments”NŒstrip_elements_with_classes”NŒ strip_classes”NŒ report_level”KŒ halt_level”KŒexit_status_level”KŒdebug”NŒwarning_stream”NŒ traceback”ˆŒinput_encoding”Œ utf-8-sig”Œinput_encoding_error_handler”Œstrict”Œoutput_encoding”Œutf-8”Œoutput_encoding_error_handler”jàŒerror_encoding”ŒUTF-8”Œerror_encoding_error_handler”Œbackslashreplace”Œ language_code”Œen”Œrecord_dependencies”NŒconfig”NŒ id_prefix”hŒauto_id_prefix”Œid”Œ dump_settings”NŒdump_internals”NŒdump_transforms”NŒdump_pseudo_xml”NŒexpose_internals”NŒstrict_visitor”NŒ_disable_config”NŒ_source”h¶Œ _destination”NŒ _config_files”]”Œ7/var/lib/git/docbuild/linux/Documentation/docutils.conf”aŒpep_references”NŒ pep_base_url”Œhttps://peps.python.org/”Œpep_file_url_template”Œpep-%04d”Œrfc_references”NŒ rfc_base_url”Œ&https://datatracker.ietf.org/doc/html/”Œ tab_width”KŒtrim_footnote_reference_space”‰Œfile_insertion_enabled”ˆŒ raw_enabled”KŒline_length_limit”M'Œsyntax_highlight”Œlong”Œ smart_quotes”ˆŒsmartquotes_locales”]”Œcharacter_level_inline_markup”‰Œdoctitle_xform”‰Œ docinfo_xform”KŒsectsubtitle_xform”‰Œ image_loading”Œlink”Œembed_stylesheet”‰Œcloak_email_addresses”ˆŒsection_self_link”‰Œ embed_images”‰Œenv”NubŒreporter”NŒindirect_targets”]”Œsubstitution_defs”}”Œsubstitution_names”}”Œrefnames”}”Œrefids”}”Œnameids”}”(jºj·j%j"j„jjèjåjjjDjAj²j¯uŒ nametypes”}”(jºNj%Nj„NjèNjNjDNj²Nuh}”(j·h£j"h·jj(jåj‡jjëjAj j¯jGuŒ footnote_refs”}”Œ citation_refs”}”Œ autofootnotes”]”Œautofootnote_refs”]”Œsymbol_footnotes”]”Œsymbol_footnote_refs”]”Œ footnotes”]”Œ citations”]”Œautofootnote_start”KŒsymbol_footnote_start”KŒ id_counter”Œ collections”ŒCounter”“”}”…”R”Œparse_messages”]”Œtransform_messages”]”Œ transformer”NŒ include_log”]”Œ decoration”Nhžhub.