diff --git a/llvm/docs/DirectX/DXContainer.rst b/llvm/docs/DirectX/DXContainer.rst new file mode 100644 index 0000000000000..36e670a1c164d --- /dev/null +++ b/llvm/docs/DirectX/DXContainer.rst @@ -0,0 +1,402 @@ +================= +DirectX Container +================= + +.. contents:: + :local: + +.. toctree:: + :hidden: + +Overview +======== + +The DirectX Container (DXContainer) file format is the binary file format for +compiled shaders targeting the DirectX runtime. The file format is also called +the DXIL Container or DXBC file format. Because the file format can be used to +include either DXIL or DXBC compiled shaders, the nomenclature in LLVM is simply +DirectX Container. + +DirectX Container files are read by the compiler and associated tools as well as +the DirectX runtime, profiling tools and other users. This document serves as a +companion to the implementation in LLVM to more completely document the file +format for its many users. + +Basic Structure +=============== + +A DXContainer file begins with a header, and is then followed by a sequence of +"parts", which are analogous to object file sections. Each part contains a part +header, and some number of bytes of data after the header in a defined format. + +DX Container data structures are encoded little-endian in the binary file. + +The LLVM versions of all data structures described and/or referenced in this +file are defined in +`llvm/include/llvm/BinaryFormat/DXContainer.h +`_. +Some pseudo code is provided in blocks below to ease understanding of this +document, but reading it with the header available will provide the most +clarity. + +File Header +----------- + +.. code-block:: c + + struct Header { + uint8_t Magic[4]; + uint8_t Digest[16]; + uint16_t MajorVersion; + uint16_t MinorVersion; + uint32_t FileSize; + uint32_t PartCount; + }; + +The DXContainer header matches the pseudo-definition above. It begins with a +four character code (magic number) with the value ``DXBC`` to denote the file +format. + +The ``Digest`` is a 128bit hash digest computed with a proprietary algorithm and +encoded in the binary by the bytecode validator. + +The ``MajorVersion`` and ``MinorVersion`` encode the file format version +``1.0``. + +The remaining fields encode 32-bit unsigned integers for the file size and +number of parts. + +Following the part header is an array of ``PartCount`` 32-bit unsigned integers +specifying the offsets of each part header. + +Part Data +--------- + +.. code-block:: c + + struct PartHeader { + uint8_t Name[4]; + uint32_t Size; + } + +Each part begins with a part header. A part header includes the 4-character part +name, and a 32-bit unsigned integer specifying the size of the part data. The +part header is followed by ``Size`` bytes of data comprising the part. The +format does not explicitly require 32-bit alignment of parts, although LLVM does +implement this restriction in the writer code (because it's a good idea). The +LLVM object reader code does not assume inputs are correctly aligned to avoid +undefined behavior caused by misaligned inputs generated by other compilers. + +Part Formats +============ + +The part name indicates the format of the part data. There are 24 part headers +used by DXC and FXC. Not all compiled shaders contain all parts. In the list +below parts generated only by DXC are marked with †, and parts generated only by +FXC are marked with \*. + +#. `DXIL`_† - Stores the DXIL bytecode. +#. `HASH`_† - Stores the shader MD5 hash. +#. ILDB† - Stores the DXIL bytecode with LLVM Debug Information embedded in the module. +#. ILDN† - Stores shader debug name for external debug information. +#. `ISG1`_ - Stores the input signature for Shader Model 5.1+. +#. ISGN\* - Stores the input signature for Shader Model 4 and earlier. +#. `OSG1`_ - Stores the output signature for Shader Model 5.1+. +#. OSG5\* - Stores the output signature for Shader Model 5. +#. OSGN\* - Stores the output signature for Shader Model 4 and earlier. +#. PCSG\* - Stores the patch constant signature for Shader Model 5.1 and earlier. +#. PDBI† - Stores PDB information. +#. PRIV - Stores arbitrary private data (Not encoded by either FXC or DXC). +#. `PSG1`_ - Stores the patch constant signature for Shader Model 6+. +#. `PSV0`_ - Stores Pipeline State Validation data. +#. RDAT† - Stores Runtime Data. +#. RDEF\* - Stores resource definitions. +#. RTS0 - Stores compiled root signature. +#. `SFI0`_ - Stores shader feature flags. +#. SHDR\* - Stores compiled DXBC bytecode. +#. SHEX\* - Stores compiled DXBC bytecode. +#. DXBC\* - Stores compiled DXBC bytecode. +#. SRCI† - Stores shader source information. +#. STAT† - Stores shader statistics. +#. VERS† - Stores shader compiler version information. + +DXIL Part +--------- +.. _DXIL: + +The DXIL part is comprised of three data structures: the ``ProgramHeader``, the +``BitcodeHeader`` and the bitcode serialized LLVM 3.7 IR Module. + +The ``ProgramHeader`` contains the shader model version and pipeline stage +enumeration value. This identifies the target profile of the contained shader +bitcode. + +The ``BitcodeHeader`` contains the DXIL version information and refers to the +start of the bitcode data. + +HASH Part +--------- +.. _HASH: + +The HASH part contains a 32-bit unsigned integer with the shader hash flags, and +a 128-bit MD5 hash digest. The flags field can either have the value ``0`` to +indicate no flags, or ``1`` to indicate that the file hash was computed +including the source code that produced the binary. + +Program Signature (SG1) Parts +----------------------------- +.. _ISG1: +.. _OSG1: +.. _PSG1: + +.. code-block:: c + + struct ProgramSignatureHeader { + uint32_t ParamCount; + uint32_t FirstParamOffset; + } + +The program signature parts (ISG1, OSG1, & PSG1) all use the same data +structures to encode inputs, outputs and patch information. The +``ProgramSignatureHeader`` includes two 32-bit unsigned integers to specify the +number of signature parameters and the offset of the first parameter. + +Beginning at ``FirstParamOffset`` bytes from the start of the +``ProgramSignatureHeader``, ``ParamCount`` ``ProgramSignatureElement`` +structures are written. Following the ``ProgramSignatureElements`` is a string +table of null terminated strings padded to 32-byte alignment. This string table +matches the DWARF string table format as implemented by LLVM. + +Each ``ProgramSignatureElement`` encodes a ``NameOffset`` value which specifies +the offset into the string table. A value of ``0`` denotes no name. The offsets +encoded here are from the beginning of the ``ProgramSignatureHeader`` not the +beginning of the string table. + +The ``ProgramSignatureElement`` contains several enumeration fields which are +defined in `llvm/include/llvm/BinaryFormat/DXContainerConstants.def `_. +These fields encode the D3D system value, the type of data and its precision +requirements. + +PSV0 Part +--------- +.. _PSV0: + +The Pipeline State Validation data encodes versioned runtime information +structures. These structures use a scheme where in lieu of encoding a version +number, they encode the size of the structure and each new version of the +structure is additive. This allows readers to infer the version of the structure +by comparing the encoded size with the size of known structures. If the encoded +size is larger than any known structure, the largest known structure can validly +parse the data represented in the known structure. + +In LLVM we represent the versions of the associated data structures with +versioned namespaces under the ``llvm::dxbc::PSV`` namespace (e.g. ``v0``, +``v1``). Each structure in the ``v0`` namespace is the base version, the +structures in the ``v1`` namespace inherit from the ``v0`` namespace, and the +``v2`` structures inherit from the ``v1`` structures, and so on. + +The high-level structure of the PSV data is: + +#. ``RuntimeInfo`` structure +#. Resource bindings +#. Signature elements +#. Mask Vectors (Output, Input, InputPatch, PatchOutput) + +Immediately following the part header for the PSV0 part is a 32-bit unsigned +integer specifying the size of the ``RuntimeInfo`` structure that follows. + +Immediately following the ``RuntimeInfo`` structure is a 32-bit unsigned integer +specifying the number of resource bindings. If the number of resources is +greater than zero, another unsigned 32-bit integer follows to specify the size +of the ``ResourceBindInfo`` structure. This is followed by the specified number +of structures of the specified size (which infers the version of the structure). + +For version 0 of the data this ends the part data. + +PSV0 Signature Elements +~~~~~~~~~~~~~~~~~~~~~~~ + +The signature elements are conceptually a single concept but the data is encoded +in three different blocks. The first block is a string table, the second block +is an index table, and the third block is the elements themselves, which in turn +are separeated by input, output and patch constant or primitive elements. + +Signature elements capture much of the same data captured in the :ref:`SG1 +` parts. The use of an index table allows de-duplciation of data for a more +compact final representation. + +The string table begins with a 32-bit unsigned integer specifying the table +size. This string table uses the DXContainer format as implemented in LLVM. This +format prefixes the string table with a null byte so that offset ``0`` is a null +string, and pads to 32-byte alignment. + +The index table begins with a 32-bit unsigned integer specifying the size of the +table, and is followed by that many 32-bit unsigned integers representing the +table. The index table may or may not deduplicate repeated sequences (both DXC +and Clang do). The indices signify the indices in the flattened aggregate +representation which the signature element describes. A single semantic may have +more than one entry in this table to denote the different attributes of its +members. + +For example given the following code: + +.. code-block:: c + + struct VSOut_1 + { + float4 f3 : VOUT2; + float3 f4 : VOUT3; + }; + + + struct VSOut + { + float4 f1 : VOUT0; + float2 f2[4] : VOUT1; + VSOut_1 s; + int4 f5 : VOUT4; + }; + + void main(out VSOut o1 : A) { + } + +The semantic ``A`` gets expanded into 5 output signature elements. Those +elements are: + +.. note:: + + In the example below, it is a coincidence that the rows match the indices, in + more complicated examples with multiple semantics this is not the case. + +#. Index 0 starts at row 0, contains 4 columns, and is float32. This represents + ``f1`` in the source. +#. Index 1, 2, 3, and 4 starts at row 1, contains two columns and is float32. + This represents ``f2`` in the source, and it spreads across rows 1 - 4. +#. Index 5 starts at row 5, contains 4 columns, and is float32. This represents + ``f3`` in the source. +#. Index 6 starts at row 6, contains 3 columns, and is float32. This represents + ``f4``. +#. Index 7 starts at row 7, contains 4 columns, and is signed 32-bit integer. + This represents ``f5`` in the source. + +The LLVM ``obj2yaml`` tool can parse this data out of the PSV and present it in +human readable YAML. For the example above it produces the output: + +.. code-block:: YAML + + SigOutputElements: + - Name: A + Indices: [ 0 ] + StartRow: 0 + Cols: 4 + StartCol: 0 + Allocated: true + Kind: Arbitrary + ComponentType: Float32 + Interpolation: Linear + DynamicMask: 0x0 + Stream: 0 + - Name: A + Indices: [ 1, 2, 3, 4 ] + StartRow: 1 + Cols: 2 + StartCol: 0 + Allocated: true + Kind: Arbitrary + ComponentType: Float32 + Interpolation: Linear + DynamicMask: 0x0 + Stream: 0 + - Name: A + Indices: [ 5 ] + StartRow: 5 + Cols: 4 + StartCol: 0 + Allocated: true + Kind: Arbitrary + ComponentType: Float32 + Interpolation: Linear + DynamicMask: 0x0 + Stream: 0 + - Name: A + Indices: [ 6 ] + StartRow: 6 + Cols: 3 + StartCol: 0 + Allocated: true + Kind: Arbitrary + ComponentType: Float32 + Interpolation: Linear + DynamicMask: 0x0 + Stream: 0 + - Name: A + Indices: [ 7 ] + StartRow: 7 + Cols: 4 + StartCol: 0 + Allocated: true + Kind: Arbitrary + ComponentType: SInt32 + Interpolation: Constant + DynamicMask: 0x0 + Stream: 0 + +The number of signature elements of each type is encoded in the +``llvm::dxbc::PSV::v1::RuntimeInfo`` structure. If any of the element count +values are non-zero, the size of the ``ProgramSignatureElement`` structure is +encoded next to allow versioning of that structure. Today there is only one +version. Following the size field is the specified number of signature elements +in the order input, output, then patch constant or primitive. + +Following the signature elements is a sequence of mask vectors encoded as a +series of 32-bit integers. Each 32-bit integer in the mask encodes values for 8 +input/output/patch or primitive elements. The mask vector is filled from least +significant bit to most significant bit with each added element shifting the +previous elements left. A reader needs to consult the total number of vectors +encoded in the ``RuntimeInfo`` structure to know how to read the mask vector. + +If the shader has ``UsesViewID`` enabled in the ``RuntimeInfo`` an output mask +vector will be included. The output mask vector is four arrays of 32-bit +unsigned integers. Each of the four arrays corresponds to an output stream. +Geometry shaders have a maximum of four output streams, all other shader stages +only support one output stream. Each bit in the mask vector identifies one +column of an output from the output signature depends on the ViewID. + +If the shader has ``UsesViewID`` enabled, it is a hull shader, and it has patch +constant or primitive vector elements, a patch constant or primitive vector mask +will be included. It is identical in structure to the output mask vector. Each +bit in the mask vector identifies one column of a patch constant output which +depends on the ViewID. + +The next series of mask vectors are similar in structure to the output mask +vector, but they contain an extra dimension. + +The output/input map is encoded next if the shader has inputs and outputs. The +output/input mask encodes which outputs are impacted by each column of each +input. The size for each mask vector is the size of the output max vector * the +number of inputs * 4 (for each component). Each bit in the mask vector +identifies one column of an output and a column of an input. A value of 1 means +the output is impacted by the input. + +If the shader is a hull shader, and it has inputs and patch outputs, an input to +patch map will be included next. This is identical in structure to the +output/input map. The dimensions are defined by the size of the patch constant +or primitive vector mask * the number of inputs * 4 (for each component). Each +bit in the mask vector identifies one column of a patch constant output and a +column of an input. A value of 1 means the output is impacted by the input. + +If the shader is a domain shader, and it has outputs and patch outputs, an +output patch map will be included next. This is identical in structure to the +output/input map. The dimensions are defined by the size of the patch constant +or primitive vector mask * the number of outputs * 4 (for each component). Each +bit in the mask vector identifies one column of a patch constant input and a +column of an output. A value of 1 means the output is impacted by the primitive +input. + +SFI0 Part +--------- +.. _SFI0: + +The SFI0 part encodes a 64-bit unsigned integer bitmask of the feature flags. +This denotes which optional features the shader requires. The flag values are +defined in `llvm/include/llvm/BinaryFormat/DXContainerConstants.def `_. diff --git a/llvm/docs/DirectXUsage.rst b/llvm/docs/DirectXUsage.rst index 79543e19bd34b..dd7796e1fdc96 100644 --- a/llvm/docs/DirectXUsage.rst +++ b/llvm/docs/DirectXUsage.rst @@ -14,6 +14,7 @@ User Guide for the DirectX Target :hidden: DirectX/DXILArchitecture + DirectX/DXContainer Introduction ============ @@ -81,6 +82,8 @@ code generation targets in LLVM, the LLVM codebase uses a more neutral name, The ``DXContainer`` format is sparsely documented in the functional specification, but a reference implementation exists in the `DirectXShaderCompiler. `_. +The format is documented in the LLVM project docs as well (see +:doc:`DirectX/DXContainer`). Support for generating ``DXContainer`` files in LLVM, is being added to the LLVM MC layer for object streamers and writers, and to the Object and ObjectYAML