Investigate d3d12 vs. metal alignment/padding differences in sysgpu

[hordurj]

Running textured-quad, sprite2d sysgpu examples on Windows / D3D12 fails.

zig build run-core-sysgpu-textured-quad

mach\zig-out\bin\Shader@0x0000015909B90800(7,25): warning X3578: Output value 'vertex_main' is not completely initialized

If I change the shader return to
return VertexOutput(vec4(position, 0.0, 1.0), uv);

I get:

thread 23028 panic: Expr: sysgpu.shader.Air.Inst{ .struct_construct = sysgpu.shader.Air.Inst.StructConstruct{ .struct = sysgpu.shader.Air.InstIndex(6), .members = sysgpu.shader.Air.RefIndex(6) } }
.\mach\src\sysgpu\shader\codegen\hlsl.zig:714:39: 0xee82da in emitExpr (core-sysgpu-textured-quad.exe.obj)
        else => |inst| std.debug.panic("Expr: {}", .{inst}), // TODO
                                      ^
.\mach\src\sysgpu\shader\codegen\hlsl.zig:641:26: 0xf97324 in emitReturn (core-sysgpu-textured-quad.exe.obj)
        try hlsl.emitExpr(inst_idx);
                         ^
.\mach\src\sysgpu\shader\codegen\hlsl.zig:576:60: 0xeed216 in emitStatement (core-sysgpu-textured-quad.exe.obj)
        .@"return" => |return_inst_idx| try hlsl.emitReturn(return_inst_idx),
                                                           ^
.\mach\src\sysgpu\shader\codegen\hlsl.zig:545:35: 0xeecadf in emitFn (core-sysgpu-textured-quad.exe.obj)
            try hlsl.emitStatement(statement);
                                  ^
.\mach\src\sysgpu\shader\codegen\hlsl.zig:41:38: 0xeeda63 in gen (core-sysgpu-textured-quad.exe.obj)
            .@"fn" => try hlsl.emitFn(inst_idx),
                                     ^
.\mach\src\sysgpu\shader\CodeGen.zig:155:29: 0xf02d42 in generate (core-sysgpu-textured-quad.exe.obj)
        .hlsl => try genHlsl(allocator, air, debug_info),
                            ^
.\mach\src\sysgpu\d3d12.zig:2496:54: 0xf03215 in compile (core-sysgpu-textured-quad.exe.obj)
            .air => |air| try shader.CodeGen.generate(allocator, air, .hlsl, false, .{ .emit_source_file = "" }, null, null, null),
                                                     ^
.\mach\src\sysgpu\d3d12.zig:2658:56: 0xf03a34 in init (core-sysgpu-textured-quad.exe.obj)
        const vertex_shader = try vertex_module.compile(desc.vertex.entry_point, "vs_5_1");
                                                       ^
.\mach\src\sysgpu\d3d12.zig:464:35: 0xf05f57 in createRenderPipeline (core-sysgpu-textured-quad.exe.obj)
        return RenderPipeline.init(device, desc);
                                  ^
.\mach\src\core\examples\sysgpu\textured-quad\main.zig:73:54: 0xf1842c in init (core-sysgpu-textured-quad.exe.obj)
    const pipeline = core.device.createRenderPipeline(&pipeline_descriptor);
                                                     ^
.\mach\src\core\platform\native_entrypoint.zig:38:17: 0xf1ff39 in main (core-sysgpu-textured-quad.exe.obj)
    try app.init();

...
                ^
``

[hordurj]

I was testing this again and core-sysgpu-textured-quad only shows the warning and does not crash. But sprite2d is still crashing.

One thing with the sprite2d example is that is tries to load the sprite sheet from ../../examples/sprite2d/sprites.json but the file is actually under ../../src/core/examples/sysgpu/sprite2d/sprites.json. Not sure if the path should be changed or the sample file moved. I guess this is because of project reorganization.

I have the Debug layer now enabled. The output I get from that is:
[40476] D3D12 ERROR: ID3D12Device::CreateGraphicsPipelineState: Vertex Shader - Pixel Shader linkage error: Signatures between stages are incompatible. Semantic 'ATTR' is defined for mismatched hardware registers between the output stage and input stage. [ STATE_CREATION ERROR #660: CREATEGRAPHICSPIPELINESTATE_SHADER_LINKAGE_REGISTERINDEX]
[40476] D3D12 ERROR: ID3D12Device::CreateGraphicsPipelineState: Vertex Shader - Pixel Shader linkage error: Signatures between stages are incompatible. Semantic 'ATTR' of the input stage has a hardware register component mask that is not a subset of the output of the previous stage. [ STATE_CREATION ERROR #662: CREATEGRAPHICSPIPELINESTATE_SHADER_LINKAGE_REGISTERMASK]

[hordurj]

The sprite2d example works if the spriteIndex is changed from a f32 to vec3. But it does not solve the underlying problem. I also tested examples/sprite which uses mach.gfx.sprite_modules. That does not crash but the sprite does not show properly.

 struct VertexOutput {
   @builtin(position) Position : vec4<f32>,
   @location(0) fragUV : vec2<f32>,
-  @location(1) spriteIndex : f32,
+  @location(1) spriteIndex : vec3<f32>,
 };

[hordurj]

I found out why the mach.Core sprite example is not working on sysgpu / d3d12.

1. Texture.GetWidth returns height
2. Transfrom and transform_uv need to be transposed
3. Had to increase transform_uv to mat4x4. Probably because of this storage order issue.

[slimsag]

additional context

https://maraneshi.github.io/HLSL-ConstantBufferLayoutVisualizer/HLSL%20matrix%20storage%20order.png

in some modes not all buffers have the same column ordering

the default in HLSL is column major. Given that is what mach uses I think that is for the best to just use that.