Remove vertex input - #411

@sebavan too

I love this galaxy-brain idea, though it will likely be a porting pain.

I like that we can add it back later, and we don't even have to commit to an MVP without it.

I don't love that it makes it harder to directly emulate on non-large-uav systems, but like Kai says it's probably possible, just more work.

My concerns are mostly about some formats not being representable in structured buffers, especially those which have weird strides. For instance, can we support F32_R8G8B8, which has a bizarre stride of 24 bytes? What about R10G10B10A2?

I'd be happy to try to port my code to this if given a prerelease of WebGPU.

8-bit and 16-bit access to storage buffers in Vulkan: VK_KHR_8bit_storage, VK_KHR_16bit_storage, storageBuffer16BitAccess, etc.

D3D12 I only sort of understand D3D12, but I can't imagine it's more restrictive than Vulkan here.

I think Metal supports everything here.

@magcius F32_R8G8B8 and R10G10B10A2 are not in the list of our GPUVertexFormat today, so they aren't related to this PR. Are you saying that we'll have an issue if we decide at some point that we'll need them?

If you have 8bit storage buffer access or use an 8-bit texture, R8G8B8 (noninterleaved) can be done with a uint8_t[]. Only a little annoying. Issues come up when you want to interleave (as in James's observation that I edited into the original post).

(Sorry, what is F32_R8G8B8? float 32 r8g8b8?)

@kvark what I call F32_R8G8B8 is supported -- it's simply called "float3" in WebGPU, and was removed by this PR. I suppose you can emulate it with three float buffer loads, but you typically want to use the native load for that...

R10G10B10A2_UNORM is common for 4-bone vertex blend weights, since you get three 10-bit weights and the fourth can easily be reconstructed. It's perhaps not too difficult to read in the shader, though.

I have no qualms as long as we think we can support the esoteric vertex formats with application pulling.

I discussed this with our Metal team.

On some hardware, both old and new, it's reported that using vertex fetch is faster than doing it in the shader. I'd like to verify this by creating a benchmark. I have received guidance from the Metal team about how to best test the performance of vertex fetch.

The biggest benefit of vertex fetch is for content management at authoring time, not runtime. Changing the layout of vertices during an app's development is done in response to the changing type and amount of geometry. We've measured that using more compact attribute types improves both memory use and throughput to the GPU, but it's only possible to use these packed formats when you don't need the full precision. The same goes for index buffers - smaller index formats save memory, but you can only use them when you don't have too many vertices. Therefore, some layouts are better than others, and apps should be free to change their layouts to best fit their content.

Many engines have hundreds or thousands of shaders. When a change is made to the type / amount of geometry, having to rewrite all of them, or even create new variants of all of them, is difficult and tedious. Conceptually, the separation of concerns between vertex inputs (geometry) and shaders (materials, lighting, etc.) is important.

Also, when the Metal framework was being developed, we designed it with input from engine developers. We added vertex fetch because it was "a high priority feature request from engine developers".

This feedback is really valuable, thank you!

Couldn't the authoring concern be mostly addressed with a user-space (polyfill?) library. The user would, for example, dedicate a bing group index in the pipeline layout to this library, then provide it with the very same vertex buffer input structure that we currently have (which this PR removes) as well as the shader, which the library patches. Then the library would expose the very same setVertexBuffers call, which it would translate into a bind group creation (supposedly, cached) and binding.

Similar approach could be taken by the engines in question that have thousands of shaders - they just abstract away vertex inputs behind a function, which needs to be modified when the authors experiment with the vertex layout. To clarify: vertex pulling doesn't prevent them to use any formats they could be using with the native vertex data - it just makes some formats less convenient to work with (e.g. sRGB, mentioned, RGB10A2, etc).

This is all quite hand-wavy, of course. From this point of view, the vertex fetch stage is just another level of indirection, which, according to David Wheeler, can go a long way solving users problems :)

Reminder that changing shaders isn't magical. There's still a compilation process that needs to happen. Admittedly, this is already the case with some APIs today, as you need to create a pipeline layout for a second vertex layout, but not all IHVs will do this (NVIDIA supports different vertex layouts all in hardware).

What might be more of a pressing web concern -- if one wanted to e.g. pre-compile shaders to SPIR-V offline, we now have to have different variants for different vertex layouts, which means bloating our shader binary archives.

This is a scary change because we don't know that shader-based vertex pulling will be as efficient (or close enough) to the hardware paths, especially on mobile GPUs. For example ARM's latest GPU generation, Valhalla has something they call an "attribute unit" that sounds like it is a fixed-function vertex fetch (see here) and could be more efficient performance-wise and power-wise.

Somewhat related feedback from @DagAgren who uses "wgpu-native":

Using vertex buffers for all per-instance data seems very ugly and annoying if you have a lot of it.
Well there is a lot of manual setup, rather than just having matching structs in the shader and CPU code.

And indeed: describing all your data in vertex attributes is less user friendly than just making sure the shader sees the same data as your CPU. It's an ergonomic aspect that I previously didn't think of.

@litherum re: feedback from Metal team. I just remembered that on Metal we are not going to be able to use vertex fetch at all for indexed draw calls due to the lack of robust buffer access. This means for some draws on Metal we don't even have the option to get at that hardware performance. Is this a helpful argument in favor of adding robust buffer access to Metal?

@kainino0x Remind me again why we can't validate indices at runtime on the CPU or in a compute shader? If the answer is "performance," I'd like to measure it before disregarding the option.

Yeah, it's about performance. And you're right we should measure it - it seems very likely that vertex pulling is a smaller overhead than validating, but there could be hardware + workload combos where that isn't true. Workloads aren't trivial to find, but at least this one should be easier than some other performance tests we have wanted to do in the past.

And similar to those, it would be handy to just ask IHVs.

And indeed: describing all your data in vertex attributes is less user friendly than just making sure the shader sees the same data as your CPU. It's an ergonomic aspect that I previously didn't think of.

Be careful about this; you need to make sure to use indexed draws and index exclusively by SV_VertexID/SV_InstanceID (or similar) to get good vertex cache behavior.

@magcius I don't think I understand. Whatever you use to sample from those buffers will end up in the vertex cache associated with the SV_VertexID. Whether you use SV_VertexID directly to fetch data or something else matters as much as the coordinates you'd use for vertex texture sampling, i.e. it may affect performance but it's not related to the vertex cache specifically.

Yes, I'm saying it's an education problem -- upon moving to "no vertex input", it's easy to think "oh, I don't need a hardware index buffer, I can use all sorts of other data structures instead". But you really do want to keep around the hardware index buffer for cache reasons.

My concerns are mostly about some formats not being representable in structured buffers, especially those which have weird strides. For instance, can we support F32_R8G8B8, which has a bizarre stride of 24 bytes? What about R10G10B10A2?

If you're going to do this (not only in this PR but in Robust Buffer Access emulation), you must use texel buffer objects instead of SSBOs, the reason is that TBOs go through the texture cache path (not because you get free conversions to float4/vec4 from packed formats or small integers) which will save you when you're replacing the index buffer.

As soon as you introduce an index buffer, it will break memory coalescing, caching, etc. texture cache is much more used to random access by items in a execution subgroup.

@magcius is 100% right about needing a hardware index buffer, it may mitigate the "dependent read" cycle stalls/latencies if you read a value from an SSBO and then use that to compute a TBO offset to fetch from.

It is very possible that the hardware index buffer gets pre-fetched waay ahead of time and the index is ready in the register or a cache before the vertex shader even starts the invocation.

Also how does this play with #383 if you were to implement Vx pulls via TBO?

This confusion comes again and again. I don't think anybody suggests to remove the "hardware" index buffer. We are only talking about vertex buffers here.

I made a benchmark in Metal to test vertex pulling vs vertex fetch. It simply sets up the maximum number of attributes (31), makes them as fat as possible (float4), and adds them all together in the vertex shader.

Here are the results on my laptop (Intel(R) HD Graphics 5300):

Vertex pulling is 24% slower than using attributes on this machine.

@litherum funny, I just ran your test on Intel Iris 550 and got:

• true: 0.62 ms to 0.74 ms
• false: 0.49 ms to 0.56 ms

Given that isAttributes == true is for the "classical" vertex attribute version, and that the metric is the time... The vertex pulling appears to be 20% to 25% faster 🎃

You actually ran it!!! 😍😍😍

The advice I heard from the Metal team is that there is likely to be a performance difference on old Intel GPUs, so that's the GPU I tested on. Here's the graph of the raw data for the above chart in my previous post:

I also ran it on another newer Intel GPU (Intel(R) HD Graphics 630) and found that there wasn't any performance affect there:

This is consistent with the hypothesis from our Metal team that old Intel GPUs would have a performance hit, but new Intel GPUs wouldn't.

One interesting thing to notice is that the Y axis scales are different between my runs and @kvark's runs: on the Intel GPUs I've been running this on, the times are generally between 1-3ms. However, @kvark's runs are getting around half a millisecond. I wonder if there's something else going on here...

Interestingly enough, if you remove that one outlier in the Intel(R) HD Graphics 630 graphs above, the results are that using attributes is a statistically significant 4.5% progression against vertex pulling (P = 0.0451)

I tried to run it yesterday... (also suspected I might see different results). But it's not possible on macOS 10.14 ^_^ (gpuStartTime/gpuEndTime)

@litherum could you share the workflow of visualizing the output of your program as these nice graphs?

Interestingly, I modified the code to use array-of-structures (a single interleaved vertex buffer) instead of structure-of-arrays, and the results on the same machine were completely opposite: interleaved attributes are about 40% faster than pulling.

main-interleaved.swift.txt

@litherum could you share the workflow of visualizing the output of your program as these nice graphs?

Sure!

1. Run the benchmark, save the output to a file
2. $ grep true file.txt | cut -d " " -f 2 | pbcopy
3. Paste into Numbers
4. Repeat step 2, except with false instead of true
5. Paste into another column in Numbers
6. Add another row, where the cells are set to AVERAGE(column)
7. Select those new cells, and click the "bar graph" button

Here's an example: Vertex Pulling Benchmark.zip

~5% of Vulkan gpuinfo.org reports claim less than 16 (instead 4) for maxPerStageDescriptorStorageBuffers:
https://vulkan.gpuinfo.org/displaydevicelimit.php?name=maxPerStageDescriptorStorageBuffers

It's hard to extrapolate that to market share, but some of us might have more insight there. :)

Off the cuff, looking at these devices, I think we could consider requiring 16 here. I'm much more hesitant to remove vertex input (which always supports 16+ buffer bindings) if there's only 4 vertex storage buffer/uav slots.

40% worst-case perf hit does sound bad, if we think that's the best we can do there. It's a fairly synthetic measurement though, so I imagine real world unlikely to be drastic.

All told, I think I'd be comfortable shipping an MVP without vertex input, though we might decide the perf benefits on some hardware make it worth exposing. I want to play around with the benchmark more. Thanks for providing it! (and the nice graphs!)

Related, some ISVs are already switching to vertex pulling for flexibility: https://ourmachinery.com/post/vertex-assembly-and-skinning/

The question is not whether or not to perform vertex pulling; the question in whether or not to require vertex pulling to be performed. If vertex pulling is faster on some hardware, that hardware can use it even if the API exposed to the web content uses vertex descriptors.

The question in this PR is sort of about the opposite: "Should we remove fixed function vertex input, requiring developers to use vertex pulling?"

Playing with the interleaved example, I consistently saw best-case vertex pulling 100% slower than fixed function for float4[31] inputs. (GeForce GT 750M, late '13 15" MBP) These are huge inputs though, larger than some webgpu devices will support. I do suspect that NV is as-suspected doing something clever with (copying and?) caching interleaved vertex attrib reads, at least on this relatively ancient hardware.

some of us might have more insight there. :)

It's been really hard to get good data there. vulkan.gpuinfo.org isn't enough and I don't think we feature-grained stats for Android. I'll investigate more but don't hold your breath.

All told, I think I'd be comfortable shipping an MVP without vertex input, though we might decide the perf benefits on some hardware make it worth exposing. I want to play around with the benchmark more. Thanks for providing it! (and the nice graphs!)

This makes me very uneasy for several reasons:

• Removing vertex input means developers have to do it in the shader by themselves, which a minority of developers do today (at least that's my intuition). In particular none of the WebGL developers are doing vertex pulling because they don't have storage buffers.
• Having just 4 vertex buffers is a really small amount, probably sufficient for most cases but I'm sure folks will run into issues. (and it gets much worse if we do Metal-style tessellation but that adds new input step modes).
• We don't know how the performance will fare: in this image taken from this article about the latest ARM GPU architecture you can see an "Attribute Unit" that in the previous generation (Bifrost) "Handles attribute indexing and ddressing Defers to load/store for actual memory access" as explained slide 29 of this deck.

I think it's too risky for WebGPU to innovate in that part of the API.

@Kangz can we reach out to ISVs in our mailing list and ask?

We don't know how the performance will fare: in this image taken from this article about the latest ARM GPU architecture you can see an "Attribute Unit" that in the previous generation (Bifrost) "Handles attribute indexing and ddressing Defers to load/store for actual memory access" as explained slide 29 of this deck.

That's at least one reason not to do it.

I'm all for vertex pulling if necessary for emulating ``robustness extensions", but not as a one-size-fits-all permament solution.

It sounds like we're reaching consensus.

Interestingly, I modified the code to use array-of-structures (a single interleaved vertex buffer) instead of structure-of-arrays, and the results on the same machine were completely opposite: interleaved attributes are about 40% faster than pulling.

main-interleaved.swift.txt

    uint offset = vertexID * 32;

shouldn't this be * 31? (same in several other places)

* 32 seems fine to me. Metal's limit is 31, hence 31 elsewhere, but it shouldn't matter whether it's tightly packed to 31 or padded to 32.

Discussed in the October 28th 2019 call https://docs.google.com/document/d/1vjEeT_CO2zlHZ2K5SiNMdROVDk6ag8skSgN-ZEO4evg/edit

The group resolved today to keep vertex input (thereby closing this pull request no change).