Atom Tools: remove unnecessary Asset Processor work from a graph edit, and bound the wait - #20109

AtomToolsFramework: stop a graph edit waiting on Asset Processor work that did not need doing

A graph edit in an Atom tool spends most of its time between the edit and the viewport waiting on the
Asset Processor. Much of that wait is for work that should never have been queued, and some of it is
sleep the code chooses rather than work anything is doing.

Six commits, independent of each other in effect but sharing one theme. Four remove or shorten a
wait, one fixes a build race that shows up as a hard shader compile failure, one adds a test.

The branch is self-contained: it depends on nothing else and nothing here needs a caller that does
not already exist.

Rebuilds that should not have been triggered

Generated output was not deterministic. SortNodesInExecutionOrder scored nodes by input slots,
output slots and depth, then sorted stably. Those components tie readily — two nodes with the same
slot shape at the same depth is the ordinary case for sibling branches — and a stable sort leaves
ties in whatever order the caller supplied. Callers build their containers by iterating
GraphModel::Graph::GetNodes(), an unordered_map, so inserting any node rehashes it and siblings
come back the other way round.

The generated shader source then changes line order without the graph meaning anything different,
the Asset Processor sees a changed file, and every shader generated for the material is rebuilt.
Dropping an unconnected node on the canvas was enough. Adding the node ID as the last score
component makes the ordering total, so the output depends only on the graph.

Unchanged files were rewritten anyway. GraphTemplateFileData::Save always wrote. Rewriting a
file with identical content still moves its modification time and content hash, which re-runs every
job listing it as a source dependency — again, every shader generated for the material.

Many edits regenerate identical text for most or all of the template files: moving a node, changing
the selection, editing a branch that does not feed the output. Comparing against what is on disk
removes the entire asset pipeline round trip for that whole class of edit. Save gains an optional
out parameter reporting whether it actually wrote, so a caller that needs to know can ask rather than
assume.

A dependency the Asset Processor could not see

GetListOfIncludedFiles resolves #include lines textually, so it cannot follow an include whose
path is a macro. Material pipeline generated shaders reach their parameter struct exactly that way:
MaterialTypeBuilder emits #define MATERIAL_PARAMETERS_AZSLI_FILE_PATH "<name>_parameters.azsli"
and the shared material azsli does #include MATERIAL_PARAMETERS_AZSLI_FILE_PATH.

With no dependency registered, the Asset Processor does not know the two belong together. The
generated surface evaluation azsli beside the graph is a plain include and therefore is a
registered dependency, so editing a graph retriggers the shader job immediately — racing the
Material Type Builder pipeline stage job that regenerates the parameter struct. The shader compiles
new graph code against the old struct and fails with no member named ... in 'MaterialParameters'.

The dependency is registered by reading the define back out of the generated AZSL, so it is scoped to
shaders that actually contain it. Hand-written shaders are unaffected.

Waiting longer than the work takes

Product replacement retried on a flat 250 ms. The first failure is usually not contention that
needs waiting out. Immediately before removing a product the Asset Processor broadcasts
JobFileClaimed to every connected client, and AzFramework's handler answers by queuing a streamer
FlushCache. The handle is dropped a few milliseconds later on the streamer thread and nothing
acknowledges it back, so the remove races that flush and loses — and a flat quarter second charges
full price for a wait that was over almost immediately. With an editor open on the assets being
built this is the normal path, per product.

Retries now start at 5 ms and double to the same 250 ms ceiling. A file genuinely held open by
something else reaches the old pacing within a few hundred milliseconds; the common case costs about
as little as the platform's sleep granularity allows. The overall timeout is unchanged.

Files with no builder were waited on. An azsli has no jobs, so querying its status costs a full
round trip that can never return anything but an empty list. ShouldReportGeneratedFileStatus is
virtual so a derived compiler can also exclude files whose readiness its own consumer handles. The
files stay in m_generatedFiles, which other systems read in full.

One settled path was drained per update. The driving thread sleeps 10 ms between iterations, so
stepping a single path at a time imposed a floor of 10 ms times the number of reported paths on every
compile, even when every job had long since finished. It now drains everything already settled.

The wait was unbounded. AssetStatusReporter walks its paths with an index that only moves
forward, so a path it is sitting on must reach a terminal job state or it waits there forever — which
the Asset Processor can fail to deliver, for instance when a duplicated intermediate entry
(#19642) leaves paths that never settle. The wait is now bounded, 15 seconds by default and
configurable through /O3DE/AtomToolsFramework/GraphCompiler/AssetStatusTimeoutMs, with 0
restoring the original unbounded behaviour. Giving up reports the compile complete and leaves the
consumer to pick the assets up from the catalog. AssetStatusReporterSystem gains
GetStatusMessage so the warning can name the path it gave up on.

Cancellable compilation

A graph compile ran on a background job and could not be stopped, so a second edit arriving mid-flight
had to wait out the first, and the compiler could sit in Processing indefinitely if the Asset
Processor never settled.

Reset now reserves the compiler synchronously when idle — closing the window where a second edit
observes a terminal state and dispatches another job before the first worker has started — and
requests cancellation when a job is already active, so the caller can leave the replacement queued.
FinishCompile publishes exactly one terminal state and releases the reservation, with a
cancellation request always winning over the requested state. CompileGraph still supports direct
callers by creating and consuming its own reservation when none exists.

Both GraphCompiler subclasses are converted to publish their terminal state through
FinishCompile. This is the part to review closely: the reservation is released there and nowhere
else, so a subclass reporting Complete or Failed by calling SetState directly holds the
compiler reserved forever and never compiles again — with no error, and no symptom until the tool
stops responding to edits.

The final commit tests that contract: a compile reserves the compiler, a second reservation attempt
requests cancellation and leaves the replacement queued, a cancellation request wins over the
worker's terminal state, and the completion callback does not run for a cancelled compile.

Compatibility

• The ordering change alters generated line order once, on the first build after merging, and is
  stable from then on.
• Skipping unchanged writes is observable only as absent Asset Processor work.
• The new dependency is registered only for shaders containing the define.
• The retry backoff reaches the previous pacing for anything genuinely contended; the overall timeout
  is unchanged.
• The status wait bound is configurable, and 0 restores the previous unbounded behaviour.