Add service extension getSemanticsTree by hannah-hyj · Pull Request #189635 · flutter/flutter

hannah-hyj · 2026-07-17T01:02:44Z

tracking issue: flutter/devtools#9893

Add a service extension ,it will be used to visualize semantics tree in devtool.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the AI contribution guidelines and understand my responsibilities, or I am not using AI tools.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.

tracking issue: flutter/devtools#9893

It will used to visualize semantics tree in devtool

I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
I followed the breaking change policy and added Data Driven Fixes where supported.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

gemini-code-assist

Code Review

This pull request adds a new service extension, getSemanticsTree, to the WidgetInspectorService to retrieve and serialize the semantics tree as a JSON map. Feedback suggests enhancing the serialization by including additional properties (such as tooltips, actions, and transform matrices), prioritizing the non-deprecated rootPipelineOwner in pipeline owner lookup, and utilizing ensureVisualUpdate instead of scheduleWarmUpFrame for safer frame scheduling.

elliette · 2026-07-17T16:57:48Z

Thanks for taking this on, I'm very excited for the accessibility panel in DevTools! I think my main concern is I'm not sure that adding these new service extensions to the existing WidgetInspector is the right approach. I think it will be more maintainable in the long run if the semantics/accessibility extensions are registered under a separate namespace and implemented in a separate class (maybe AccessibilityInspector or SemanticsInspector) especially since I know we are planning on adding other service extensions as well. Let me know what you think - curious what @kenzieschmoll and @chunhtai thoughts are on this as well. Thanks!

hannah-hyj · 2026-07-17T18:38:26Z

Yeah adding new AccessibilityInspector should be easier for future maintenance.
The a11y page needs more service extension like ScaleFactorOverride, boldTextOverride, highContrastOverride, etc. I was thinking about adding them to WidgetInspector or FoundationServiceExtensions (the location brightnessOverride already lives), but I thought about it again, Yyah it makes more sense if all new services for a11y reside in AccessibilityInspector.

kenzieschmoll · 2026-07-17T21:21:35Z

Thanks for taking this on, I'm very excited for the accessibility panel in DevTools! I think my main concern is I'm not sure that adding these new service extensions to the existing WidgetInspector is the right approach. I think it will be more maintainable in the long run if the semantics/accessibility extensions are registered under a separate namespace and implemented in a separate class (maybe AccessibilityInspector or SemanticsInspector) especially since I know we are planning on adding other service extensions as well. Let me know what you think - curious what @kenzieschmoll and @chunhtai thoughts are on this as well. Thanks!

Completely agree with @elliette.

elliette · 2026-07-21T18:42:05Z

+  Future<Map<String, dynamic>> _getSemanticsTree(Map<String, String> parameters) async {
+    _semanticsHandle ??= SemanticsBinding.instance.ensureSemantics();
+
+    PipelineOwner? findPipelineOwner() {


Is there a reason for having this declared as function inside _getSemanticsTree vs as a separate private method?

No particular reasons, I've moved it out as a private method for readability

elliette · 2026-07-21T18:43:37Z

+      };
+    }
+
+    final SemanticsOwner semanticsOwner = pipelineOwner.semanticsOwner!;


Should we guard against the case where semanticsOwner is null here? (Maybe by returning error response like above?)

added the error response

elliette · 2026-07-21T18:44:28Z

+      return <String, dynamic>{'error': 'rootSemanticsNode is null', 'needsFrame': true};
+    }
+
+    Map<String, dynamic> toJsonMap(SemanticsNode node) {


Consider implementing toJsonMap on SemanticsNode instead

moved to SemanticsNode

victorsanni · 2026-07-21T22:03:01Z

+
+    PipelineOwner? findPipelineOwner() {
+      for (final RenderView renderView in RendererBinding.instance.renderViews) {
+        if (renderView.owner?.semanticsOwner != null) {


Looks like this is returning the first semanticsOwner of any renderView. Could there be multiple such renderViews? Maybe with multi-view?

Actually I haven't really thought about how multi-view semantics tree UI should look like in the dev tool tab. I will support single view for now, and add a todo about multi view here.

victorsanni · 2026-07-21T22:06:24Z

+  }
+
+  Future<Map<String, dynamic>> _getSemanticsTree(Map<String, String> parameters) async {
+    _semanticsHandle ??= SemanticsBinding.instance.ensureSemantics();


Could this be initialized but undisposed? It is disposed only in resetAllState, which is test-only.

Good catch!

Added disposeSemantics similar to widget inspector's disposeGroup call

chunhtai

We have existing vmservice for semantics tree dump, do you think we can repurpose it instead?

flutter/packages/flutter/lib/src/rendering/binding.dart

Line 199 in 0f02463

    
           name: RenderingServiceExtensions.debugDumpSemanticsTreeInInverseHitTestOrder.name,

Edit: just realize this is tree dump in string representation, so i guess not then.

chunhtai · 2026-07-28T12:22:23Z

+  }
+
+  Future<Map<String, dynamic>> _getSemanticsTree(Map<String, String> parameters) async {
+    _semanticsHandle ??= SemanticsBinding.instance.ensureSemantics();


we should probably separate out the ensuresemantics to its own service extension incase we have have other use case we want to enable semantics but not get the semantics tree.

since now enablesemantics is a separate endpoint, we should just return something like semantics not enabled

chunhtai · 2026-07-28T12:37:25Z

+  Map<String, dynamic> toJsonMap({
+    DebugSemanticsDumpOrder childOrder = DebugSemanticsDumpOrder.traversalOrder,
+  }) {
+    final SemanticsData data = getSemanticsData();


consider implement toJson method in SemanticsData directly so it will be more discoverable when adding new property.

also vmservice will call toJson on complex object directly if you throw it into the response IIRC. so there is really to have to call toJsonMap directly

chunhtai · 2026-07-28T12:42:52Z

+      }
+    }
+    final children = <Map<String, dynamic>>[
+      for (final SemanticsNode child in debugListChildrenInOrder(childOrder))


this is in traversal order, should also find a way to return hittest order or both at the same time.

can also consider return a flat map where key is id, and the children list will just be a list of id instead of recursive object.

also the transform here is hittest transform, traversal transform is computed before addtoupdate. if we want to include both, we probably need to refactor something out

chunhtai · 2026-07-28T12:44:42Z

+      return <String, dynamic>{'error': 'rootSemanticsNode is null', 'needsFrame': true};
+    }
+
+    return root.toJsonMap();


should put this in a more standard format like

{'data': root} incase we want to send more thing besides the tree in the future

chunhtai · 2026-07-28T12:46:41Z

+    }
+    final PipelineOwner rootOwner = RendererBinding.instance.rootPipelineOwner;
+    if (rootOwner.semanticsOwner != null) {
+      return rootOwner;


root is always empty i think

chunhtai · 2026-07-29T09:04:18Z

+        'width': rect.width,
+        'height': rect.height,
+      },
+      if (transform != null) 'transform': transform!.storage.toList(),


not something need to be address, just note that this is hittest transform.

chunhtai · 2026-07-29T09:06:43Z

+  }
+
+  Future<Map<String, dynamic>> _getSemanticsTree(Map<String, String> parameters) async {
+    _semanticsHandle ??= SemanticsBinding.instance.ensureSemantics();


since now enablesemantics is a separate endpoint, we should just return something like semantics not enabled

elliette · 2026-07-29T22:55:13Z


 export 'foundation.dart' show Brightness, UniqueKey;
 export 'rendering.dart' show TextSelectionHandleType;
+export 'src/widgets/accessibility_inspector.dart';


Why is this being exported? I'm not sure it needs to be public.

I was following the practice of widget_inspector.dart.
But yeah accessibility_inspector.dart doesn't really need to be exported, it doesn't define public widgets or anything, i will revert the export here

chunhtai

mostly lgtm, just some question about error handling

chunhtai · 2026-07-30T03:00:28Z

+    final PipelineOwner? pipelineOwner = _findPipelineOwner();
+    final SemanticsOwner? semanticsOwner = pipelineOwner?.semanticsOwner;
+    if (semanticsOwner == null) {
+      return <String, dynamic>{'error': 'No PipelineOwner with SemanticsOwner found'};


Hi @bkonyi, when we want to return error, should we throw directly? looking at the registerServiceExtension, looks like it will convert error throw into a error response to the call

It really depends. If you throw, it's going to be reported as a generic extension error and also go through FlutterError.reportError. BindingBase.registerServiceExtension's implementation unfortunately takes away a lot of control over error responses.

FWIW, I haven't found other service extensions in the framework that throw or explicitly return error state. You could always consider just returning empty responses and/or add a "success" property to indicate whether or not the request was successful.

I will just keep the current approach and not throw then.

bkonyi · 2026-07-31T17:43:00Z

+    final PipelineOwner? pipelineOwner = _findPipelineOwner();
+    final SemanticsOwner? semanticsOwner = pipelineOwner?.semanticsOwner;
+    if (semanticsOwner == null) {
+      return <String, dynamic>{'error': 'No PipelineOwner with SemanticsOwner found'};


It really depends. If you throw, it's going to be reported as a generic extension error and also go through FlutterError.reportError. BindingBase.registerServiceExtension's implementation unfortunately takes away a lot of control over error responses.

FWIW, I haven't found other service extensions in the framework that throw or explicitly return error state. You could always consider just returning empty responses and/or add a "success" property to indicate whether or not the request was successful.

elliette

LGTM with one note about the formatting change

elliette · 2026-08-06T22:07:37Z

-typedef ChildSemanticsConfigurationsDelegate = ChildSemanticsConfigurationsResult Function(
-  List<SemanticsConfiguration>,
-);
+typedef ChildSemanticsConfigurationsDelegate =


I think we are trying to avoid formatting changes until #187204 is resolved so that we can re-format as part of the Dart bump

flutter-dashboard Bot added the CICD Run CI/CD label Jul 17, 2026

github-actions Bot added the framework flutter/packages/flutter repository. See also f: labels. label Jul 17, 2026

gemini-code-assist Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread packages/flutter/lib/src/widgets/widget_inspector.dart Outdated

hannah-hyj requested review from elliette and victorsanni July 17, 2026 01:08

github-actions Bot added the a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) label Jul 17, 2026

hannah-hyj mentioned this pull request Jul 17, 2026

[Feature] Implement A11y page in devtool flutter/devtools#9893

Open

2 of 11 tasks

kenzieschmoll reviewed Jul 17, 2026

View reviewed changes

Comment thread packages/flutter/test/widgets/widget_inspector_test.dart Outdated

kenzieschmoll reviewed Jul 17, 2026

View reviewed changes

Comment thread packages/flutter/lib/src/widgets/binding.dart

hannah-hyj requested a review from kenzieschmoll July 17, 2026 23:23

elliette reviewed Jul 21, 2026

View reviewed changes

victorsanni reviewed Jul 21, 2026

View reviewed changes

hannah-hyj force-pushed the getSemanticsTree branch 2 times, most recently from 62ac6f8 to 14c3c1f Compare July 28, 2026 03:21

hannah-hyj requested a review from chunhtai July 28, 2026 04:20

chunhtai reviewed Jul 28, 2026

View reviewed changes

hannah-hyj requested review from chunhtai, elliette and victorsanni July 28, 2026 21:22

chunhtai reviewed Jul 29, 2026

View reviewed changes

hannah-hyj requested a review from chunhtai July 29, 2026 22:46

elliette reviewed Jul 29, 2026

View reviewed changes

chunhtai reviewed Jul 30, 2026

View reviewed changes

hannah-hyj requested a review from bkonyi July 30, 2026 20:36

bkonyi reviewed Jul 31, 2026

View reviewed changes

Update service_extensions_test.dart

095dc5c

hannah-hyj dismissed stale reviews from chunhtai and elliette via 095dc5c August 6, 2026 20:19

hannah-hyj force-pushed the getSemanticsTree branch from 63cabe7 to 095dc5c Compare August 6, 2026 20:19

hannah-hyj requested review from chunhtai and elliette August 6, 2026 20:20

elliette previously approved these changes Aug 6, 2026

View reviewed changes

revert format changes

d6c7ba9

hannah-hyj dismissed elliette’s stale review via d6c7ba9 August 6, 2026 22:11

hannah-hyj force-pushed the getSemanticsTree branch from 3a7e481 to d6c7ba9 Compare August 6, 2026 22:11

Merge branch 'master' into getSemanticsTree

3e9942f

elliette approved these changes Aug 6, 2026

View reviewed changes

hannah-hyj added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 6, 2026

auto-submit Bot added this pull request to the merge queue Aug 7, 2026

Merged via the queue into flutter:master with commit 198e1ce Aug 7, 2026
11 checks passed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Aug 7, 2026

Conversation

hannah-hyj commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

elliette commented Jul 17, 2026

Uh oh!

hannah-hyj commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kenzieschmoll commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chunhtai left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chunhtai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hannah-hyj commented Jul 17, 2026 •

edited

Loading

hannah-hyj commented Jul 17, 2026 •

edited

Loading

chunhtai left a comment •

edited

Loading