Allow resetting test invariants in addTearDown - #192082
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors TestWidgetsFlutterBinding to run post-test invariant checks inside postTest using a try-finally block, ensuring cleanup is executed even when checks fail, and skips these checks if the test body itself throws an exception. The review feedback suggests avoiding potential exception masking in the finally blocks of AutomatedTestWidgetsFlutterBinding and LiveTestWidgetsFlutterBinding by only running assertions if super.postTest() completes successfully. Additionally, it recommends registering debug variable cleanups in addTearDown within the new tests to prevent state leakage in case of test failures.
| test('direct runTest with failed test body skips invariant check in postTest', () async { | ||
| final TestWidgetsFlutterBinding binding = TestWidgetsFlutterBinding.ensureInitialized(); | ||
| final TestExceptionReporter oldReporter = reportTestException; | ||
| reportTestException = (FlutterErrorDetails details, String testDescription) {}; | ||
| addTearDown(() { | ||
| reportTestException = oldReporter; | ||
| }); | ||
|
|
||
| await binding.runTest(() async { | ||
| debugDefaultTargetPlatformOverride = TargetPlatform.macOS; | ||
| throw Exception('test failed'); | ||
| }, () {}); | ||
|
|
||
| // postTest should NOT throw invariant error because the test body failed | ||
| expect(() => binding.postTest(), returnsNormally); | ||
| expect(binding.inTest, isFalse); | ||
| debugDefaultTargetPlatformOverride = null; | ||
| }); |
There was a problem hiding this comment.
If the expect block or any other assertion in this test fails, the cleanup line debugDefaultTargetPlatformOverride = null; will be skipped, leaking the override to subsequent tests.
Register the cleanup in the existing addTearDown block to ensure it always runs.
| test('direct runTest with failed test body skips invariant check in postTest', () async { | |
| final TestWidgetsFlutterBinding binding = TestWidgetsFlutterBinding.ensureInitialized(); | |
| final TestExceptionReporter oldReporter = reportTestException; | |
| reportTestException = (FlutterErrorDetails details, String testDescription) {}; | |
| addTearDown(() { | |
| reportTestException = oldReporter; | |
| }); | |
| await binding.runTest(() async { | |
| debugDefaultTargetPlatformOverride = TargetPlatform.macOS; | |
| throw Exception('test failed'); | |
| }, () {}); | |
| // postTest should NOT throw invariant error because the test body failed | |
| expect(() => binding.postTest(), returnsNormally); | |
| expect(binding.inTest, isFalse); | |
| debugDefaultTargetPlatformOverride = null; | |
| }); | |
| test('direct runTest with failed test body skips invariant check in postTest', () async { | |
| final TestWidgetsFlutterBinding binding = TestWidgetsFlutterBinding.ensureInitialized(); | |
| final TestExceptionReporter oldReporter = reportTestException; | |
| reportTestException = (FlutterErrorDetails details, String testDescription) {}; | |
| addTearDown(() { | |
| reportTestException = oldReporter; | |
| debugDefaultTargetPlatformOverride = null; | |
| }); | |
| await binding.runTest(() async { | |
| debugDefaultTargetPlatformOverride = TargetPlatform.macOS; | |
| throw Exception('test failed'); | |
| }, () {}); | |
| // postTest should NOT throw invariant error because the test body failed | |
| expect(() => binding.postTest(), returnsNormally); | |
| expect(binding.inTest, isFalse); | |
| }); |
There was a problem hiding this comment.
I guess the point is that you're not resetting debugDefaultTargetPlatformOverride in a tear down though?
There was a problem hiding this comment.
I don't think it matters much, since this test file is expected to... not throw. If it throws then we have to fix it anyway.
Also I've significantly changed the structure of this test file, where this problem should no longer exist.
| test('direct runTest with failed test body skips invariant check in postTest', () async { | ||
| final TestWidgetsFlutterBinding binding = TestWidgetsFlutterBinding.ensureInitialized(); | ||
| final TestExceptionReporter oldReporter = reportTestException; | ||
| reportTestException = (FlutterErrorDetails details, String testDescription) {}; | ||
| addTearDown(() { | ||
| reportTestException = oldReporter; | ||
| }); | ||
|
|
||
| await binding.runTest(() async { | ||
| debugDefaultTargetPlatformOverride = TargetPlatform.macOS; | ||
| throw Exception('test failed'); | ||
| }, () {}); | ||
|
|
||
| // postTest should NOT throw invariant error because the test body failed | ||
| expect(() => binding.postTest(), returnsNormally); | ||
| expect(binding.inTest, isFalse); | ||
| debugDefaultTargetPlatformOverride = null; | ||
| }); |
There was a problem hiding this comment.
I guess the point is that you're not resetting debugDefaultTargetPlatformOverride in a tear down though?
justinmc
left a comment
There was a problem hiding this comment.
Sorry I meant to stamp it 👍
chunhtai
left a comment
There was a problem hiding this comment.
overall LGTM, just some minor comment
| _verifyErrorWidgetBuilderUnset(errorWidgetBuilderBeforeTest); | ||
| _verifyShouldPropagateDevicePointerEventsUnset(shouldPropagateDevicePointerEventsBeforeTest); | ||
| _verifyInvariants(); | ||
| _shouldVerifyInvariants = true; |
There was a problem hiding this comment.
so this is here to ensure we only verify invariant when test body run successfully?
There was a problem hiding this comment.
Yes. I've added a paragraph in the PR description to explain it. See the section "3. Preserving Existing Behavior on Failed Tests (_shouldVerifyInvariants)".
| _clock = null; | ||
| _currentFakeAsync = null; | ||
| try { | ||
| super.postTest(); |
There was a problem hiding this comment.
I think we should set the contract that parent method should handle the try catch and each override handle their own try catch if they introduce new way things can crash.
There was a problem hiding this comment.
Are you suggesting that we should not try here and this method should assume that super.postTest does not throw? That's a good idea.
There was a problem hiding this comment.
I've made the changes, so that postTest are defined to never throw. I've updated the PR description for some detailed explanation. Thanks!
There was a problem hiding this comment.
yes, I think that will be cleaner instead of each subclass has to do a try catch
There was a problem hiding this comment.
I've made non-trivial changes to this PR.
- As recommended by @chunhtai ,
postTestis defined to no longer throw. This does not break the current behavior in any means, but does require a slightly different way to writepostTest. - I've added more comprehensive tests to
bindings_invariants_test.dart. - I've also added more detailed explanation to the PR description, partly because I think the test harness is not wildly understood and it'd be great to document my understandings. :)
| _clock = null; | ||
| _currentFakeAsync = null; | ||
| try { | ||
| super.postTest(); |
There was a problem hiding this comment.
I've made the changes, so that postTest are defined to never throw. I've updated the PR description for some detailed explanation. Thanks!
| test('direct runTest with failed test body skips invariant check in postTest', () async { | ||
| final TestWidgetsFlutterBinding binding = TestWidgetsFlutterBinding.ensureInitialized(); | ||
| final TestExceptionReporter oldReporter = reportTestException; | ||
| reportTestException = (FlutterErrorDetails details, String testDescription) {}; | ||
| addTearDown(() { | ||
| reportTestException = oldReporter; | ||
| }); | ||
|
|
||
| await binding.runTest(() async { | ||
| debugDefaultTargetPlatformOverride = TargetPlatform.macOS; | ||
| throw Exception('test failed'); | ||
| }, () {}); | ||
|
|
||
| // postTest should NOT throw invariant error because the test body failed | ||
| expect(() => binding.postTest(), returnsNormally); | ||
| expect(binding.inTest, isFalse); | ||
| debugDefaultTargetPlatformOverride = null; | ||
| }); |
There was a problem hiding this comment.
I don't think it matters much, since this test file is expected to... not throw. If it throws then we have to fix it anyway.
Also I've significantly changed the structure of this test file, where this problem should no longer exist.
| try { | ||
| if (_shouldVerifyInvariants) { | ||
| _shouldVerifyInvariants = false; | ||
| FlutterErrorDetails? invariantError; |
There was a problem hiding this comment.
To explain:
- Previously, verification errors were caught, and after cleanups, rethrowed to outside. The outside harness will catch these errors and post them to
reportTestException. - Now, verifications errors were caught, and after cleanups, posted to
reportTestExceptiondirectly.
In this way, the behavior is kept the same, while postTest itself does not throw.
|
autosubmit label was removed for flutter/flutter/192082, because - The status or check suite Tree_analyze has failed. Please fix the issues identified (or deflake) before re-applying this label. |
|
Update: There's a failure in google tests. It appears that this PR revealed some tests that are written incorrectly (lacking necessary |
|
Reason for revert: Breaks tree with tool_integration_tests failures. Example: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_2/1329/overview |
|
Successfully created revert PR: #193383 |
) Reverts: [Allow resetting test invariants in `addTearDown`](flutter#192082) Initiated by: @b-luk Reason for reverting: Breaks tree with tool_integration_tests failures. Example: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_2/1329/overview Original PR Author: @dkwingsmt Reviewed By: @justinmc The original PR description is provided below: Fixes flutter#110488. ### Background: The `testWidgets` Lifecycle Understanding this change requires distinguishing between two layers of `Future`s and two distinct verification steps in `testWidgets`: #### 1. Two Layers of `Future`s * **`testBody()` Future**: The closure containing the user's test logic. * **`runTest()` Future**: The overall async harness returned by `binding.runTest()`. It manages the test zone, unmounts the widget tree, and resolves only after `testBody()` and test-specific verifications finish. #### 2. Two Distinct Verification Phases * **Test-specific verifications (`invariantTester`)**: Passed as a callback to `runTest` and executed synchronously inside `_runTestBody` immediately after the `testBody()` Future completes and the widget tree is unmounted. In `testWidgets`, this runs `tester._endOfTestVerifications` to verify that tickers and semantics handles were disposed. * **Framework-level invariant checks**: Verifies global framework state, including ensuring foundation/rendering/widgets debug flags were reset and no non-periodic timers or animations remain. #### 3. The Teardown Lifecycle & Root Cause of flutter#110488 When `testWidgets` runs: 1. `testWidgets` registers `test_package.addTearDown(binding.postTest)` with `package:test`. 2. `testWidgets` calls and returns `binding.runTest(...)`. 3. Inside `testBody()`, users may register cleanups using `addTearDown` (for example, `addTearDown(() => debugDefaultTargetPlatformOverride = null)`). 4. `package:test` executes teardowns in **LIFO (reverse registration) order** only *after* the Future returned by `runTest` completes. Previously, framework-level invariant checks ran at the end of `_runTestBody` *before* the `runTest` Future completed. As a result, they ran before `package:test` executed any user-registered `addTearDown` callbacks, causing tests that clean up debug flags via `addTearDown` to fail invariant verification (flutter#110488). The following diagram shows the code flow after this PR: ``` testWidgets(description, (tester) async { ... }) │ ├── 1. Registration Phase │ └── test_package.addTearDown(binding.postTest) [Teardown Stack: #1] │ ├── 2. Test Execution Phase: binding.runTest(...) │ │ │ │ [ Future 1: testBody() ] │ ├──► Run testBody(tester) │ │ └── User calls addTearDown(...) [Teardown Stack: #2] │ │ │ │ [ Post-testBody cleanup (inside runTest) ] │ ├──► Unmount widget tree (runApp) │ ├──► Verification 1: invariantTester() (tickers, semantics) │ └──► Set _shouldVerifyInvariants = true │ │ [ Future 2: runTest() completes ] │ └── 3. Teardown Phase: package:test executes in LIFO order │ ├──► [Step 1] User Teardowns (Stack #2) │ └── e.g., debugDefaultTargetPlatformOverride = null │ └──► [Step 2] binding.postTest() (Stack #1) ├── Verification 2: Framework Invariants (debug vars, timers) ├── Reset Binding State (_currentFakeAsync, etc.) └── reportTestException(...) (if Verification 2 failed) ``` ### Changes & Design Rationale #### 1. Defer Framework Invariant Checks to `postTest` Framework invariant checks (`_verifyInvariants()`, `_verifyAutoUpdateGoldensUnset`, etc.) are moved from `_runTestBody` into `TestWidgetsFlutterBinding.postTest`. Because `testWidgets` registers `binding.postTest` before running the test body, `package:test`'s LIFO teardown ordering guarantees that all user-registered `addTearDown` callbacks execute before `binding.postTest`. #### 2. Clarifying the `postTest` Error Contract Previously, `postTest` only performed state cleanup and did not check invariants. Moving invariant checks into `postTest` introduced the question of how invariant failures should be propagated: allowing `postTest` to throw, or absorbing errors and reporting them via `reportTestException`. We explicitly define the contract of `postTest` to **not throw exceptions**, reporting any invariant failures via `reportTestException`: * **Rationale & Trade-offs**: * *Alternative considered (throwing)*: If `super.postTest()` threw on an invariant failure, `package:test` would still capture the failure. However, subclasses (`AutomatedTestWidgetsFlutterBinding`, `LiveTestWidgetsFlutterBinding`, or custom bindings) override `postTest` to reset their own state (e.g. nulling `_currentFakeAsync` / `_clock`, or setting `_inTest = false`). Allowing `super.postTest()` to throw would require every subclass to defensively wrap `super.postTest()` in `try ... finally`. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading `!inTest` failures in subsequent tests. * *Chosen approach (non-throwing)*: Absorbing errors in `postTest` and forwarding them to `reportTestException` guarantees that all cleanup in base and subclass implementations executes sequentially without defensive `try ... finally` boilerplate, while still cleanly failing the test. **Why this does not break existing behavior:** In the previous implementation, when invariant assertions failed inside `_runTestBody`, they were caught by `_testZone.handleUncaughtError` and forwarded to `reportTestException` (which delegates to `package:test.registerException`). They never escaped `runTest` as unhandled thrown exceptions. Catching invariant failures in `postTest` and routing them to `reportTestException` preserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution. #### 3. Preserving Existing Behavior on Failed Tests (`_shouldVerifyInvariants`) Historically, `_runTestBody` only verified invariants if the test body did not fail: ```dart if (_pendingExceptionDetails == null) { // We only try to clean up and verify invariants if we didn't already // fail. If we got an exception already, then we instead leave everything // alone so that we don't cause more spurious errors. ... _verifyInvariants(); } ``` When invariant checks are moved to `postTest`, `_pendingExceptionDetails` is no longer available because it has already been reported and cleared by `testCompletionHandler` when `runTest` completed. To preserve this exact existing behavior across the lifecycle boundary, `_shouldVerifyInvariants` is introduced. It is set to `true` at the end of `_runTestBody` only when `_pendingExceptionDetails == null`. If a test fails in its body, `_shouldVerifyInvariants` remains `false`, and `postTest` skips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure. #### 4. Documentation Updates Updated the doc comments on: * `TestWidgetsFlutterBinding.postTest`: Documents the non-throwing contract, the `@mustCallSuper` requirement, and error reporting via `reportTestException`. * `TestWidgetsFlutterBinding.runTest`: Clarifies the distinction between `invariantTester` (which runs before `runTest` completes) and framework invariant checks (which run in `postTest` after `runTest` and user teardowns complete). ### Tests - Added unit tests in `packages/flutter_test/test/bindings_invariants_test.dart` verifying that foundation variables (`debugDefaultTargetPlatformOverride`, `debugDoublePrecision`, `debugBrightnessOverride`), `autoUpdateGoldenFiles`, `ErrorWidget.builder`, and `shouldPropagateDevicePointerEvents` can all be reset using `addTearDown`. - Added test cases verifying invariant failures in `postTest` and ensuring invariant checks are skipped when the test body throws. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Fixes flutter#110488. ### Background: The `testWidgets` Lifecycle Understanding this change requires distinguishing between two layers of `Future`s and two distinct verification steps in `testWidgets`: #### 1. Two Layers of `Future`s * **`testBody()` Future**: The closure containing the user's test logic. * **`runTest()` Future**: The overall async harness returned by `binding.runTest()`. It manages the test zone, unmounts the widget tree, and resolves only after `testBody()` and test-specific verifications finish. #### 2. Two Distinct Verification Phases * **Test-specific verifications (`invariantTester`)**: Passed as a callback to `runTest` and executed synchronously inside `_runTestBody` immediately after the `testBody()` Future completes and the widget tree is unmounted. In `testWidgets`, this runs `tester._endOfTestVerifications` to verify that tickers and semantics handles were disposed. * **Framework-level invariant checks**: Verifies global framework state, including ensuring foundation/rendering/widgets debug flags were reset and no non-periodic timers or animations remain. #### 3. The Teardown Lifecycle & Root Cause of flutter#110488 When `testWidgets` runs: 1. `testWidgets` registers `test_package.addTearDown(binding.postTest)` with `package:test`. 2. `testWidgets` calls and returns `binding.runTest(...)`. 3. Inside `testBody()`, users may register cleanups using `addTearDown` (for example, `addTearDown(() => debugDefaultTargetPlatformOverride = null)`). 4. `package:test` executes teardowns in **LIFO (reverse registration) order** only *after* the Future returned by `runTest` completes. Previously, framework-level invariant checks ran at the end of `_runTestBody` *before* the `runTest` Future completed. As a result, they ran before `package:test` executed any user-registered `addTearDown` callbacks, causing tests that clean up debug flags via `addTearDown` to fail invariant verification (flutter#110488). The following diagram shows the code flow after this PR: ``` testWidgets(description, (tester) async { ... }) │ ├── 1. Registration Phase │ └── test_package.addTearDown(binding.postTest) [Teardown Stack: #1] │ ├── 2. Test Execution Phase: binding.runTest(...) │ │ │ │ [ Future 1: testBody() ] │ ├──► Run testBody(tester) │ │ └── User calls addTearDown(...) [Teardown Stack: #2] │ │ │ │ [ Post-testBody cleanup (inside runTest) ] │ ├──► Unmount widget tree (runApp) │ ├──► Verification 1: invariantTester() (tickers, semantics) │ └──► Set _shouldVerifyInvariants = true │ │ [ Future 2: runTest() completes ] │ └── 3. Teardown Phase: package:test executes in LIFO order │ ├──► [Step 1] User Teardowns (Stack #2) │ └── e.g., debugDefaultTargetPlatformOverride = null │ └──► [Step 2] binding.postTest() (Stack #1) ├── Verification 2: Framework Invariants (debug vars, timers) ├── Reset Binding State (_currentFakeAsync, etc.) └── reportTestException(...) (if Verification 2 failed) ``` ### Changes & Design Rationale #### 1. Defer Framework Invariant Checks to `postTest` Framework invariant checks (`_verifyInvariants()`, `_verifyAutoUpdateGoldensUnset`, etc.) are moved from `_runTestBody` into `TestWidgetsFlutterBinding.postTest`. Because `testWidgets` registers `binding.postTest` before running the test body, `package:test`'s LIFO teardown ordering guarantees that all user-registered `addTearDown` callbacks execute before `binding.postTest`. #### 2. Clarifying the `postTest` Error Contract Previously, `postTest` only performed state cleanup and did not check invariants. Moving invariant checks into `postTest` introduced the question of how invariant failures should be propagated: allowing `postTest` to throw, or absorbing errors and reporting them via `reportTestException`. We explicitly define the contract of `postTest` to **not throw exceptions**, reporting any invariant failures via `reportTestException`: * **Rationale & Trade-offs**: * *Alternative considered (throwing)*: If `super.postTest()` threw on an invariant failure, `package:test` would still capture the failure. However, subclasses (`AutomatedTestWidgetsFlutterBinding`, `LiveTestWidgetsFlutterBinding`, or custom bindings) override `postTest` to reset their own state (e.g. nulling `_currentFakeAsync` / `_clock`, or setting `_inTest = false`). Allowing `super.postTest()` to throw would require every subclass to defensively wrap `super.postTest()` in `try ... finally`. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading `!inTest` failures in subsequent tests. * *Chosen approach (non-throwing)*: Absorbing errors in `postTest` and forwarding them to `reportTestException` guarantees that all cleanup in base and subclass implementations executes sequentially without defensive `try ... finally` boilerplate, while still cleanly failing the test. **Why this does not break existing behavior:** In the previous implementation, when invariant assertions failed inside `_runTestBody`, they were caught by `_testZone.handleUncaughtError` and forwarded to `reportTestException` (which delegates to `package:test.registerException`). They never escaped `runTest` as unhandled thrown exceptions. Catching invariant failures in `postTest` and routing them to `reportTestException` preserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution. #### 3. Preserving Existing Behavior on Failed Tests (`_shouldVerifyInvariants`) Historically, `_runTestBody` only verified invariants if the test body did not fail: ```dart if (_pendingExceptionDetails == null) { // We only try to clean up and verify invariants if we didn't already // fail. If we got an exception already, then we instead leave everything // alone so that we don't cause more spurious errors. ... _verifyInvariants(); } ``` When invariant checks are moved to `postTest`, `_pendingExceptionDetails` is no longer available because it has already been reported and cleared by `testCompletionHandler` when `runTest` completed. To preserve this exact existing behavior across the lifecycle boundary, `_shouldVerifyInvariants` is introduced. It is set to `true` at the end of `_runTestBody` only when `_pendingExceptionDetails == null`. If a test fails in its body, `_shouldVerifyInvariants` remains `false`, and `postTest` skips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure. #### 4. Documentation Updates Updated the doc comments on: * `TestWidgetsFlutterBinding.postTest`: Documents the non-throwing contract, the `@mustCallSuper` requirement, and error reporting via `reportTestException`. * `TestWidgetsFlutterBinding.runTest`: Clarifies the distinction between `invariantTester` (which runs before `runTest` completes) and framework invariant checks (which run in `postTest` after `runTest` and user teardowns complete). ### Tests - Added unit tests in `packages/flutter_test/test/bindings_invariants_test.dart` verifying that foundation variables (`debugDefaultTargetPlatformOverride`, `debugDoublePrecision`, `debugBrightnessOverride`), `autoUpdateGoldenFiles`, `ErrorWidget.builder`, and `shouldPropagateDevicePointerEvents` can all be reset using `addTearDown`. - Added test cases verifying invariant failures in `postTest` and ensuring invariant checks are skipped when the test body throws. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
) Reverts: [Allow resetting test invariants in `addTearDown`](flutter#192082) Initiated by: @b-luk Reason for reverting: Breaks tree with tool_integration_tests failures. Example: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_2/1329/overview Original PR Author: @dkwingsmt Reviewed By: @justinmc The original PR description is provided below: Fixes flutter#110488. ### Background: The `testWidgets` Lifecycle Understanding this change requires distinguishing between two layers of `Future`s and two distinct verification steps in `testWidgets`: #### 1. Two Layers of `Future`s * **`testBody()` Future**: The closure containing the user's test logic. * **`runTest()` Future**: The overall async harness returned by `binding.runTest()`. It manages the test zone, unmounts the widget tree, and resolves only after `testBody()` and test-specific verifications finish. #### 2. Two Distinct Verification Phases * **Test-specific verifications (`invariantTester`)**: Passed as a callback to `runTest` and executed synchronously inside `_runTestBody` immediately after the `testBody()` Future completes and the widget tree is unmounted. In `testWidgets`, this runs `tester._endOfTestVerifications` to verify that tickers and semantics handles were disposed. * **Framework-level invariant checks**: Verifies global framework state, including ensuring foundation/rendering/widgets debug flags were reset and no non-periodic timers or animations remain. #### 3. The Teardown Lifecycle & Root Cause of flutter#110488 When `testWidgets` runs: 1. `testWidgets` registers `test_package.addTearDown(binding.postTest)` with `package:test`. 2. `testWidgets` calls and returns `binding.runTest(...)`. 3. Inside `testBody()`, users may register cleanups using `addTearDown` (for example, `addTearDown(() => debugDefaultTargetPlatformOverride = null)`). 4. `package:test` executes teardowns in **LIFO (reverse registration) order** only *after* the Future returned by `runTest` completes. Previously, framework-level invariant checks ran at the end of `_runTestBody` *before* the `runTest` Future completed. As a result, they ran before `package:test` executed any user-registered `addTearDown` callbacks, causing tests that clean up debug flags via `addTearDown` to fail invariant verification (flutter#110488). The following diagram shows the code flow after this PR: ``` testWidgets(description, (tester) async { ... }) │ ├── 1. Registration Phase │ └── test_package.addTearDown(binding.postTest) [Teardown Stack: #1] │ ├── 2. Test Execution Phase: binding.runTest(...) │ │ │ │ [ Future 1: testBody() ] │ ├──► Run testBody(tester) │ │ └── User calls addTearDown(...) [Teardown Stack: #2] │ │ │ │ [ Post-testBody cleanup (inside runTest) ] │ ├──► Unmount widget tree (runApp) │ ├──► Verification 1: invariantTester() (tickers, semantics) │ └──► Set _shouldVerifyInvariants = true │ │ [ Future 2: runTest() completes ] │ └── 3. Teardown Phase: package:test executes in LIFO order │ ├──► [Step 1] User Teardowns (Stack #2) │ └── e.g., debugDefaultTargetPlatformOverride = null │ └──► [Step 2] binding.postTest() (Stack #1) ├── Verification 2: Framework Invariants (debug vars, timers) ├── Reset Binding State (_currentFakeAsync, etc.) └── reportTestException(...) (if Verification 2 failed) ``` ### Changes & Design Rationale #### 1. Defer Framework Invariant Checks to `postTest` Framework invariant checks (`_verifyInvariants()`, `_verifyAutoUpdateGoldensUnset`, etc.) are moved from `_runTestBody` into `TestWidgetsFlutterBinding.postTest`. Because `testWidgets` registers `binding.postTest` before running the test body, `package:test`'s LIFO teardown ordering guarantees that all user-registered `addTearDown` callbacks execute before `binding.postTest`. #### 2. Clarifying the `postTest` Error Contract Previously, `postTest` only performed state cleanup and did not check invariants. Moving invariant checks into `postTest` introduced the question of how invariant failures should be propagated: allowing `postTest` to throw, or absorbing errors and reporting them via `reportTestException`. We explicitly define the contract of `postTest` to **not throw exceptions**, reporting any invariant failures via `reportTestException`: * **Rationale & Trade-offs**: * *Alternative considered (throwing)*: If `super.postTest()` threw on an invariant failure, `package:test` would still capture the failure. However, subclasses (`AutomatedTestWidgetsFlutterBinding`, `LiveTestWidgetsFlutterBinding`, or custom bindings) override `postTest` to reset their own state (e.g. nulling `_currentFakeAsync` / `_clock`, or setting `_inTest = false`). Allowing `super.postTest()` to throw would require every subclass to defensively wrap `super.postTest()` in `try ... finally`. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading `!inTest` failures in subsequent tests. * *Chosen approach (non-throwing)*: Absorbing errors in `postTest` and forwarding them to `reportTestException` guarantees that all cleanup in base and subclass implementations executes sequentially without defensive `try ... finally` boilerplate, while still cleanly failing the test. **Why this does not break existing behavior:** In the previous implementation, when invariant assertions failed inside `_runTestBody`, they were caught by `_testZone.handleUncaughtError` and forwarded to `reportTestException` (which delegates to `package:test.registerException`). They never escaped `runTest` as unhandled thrown exceptions. Catching invariant failures in `postTest` and routing them to `reportTestException` preserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution. #### 3. Preserving Existing Behavior on Failed Tests (`_shouldVerifyInvariants`) Historically, `_runTestBody` only verified invariants if the test body did not fail: ```dart if (_pendingExceptionDetails == null) { // We only try to clean up and verify invariants if we didn't already // fail. If we got an exception already, then we instead leave everything // alone so that we don't cause more spurious errors. ... _verifyInvariants(); } ``` When invariant checks are moved to `postTest`, `_pendingExceptionDetails` is no longer available because it has already been reported and cleared by `testCompletionHandler` when `runTest` completed. To preserve this exact existing behavior across the lifecycle boundary, `_shouldVerifyInvariants` is introduced. It is set to `true` at the end of `_runTestBody` only when `_pendingExceptionDetails == null`. If a test fails in its body, `_shouldVerifyInvariants` remains `false`, and `postTest` skips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure. #### 4. Documentation Updates Updated the doc comments on: * `TestWidgetsFlutterBinding.postTest`: Documents the non-throwing contract, the `@mustCallSuper` requirement, and error reporting via `reportTestException`. * `TestWidgetsFlutterBinding.runTest`: Clarifies the distinction between `invariantTester` (which runs before `runTest` completes) and framework invariant checks (which run in `postTest` after `runTest` and user teardowns complete). ### Tests - Added unit tests in `packages/flutter_test/test/bindings_invariants_test.dart` verifying that foundation variables (`debugDefaultTargetPlatformOverride`, `debugDoublePrecision`, `debugBrightnessOverride`), `autoUpdateGoldenFiles`, `ErrorWidget.builder`, and `shouldPropagateDevicePointerEvents` can all be reset using `addTearDown`. - Added test cases verifying invariant failures in `postTest` and ensuring invariant checks are skipped when the test body throws. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Fixes flutter#110488. ### Background: The `testWidgets` Lifecycle Understanding this change requires distinguishing between two layers of `Future`s and two distinct verification steps in `testWidgets`: #### 1. Two Layers of `Future`s * **`testBody()` Future**: The closure containing the user's test logic. * **`runTest()` Future**: The overall async harness returned by `binding.runTest()`. It manages the test zone, unmounts the widget tree, and resolves only after `testBody()` and test-specific verifications finish. #### 2. Two Distinct Verification Phases * **Test-specific verifications (`invariantTester`)**: Passed as a callback to `runTest` and executed synchronously inside `_runTestBody` immediately after the `testBody()` Future completes and the widget tree is unmounted. In `testWidgets`, this runs `tester._endOfTestVerifications` to verify that tickers and semantics handles were disposed. * **Framework-level invariant checks**: Verifies global framework state, including ensuring foundation/rendering/widgets debug flags were reset and no non-periodic timers or animations remain. #### 3. The Teardown Lifecycle & Root Cause of flutter#110488 When `testWidgets` runs: 1. `testWidgets` registers `test_package.addTearDown(binding.postTest)` with `package:test`. 2. `testWidgets` calls and returns `binding.runTest(...)`. 3. Inside `testBody()`, users may register cleanups using `addTearDown` (for example, `addTearDown(() => debugDefaultTargetPlatformOverride = null)`). 4. `package:test` executes teardowns in **LIFO (reverse registration) order** only *after* the Future returned by `runTest` completes. Previously, framework-level invariant checks ran at the end of `_runTestBody` *before* the `runTest` Future completed. As a result, they ran before `package:test` executed any user-registered `addTearDown` callbacks, causing tests that clean up debug flags via `addTearDown` to fail invariant verification (flutter#110488). The following diagram shows the code flow after this PR: ``` testWidgets(description, (tester) async { ... }) │ ├── 1. Registration Phase │ └── test_package.addTearDown(binding.postTest) [Teardown Stack: #1] │ ├── 2. Test Execution Phase: binding.runTest(...) │ │ │ │ [ Future 1: testBody() ] │ ├──► Run testBody(tester) │ │ └── User calls addTearDown(...) [Teardown Stack: #2] │ │ │ │ [ Post-testBody cleanup (inside runTest) ] │ ├──► Unmount widget tree (runApp) │ ├──► Verification 1: invariantTester() (tickers, semantics) │ └──► Set _shouldVerifyInvariants = true │ │ [ Future 2: runTest() completes ] │ └── 3. Teardown Phase: package:test executes in LIFO order │ ├──► [Step 1] User Teardowns (Stack #2) │ └── e.g., debugDefaultTargetPlatformOverride = null │ └──► [Step 2] binding.postTest() (Stack #1) ├── Verification 2: Framework Invariants (debug vars, timers) ├── Reset Binding State (_currentFakeAsync, etc.) └── reportTestException(...) (if Verification 2 failed) ``` ### Changes & Design Rationale #### 1. Defer Framework Invariant Checks to `postTest` Framework invariant checks (`_verifyInvariants()`, `_verifyAutoUpdateGoldensUnset`, etc.) are moved from `_runTestBody` into `TestWidgetsFlutterBinding.postTest`. Because `testWidgets` registers `binding.postTest` before running the test body, `package:test`'s LIFO teardown ordering guarantees that all user-registered `addTearDown` callbacks execute before `binding.postTest`. #### 2. Clarifying the `postTest` Error Contract Previously, `postTest` only performed state cleanup and did not check invariants. Moving invariant checks into `postTest` introduced the question of how invariant failures should be propagated: allowing `postTest` to throw, or absorbing errors and reporting them via `reportTestException`. We explicitly define the contract of `postTest` to **not throw exceptions**, reporting any invariant failures via `reportTestException`: * **Rationale & Trade-offs**: * *Alternative considered (throwing)*: If `super.postTest()` threw on an invariant failure, `package:test` would still capture the failure. However, subclasses (`AutomatedTestWidgetsFlutterBinding`, `LiveTestWidgetsFlutterBinding`, or custom bindings) override `postTest` to reset their own state (e.g. nulling `_currentFakeAsync` / `_clock`, or setting `_inTest = false`). Allowing `super.postTest()` to throw would require every subclass to defensively wrap `super.postTest()` in `try ... finally`. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading `!inTest` failures in subsequent tests. * *Chosen approach (non-throwing)*: Absorbing errors in `postTest` and forwarding them to `reportTestException` guarantees that all cleanup in base and subclass implementations executes sequentially without defensive `try ... finally` boilerplate, while still cleanly failing the test. **Why this does not break existing behavior:** In the previous implementation, when invariant assertions failed inside `_runTestBody`, they were caught by `_testZone.handleUncaughtError` and forwarded to `reportTestException` (which delegates to `package:test.registerException`). They never escaped `runTest` as unhandled thrown exceptions. Catching invariant failures in `postTest` and routing them to `reportTestException` preserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution. #### 3. Preserving Existing Behavior on Failed Tests (`_shouldVerifyInvariants`) Historically, `_runTestBody` only verified invariants if the test body did not fail: ```dart if (_pendingExceptionDetails == null) { // We only try to clean up and verify invariants if we didn't already // fail. If we got an exception already, then we instead leave everything // alone so that we don't cause more spurious errors. ... _verifyInvariants(); } ``` When invariant checks are moved to `postTest`, `_pendingExceptionDetails` is no longer available because it has already been reported and cleared by `testCompletionHandler` when `runTest` completed. To preserve this exact existing behavior across the lifecycle boundary, `_shouldVerifyInvariants` is introduced. It is set to `true` at the end of `_runTestBody` only when `_pendingExceptionDetails == null`. If a test fails in its body, `_shouldVerifyInvariants` remains `false`, and `postTest` skips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure. #### 4. Documentation Updates Updated the doc comments on: * `TestWidgetsFlutterBinding.postTest`: Documents the non-throwing contract, the `@mustCallSuper` requirement, and error reporting via `reportTestException`. * `TestWidgetsFlutterBinding.runTest`: Clarifies the distinction between `invariantTester` (which runs before `runTest` completes) and framework invariant checks (which run in `postTest` after `runTest` and user teardowns complete). ### Tests - Added unit tests in `packages/flutter_test/test/bindings_invariants_test.dart` verifying that foundation variables (`debugDefaultTargetPlatformOverride`, `debugDoublePrecision`, `debugBrightnessOverride`), `autoUpdateGoldenFiles`, `ErrorWidget.builder`, and `shouldPropagateDevicePointerEvents` can all be reset using `addTearDown`. - Added test cases verifying invariant failures in `postTest` and ensuring invariant checks are skipped when the test body throws. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
) Reverts: [Allow resetting test invariants in `addTearDown`](flutter#192082) Initiated by: @b-luk Reason for reverting: Breaks tree with tool_integration_tests failures. Example: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_2/1329/overview Original PR Author: @dkwingsmt Reviewed By: @justinmc The original PR description is provided below: Fixes flutter#110488. ### Background: The `testWidgets` Lifecycle Understanding this change requires distinguishing between two layers of `Future`s and two distinct verification steps in `testWidgets`: #### 1. Two Layers of `Future`s * **`testBody()` Future**: The closure containing the user's test logic. * **`runTest()` Future**: The overall async harness returned by `binding.runTest()`. It manages the test zone, unmounts the widget tree, and resolves only after `testBody()` and test-specific verifications finish. #### 2. Two Distinct Verification Phases * **Test-specific verifications (`invariantTester`)**: Passed as a callback to `runTest` and executed synchronously inside `_runTestBody` immediately after the `testBody()` Future completes and the widget tree is unmounted. In `testWidgets`, this runs `tester._endOfTestVerifications` to verify that tickers and semantics handles were disposed. * **Framework-level invariant checks**: Verifies global framework state, including ensuring foundation/rendering/widgets debug flags were reset and no non-periodic timers or animations remain. #### 3. The Teardown Lifecycle & Root Cause of flutter#110488 When `testWidgets` runs: 1. `testWidgets` registers `test_package.addTearDown(binding.postTest)` with `package:test`. 2. `testWidgets` calls and returns `binding.runTest(...)`. 3. Inside `testBody()`, users may register cleanups using `addTearDown` (for example, `addTearDown(() => debugDefaultTargetPlatformOverride = null)`). 4. `package:test` executes teardowns in **LIFO (reverse registration) order** only *after* the Future returned by `runTest` completes. Previously, framework-level invariant checks ran at the end of `_runTestBody` *before* the `runTest` Future completed. As a result, they ran before `package:test` executed any user-registered `addTearDown` callbacks, causing tests that clean up debug flags via `addTearDown` to fail invariant verification (flutter#110488). The following diagram shows the code flow after this PR: ``` testWidgets(description, (tester) async { ... }) │ ├── 1. Registration Phase │ └── test_package.addTearDown(binding.postTest) [Teardown Stack: #1] │ ├── 2. Test Execution Phase: binding.runTest(...) │ │ │ │ [ Future 1: testBody() ] │ ├──► Run testBody(tester) │ │ └── User calls addTearDown(...) [Teardown Stack: #2] │ │ │ │ [ Post-testBody cleanup (inside runTest) ] │ ├──► Unmount widget tree (runApp) │ ├──► Verification 1: invariantTester() (tickers, semantics) │ └──► Set _shouldVerifyInvariants = true │ │ [ Future 2: runTest() completes ] │ └── 3. Teardown Phase: package:test executes in LIFO order │ ├──► [Step 1] User Teardowns (Stack #2) │ └── e.g., debugDefaultTargetPlatformOverride = null │ └──► [Step 2] binding.postTest() (Stack #1) ├── Verification 2: Framework Invariants (debug vars, timers) ├── Reset Binding State (_currentFakeAsync, etc.) └── reportTestException(...) (if Verification 2 failed) ``` ### Changes & Design Rationale #### 1. Defer Framework Invariant Checks to `postTest` Framework invariant checks (`_verifyInvariants()`, `_verifyAutoUpdateGoldensUnset`, etc.) are moved from `_runTestBody` into `TestWidgetsFlutterBinding.postTest`. Because `testWidgets` registers `binding.postTest` before running the test body, `package:test`'s LIFO teardown ordering guarantees that all user-registered `addTearDown` callbacks execute before `binding.postTest`. #### 2. Clarifying the `postTest` Error Contract Previously, `postTest` only performed state cleanup and did not check invariants. Moving invariant checks into `postTest` introduced the question of how invariant failures should be propagated: allowing `postTest` to throw, or absorbing errors and reporting them via `reportTestException`. We explicitly define the contract of `postTest` to **not throw exceptions**, reporting any invariant failures via `reportTestException`: * **Rationale & Trade-offs**: * *Alternative considered (throwing)*: If `super.postTest()` threw on an invariant failure, `package:test` would still capture the failure. However, subclasses (`AutomatedTestWidgetsFlutterBinding`, `LiveTestWidgetsFlutterBinding`, or custom bindings) override `postTest` to reset their own state (e.g. nulling `_currentFakeAsync` / `_clock`, or setting `_inTest = false`). Allowing `super.postTest()` to throw would require every subclass to defensively wrap `super.postTest()` in `try ... finally`. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading `!inTest` failures in subsequent tests. * *Chosen approach (non-throwing)*: Absorbing errors in `postTest` and forwarding them to `reportTestException` guarantees that all cleanup in base and subclass implementations executes sequentially without defensive `try ... finally` boilerplate, while still cleanly failing the test. **Why this does not break existing behavior:** In the previous implementation, when invariant assertions failed inside `_runTestBody`, they were caught by `_testZone.handleUncaughtError` and forwarded to `reportTestException` (which delegates to `package:test.registerException`). They never escaped `runTest` as unhandled thrown exceptions. Catching invariant failures in `postTest` and routing them to `reportTestException` preserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution. #### 3. Preserving Existing Behavior on Failed Tests (`_shouldVerifyInvariants`) Historically, `_runTestBody` only verified invariants if the test body did not fail: ```dart if (_pendingExceptionDetails == null) { // We only try to clean up and verify invariants if we didn't already // fail. If we got an exception already, then we instead leave everything // alone so that we don't cause more spurious errors. ... _verifyInvariants(); } ``` When invariant checks are moved to `postTest`, `_pendingExceptionDetails` is no longer available because it has already been reported and cleared by `testCompletionHandler` when `runTest` completed. To preserve this exact existing behavior across the lifecycle boundary, `_shouldVerifyInvariants` is introduced. It is set to `true` at the end of `_runTestBody` only when `_pendingExceptionDetails == null`. If a test fails in its body, `_shouldVerifyInvariants` remains `false`, and `postTest` skips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure. #### 4. Documentation Updates Updated the doc comments on: * `TestWidgetsFlutterBinding.postTest`: Documents the non-throwing contract, the `@mustCallSuper` requirement, and error reporting via `reportTestException`. * `TestWidgetsFlutterBinding.runTest`: Clarifies the distinction between `invariantTester` (which runs before `runTest` completes) and framework invariant checks (which run in `postTest` after `runTest` and user teardowns complete). ### Tests - Added unit tests in `packages/flutter_test/test/bindings_invariants_test.dart` verifying that foundation variables (`debugDefaultTargetPlatformOverride`, `debugDoublePrecision`, `debugBrightnessOverride`), `autoUpdateGoldenFiles`, `ErrorWidget.builder`, and `shouldPropagateDevicePointerEvents` can all be reset using `addTearDown`. - Added test cases verifying invariant failures in `postTest` and ensuring invariant checks are skipped when the test body throws. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
…#13060) Manual roll Flutter from 4fcd90be0045 to c5a061b18fa2 (222 revisions) Manual roll requested by quncheng@google.com flutter/flutter@4fcd90b...c5a061b 2026-09-28 brackenavaron@gmail.com [cross imports] navigator_test.dart (flutter/flutter#190954) 2026-09-28 154381524+flutteractionsbot@users.noreply.github.com Revert: Reland: Only render views that need to be rendered (flutter/flutter#193360) 2026-09-28 codefu@google.com ci(bringup): android_intent_security_test is green (flutter/flutter#193469) 2026-09-28 154381524+flutteractionsbot@users.noreply.github.com Sync CHANGELOG.md from stable (flutter/flutter#193020) 2026-09-28 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from ukukV5lEkKabtOATk... to QdgqP02_cYpRQQQNN... (flutter/flutter#193454) 2026-09-28 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#193460) 2026-09-28 victorsanniay@gmail.com Un-nest sceneBuildDuration and windowRenderDuration in web SceneBuilderRecorder (flutter/flutter#193260) 2026-09-28 zarah@google.com Fix unresolved doc comment references in dev/integration_tests and dev/benchmarks (flutter/flutter#193433) 2026-09-28 engine-flutter-autoroll@skia.org Roll Packages from e55e7ac to ba0364a (9 revisions) (flutter/flutter#193450) 2026-09-28 bkonyi@google.com [tool] Require explicit dependency injection for FlutterDevice and FlutterDevice.create (flutter/flutter#192830) 2026-09-28 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#193442) 2026-09-28 dacoharkes@google.com [flutter_tools] Include data assets from hooks when pubspec.yaml is empty (flutter/flutter#193434) 2026-09-28 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#193438) 2026-09-28 zarah@google.com Fix unresolved doc comment references in Material and Cupertino (flutter/flutter#193283) 2026-09-28 zarah@google.com Fix doc references to Material and Cupertino in the widgets library and its tests (flutter/flutter#193334) 2026-09-26 kevmoo@users.noreply.github.com wasm: enforce WasmGC opt-in capability checks and Firefox < 147 guard (flutter/flutter#193180) 2026-09-26 43054281+camsim99@users.noreply.github.com [Android] Refuse external setters of engine entrypoint and cached engine arguments via `Intent`s (flutter/flutter#190249) 2026-09-26 1961493+harryterkelsen@users.noreply.github.com [web] Unskip TextPainter.getWordBoundary test (flutter/flutter#193378) 2026-09-26 bkonyi@google.com Specify non-obvious types in pattern variable declarations (flutter/flutter#192632) 2026-09-26 30870216+gaaclarke@users.noreply.github.com Removes feedback loop from advanced filters without offscreen msaa and framebufferfetch (flutter/flutter#193306) 2026-09-26 1961493+harryterkelsen@users.noreply.github.com [web] Unskip 8 passing image tests in painting and rendering (flutter/flutter#193377) 2026-09-26 154381524+flutteractionsbot@users.noreply.github.com Revert: Allow resetting test invariants in `addTearDown` (flutter/flutter#193383) 2026-09-25 bkonyi@google.com [tool] Migrate CoverageCollector to modular dependency injection (flutter/flutter#192924) 2026-09-25 zarah@google.com Fix more unresolved doc comment references (flutter/flutter#193336) 2026-09-25 dkwingsmt@users.noreply.github.com Allow resetting test invariants in `addTearDown` (flutter/flutter#192082) 2026-09-25 jhy03261997@gmail.com [a11y] Add a semantics role for slider (flutter/flutter#193324) 2026-09-25 stuartmorgan@google.com Fix plugin tests after Pigeon plugin template changes (flutter/flutter#193302) 2026-09-25 jesswon@google.com [Android 17] Bump Standard Test Apps in flutter/flutter to AGP 9.3.1 (flutter/flutter#193263) 2026-09-25 97480502+b-luk@users.noreply.github.com SSBO-based gradients in UberSDF (flutter/flutter#192962) 2026-09-25 robert.ancell@canonical.com [Linux] Handle FlView being destroyed before rendering is complete. (flutter/flutter#193268) 2026-09-25 bkonyi@google.com [flutter_tools] Handle Windows reserved characters in test target path (flutter/flutter#191900) 2026-09-25 sneurlax@gmail.com docs(tools): nit: say hook/build.dart in CMake native assets comment (flutter/flutter#192205) 2026-09-25 kevmoo@users.noreply.github.com [web] Enable Firefox Skwasm UI CI suites, configure COI configs, and fail fast on loader rejections (flutter/flutter#193187) 2026-09-25 jesswon@google.com [Android 17] Bumped Engine Dependencies to 9.3.1 (flutter/flutter#193265) 2026-09-25 kevmoo@users.noreply.github.com [web] Omit group role on menu scrollables and assign region role to named routes (flutter/flutter#192965) 2026-09-25 engine-flutter-autoroll@skia.org Roll Skia from 8eedeed98e79 to f441ca223b2b (5 revisions) (flutter/flutter#193353) 2026-09-25 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from EbPpoJW-Lnsu-8dyZ... to ukukV5lEkKabtOATk... (flutter/flutter#193326) 2026-09-25 engine-flutter-autoroll@skia.org Roll Packages from 431ea69 to e55e7ac (6 revisions) (flutter/flutter#193351) 2026-09-25 139053348+shikharish@users.noreply.github.com Support DynamicLibrary.codeAsset in Flutter (flutter/flutter#188947) 2026-09-25 engine-flutter-autoroll@skia.org Roll Dart SDK from 75d9e87d4e3d to 0e7642b85457 (1 revision) (flutter/flutter#193346) 2026-09-25 15619084+vashworth@users.noreply.github.com Exclude iOS PRs from macOS link (flutter/flutter#193315) 2026-09-25 engine-flutter-autoroll@skia.org Roll Skia from 6357608543ec to 8eedeed98e79 (4 revisions) (flutter/flutter#193341) 2026-09-25 engine-flutter-autoroll@skia.org Roll Dart SDK from d3aedb186aab to 75d9e87d4e3d (3 revisions) (flutter/flutter#193335) 2026-09-25 nickolasdeluca@live.com Cache the paint offset adjusted line metrics in TextPainter (flutter/flutter#191223) ...
) This PR relands flutter#192082 after fixing the error in integration tests as described in flutter#193383. I've manually verified that the integration test fails on flutter#192082 and passes now. ### Why it failed This integration test runs `dev/automated_tests/flutter_test/ticker_test.dart`, which leaves a `Ticker` running. It expects that test to fail with an "An animation is still running even after the widget tree was disposed." error. After flutter#192082, `ticker_test.dart` passed instead. The leaked ticker is caught by `SchedulerBinding.debugAssertNoTransientCallbacks`. Unlike most invariant checks, it doesn't throw; it reports the failure through `FlutterError.reportError`. During a test, `FlutterError.onError` stores reported errors in `_pendingExceptionDetails`, and the test completion handler reports them when `runTest` finishes. Changing the behavior of this method would be a breaking change to the public API in the framework, so instead the test framework should be changed to pick up this error. Before flutter#192082, the check ran inside `_runTestBody`, so the completion handler picked up the error. flutter#192082 moved the check to `postTest`, which runs after the completion handler. The `try/catch` in `postTest` only catches thrown errors, and `postTest` then clears `_pendingExceptionDetails`, so the error was silently dropped and the test passed. #### What this reland changes See the [2nd commit](flutter@c617498) for the list of changes. - After running the invariant checks, `postTest` also picks up `_pendingExceptionDetails` and reports it. If a check also threw, the thrown error is reported and the other one is printed to the console so it isn't lost. - `postTest` now passes the test description to `reportTestException` instead of `''`, so the output includes `The test description was: ...` again. The integration test's expected output relies on that line. - The four `_verify*Unset` helpers go back to `FlutterError.reportError`, which is what they used before flutter#192082. Now that `postTest` handles reported errors, they don't need to throw, and they behave the same as before. flutter#192082 made this change with the hope to simplify and unify the code flow, but now that we have to pick up `_pendingExceptionDetails` anyway, there's no reason to make this change. #### Tests - Added a test to `bindings_invariants_test.dart`. It leaks a `Ticker` and checks that `postTest` reports the error along with the test description. - `test_test.dart: flutter test should report a nice error when a Ticker is left running` passes again. ### What the 3rd commit changes The [3rd commit](flutter@6866134) refactors the `bindings_invariants_test.dart` to ensure that the `postTest` is run (and invariants are reset) even when errors occur during the test body. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant in-code documentation (doc comments with `///`). - [ ] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Fixes #110488.
Background: The
testWidgetsLifecycleUnderstanding this change requires distinguishing between two layers of
Futures and two distinct verification steps intestWidgets:1. Two Layers of
FuturestestBody()Future: The closure containing the user's test logic.runTest()Future: The overall async harness returned bybinding.runTest(). It manages the test zone, unmounts the widget tree, and resolves only aftertestBody()and test-specific verifications finish.2. Two Distinct Verification Phases
invariantTester): Passed as a callback torunTestand executed synchronously inside_runTestBodyimmediately after thetestBody()Future completes and the widget tree is unmounted. IntestWidgets, this runstester._endOfTestVerificationsto verify that tickers and semantics handles were disposed.3. The Teardown Lifecycle & Root Cause of #110488
When
testWidgetsruns:testWidgetsregisterstest_package.addTearDown(binding.postTest)withpackage:test.testWidgetscalls and returnsbinding.runTest(...).testBody(), users may register cleanups usingaddTearDown(for example,addTearDown(() => debugDefaultTargetPlatformOverride = null)).package:testexecutes teardowns in LIFO (reverse registration) order only after the Future returned byrunTestcompletes.Previously, framework-level invariant checks ran at the end of
_runTestBodybefore therunTestFuture completed. As a result, they ran beforepackage:testexecuted any user-registeredaddTearDowncallbacks, causing tests that clean up debug flags viaaddTearDownto fail invariant verification (#110488).The following diagram shows the code flow after this PR:
Changes & Design Rationale
1. Defer Framework Invariant Checks to
postTestFramework invariant checks (
_verifyInvariants(),_verifyAutoUpdateGoldensUnset, etc.) are moved from_runTestBodyintoTestWidgetsFlutterBinding.postTest.Because
testWidgetsregistersbinding.postTestbefore running the test body,package:test's LIFO teardown ordering guarantees that all user-registeredaddTearDowncallbacks execute beforebinding.postTest.2. Clarifying the
postTestError ContractPreviously,
postTestonly performed state cleanup and did not check invariants. Moving invariant checks intopostTestintroduced the question of how invariant failures should be propagated: allowingpostTestto throw, or absorbing errors and reporting them viareportTestException.We explicitly define the contract of
postTestto not throw exceptions, reporting any invariant failures viareportTestException:super.postTest()threw on an invariant failure,package:testwould still capture the failure. However, subclasses (AutomatedTestWidgetsFlutterBinding,LiveTestWidgetsFlutterBinding, or custom bindings) overridepostTestto reset their own state (e.g. nulling_currentFakeAsync/_clock, or setting_inTest = false). Allowingsuper.postTest()to throw would require every subclass to defensively wrapsuper.postTest()intry ... finally. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading!inTestfailures in subsequent tests.postTestand forwarding them toreportTestExceptionguarantees that all cleanup in base and subclass implementations executes sequentially without defensivetry ... finallyboilerplate, while still cleanly failing the test.Why this does not break existing behavior:
In the previous implementation, when invariant assertions failed inside
_runTestBody, they were caught by_testZone.handleUncaughtErrorand forwarded toreportTestException(which delegates topackage:test.registerException). They never escapedrunTestas unhandled thrown exceptions. Catching invariant failures inpostTestand routing them toreportTestExceptionpreserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution.3. Preserving Existing Behavior on Failed Tests (
_shouldVerifyInvariants)Historically,
_runTestBodyonly verified invariants if the test body did not fail:When invariant checks are moved to
postTest,_pendingExceptionDetailsis no longer available because it has already been reported and cleared bytestCompletionHandlerwhenrunTestcompleted.To preserve this exact existing behavior across the lifecycle boundary,
_shouldVerifyInvariantsis introduced. It is set totrueat the end of_runTestBodyonly when_pendingExceptionDetails == null. If a test fails in its body,_shouldVerifyInvariantsremainsfalse, andpostTestskips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure.4. Documentation Updates
Updated the doc comments on:
TestWidgetsFlutterBinding.postTest: Documents the non-throwing contract, the@mustCallSuperrequirement, and error reporting viareportTestException.TestWidgetsFlutterBinding.runTest: Clarifies the distinction betweeninvariantTester(which runs beforerunTestcompletes) and framework invariant checks (which run inpostTestafterrunTestand user teardowns complete).Tests
packages/flutter_test/test/bindings_invariants_test.dartverifying that foundation variables (debugDefaultTargetPlatformOverride,debugDoublePrecision,debugBrightnessOverride),autoUpdateGoldenFiles,ErrorWidget.builder, andshouldPropagateDevicePointerEventscan all be reset usingaddTearDown.postTestand ensuring invariant checks are skipped when the test body throws.Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.