Update Test Exemption Process - #26
eyebrowsoffire wants to merge 3 commits into
Conversation
Piinks
left a comment
There was a problem hiding this comment.
This is awesome thank you!
|
|
||
| * Change the comment posted in the PR to accurately match the process according to the solution we choose. | ||
| * The bot should add a `missing-tests` label if a PR is missing tests, and keep the state of that label up-to-date which each push. That is to say, the label should be removed if a subsequent push adds tests, or if the change no longer touches code which requires tests. | ||
| * Whoever is responsible for granting test exemptions is responsible for adding a `test-exempt` label. This will allow the test exemption process to be more auditable, since the GitHub history will reflect who added the label. |
There was a problem hiding this comment.
Possible, but probably unlikely, any flutter-hacker will be able to apply this label.
There was a problem hiding this comment.
This is true. However, I'm personally more of a fan of gradually adding restrictions for things like this rather than making things completely air-tight out of the gate. If it turns out that every member of flutter-hackers technically having the capability of adding the test-exempt is something that people end up abusing, we can consider being more strict with this. But I'd imagine that we can hope people would act in good faith, especially because the audit log at least keeps them somewhat honest here.
There was a problem hiding this comment.
I was going to make the same argument as Kate: anyone with write access to add or remove these labels. But we'll have a paper trail. Will we have tools for auditing that paper trail? something we could run during org triage (or any other triage designated owner)?
| created: 2026-09-24T00:00:00Z | ||
| updated: 2026-09-24T00:00:00Z | ||
| tags: | ||
| - 000-meta |
There was a problem hiding this comment.
000 is for RFC Process & Templates - how to change the current RFC and design process. Maybe we just add a simple "Policy" group.
|
|
||
| ## Overview | ||
|
|
||
| If a PR needs an exemption from our testing guidelines, those exemptions are currently granted by a `@test-exemption-reviewer` role on Discord. Originally, this responsibility was managed by `@Hixie`. Upon his departure from the team, the role was taken over by `@stuartmorgan`, who is also leaving. With his departure, we are looking at reworking this system to be more sustainable and distribute this responsibility more broadly so as to avoid a single point of failure. |
There was a problem hiding this comment.
Nit: Could read as "Originally this responsibility was managed by a single engineer. This represents an unacceptable bus factor." <... we're looking at reworking...>
There was a problem hiding this comment.
My original draft actually used the term "bus factor" but I changed that because (1) it's kind of a jargony term that not everyone knows and (2) it's actually kind of morbid and it felt weird putting into a doc.
I think the short history here of who has carried the torch for this role is useful. I also don't really consider it "unacceptable" as we've lived with this single point of failure for many years, but it can definitely be improved. Everyone I've talked to about this is already open to the idea of adjusting the policy here, so I don't think we need to use needlessly aggressive language to get our point across. The doc is really just about the pros and cons of our different options moving forward.
There was a problem hiding this comment.
I don't think it matters enough to rewrite the text, but FWIW just to capture the history here: there have been between one and three (I think that was the max) engineers active in the role at various times. Hixie and I were both actively granting exceptions for a while, for example.
|
|
||
| * **000:** RFC Process & Templates | ||
| * **010:** Governance & Steering Committees | ||
| * **010:** General Team Process & Policy |
There was a problem hiding this comment.
you should update 000.00001 as well.
|
|
||
| If a PR needs an exemption from our testing guidelines, those exemptions are currently granted by a `@test-exemption-reviewer` role on Discord. Originally, this responsibility was managed by `@Hixie`. Upon his departure from the team, the role was taken over by `@stuartmorgan-g`, who is also leaving. With his departure, we are looking at reworking this system to be more sustainable and distribute this responsibility more broadly so as to avoid a single point of failure. | ||
|
|
||
| ## Possible solutions |
There was a problem hiding this comment.
Before this lands, we should express an opinion of which route we're taking.
A document for discussion on how to update our test exemption process.
Tracking issue: flutter/flutter#00000
Pre-launch Checklist
design doc.dart run bin/assign_rfc_number.dartlocally when instructed).If you need help, consider asking for advice on the #hackers channel on Discord.