Repository navigation
Feature request: test runner reporters #45648
Description
Activity
cc: @manekinekko since you expressed interest in this. In the TAP parser PR, you initially had some colorful output that I asked you to remove. I think reporters would be the time and place to introduce that 😄
Reacted by Wassim CheghamThank you @cjihrig for putting this together so quickly. Gonna draft a rough design for the reporter API. Feel free to assign this work to me.
cc: @manekinekko since you expressed interest in this. In the TAP parser PR, you initially had some colorful output that I asked you to remove. I think reporters would be the time and place to introduce that 😄
Of course, I will 😂
+1 for me.
A few thoughts regarding the implementation:
- we already fire some of the data through
tapStream- we can either use that class to fire all the additional data needed for reporters to work, or use a new/different API - in that case, we might want to remove the events fromtapStreamfor the sake of simplicity - do we want to support using reporters when using
node:testwithout the--testflag? seems like a valid use case, and we already stream TAP tostdout. - another interesting use-case I know of is having multiple reporters streamed to different locations, for example, a
specreporter streamed tostdoutand ajUnitreporter to a file. even if we do not support that directly - we should think about what API to expose that will allow that
Reacted by Wassim Chegham- we already fire some of the data through
cc @nodejs/test_runner
do we want to support using reporters when using node:test without the --test flag?
Yes, we should. Since both ways produce TAP, it should be feasible.
another interesting use-case I know of is having multiple reporters streamed to different locations
Yes, we should support this too. We can make a new
--test-reporterCLI flag that can be specified multiple times.Reacted by Wassim CheghamYes, we should support this too. We can make a new --test-reporter CLI flag that can be specified multiple times.
but how can we specify the target stream for each flag?
but how can we specify the target stream for each flag?
To be determined 😅. There are different approaches we could try - maybe part of the CLI flag (
--test-reporter=reporter-name,stderr), preloaded modules, etc. Maybe the simplest thing would be to only support multiple reporters via the programmatic config (run()). I'm open to ideas.Reacted by Moshe Atlowit should definitely be supported through
run.
my concern is this use case is very common, and we should also provide an even simpler way to do this.- maybe we do finally introduce a configuration file, even if it is just for test runner configuration?
- maybe each reporter can define a default destination
anyway for a first iteration supporting onlyrun()for multiple reporters sounds good enough
Here is one possible way to do it: https://github.com/hapijs/lab/blob/master/API.md#multiple-reporters (also note the Custom Reporters section immediately below it)
Reacted by Moshe Atlowbut how can we specify the target stream for each flag?
We should be flexible and support both
--test-reporter=abc --test-reporter=xyzand as @cjihrig mentioned--test-reporter=abc,xyzmaybe we do finally introduce a configuration file, even if it is just for test runner configuration?
Maintaining a configuration file can be hard sometimes. How about reusing
package.jsonand introducing a new property?- added a commit that references this issue
on Dec 19, 2022 Maintaining a configuration file can be hard sometimes. How about reusing
package.jsonand introducing a new property?Hi, just discovering this issue. Going forward, please always tag @nodejs/modules and @nodejs/loaders for any discussion around CLI flags that load files,
package.jsonstuff and config files. These are all hot topics related to module loading and tied into other designs such as the Loaders API.Reacted by Benjamin Gruenbaum1 remaining item
- added 5 commits that reference this issue
on Feb 6, 2023 - added 2 commits that reference this issue
on Mar 3, 2023
What is the problem this feature will solve?
The test runner currently only generates TAP output. Users would like a way to author and use reporters in a format other than TAP.
What is the feature you are proposing to solve the problem?
Now that the TAP parser has landed, it would be nice to define some type of reporter API built on top of the parser output. Node should probably ship one reporter that is easier on the eyes than TAP, and can serve as an example of how to create a reporter.
I think at a minimum we would want an API that:
'test'events. This would include the test name, result, and meta information such as the test duration, comments/user logs, and error information if the test fails.test()anddescribe()/it()style tests.test()s, or may want to count nested tests in different ways.What alternatives have you considered?
There are some existing reporters on npm already. However, I think they have a few drawbacks:
tap-arcwas looking for specific fields in the TAP output. Neither Node nortap-arcwere wrong here andtap-archas an open issue to work better with Node's test runner. It would be nice to build something that is guaranteed to work with Node's runner.node --test --test-reporter=my-reporterand have it just work.