Sitelet https://web.archive.org/web/20220318033738/https://github.com/ploomber/ploomber/issues/647
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to customise names of grid tasks. #647

Open
vgn1 opened this issue Mar 10, 2022 · 3 comments
Open

Add option to customise names of grid tasks. #647

vgn1 opened this issue Mar 10, 2022 · 3 comments

Comments

@vgn1
Copy link

@vgn1 vgn1 commented Mar 10, 2022

When running tasks in a grid, a number is appended at the end of the tasks product names. For example, the following pipeline.yaml file:

- source: example_task.py
   name: example-task
   product:
       data: output/output_dataframe.csv
   grid:
       input_dataframe: ['birds.csv', 'fish.csv', 'flowers.csv']

would result in 3 products: output_dataframe-1.csv, output_dataframe-2.csv and output_dataframe-3.csv.

I would like to have an option to replace -1, -2 and -3 in the filenames with for example -birds, -fish and -flowers.

@edublancas
Copy link
Contributor

@edublancas edublancas commented Mar 10, 2022 •

Thanks for your feedback!

Any suggestions on how this could look like? I'm thinking of having a product_prefix like this:

- source: example_task.py
   name: example-task
   product:
       data: output/output_dataframe.csv
   grid:
       input_dataframe: ['birds.csv', 'fish.csv', 'flowers.csv']
   product_suffix: '_{{input_dataframe}}'

Then, the outputs would be 'output/output_dataframe_birds.csv', etc.

@vgn1
Copy link
Author

@vgn1 vgn1 commented Mar 10, 2022 •

Yes! Something like that would be great.

I forgot this in the original issue description, but I think the suffix also should be (optionally) added to the task name. So that the name can be the same as the products. So maybe task_suffix or task_name_suffix would be better. And I think the suffix field should be able to take a separate list like:
task_name_suffix: ['birds', 'fish', 'flowers']

Edit: Perhaps a task_name_prefix field could be added as well.

@hornste
Copy link

@hornste hornste commented Mar 11, 2022

I like the product_suffix as well. Would be nice to apply to task_suffix as well.

That way if you had something like:

grid:
   param1: [1,2,3,4]
   param2: [a,b,c,d]

you could do, product_suffix: '-{{param1}}-{{param2}}'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants