Sitelet https://web.archive.org/web/20220515000523/https://github.com/huggingface/transformers/issues/17208
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Visual Question Answering (VQA) pipeline #17208

Open
NielsRogge opened this issue May 12, 2022 · 5 comments
Open

Add Visual Question Answering (VQA) pipeline #17208

NielsRogge opened this issue May 12, 2022 · 5 comments
Labels
Good First Issue

Comments

@NielsRogge
Copy link
Contributor

@NielsRogge NielsRogge commented May 12, 2022 •
edited

Feature request

We currently have ViLT in the library, which, among other tasks, is capable of performing visual question answering (VQA).

It would be great to have a pipeline for this task, with the following API:

from transformers import pipeline

pipe = pipeline("vqa")
pipe("cats.png", "how many cats are there?")

This pipeline could default to the https://huggingface.co/dandelin/vilt-b32-finetuned-vqa checkpoint. Also check out the Space that showcases the model.

This can be implemented similar to other pipelines. For an example PR that added a pipeline, see #11598.

Motivation

A pipeline is required in order to have inference widgets + a task defined at hf.co/tasks.

Moreover, it would be great to do VQA in two lines of code.

Your contribution

I can definitely assist in this, together with @Narsil, who's the pipeline expert.
@Narsil
Copy link
Contributor

@Narsil Narsil commented May 12, 2022

Tagging @mishig25 for the widget

@LysandreJik
Copy link
Member

@LysandreJik LysandreJik commented May 12, 2022

Also LXMERT should handle this task, but likely has a very different API.

@mishig25
Copy link
Contributor

@mishig25 mishig25 commented May 12, 2022

This sounds amazing. Happy to contribute in anyway I can

@sijunhe
Copy link

@sijunhe sijunhe commented May 13, 2022

I'd love to pick this up!

@sabarish-srinivasan
Copy link

@sabarish-srinivasan sabarish-srinivasan commented May 14, 2022

Hey @sijunhe, I'm just starting out in open-source, but I'd like to help out however I can!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Good First Issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@Narsil @mishig25 @sijunhe @LysandreJik @NielsRogge @sabarish-srinivasan