Generating natural questions about images. This project task is to generate questions which are more likely to be asked by human when shown an image or a scheme. Most of the questions are not describing the visual objects but inferring deeper concepts and events which canbe used to start a conversation in a human-machine interaction.
Download datasets.ipynb
A Jupyter Notebook that downloads images and questions from the dataset link
here
The dataset is orgenised by the source of the image: Bing, MSCOCO, or Flickr, then by type of dataset: train, val and test.
This notebook is mainly inspired by this repo
here.
After fully running this Notebook you can procceed to the next notebook VGQ-PyTorch.ipynb to train and see some results.
VQG-PyTorch.ipynb
This Jupyter Notebook can run in 2 mode: train, predict.
Feel free to change the to_train variable in the second cell to False if you no longer need to train.
This notebook has mainely three parts:
- Building the train, validation, test set by encoding the images and join them with the question.
- Training the GRNN with multiple variables.
- Predict and show the results using beam search.
[1] Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He and Lucy VanderwendeGenerating Natural Questions About an Image