Why are the instructions always in English?

[thesofakillers]

I was quite surprised to find out that tasks where both the input and output are not in English, the instructions remain in English.

Is there a specific design reason for this other than accessibility?

The instructions are technically part of the training input for most of the best performing models in Table 4 of the Tk-instruct paper, this way the input is ultimately always at least partly in English.

This makes the dataset dependent on the presence of English capabilities of the model, making it hard to evaluate performance on a given non-english language completely independently.

I think it would be of value, perhaps for v3, to provide the instructions also in the input and output languages besides English. This would make the dataset more flexible and allow experimenters more freedom.

[danyaljj]

Hi @thesofakillers 👋

It is true almost all tasks with non-English input/outputs have English instructions.
This was mainly because we didn't have native speakers that could rewrite instructions in other languages (though we're open to it. people can absolutely send pull-requests with non-English instructions).

I should also add that, when it comes to building multilingual models, less diverse inputs is often less of a concern -- multi-lingual LMs often can parse inputs of various language (e.g., a multilingual LM trained on SQuAD can reasonably generalize to similar questions in other lang). The main bottleneck is being able to generate text in various scripts which depends on outputs being in diverse languages (which we support).

FYI @yizhongw

[thesofakillers]

I recognise that this is less of an issue for multilingual models, but consider that the non-english instructions you have collected here may have been of great use for research on _mono_lingual models in other languages, or research, for instance, on cross-lingual transfer of monolingual models, which are both valid areas of research that greatly suffer from the lack of non-english datasets. This is why i was underlining a potential increase in flexibility and widespread adoption of your dataset.

But i totally understand that everyone has limits, and I appreciate the openness. Just a small suggestion to consider for a v3 :)