Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support NN models with multiple inputs #161

Open
sashuIya opened this issue Feb 27, 2021 · 2 comments · May be fixed by #163
Open

Support NN models with multiple inputs #161

sashuIya opened this issue Feb 27, 2021 · 2 comments · May be fixed by #163

Comments

@sashuIya
Copy link
Contributor

Additionally to the sequence, we'd like to provide some other input (of some different size) to the model. A simple basic example to illustrate:

class SimpleConv(nn.Module):
    def __init__(self):
        self.conv_net = nn.Sequential(
            nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
        )

        self.fc_net = nn.Sequential(
            nn.Linear(channels_in, channels_out),
            nn.ReLU(inplace=True),
        )
 

    def forward(self, x: List[np.ndarray]):
        y1 = self.conv_net(x[0])
        y2 = self.fc_net(x[1])
        y = torch.cat((y1, y2), 1)
        return y

Do you think we could modify the _get_batch() function to return a tuple(List[np.ndarray], np.ndarray)?
https://github.com/FunctionLab/selene/blob/master/selene_sdk/train_model.py#L346-L355

Maybe we could wrap the
https://github.com/FunctionLab/selene/blob/master/selene_sdk/train_model.py#L453-L464
into some function, which will return either a single Tensor or a List[Tensor] for the inputs, based on the provided inputs type? Or would be there a better design solution?
@jzthree
Copy link
Contributor

jzthree commented Mar 1, 2021

Thank you for bringing up the need for multiple inputs! As discussed with @kathyxchen we are planning to support multiple inputs as well as multiple targets. Currently, we have a couple of major updates underway(adding support for parallelized data loading, and improving the support for custom targets) that will involve changes to Sampler and Target, so we plan to introduce it after these updates.

Our current plan for the API change is to support multiple types of output from sampler.sample():

numpy.ndarray, numpy.ndarray (single input, single target)
tuple(numpy.ndarray), numpy.ndarray  (multiple input, single target)
numpy.ndarray, tuple(numpy.ndarray)  (single input, multiple targets)
tuple(array), tuple(numpy.ndarray) (multiple input, multiple targets)

where both input and targets can be either an array or a tuple of arrays. TrainModel will handle the transformation of numpy array to pytorch tensor, and then it's up to the user to make sure Model and Criterion handles the tuple input correctly.

@sashuIya
Copy link
Contributor Author

sashuIya commented Mar 1, 2021

Hi @jzthree. I've been working on a PR for this today, let me send it to you for a review tomorrow.

@arlapin arlapin linked a pull request Mar 2, 2021 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support multi-inputs arlapin/selene
2 participants
@sashuIya @jzthree