Paired multimodal autoencoders (with tests!) #393
Conversation
ml4h/plots.py
Outdated
| figure_path = f'{prefix}/{dtm.name}_{feature}_transform_scalar_{scalar}.png' | ||
| if not os.path.exists(os.path.dirname(figure_path)): | ||
| os.makedirs(os.path.dirname(figure_path)) | ||
| plt.savefig(figure_path) |
There was a problem hiding this comment.
4got pre-commit
ml4h/plots.py
Outdated
| axes[i, 0].set_xticks(()) | ||
| axes[i, 0].set_yticks(()) | ||
| axes[i, 1].set_xticks(()) | ||
| axes[i, 1].set_yticks(()) |
There was a problem hiding this comment.
Is this equivalent to axes[i, 0].axis('off')?
ml4h/models.py
Outdated
|
|
||
| multimodal_activation = Concatenate()(multimodal_activations) | ||
| multimodal_activation = Dense(units=kwargs['dense_layers'][0])(multimodal_activation) | ||
| #multimodal_activation = _activation_layer(kwargs['activation'])(multimodal_activation) |
ml4h/models.py
Outdated
| for left, right in pairs: | ||
| kwargs['tensor_maps_in'] = [left] | ||
| left_model = make_multimodal_multitask_model(**kwargs) | ||
| encode_left = make_hidden_layer_model(left_model, [left], kwargs['hidden_layer']) | ||
| h_left = encode_left(inputs[left]) | ||
|
|
||
| kwargs['tensor_maps_in'] = [right] | ||
| right_model = make_multimodal_multitask_model(**kwargs) | ||
| encode_right = make_hidden_layer_model(right_model, [right], kwargs['hidden_layer']) | ||
| h_right = encode_right(inputs[right]) | ||
|
|
||
| if pair_loss == 'cosine': | ||
| loss_layer = CosineLossLayer(1.0) | ||
| elif pair_loss == 'euclid': | ||
| loss_layer = L2LossLayer(1.0) | ||
|
|
||
| paired_embeddings = loss_layer([h_left, h_right]) | ||
| multimodal_activations.extend(paired_embeddings) |
There was a problem hiding this comment.
Could the logic of this function be written as a new bottleneck type?
Maybe the problem is that the model factory doesn't return the encoders and decoders?
There was a problem hiding this comment.
right, I think it would be possible, maybe cleaner. I'm seeing a weird bug when I try triplet style models so I think this code is still changing quickly
|
Lots of cleanup, now this works with multiple pairs and cosine/euclid distance as the loss. Also added tests for the semi-supervised setup. Resurrected the |
Adds recipe `train_paired` for multimodal models with losses to encourage the same embeddings of different modalities.
Adds recipe `train_paired` for multimodal models with losses to encourage the same embeddings of different modalities.
Adds recipe `train_paired` for multimodal models with losses to encourage the same embeddings of different modalities.
Adds recipe `train_paired` for multimodal models with losses to encourage the same embeddings of different modalities.
No description provided.