Sidecar JSON file is not recognized by dandi upload when the dataset_description.json file is missing

[kabilar]

Description

1. When uploading a new or edited JSON file that accompanies a Zarr file in a BIDS inspired dataset, the JSON file does not get recognized by the dandi upload command when the dataset_description.json file is missing at the top-level directory of the Dandiset. Do you know why this might be the case?
2. The dataset_description.json file is required for a BIDS dataset (see BIDS Specification), but we don't specify that this is a requirement for a Dandiset.

Thank you.

cc @dstansby @aaronkanzer

[yarikoptic]

If it isn't bids dataset, then we upload only nwb, zarr, and video files "by default"... Eg as documented at

dandi-cli/dandi/cli/cmd_upload.py

Line 90 in e975d9a

By default all .nwb, .zarr, and .ngff assets in the Dandiset (ignoring

. What's the use case here?

[kabilar]

Thanks @yarikoptic. For the LINC project, we are working towards uploading BIDS compliant datasets.

1. Sometimes, I have forgotten to add the dataset_description.json file for a new dataset/Dandiset.
2. Other times, a user wants to change the metadata in the sidecar JSON file.
   1. They don't have the entire dataset/Dandiset stored locally because the Zarr files are >1 TB.
   2. So they download the dandiset.yaml file, download the sidecar JSON file, create the dataset/Dandiset directory tree, place the JSON file in the correct location within the tree, and modify the JSON file.
   3. They then try to upload the modified JSON file but dandi upload doesn't recognize the (changed) JSON file, since they didn't know that they need to download the dataset_description.json file.

[yarikoptic]

heh, we might want to change behaviors here, but should also discuss since .json is nohow specific here, and in general such partial uploads forbid validation of the entire dandiset (unless we validate against full list/content in the archive). Meanwhile workaround is 'ad-hoc' - should do smth like

DANDI_DEVEL=1 dandi upload --allow-any-path whatever/path/to.json

since --allow-any-path is available only when DANDI_DEVEL=1

[kabilar]

Thanks for the workaround, Yarik.

[yarikoptic]

For now, as long as we do not decide that "DANDI layout accepts/expects sidecar .json files", I think there is nothing to act on here to allow for download of those. But I think we might want to explicitly inform user , hence

• upload: if any file/path is skipped due to not part of DANDI layout -- log WARNING upon completion or ask #1493