Skip to content

Commit

Permalink
Merge pull request #25 from mcwdsi/use-templates
Browse files Browse the repository at this point in the history 
Use templates to build DrOn
  • Loading branch information
@jamesaoverton
jamesaoverton authored May 16, 2024
2 parents d81061b + c6d9187 commit b092a5b
Show file tree
Hide file tree
Showing 22 changed files with 871,127 additions and 164 deletions.
3 changes: 2 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,9 @@ dron-base.*
dron-full.*

src/ontology/archive
src/ontology/components
src/patterns
dron-lite.json
dron-lite.obo
dron-lite.owl
/site
/site
44 changes: 14 additions & 30 deletions docs/DronReleaseWorkflow.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,50 +2,34 @@

To create a new Dron release:

1. [Update the external components](#Update-the-external-components)
2. [Import the external components](#Import-the-external-components)
3. [Build the release files](#Build-the-release-files-and-create-a-GitHub-release)
4. [Create a GitHub release](#Build-the-release-files-and-create-a-GitHub-release)
1. [Update the components](#Update-the-components)
2. [Build the release files](#Build-the-release-files-and-create-a-GitHub-release)
3. [Create a GitHub release](#Build-the-release-files-and-create-a-GitHub-release)

## Update the external components
DrOn has three large external components:
## Update the components

DrOn has three components:

1. dron-rxnorm
2. dron-ndc
3. dron-ingredient

Due to their size, they are not in GitHub version control.

The process of preparing a release works as follows:

- Build the components using the DrOn builder application (as of 2021, done by Matt McConnell).
- Upload the components to their s3 bucket. *Important: After the upload, all three components must be available from the following locations*:
- https://drugontology.s3.amazonaws.com/dron-ingredient.owl
- https://drugontology.s3.amazonaws.com/dron-ndc.owl
- https://drugontology.s3.amazonaws.com/dron-rxnorm.owl
- (If the location has to change, make sure that update the `DRON_RELEASE_LOCATION` variable in `src/ontology/dron.Makefile` before running an ODK release.)

The Dron Builder code is currently still bundled in a Scala module managed by Matt McConnell.

## Import the external components
They are generated from the templates in `src/templates/`.
The templates are updated from RxNorm monthly releases.

Before running the normal ODK release workflow, the release manager (as of 2021, Bill Hogan) has to download the components into their local DrOn repo. This is achieved as follows:
TODO: RxNorm updates are not implemented in this version of the code.
That functionality will be added soon!

- In your Terminal, go to your DrOn edit directory, for example:
Build them with the `all_components` task:

```
cd dron/src/ontology
sh run.sh make all_components -B
```

- To download the external components previously uploaded to their s3 bucket:

```
sh run.sh make download_components
```

This command will now download dron-ingredient.owl, dron-ndc.owl, dron-rxnorm.owl to your local machine (and overwrite any old versions of these three files), so that the ODK will recognise them as part of the upcoming release.
The files will be created in `src/ontology/components/` and will be imported into `dron-edit.owl`.

## 3+4. Build the release files and create a GitHub release
## 2+3. Build the release files and create a GitHub release

Follow the [default ODK release instructions](odk-workflows/ReleaseWorkflow.md).

Expand Down
233 changes: 100 additions & 133 deletions src/ontology/dron.Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,6 @@
#### Custom release artefacts ####
##################################

define swap_chebi
$(ROBOT) rename -i $(1) --mappings mappings/dron-chebi-mapping.csv --allow-missing-entities true --allow-duplicates true convert -f ofn -o $(1)
endef

# The following describes the definition of the dron-lite release.
# In essence we merge rxnorm, dron-ingredient, all imports and the edit file
# (no NDC) and the run a regular full release.
Expand All @@ -27,132 +23,103 @@ dron-lite.owl: $(TMPDIR)/dron-edit_lite.owl
reduce -r ELK \
annotate --ontology-iri $(ONTBASE)/$@ $(ANNOTATE_ONTOLOGY_VERSION) --output $@.tmp.owl && mv $@.tmp.owl $@


# export: $(TMPDIR)/export_dron-hand.tsv
#
# $(TMPDIR)/export_dron-%.tsv: $(COMPONENTSDIR)/dron-%.owl | $(TMPDIR)
# $(ROBOT) export --input $< \
# --header "ID|Type|LABEL|has_obo_namespace|SubClass Of|definition|hasDbXref|comment|hasExactSynonym|has_narrow_synonym|hasRelatedSynonym|created_by|creation_date|in_subset|has_alternative_id" \
# --include "classes properties" \
# --export $@
# reports/dron-%.tsv: $(COMPONENTSDIR)/dron-%.owl
# $(ROBOT) query -i $< --query ../sparql/dron-$*.sparql $@
# cat $@ | sed 's|http://purl.obolibrary.org/obo/DRON_|DRON:|g' | sed 's|http://purl.obolibrary.org/obo/CHEBI_|CHEBI:|g' | sed 's|http://purl.obolibrary.org/obo/OBI_|OBI:|g' | sed 's|http://www.w3.org/2002/07/owl#|owl:|g' | sed 's/[<>]//g' > [email protected] && mv [email protected] $@
#
# reports/template-dron-ingredient.tsv: reports/dron-ingredient.tsv
# sed -i '1d' reports/dron-ingredient.tsv > $@
# echo "ID LABEL TYPE PARENT RXCUI p1_bfo53 p1_bfo71 p1_genus BEARER\nID LABEL TYPE SC % A DRON:00010000 SC 'is bearer of' some % SPLIT=|" | cat - reports/dron-ingredient.tsv > [email protected] && mv [email protected] $@
# echo "BFO:0000053 is bearer of owl:ObjectProperty " >> $@
#
# reports/template-dron-ndc.tsv: reports/dron-ndc.tsv
# sed -i '1d' reports/dron-ndc.tsv > $@
# echo "ID LABEL TYPE PARENT BEARER\nID LABEL TYPE SC % SC 'has_proper_part' some % SPLIT=|" | cat - reports/dron-ndc.tsv > [email protected] && mv [email protected] $@
# echo "http://www.obofoundry.org/ro/ro.owl#has_proper_part has_proper_part owl:ObjectProperty " >> $@
#
# reports/template-dron-rxnorm.tsv: reports/dron-rxnorm.tsv
# sed -i '1d' reports/dron-rxnorm.tsv > $@
# echo "ID LABEL TYPE PARENT RXCUI BEARER\nID LABEL TYPE SC % A DRON:00010000 SC 'is bearer of' some % SPLIT=|" | cat - reports/dron-rxnorm.tsv > [email protected] && mv [email protected] $@
# echo "BFO:0000053 is bearer of owl:ObjectProperty " >> $@
#
# reports/template-%.owl: reports/template-%.tsv
# $(ROBOT) template --template $< \
# --ontology-iri "$(ONTBASE)/$@" \
# --output $@
#
# tables: reports/dron-rxnorm.tsv reports/dron-ingredient.tsv reports/dron-ndc.tsv
#
# tmp/unmerge-%.owl: $(COMPONENTSDIR)/dron-%.owl reports/template-dron-%.owl
# $(ROBOT) merge -i $< unmerge -i reports/template-dron-$*.owl convert -f ofn -o $@
#
# tmp/unmerge-%.ttl: tmp/unmerge-%.owl
# $(ROBOT) convert -i $< -f ttl -o $@
#
# tmp/unmerge-ingredient.owl:
# echo "skipped"
#
# unmerge: tmp/unmerge-ingredient.owl tmp/unmerge-ingredient.ttl
# unmerge: tmp/unmerge-ndc.owl tmp/unmerge-ndc.owl tmp/unmerge-rxnorm.owl
#
# tmp/rename-%.owl: tmp/%.owl
# $(ROBOT) rename -i $< --mappings config/rename.tsv -o $@
#
# ALL_PATTERNS=$(patsubst ../patterns/dosdp-patterns/%.yaml,%,$(wildcard ../patterns/dosdp-patterns/[a-z]*.yaml))
#
# .PHONY: matches
# matches:
# dosdp-tools query --ontology=tmp/rename-unmerge-ingredient.owl --catalog=catalog-v001.xml --reasoner=elk --obo-prefixes=true --batch-patterns="$(ALL_PATTERNS)" --template="../patterns/dosdp-patterns" --outfile="../patterns/data/matches/"

DRON_RELEASE_LOCATION=https://drugontology.s3.amazonaws.com
DRON_NDC_RELEASE=$(DRON_RELEASE_LOCATION)/dron-ndc.owl
DRON_RXNORM_RELEASE=$(DRON_RELEASE_LOCATION)/dron-rxnorm.owl
DRON_INGREDIENTS_RELEASE=$(DRON_RELEASE_LOCATION)/dron-ingredient.owl

download_components:
echo "Resetting components to the last release. This is usually only necessary when the project is cloned on a new machine."
mkdir -p $(COMPONENTSDIR)
wget $(DRON_NDC_RELEASE) -O $(COMPONENTSDIR)/dron-ndc.owl
$(call swap_chebi,$(COMPONENTSDIR)/dron-ndc.owl)
wget $(DRON_RXNORM_RELEASE) -O $(COMPONENTSDIR)/dron-rxnorm.owl
$(call swap_chebi,$(COMPONENTSDIR)/dron-rxnorm.owl)
wget $(DRON_INGREDIENTS_RELEASE) -O $(COMPONENTSDIR)/dron-ingredient.owl
$(call swap_chebi,$(COMPONENTSDIR)/dron-ingredient.owl)
grep -v "SubClassOf.*CHEBI_.*/OBI_0000047" components/dron-ingredient.owl > components/dron-ingredient-remediated.owl
mv components/dron-ingredient.owl components/dron-ingredient-initial.owl
mv components/dron-ingredient-remediated.owl components/dron-ingredient.owl

################################
## From March 2021 Migration ###
################################

# tmp/dron-edit-external.ofn: $(SRC)
# $(ROBOT) filter --input $< \
# --select "DRON:*" \
# --select complement \
# --preserve-structure false \
# --output $@
#
#
# unmerge_src:
# $(ROBOT) merge -i $(SRC) --collapse-import-closure false unmerge -i tmp/dron-edit-external.ofn convert -f ofn -o $(SRC)
#
#
# merge_release:
# $(ROBOT) merge -i $(SRC) -i $(COMPONENTSDIR)/dron-hand.owl -i $(COMPONENTSDIR)/dron-upper.owl --collapse-import-closure false -o $(SRC).ofn

##### Diff #####

tmp/$(ONT)-build.owl:
cp ../../$(ONT).owl $@

tmp/$(ONT)-merged.owl: $(SRC)
$(ROBOT) merge -i $< -o $@

tmp/$(ONT)-release.owl:
$(ROBOT) merge -I http://purl.obolibrary.org/obo/$(ONT).owl -o $@

reports/release-diff-%.md: tmp/$(ONT)-release.owl tmp/$(ONT)-%.owl
$(ROBOT) diff --left $< --right tmp/$(ONT)-$*.owl -f markdown -o $@

reports/release-diff-%.txt: tmp/$(ONT)-release.owl tmp/$(ONT)-%.owl
$(ROBOT) diff --left $< --right tmp/$(ONT)-$*.owl -o $@

reports/dron-release-diff-%.txt: reports/release-diff-%.txt
grep DRON_ $< > $@

release_diff: reports/release-diff-build.md reports/release-diff-build.txt
release_diff: reports/release-diff-merged.md reports/release-diff-merged.txt
release_diff: reports/dron-release-diff-build.txt
release_diff: reports/dron-release-diff-merged.txt

.PHONY: unsat
unsat: tmp/dron_unsat.ofn

tmp/dron_unsat.ofn: $(SRC)
robot merge --input $< explain --reasoner ELK \
-M unsatisfiability --unsatisfiable all --explanation [email protected] \
annotate --ontology-iri "$(ONTBASE)/$@" \
--output $@

.PHONY: swap_chebi_in_edit
swap_chebi_in_edit:
$(call swap_chebi,$(SRC))
###################################
#### Create OWL from templates ####
###################################

$(TMPDIR)/ldtab.jar: | $(TMPDIR)
wget https://github.com/ontodev/ldtab.clj/releases/download/v2024-05-14/ldtab.jar -O $@

LDTAB := java -jar $(TMPDIR)/ldtab.jar

# Load DrOn templates into SQLite.
$(TMPDIR)/dron.db: $(SCRIPTSDIR)/create-dron-tables.sql $(SCRIPTSDIR)/load-dron-tables.sql
rm -f $@
sqlite3 $@ < $<
sqlite3 $@ < $(word 2,$^)

# Convert DrOn template tables to LDTab format tables.
$(TMPDIR)/ldtab.db: $(TMPDIR)/dron.db $(SCRIPTSDIR)/prefix.tsv $(SCRIPTSDIR)/convert-dron-ldtab.sql | $(TMPDIR)/ldtab.jar
$(eval DB=$@)
rm -f $@
$(LDTAB) init $(DB) --table dron_ingredient
$(LDTAB) init $(DB) --table dron_rxnorm
$(LDTAB) init $(DB) --table dron_ndc
$(LDTAB) prefix $@ $(word 2,$^)
sqlite3 $@ < $(word 3,$^)

# Export an LDTab table to Turtle format.
$(COMPONENTSDIR)/dron-%.ttl: $(TMPDIR)/ldtab.db | $(COMPONENTSDIR)
rm -f $@
$(LDTAB) export $< $@ --table dron_$*

# Convert a Turtle file to an OWL file in RDFXML format.
$(COMPONENTSDIR)/%.owl: $(COMPONENTSDIR)/%.ttl
$(ROBOT) convert -i $< -o $@

# Override the all_components task.
all_components: $(COMPONENTSDIR)/dron-ingredient.ttl $(COMPONENTSDIR)/dron-rxnorm.ttl $(COMPONENTSDIR)/dron-ndc.ttl $(COMPONENTSDIR)/dron-ingredient.owl $(COMPONENTSDIR)/dron-rxnorm.owl $(COMPONENTSDIR)/dron-ndc.owl

###################################
#### Create templates from OWL ####
###################################

# Convert component OWL files to DrOn templates.
# This should be a one-time action
# when migrating away from the old Scala DrOn build system,
# but it could also be helpful for testing changes to new build scripts.

$(TMPDIR)/reverse/:
mkdir -p $@

# Convert components from OFN to OWL for LDTab to import.
$(TMPDIR)/reverse/dron-%.owl: components.bk/dron-%.owl | $(TMPDIR)/reverse/
$(ROBOT) convert -i $< -o $@

$(TMPDIR)/reverse.db: $(SCRIPTSDIR)/prefix.tsv $(SCRIPTSDIR)/create-dron-tables.sql $(SCRIPTSDIR)/convert-ldtab-dron.sql $(SCRIPTSDIR)/save-dron-tables.sql $(TMPDIR)/reverse/dron-ingredient.owl $(TMPDIR)/reverse/dron-rxnorm.owl $(TMPDIR)/reverse/dron-ndc.owl | $(TMPDIR)/ldtab.jar
$(eval DB=$@)
rm -f $(DB)
$(LDTAB) init $(DB) --table dron_ingredient
$(LDTAB) init $(DB) --table dron_rxnorm
$(LDTAB) init $(DB) --table dron_ndc
$(LDTAB) prefix $(DB) $<
sqlite3 $(DB) < $(word 2,$^)
$(LDTAB) import $(DB) --table dron_ingredient $(TMPDIR)/reverse/dron-ingredient.owl
$(LDTAB) import $(DB) --table dron_rxnorm $(TMPDIR)/reverse/dron-rxnorm.owl
$(LDTAB) import $(DB) --table dron_ndc $(TMPDIR)/reverse/dron-ndc.owl

.PHONY: reverse
reverse: $(TMPDIR)/reverse.db
# Convert LDTab tables to DrOn templates
sqlite3 $(DB) < $(word 3,$^)
# Save template tables to TSV
cd $(TMPDIR)/reverse/ && sqlite3 ../../$(DB) < ../../$(word 4,$^)
# Copy to src/templates/
cp $(TMPDIR)/reverse/*.tsv $(TEMPLATEDIR)

###########################
#### Roundtrip testing ####
###########################

# Export LDTab tables as TSV and sort them for diffing.

.PRECIOUS: $(COMPONENTSDIR)/dron-%.tsv
$(COMPONENTSDIR)/dron-%.tsv: $(COMPONENTSDIR)/dron-%.owl
$(eval DB=$(TMPDIR)/ldtab.db)
rm -f $@
$(LDTAB) export $(DB) $@ --table dron_$* --sort

.PRECIOUS: $(TMPDIR)/reverse/dron-%.tsv
$(TMPDIR)/reverse/dron-%.tsv: | $(TMPDIR)/reverse.db
$(eval DB=$|)
rm -f $@
$(LDTAB) export $(DB) $@ --table dron_$* --sort

$(TMPDIR)/%.tsv.diff: $(TMPDIR)/reverse/%.tsv $(COMPONENTSDIR)/%.tsv
-diff -u $^ > $@

$(TMPDIR)/%.owl.diff: $(TMPDIR)/reverse/%.owl $(COMPONENTSDIR)/%.owl
-diff -u $^ > $@

.PHONY: roundtrip
roundtrip: $(COMPONENTSDIR)/dron-ingredient.ttl $(COMPONENTSDIR)/dron-rxnorm.ttl $(COMPONENTSDIR)/dron-ndc.ttl $(TMPDIR)/dron-ingredient.owl.diff $(TMPDIR)/dron-ingredient.tsv.diff $(TMPDIR)/dron-rxnorm.owl.diff $(TMPDIR)/dron-rxnorm.tsv.diff $(TMPDIR)/dron-ndc.owl.diff $(TMPDIR)/dron-ndc.tsv.diff
Loading

0 comments on commit b092a5b

Please sign in to comment.