Configure ctp anonymizer basic beta #2

maxicheong · 2017-05-08T05:53:40Z

Finally circled back on this one! Added and changed all the files needed to perform a very solid, basic anonymization of DICOM data using the DICOM Supplement 142 Basic Anonymization Profile. Also, enabled patient ID and name replacement lookups via Lookup.properties. Lookup.properties is the file that determines which patient ID and name to change to. We'll get these IDs from STRIDE.

Also, changed port to 104 which works better with the firewall rules in the hospital network.

…me lookup using Lookup.Properties, turned on Remove Unchecked Elements for stronger anonymization

vsoch · 2017-05-08T06:05:41Z

Great! Have you tested that when building and bringing up the image on som-ctp it is functioning and complete?

maxicheong · 2017-05-08T06:07:07Z

...not yet - just a local instance of CTP I tested on. Can you remind me the steps? : <

…

On Sun, May 7, 2017 at 11:05 PM, Vanessa Sochat ***@***.***> wrote: Great! Have you tested that when building and bringing up the image on som-ctp it is functioning and complete? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZZsMzItX82ODWUBezznVrNaloRnKS8oks5r3rC2gaJpZM4NTg4e> .

vsoch · 2017-05-08T06:13:58Z

I don't see the updates on the server or in the logs:

vsoch · 2017-05-08T06:29:59Z

I'm logged into the server I will pull your changes and test, with complete instructions here so you know what to do. To look at currently running docker images:

docker ps

you can then start an interactive shell using the image id, which is the first 12 characters of a much longe r string:

 docker exec -it e800e956dd66 bash

This is how I "shelled" into the image above to see that it hadn't been updated. Now, we want to rebuild the image to test the changes that you've made:

 docker build -t vanessa/ctp .

Note the . at the end, it's easy to miss and specifies $PWD

I can tell that your build will fail because you are trying to copy a file to a place that doesn't exist - there is nowhere where "/JavaPrograms/scripts is created, and confirmed it bugs:

 ---> Running in 2078ee2b96a6
cp: cannot create regular file '/JavaPrograms/scripts/DicomServiceAnonymizer.script': No such file or directory
The command '/bin/sh -c cp /code/DicomServiceAnonymizer.script /JavaPrograms/scripts/DicomServiceAnonymizer.script' returned a non-zero code: 1

I can fix this by creating the folder using mkdir -p /JavaPrograms/scripts. I would anticipate you might also need to check permissions on running that script. First I'll stop the old Docker image:

docker stop e800e956dd66

Now I can start the new image

 docker run -d vanessa/ctp

look at with docker ps

vanessa@som-ctp:~/mirc-ctp-docker$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
4213931acafb        vanessa/ctp         "/bin/sh -c '/bin/bas"   3 seconds ago       Up 3 seconds        1080/tcp            sharp_cori

what we would need now is to add commands to the Dockerfile to fix permissions to that script, and get the server actually working so I can go to 104.196.250.197:1080 or 104.196.250.197:104 (whatever ports are set up) and verify that the CTP instance is working. How about give those fixes a try, try to get an example running on the server, and then update this PR. Thanks Maxwell!

maxicheong · 2017-05-08T06:46:59Z

Sweet! I'll give that a try tomorrow and update the PR. I'll put these pointers in Confluence so I don't forget lol Many thanks! Maxwell

…

Sent from my iPhone

On May 7, 2017, at 11:29 PM, Vanessa Sochat ***@***.***> wrote: I'm logged into the server I will pull your changes and test, with complete instructions here so you know what to do. To look at currently running docker images: docker ps you can then start an interactive shell using the image id, which is the first 12 characters of a much longe r string: docker exec -it e800e956dd66 bash This is how I "shelled" into the image above to see that it hadn't been updated. Now, we want to rebuild the image to test the changes that you've made: docker build -t vanessa/ctp . Note the . at the end, it's easy to miss and specifies $PWD I can tell that your build will fail because you are trying to copy a file to a place that doesn't exist - there is nowhere where "/JavaPrograms/scripts is created, and confirmed it bugs: ---> Running in 2078ee2b96a6 cp: cannot create regular file '/JavaPrograms/scripts/DicomServiceAnonymizer.script': No such file or directory The command '/bin/sh -c cp /code/DicomServiceAnonymizer.script /JavaPrograms/scripts/DicomServiceAnonymizer.script' returned a non-zero code: 1 I can fix this by creating the folder using mkdir -p /JavaPrograms/scripts. I would anticipate you might also need to check permissions on running that script. First I'll stop the old Docker image: docker stop e800e956dd66 Now I can start the new image docker run -d vanessa/ctp look at with docker ps ***@***.***:~/mirc-ctp-docker$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4213931acafb vanessa/ctp "/bin/sh -c '/bin/bas" 3 seconds ago Up 3 seconds 1080/tcp sharp_cori what we would need now is to add commands to the Dockerfile to fix permissions to that script, and get the server actually working so I can go to 104.196.250.197:1080 or 104.196.250.197:104 (whatever ports are set up) and verify that the CTP instance is working. How about give those fixes a try, try to get an example running on the server, and then update this PR. Thanks Maxwell! ― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

vsoch · 2017-05-08T06:53:14Z

Sounds good, thanks @maxicheong ! Ping me if you have any docker questions. And remember you can shell into the som-ctp with a gcloud command:

 gcloud compute --project "som-langlotz-lab" ssh --zone "us-west1-a" "som-ctp"

and if you ever forget, you can usually look in your history to find:

history | grep som-ctp

and if all else fails, go to the cloud console @ https://console.cloud.google.com/home/dashboard?project=som-langlotz-lab

and your credential file https://developers.google.com/identity/protocols/application-default-credentials needs to be exported to GOOGLE_APPLICATION_CREDENTIALS for the auth flow to work for gcloud. You can do this in your $HOME/.profile or $HOME/.bashrc Periodically you will need to update the google cloud components, they will tell you, like

gcloud components update

hopefully I've answered a few questions you will have in advance, haha. Keep up the good work! Remember I'm around if needed :)

vsoch · 2017-05-08T14:52:59Z

one more important tidbit! Don't forget to expose ports (in the Dockerfile) that the image needs to have listened to:

https://docs.docker.com/engine/reference/builder/#expose

But exposing doesn't mean it's available to the host - when you run you will need to do this with the -p command, and note that you don't necessarily have to map the same port numbers (eg, you might do EXPOSE 1080 but then map to the host like -p 8888:1080. Here are the docs for the command:

https://docs.docker.com/engine/reference/run/#expose-incoming-ports

If/when we have some application that runs with the image, we can expose the ports in the docker-compose.yml, like this:

https://docs.docker.com/compose/compose-file/#expose

but don't worry about that for now :)

maxicheong · 2017-05-08T15:05:58Z

Very helpful tidbit! Let me read up on it and try it out. Maxwell

…

Sent from my iPhone

On May 8, 2017, at 7:52 AM, Vanessa Sochat ***@***.***> wrote: one more important tidbit! Don't forget to expose ports (in the Dockerfile) that the image needs to have listened to: https://docs.docker.com/engine/reference/builder/#expose But exposing doesn't mean it's available to the host - when you run you will need to do this with the -p command, and note that you don't necessarily have to map the same port numbers (eg, you might do EXPOSE 1080 but then map to the host like -p 8888:1080. Here are the docs for the command: https://docs.docker.com/engine/reference/run/#expose-incoming-ports If/when we have some application that runs with the image, we can expose the ports in the docker-compose.yml, like this: https://docs.docker.com/compose/compose-file/#expose but don't worry about that for now :) ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

vsoch · 2017-05-08T23:23:20Z

Dockerfile

@@ -13,6 +13,7 @@ RUN apt-get update && apt-get -y install git \
           unzip

 RUN mkdir /JavaPrograms
+RUN mkdir /JavaPrograms/scripts
 RUN mkdir /code


for future, if you do just mkdir -p /JavaPrograms/scripts this will create both directories with one line. This is a good strategy to have for Dockerfiles, because each line is a separate layer. It's good practice to reduce your runs into as few layers as possible. Eg:

RUN mkdir -p /JavaPrograms/scripts && mkdir /code

would be more optimal, although not completely necessary :)

Good point! Noted

yeah dawg! Check this out when you have time: https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/ it's really interesting what an "image" actually is, think "sandwich layers" vs. "entire sandwich" :)

try typing:

docker images

and you'll see all the (top level) images.. then try

docker images --digests

and you'll see each image id is actually a sha256 sum! But it gets even better... do

docker images --all

and you'll see that each "image" is actually a crapton of layers, and they are only assembled when the image is actually running. Each layer is actually a diff off of the previous ones, and then at runtime the top layer is the only one that is actually writable that maintains your changes. This is a fundamentally different model than singularity, which is an actual, one file deal, image.

Fascinating concept of layers! Unlike VMs, I thought it was a one file deal.

vsoch · 2017-05-08T23:24:47Z

please post a note with a summary of changes, a URL to preview, and notification of being ready for testing/review when it's ready! Until you tell me, I'll disregard all new stuffs so no worries about me reviewing when it's not ready yet :)

maxicheong · 2017-05-09T00:05:41Z

To clarify - CTP's script folder does not contain UNIX executable scripts. They contain xml files which are used by CTP as part of its configuration (e.g. DicomServiceAnonymizer.script tells CTP which DICOM headers to keep, remove, modify)

maxicheong · 2017-05-09T00:10:04Z

I am close to getting this running. I need to know how to propagate repo's config.xml into the CTP installation at /JavaPrograms inside the Docker instance. Right now, my config.xml is being overwritten by start.sh

vsoch · 2017-05-09T00:24:48Z

I would speculate that there is a default config.xml that exists in the base /code/CTP folder that is copied from there to the JavaPrograms when start.sh is run. Specifically, this line:

mv /code/CTP/* /JavaPrograms

What I would do is have your config.xml copied there after this, so add a line after this one in start.sh to do that.

maxicheong · 2017-05-09T01:10:08Z

Cool that's what i thought too, thanks for confirming. I'll check in the code once I get back to SF. Loooooong drive home!

…

On Mon, May 8, 2017 at 5:24 PM, Vanessa Sochat ***@***.***> wrote: I would speculate that there is a default config.xml that exists in the base /code/CTP folder that is copied from there to the JavaPrograms when start.sh is run. Specifically, this line: mv /code/CTP/* /JavaPrograms What I would do is have your config.xml copied there after this, so add a line after this one in start.sh to do that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZZsM6HhfFXFfOLzlccc43EjROXiUl-Vks5r37JQgaJpZM4NTg4e> .

vsoch · 2017-05-09T01:10:52Z

haha, ok. That's nuts, I won't even leave campus for another 2 hours :P

maxicheong · 2017-05-09T01:11:53Z

#sflife #longcommute #carpooling

…

On Mon, May 8, 2017 at 6:10 PM, Vanessa Sochat ***@***.***> wrote: haha, ok. That's nuts, I won't even leave campus for another 2 hours :P — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZZsM8uYOcL_3TYctL-Bo_l2XNB2tmpkks5r370cgaJpZM4NTg4e> .

… pre-configured config.xml

maxicheong · 2017-05-09T05:33:42Z

Still cannot start CTP as a service after configuring it. Any idea? Ref: http://mircwiki.rsna.org/index.php?title=Running_CTP_as_a_Linux_Service

vsoch · 2017-05-09T07:45:06Z

this is exactly the step where I stopped because the instructions didn't work, and I thought you had done this before and would have the expertise, lol.

vsoch · 2017-05-09T07:45:45Z

How do you normally start it when you install on a server?

maxicheong · 2017-05-09T14:45:35Z

LOL totally forgot I have CTP written on my resume when I applied for this job. Ok I've added the startup (java -jar Runner.jar) now. I'll write a summary of changes this afternoon (busy morning at the Baseline Study clinic)

maxicheong added 3 commits May 7, 2017 22:44

Changed DICOM import port to 104 which is well known for DICOM traffic

1e4529a

Added DICOM S142 Basic Anonymization Profile, added patient ID and na…

fb44327

…me lookup using Lookup.Properties, turned on Remove Unchecked Elements for stronger anonymization

Really adding DICOM S142 Basic Anonymization Profile script now

7052411

maxicheong added 2 commits May 8, 2017 16:12

Created folder to store the CTP .script files

694c227

Exposing port 104 where the DICOM incoming traffic is

9a4c145

vsoch reviewed May 8, 2017

View reviewed changes

maxicheong added 2 commits May 8, 2017 22:13

During start, put in default anonymization pipeline by putting in the…

b157abb

… pre-configured config.xml

Corrected startup jar to Runner.jar

32c63bb

Added line to start CTP DICOM server and pipeline

41f5869

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure ctp anonymizer basic beta #2

Configure ctp anonymizer basic beta #2

maxicheong commented May 8, 2017

vsoch commented May 8, 2017

maxicheong commented May 8, 2017 via email

vsoch commented May 8, 2017

vsoch commented May 8, 2017

maxicheong commented May 8, 2017 via email

vsoch commented May 8, 2017

vsoch commented May 8, 2017

maxicheong commented May 8, 2017 via email

vsoch May 8, 2017

maxicheong May 8, 2017

vsoch May 8, 2017

vsoch May 8, 2017

maxicheong May 9, 2017

vsoch commented May 8, 2017

maxicheong commented May 9, 2017

maxicheong commented May 9, 2017

vsoch commented May 9, 2017

maxicheong commented May 9, 2017 via email

vsoch commented May 9, 2017

maxicheong commented May 9, 2017 via email

maxicheong commented May 9, 2017

vsoch commented May 9, 2017

vsoch commented May 9, 2017

maxicheong commented May 9, 2017

Configure ctp anonymizer basic beta #2

Are you sure you want to change the base?

Configure ctp anonymizer basic beta #2

Conversation

maxicheong commented May 8, 2017

vsoch commented May 8, 2017

maxicheong commented May 8, 2017 via email

vsoch commented May 8, 2017

vsoch commented May 8, 2017

maxicheong commented May 8, 2017 via email

vsoch commented May 8, 2017

vsoch commented May 8, 2017

maxicheong commented May 8, 2017 via email

vsoch May 8, 2017

Choose a reason for hiding this comment

maxicheong May 8, 2017

Choose a reason for hiding this comment

vsoch May 8, 2017

Choose a reason for hiding this comment

vsoch May 8, 2017

Choose a reason for hiding this comment

maxicheong May 9, 2017

Choose a reason for hiding this comment

vsoch commented May 8, 2017

maxicheong commented May 9, 2017

maxicheong commented May 9, 2017

vsoch commented May 9, 2017

maxicheong commented May 9, 2017 via email

vsoch commented May 9, 2017

maxicheong commented May 9, 2017 via email

maxicheong commented May 9, 2017

vsoch commented May 9, 2017

vsoch commented May 9, 2017

maxicheong commented May 9, 2017