Request: Removal of hot pixels #14
Comments
|
Hi, when the raw frames are read into Warp, hot pixels above 20 standard deviations are already removed, and the persistent ones should be handled by the gain reference. Do some still slip through? The pre-multiplication option never gained traction because none of the popular SPA programs band-limit the CTF correctly in cases where pre-multiplication in a bigger box makes sense (high defocus/small box). RELION lets you specify the CTF as pre-multiplied, but then uses the standard analytic CTF^2 in its calculations rather than simulating it in a bigger box and then cropping. This kills most of the information beyond the frequency where CTF aliasing starts, so it doesn't matter if the particles are pre-multiplied. However, this does make sense for sub-tomogram averaging, where the 3D CTF used by RELION is completely controlled by what Warp creates – so it can make a nice, anti-aliased CTF. This is going to be the default behavior in the next release. |
|
Hello Dimitry,
Thanks for your quick and thoughtful reply.
When I extract particles in WARP and run 2D classification in RELION, I get
some classes that look like the attached images (I hope you can see them,
let me know if you cannot). When I extract particles using RELION and have
it remove hot and dead pixels above or below 5 standard deviations, I do
not see such artifacts in the class averages. Is this evidence of WARP
missing some hot/dead pixels?
As for pre-multiplying, I thought it would be a good idea since I am using
a 192 pixel box size with data with an average defocus of 2 um. I have not
rigorously tested whether phase-flipping helps in such programs as RELION,
although anecdotal evidence from our lab suggests modest improvements in
resolution were achieved with phase-flipped particles in 3D refinement (3.3
A without phase-flip goes to 3.1A with phase-flip for 320 pixel box
size...). Thank you for your thoughts on whether this really helps given
how current programs deal with CTF correction - perhaps phase flipping has
lost its merit in these cases.
…On Thu, Jan 16, 2020 at 12:52 PM Dimitry Tegunov ***@***.***> wrote:
Hi,
when the raw frames are read into Warp, hot pixels above 20 standard
deviations are already removed, and the persistent ones should be handled
by the gain reference. Do some still slip through?
The pre-multiplication option never gained traction because none of the
popular SPA programs band-limit the CTF correctly in cases where
pre-multiplication in a bigger box makes sense (high defocus/small box).
RELION lets you specify the CTF as pre-multiplied, but then uses the
standard analytic CTF^2 in its calculations rather than simulating it in a
bigger box and then cropping. This kills most of the information beyond the
frequency where CTF aliasing starts, so it doesn't matter if the particles
are pre-multiplied.
However, this does make sense for sub-tomogram averaging, where the 3D CTF
used by RELION is completely controlled by what Warp creates – so it can
make a nice, anti-aliased CTF. This is going to be the default behavior in
the next release.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AGDIR2ZEHRRZNU4XOWNDHRTQ6DCJJA5CNFSM4KH2DHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFQC6A#issuecomment-575340920>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGDIR26R6KIYINVO4AEAUN3Q6DCJJANCNFSM4KH2DHOA>
.
|
|
Sorry about this, but the second class I sent does not appear to have the
artifact I meant to capture. The attached image does however.
…On Thu, Jan 16, 2020 at 1:50 PM David Boyer ***@***.***> wrote:
Hello Dimitry,
Thanks for your quick and thoughtful reply.
When I extract particles in WARP and run 2D classification in RELION, I
get some classes that look like the attached images (I hope you can see
them, let me know if you cannot). When I extract particles using RELION and
have it remove hot and dead pixels above or below 5 standard deviations, I
do not see such artifacts in the class averages. Is this evidence of WARP
missing some hot/dead pixels?
As for pre-multiplying, I thought it would be a good idea since I am using
a 192 pixel box size with data with an average defocus of 2 um. I have not
rigorously tested whether phase-flipping helps in such programs as RELION,
although anecdotal evidence from our lab suggests modest improvements in
resolution were achieved with phase-flipped particles in 3D refinement (3.3
A without phase-flip goes to 3.1A with phase-flip for 320 pixel box
size...). Thank you for your thoughts on whether this really helps given
how current programs deal with CTF correction - perhaps phase flipping has
lost its merit in these cases.
On Thu, Jan 16, 2020 at 12:52 PM Dimitry Tegunov ***@***.***>
wrote:
> Hi,
>
> when the raw frames are read into Warp, hot pixels above 20 standard
> deviations are already removed, and the persistent ones should be handled
> by the gain reference. Do some still slip through?
>
> The pre-multiplication option never gained traction because none of the
> popular SPA programs band-limit the CTF correctly in cases where
> pre-multiplication in a bigger box makes sense (high defocus/small box).
> RELION lets you specify the CTF as pre-multiplied, but then uses the
> standard analytic CTF^2 in its calculations rather than simulating it in a
> bigger box and then cropping. This kills most of the information beyond the
> frequency where CTF aliasing starts, so it doesn't matter if the particles
> are pre-multiplied.
>
> However, this does make sense for sub-tomogram averaging, where the 3D
> CTF used by RELION is completely controlled by what Warp creates – so it
> can make a nice, anti-aliased CTF. This is going to be the default behavior
> in the next release.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#14?email_source=notifications&email_token=AGDIR2ZEHRRZNU4XOWNDHRTQ6DCJJA5CNFSM4KH2DHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFQC6A#issuecomment-575340920>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGDIR26R6KIYINVO4AEAUN3Q6DCJJANCNFSM4KH2DHOA>
> .
>
|
|
Sorry, can't see any images here. Dropbox? |
|
… On Thu, Jan 16, 2020 at 3:21 PM Dimitry Tegunov ***@***.***> wrote:
Sorry, can't see any images here. Dropbox?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AGDIR26LGENT62R45DJV2LTQ6DTXJA5CNFSM4KH2DHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJF44CY#issuecomment-575393291>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGDIR254FJQ7TDM4EHEWSL3Q6DTXJANCNFSM4KH2DHOA>
.
|
|
Interesting! Are those from particles that were pre-multiplied by the CTF? Can you trace particles from those classes to the original micrographs and check if there is anything weird going on? It seems like the image artifacts appear at an almost fixed position relative to the particle. You can select particles from the classes in RELION and use "import particle coordinates" in Warp. |
|
These are particles straight from WARP that were not pre-multiplied. I am
working on tracing back all the particle coordinates to be able to spot
anything on the micrographs. However, when I try to run the Import
Particles job with the particles.star generated from RELION after selecting
the classes I sent you, WARP seems to crash every time I have tried. I
paste the crash error message below in case it helps. Perhaps this is
because I am using RELION 3.1-beta which outputs a different .star file
format? I checked and made sure all the paths to the particles and
micrographs (listed as the original .tif movies as per the WARP format for
outputting .star files) were correct so I am not sure what might cause this
issue. I have never used the Import Particles job before so I don't have a
reference for it working, and it seems, at least at the moment, that the
User Guide page is blank for all the 2D task dialog jobs on the WARP
website. Perhaps you have some insight? Sorry that this has become a bit of
an error finding mission; if I find another way to map the particles back
to the micrographs, I will update you. Thanks so much.
1/17/2020 12:59:58 PM:
System.InvalidOperationException: The calling thread cannot access this
object because a different thread owns it.
at System.Windows.Threading.Dispatcher.VerifyAccess()
at
Warp.Controls.Dialog2DParticleImport.<>c__DisplayClass8_0.<<ButtonWrite_OnClick>b__0>d.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Warp.Controls.Dialog2DParticleImport.<ButtonWrite_OnClick>d__8.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
168
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate
callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source,
Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
at System.Windows.Threading.Dispatcher.VerifyAccess()
at
Warp.Controls.Dialog2DParticleImport.<>c__DisplayClass8_0.<<ButtonWrite_OnClick>b__0>d.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Warp.Controls.Dialog2DParticleImport.<ButtonWrite_OnClick>d__8.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
168
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate
callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source,
Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
…On Thu, Jan 16, 2020 at 3:49 PM Dimitry Tegunov ***@***.***> wrote:
Interesting! Are those from particles that were pre-multiplied by the CTF?
Can you trace particles from those classes to the original micrographs and
check if there is anything weird going on? It seems like the image
artifacts appear at an almost fixed position relative to the particle. You
can select particles from the classes in RELION and use "import particle
coordinates" in Warp.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#14?email_source=notifications&email_token=AGDIR23MTZMBFOZCTDTT5DTQ6DXABA5CNFSM4KH2DHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJF6WCI#issuecomment-575400713>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGDIR24MPLSOMS257KC72OLQ6DXABANCNFSM4KH2DHOA>
.
|
|
I was able to look at the particles in the 2D class averages showing the
artifact and found a fair amount of particles with a hot pixel (see Dropbox
image). I am guessing this caused the ringing artifact in the class
averages I showed you?
https://www.dropbox.com/s/i6wihb1kxsz4mnm/hotpixel.png?dl=0
…On Fri, Jan 17, 2020 at 1:16 PM David Boyer ***@***.***> wrote:
These are particles straight from WARP that were not pre-multiplied. I am
working on tracing back all the particle coordinates to be able to spot
anything on the micrographs. However, when I try to run the Import
Particles job with the particles.star generated from RELION after selecting
the classes I sent you, WARP seems to crash every time I have tried. I
paste the crash error message below in case it helps. Perhaps this is
because I am using RELION 3.1-beta which outputs a different .star file
format? I checked and made sure all the paths to the particles and
micrographs (listed as the original .tif movies as per the WARP format for
outputting .star files) were correct so I am not sure what might cause this
issue. I have never used the Import Particles job before so I don't have a
reference for it working, and it seems, at least at the moment, that the
User Guide page is blank for all the 2D task dialog jobs on the WARP
website. Perhaps you have some insight? Sorry that this has become a bit of
an error finding mission; if I find another way to map the particles back
to the micrographs, I will update you. Thanks so much.
1/17/2020 12:59:58 PM:
System.InvalidOperationException: The calling thread cannot access this
object because a different thread owns it.
at System.Windows.Threading.Dispatcher.VerifyAccess()
at
Warp.Controls.Dialog2DParticleImport.<>c__DisplayClass8_0.<<ButtonWrite_OnClick>b__0>d.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
0
--- End of stack trace from previous location where exception was thrown
---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Warp.Controls.Dialog2DParticleImport.<ButtonWrite_OnClick>d__8.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
168
--- End of stack trace from previous location where exception was thrown
---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate
callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object
source, Delegate callback, Object args, Int32 numArgs, Delegate
catchHandler)
at System.Windows.Threading.Dispatcher.VerifyAccess()
at
Warp.Controls.Dialog2DParticleImport.<>c__DisplayClass8_0.<<ButtonWrite_OnClick>b__0>d.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
0
--- End of stack trace from previous location where exception was thrown
---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Warp.Controls.Dialog2DParticleImport.<ButtonWrite_OnClick>d__8.MoveNext()
in
D:\Dev\warp2\Warp\Controls\TaskDialogs\2D\Dialog2DParticleImport.xaml.cs:line
168
--- End of stack trace from previous location where exception was thrown
---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate
callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object
source, Delegate callback, Object args, Int32 numArgs, Delegate
catchHandler)
On Thu, Jan 16, 2020 at 3:49 PM Dimitry Tegunov ***@***.***>
wrote:
> Interesting! Are those from particles that were pre-multiplied by the
> CTF? Can you trace particles from those classes to the original micrographs
> and check if there is anything weird going on? It seems like the image
> artifacts appear at an almost fixed position relative to the particle. You
> can select particles from the classes in RELION and use "import particle
> coordinates" in Warp.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#14?email_source=notifications&email_token=AGDIR23MTZMBFOZCTDTT5DTQ6DXABA5CNFSM4KH2DHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJF6WCI#issuecomment-575400713>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGDIR24MPLSOMS257KC72OLQ6DXABANCNFSM4KH2DHOA>
> .
>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
I recently had an issue where due to an old K2 chip on our microscope, I needed to extract the WARP-picked particles in a program like RELION that allows hot/dead pixel removal. This unfortunately obviates the use of WARP for motion-correction because I would need to re-do motion correction using something that allows anisotropic magnification correction. It would be great if WARP could take care of hot/dead pixels for Extraction so I could stay in the WARP pipeline for pre-processing.
Also, it would be great if the on-the-fly extracted WARP particles had the option of being phase-flipped using the bigger box. I see that that is an option in the 2D task dialog, but not for the on-fly-extraction.
Thanks!
The text was updated successfully, but these errors were encountered: