Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tims tof reader #812

Open
wants to merge 87 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
d5cb13f
Started frameProxy class
Alexander-Sol Nov 18, 2023
d001aca
Added conversion functions
Alexander-Sol Nov 20, 2023
ef23825
Added useful functions to FrameProxy
Alexander-Sol Nov 21, 2023
93d47c6
Started methods for scan building, added new MsDataScan class
Alexander-Sol Nov 22, 2023
5675c5c
Minor
Alexander-Sol Nov 22, 2023
a574388
TimsTof reader is working!!!
Alexander-Sol Nov 29, 2023
57a061f
minor
Alexander-Sol Nov 30, 2023
b59ee70
Fixed merge conflicts
Alexander-Sol Nov 30, 2023
b8fe87c
Fixed merge booboo
Alexander-Sol Nov 30, 2023
6dd39d1
TimsDataScan is missing
Alexander-Sol Dec 7, 2023
a9df28f
started spectra addition stuff
Alexander-Sol Dec 8, 2023
1ae9727
Finished spectraMerger class
Alexander-Sol Dec 9, 2023
5632fa0
Reader reads in + merges all spectra/scans
Alexander-Sol Jan 12, 2024
91096ee
Parallelized, but made it worse
Alexander-Sol Jan 13, 2024
23e0a8c
Changed MS2 merger to keep list nodes
Alexander-Sol Jan 16, 2024
7e2177e
reverted MS2 merger changes
Alexander-Sol Jan 16, 2024
be4ca1b
unsure
Alexander-Sol Jan 30, 2024
b038767
Merge branch 'master' into timsTOFReader
Alexander-Sol Apr 17, 2024
cad6bae
Merge branch 'timsTOFReader' of https://github.com/Alexander-Sol/mzLi…
Alexander-Sol Jun 19, 2024
e766260
Working but slowly
Alexander-Sol Jun 19, 2024
0236386
idk
Alexander-Sol Jun 19, 2024
4ca6874
Added support for missing Pasef scans, missing precursors
Alexander-Sol Jun 20, 2024
94beb90
Making progress
Alexander-Sol Jul 8, 2024
952f021
Seems to be working
Alexander-Sol Jul 9, 2024
1d47d49
Removed empty scans
Alexander-Sol Jul 10, 2024
a9fb8b3
fixed merge conflicts
Alexander-Sol Sep 15, 2024
26285c5
Finished timsTof reader, added tests
Alexander-Sol Sep 15, 2024
d6b9e75
Updated timsdata.dll path in DllImport call
Alexander-Sol Sep 15, 2024
f6c740a
Fixed broken tests
Alexander-Sol Sep 16, 2024
21aefc9
changed bruker dll location in directory
Alexander-Sol Sep 16, 2024
6417ef4
updated references
Alexander-Sol Sep 16, 2024
26d2058
updated github actions
Alexander-Sol Sep 16, 2024
fa6be72
added depndencies
Alexander-Sol Sep 16, 2024
b1a36f3
edited workflow
Alexander-Sol Sep 16, 2024
aeae7c1
workflow
Alexander-Sol Sep 16, 2024
8c9ae65
mzLookup table
Alexander-Sol Sep 18, 2024
da39272
memoization of mz values, commented out, local tests
Alexander-Sol Sep 20, 2024
c350aa9
not immediately clear
Alexander-Sol Dec 2, 2024
276ce1a
finished index based spectra handling
Alexander-Sol Dec 3, 2024
a2eeffc
Improved parallelization
Alexander-Sol Dec 3, 2024
f2ef052
Finished performance improvements
Alexander-Sol Dec 3, 2024
df8ac01
Centroiding on mzs, not indices
Alexander-Sol Dec 4, 2024
3b1e61d
Fixed tests
Alexander-Sol Dec 5, 2024
597b175
fixed merge conflicts
Alexander-Sol Dec 5, 2024
072f2da
Fixed bugs
Alexander-Sol Dec 5, 2024
fc44991
Updated workflow
Alexander-Sol Dec 5, 2024
17b9103
Created copies of dlls
Alexander-Sol Dec 5, 2024
e8be6d8
Reorganized readers
Alexander-Sol Dec 5, 2024
ec1e7c6
idk
Alexander-Sol Dec 5, 2024
8360603
Updated worklfow
Alexander-Sol Dec 5, 2024
9de6bfe
Updated worklfow
Alexander-Sol Dec 5, 2024
db69e2c
Updated worklfow
Alexander-Sol Dec 5, 2024
6b652d0
Updated worklfow
Alexander-Sol Dec 5, 2024
55c6094
Updated worklfow
Alexander-Sol Dec 5, 2024
03bd993
Updated worklfow
Alexander-Sol Dec 5, 2024
607a9f1
Updated worklfow
Alexander-Sol Dec 5, 2024
323046b
included dependencies
Alexander-Sol Dec 5, 2024
8033e37
Pretty sure I just broke things
Alexander-Sol Dec 6, 2024
ec1cf8e
idk
Alexander-Sol Dec 6, 2024
d20d2c1
Deleted unused code, dlls
Alexander-Sol Dec 6, 2024
827318c
Fixed index out of range issue
Alexander-Sol Dec 9, 2024
253d258
Idk
Alexander-Sol Dec 9, 2024
dd4b1d3
Fixed issue where filtering was not performed
Alexander-Sol Dec 9, 2024
067189d
Things work!
Alexander-Sol Dec 13, 2024
c536184
pre-size lists
Alexander-Sol Dec 17, 2024
b59cc57
Change ProteinDBWriter in the wrong branch
Alexander-Sol Jan 6, 2025
2c8cfea
Fixed merge conflicts
Alexander-Sol Jan 15, 2025
e120c82
Refactoring so sql calls have their own methods
Alexander-Sol Jan 15, 2025
9433a17
Added functions for dynamic connections, associated tests
Alexander-Sol Jan 16, 2025
3edf877
Extended test coverage
Alexander-Sol Jan 16, 2025
52a25b1
Minor
Alexander-Sol Jan 16, 2025
cb28554
Fixed merge conflicts
Alexander-Sol Jan 16, 2025
d13c67a
Implemented try/finally blocks for calls to Marshal
Alexander-Sol Jan 16, 2025
dc18bf0
Deleted unused code, extended tests
Alexander-Sol Jan 16, 2025
7a67440
Fixed double deallocation of memory
Alexander-Sol Jan 16, 2025
8acc4d7
Deleted unintended changes
Alexander-Sol Jan 16, 2025
330eac7
Deleted unused code
Alexander-Sol Jan 16, 2025
4695762
Deleted unintended changes
Alexander-Sol Jan 16, 2025
12503cb
Edited workflow
Alexander-Sol Jan 16, 2025
b6193fd
Nixed CSV helper
Alexander-Sol Jan 16, 2025
c865b20
Fixed mobilityscan index issues
Alexander-Sol Jan 16, 2025
8854769
Added comments
Alexander-Sol Jan 16, 2025
362f654
Added comments, addressed concerns
Alexander-Sol Jan 17, 2025
4d72987
Fixed failing test
Alexander-Sol Jan 17, 2025
015f8d8
Merge branch 'master' into timsTOFReader
trishorts Jan 17, 2025
f0e20d5
Merge branch 'master' into timsTOFReader
nbollis Jan 18, 2025
a373255
Merge branch 'master' into timsTOFReader
trishorts Jan 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 26 additions & 6 deletions mzLib/MassSpectrometry/MsDataScan.cs
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,29 @@ namespace MassSpectrometry
{
public class MsDataScan
{
public MsDataScan(MzSpectrum massSpectrum, int oneBasedScanNumber, int msnOrder, bool isCentroid, Polarity polarity, double retentionTime, MzRange scanWindowRange, string scanFilter, MZAnalyzerType mzAnalyzer,
double totalIonCurrent, double? injectionTime, double[,] noiseData, string nativeId, double? selectedIonMz = null, int? selectedIonChargeStateGuess = null, double? selectedIonIntensity = null, double? isolationMZ = null,
double? isolationWidth = null, DissociationType? dissociationType = null, int? oneBasedPrecursorScanNumber = null, double? selectedIonMonoisotopicGuessMz = null, string hcdEnergy = null, string scanDescription = null)
public MsDataScan(MzSpectrum massSpectrum,
int oneBasedScanNumber,
int msnOrder,
bool isCentroid,
Polarity polarity,
double retentionTime,
MzRange scanWindowRange,
string scanFilter,
MZAnalyzerType mzAnalyzer,
double totalIonCurrent,
double? injectionTime,
double[,] noiseData,
string nativeId,
double? selectedIonMz = null,
int? selectedIonChargeStateGuess = null,
double? selectedIonIntensity = null,
double? isolationMZ = null,
double? isolationWidth = null,
DissociationType? dissociationType = null,
int? oneBasedPrecursorScanNumber = null,
double? selectedIonMonoisotopicGuessMz = null,
string hcdEnergy = null,
string scanDescription = null)
{
OneBasedScanNumber = oneBasedScanNumber;
MsnOrder = msnOrder;
Expand Down Expand Up @@ -61,7 +81,7 @@ public MsDataScan(MzSpectrum massSpectrum, int oneBasedScanNumber, int msnOrder,
/// </summary>
public MzSpectrum MassSpectrum { get; protected set; }

public int OneBasedScanNumber { get; private set; }
public int OneBasedScanNumber { get; protected set; }
public int MsnOrder { get; }
public double RetentionTime { get; }
public Polarity Polarity { get; }
Expand All @@ -70,7 +90,7 @@ public MsDataScan(MzSpectrum massSpectrum, int oneBasedScanNumber, int msnOrder,
public string ScanFilter { get; }
public string NativeId { get; private set; }
public bool IsCentroid { get; }
public double TotalIonCurrent { get; }
public double TotalIonCurrent { get; protected set; }
public double? InjectionTime { get; }
public double[,] NoiseData { get; }

Expand All @@ -82,7 +102,7 @@ public MsDataScan(MzSpectrum massSpectrum, int oneBasedScanNumber, int msnOrder,
public double? SelectedIonMZ { get; private set; } // May be adjusted by calibration
public DissociationType? DissociationType { get; }
public double? IsolationWidth { get; }
public int? OneBasedPrecursorScanNumber { get; private set; }
public int? OneBasedPrecursorScanNumber { get; protected set; }
public double? SelectedIonMonoisotopicGuessIntensity { get; private set; } // May be refined
public double? SelectedIonMonoisotopicGuessMz { get; private set; } // May be refined
public string HcdEnergy { get; private set; }
Expand Down
12 changes: 12 additions & 0 deletions mzLib/MzLibUtil/ClassExtensions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,18 @@ public static T[] SubArray<T>(this T[] data, int index, int length)
return result;
}

public static bool ToEnum<T>(this int modeInt, out T result) where T : Enum
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just (EnumType)integer?

{
Type enumType = typeof(T);
if (!Enum.IsDefined(enumType, modeInt))
{
result = default(T);
return false;
}
result = (T)Enum.ToObject(enumType, modeInt);
return true;
}

/// <summary>
/// Checks if two collections are equivalent, regardless of the order of their contents
/// </summary>
Expand Down
1 change: 0 additions & 1 deletion mzLib/MzLibUtil/MzLibUtil.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="CsvHelper" Version="32.0.3" />
<PackageReference Include="Easy.Common" Version="6.7.0" />
<PackageReference Include="MathNet.Numerics" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.ObjectPool" Version="9.0.0" />
Expand Down
4 changes: 2 additions & 2 deletions mzLib/Readers/Bruker/BrukerFileReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -378,7 +378,7 @@ private List<StepsRow> GetFullStepsTable()
/// <param name="reader">SQLiteReader object, initialized after the execution of a command.</param>
/// <returns>Return null exception if there is an error in the data format of the baf file.</returns>
/// <exception cref="ArgumentNullException"></exception>
private T SqlColumnReader<T>(SQLiteDataReader reader) where T: new()
public static T SqlColumnReader<T>(SQLiteDataReader reader) where T: new()
{
// get all the property names, then iterate over that.
// The objects should be exact 1:1 column corresponding so as
Expand Down Expand Up @@ -516,7 +516,7 @@ private static void ThrowLastBaf2SqlError()
}

/* ----------------------------------------------------------------------------------------------- */
private static byte[] ConvertStringToUTF8ByteArray(String input)
public static byte[] ConvertStringToUTF8ByteArray(String input)
{
byte[] utf8 = Encoding.UTF8.GetBytes(input);
var result = new byte[utf8.Length + 1];
Expand Down
1 change: 1 addition & 0 deletions mzLib/Readers/MsDataFileReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ public static MsDataFile GetDataFile(string filePath)
SupportedFileType.MzML => new Mzml(filePath),
SupportedFileType.Mgf => new Mgf(filePath),
SupportedFileType.BrukerD => new BrukerFileReader(filePath),
SupportedFileType.BrukerTimsTof => new TimsTofFileReader(filePath),
_ => throw new MzLibException("File type not supported"),
};
}
Expand Down
17 changes: 15 additions & 2 deletions mzLib/Readers/Readers.csproj
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Platforms>x64</Platforms>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"In a C# project file (.csproj), the "AllowUnsafeBlocks" property within a "PropertyGroup" is used to enable the compilation of code containing "unsafe" blocks, which allows the use of pointers and direct memory manipulation, but should be used with caution due to the potential for memory management issues and security risks."


<ItemGroup>
<PackageReference Include="CsvHelper" Version="32.0.3" />
<PackageReference Include="OpenMcdf" Version="2.3.1" />
<PackageReference Include="OpenMcdf.Extensions" Version="2.3.1" />
<PackageReference Include="System.Data.SQLite" Version="1.0.118" />
<PackageReference Include="System.Runtime.InteropServices" Version="4.3.0" />
</ItemGroup>

<ItemGroup>
Expand All @@ -21,7 +23,6 @@
</ItemGroup>

<ItemGroup>

<Reference Include="ThermoFisher.CommonCore.BackgroundSubtraction">
<HintPath>Thermo\ThermoFisher.CommonCore.BackgroundSubtraction.dll</HintPath>
</Reference>
Expand Down Expand Up @@ -70,6 +71,18 @@
<None Update="Thermo\ThermoFisher.CommonCore.RawFileReader.dll">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
<None Update="timsdata.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="timsdata.lib">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="timsTOF\baf2sql_c.dll">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
<None Update="timsTOF\baf2sql_c.lib">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>

</Project>
2 changes: 0 additions & 2 deletions mzLib/Readers/Thermo/ThermoRawFileReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -526,8 +526,6 @@ private static DissociationType GetDissociationType(ActivationType activationTyp
}
}



/// <summary>
/// Gets all the MS orders of all scans in a dynamic connection. This is useful if you want to open all MS1 scans
/// without loading all of the other MSn scans.
Expand Down
15 changes: 12 additions & 3 deletions mzLib/Readers/Util/SupportedFileTypes.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
ThermoRaw,
MzML,
Mgf,
BrukerD,
psmtsv,
//osmtsv
ToppicPrsm,
Expand All @@ -27,7 +26,9 @@
MsPathFinderTDecoys,
MsPathFinderTAllResults,
CruxResult,
ExperimentAnnotation
ExperimentAnnotation,
BrukerD,
BrukerTimsTof
}

public static class SupportedFileTypeExtensions
Expand All @@ -51,6 +52,7 @@
SupportedFileType.MzML => ".mzML",
SupportedFileType.Mgf => ".mgf",
SupportedFileType.BrukerD => ".d",
SupportedFileType.BrukerTimsTof => ".d",
SupportedFileType.psmtsv => ".psmtsv",
//SupportedFileType.osmtsv => ".osmtsv",
SupportedFileType.ToppicPrsm => "_prsm.tsv",
Expand All @@ -76,7 +78,14 @@
case ".raw": return SupportedFileType.ThermoRaw;
case ".mzml": return SupportedFileType.MzML;
case ".mgf": return SupportedFileType.Mgf;
case ".d": return SupportedFileType.BrukerD;
case ".d":
if(!Directory.Exists(filePath)) throw new FileNotFoundException();
var fileList = Directory.GetFiles(filePath).Select(p => Path.GetFileName(p));
if (fileList.Any(file => file == "analysis.baf"))
return SupportedFileType.BrukerD;
if (fileList.Any(file => file == "analysis.tdf"))
return SupportedFileType.BrukerTimsTof;
throw new MzLibException("Bruker file type not recognized");

Check warning on line 88 in mzLib/Readers/Util/SupportedFileTypes.cs

View check run for this annotation

Codecov / codecov/patch

mzLib/Readers/Util/SupportedFileTypes.cs#L88

Added line #L88 was not covered by tests
case ".psmtsv": return SupportedFileType.psmtsv;
//case ".osmtsv": return SupportedFileType.osmtsv;
case ".feature":
Expand Down
Loading
Loading