Problems importing GMTSAR data #73

WangJunyi2022 · 2022-04-08T10:23:03Z

Hello，
I processed the image using GMTSAR, and after processing, converted the los.grd deformation to m, and generated the los.enu data. But after I import the data, the following problem occurs.

(grond) wang@K:/media/wang/dec2022/merge$ spool --load unwrap_ll.grd
Traceback (most recent call last):
  File "/home/wang/anaconda3/envs/grond/bin/spool", line 8, in <module>
    sys.exit(main())
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/__main__.py", line 95, in main
    spool(import_file=ns.load)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/spool.py", line 405, in spool
    spool_app = Spool(*args, **kwargs)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/spool.py", line 48, in __init__
    self.importScene(import_file)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/spool.py", line 64, in importScene
    self.spool_win.model.importFile(filename)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/scene_model.py", line 174, in importFile
    self.setScene(Scene.import_data(filename))
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/scene.py", line 494, in dynclassmethod
    return func(Scene(), *args, **kwargs)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/scene.py", line 1013, in _import_data
    data = module.read(path, **kwargs)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/scene_io.py", line 815, in read
    grd = netcdf.netcdf_file(self._getDisplacementFile(path),
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/scipy/io/_netcdf.py", line 277, in __init__
    self._read()
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/scipy/io/_netcdf.py", line 598, in _read
    raise TypeError("Error: %s is not a valid NetCDF 3 file" %
TypeError: Error: /media/wang/WJY/menyuandec2022/merge/unwrap_ll.grd is not a valid NetCDF 3 file

How can I import the data?

The text was updated successfully, but these errors were encountered:

hvasbath · 2022-04-08T12:47:20Z

Hi @WangJunyi2022!

Thank you for your message!
The problem is that likely the newer GMTSAR version stores the data in netcdf4 right? Can you please check?
If that is the case then Kite would need to be extended to be able to read netcdf4, as the scipy netcdf reader (used in kite) does not support netcdf4. That would need to be added to scene_io.py line 815.
I quick workaround would be if you could somehow tell GMTSAR to store the data in netcdf3 format.

Regards!
Hannes

wasjabloch · 2022-04-08T16:16:10Z

xarray handles NetCDF4 super well. It is very powerful and mature.

WangJunyi2022 · 2022-04-14T08:54:50Z

Thanks! It is the version of InSAR that I use, and there is no problem with GMT5SAR.

hvasbath · 2022-04-14T09:04:27Z

InSAR has no version. The problem is that GMTSAR stores the data in netcdf4 format, which is currently not supported by KITE. So either you need to find out if GMTSAR can also store data in netcdf3 or you/someone need to work on a patch for KITE. As we, the maintainers of KITE are up to the brim with work- we wont have time to do that for you in the near time.

WangJunyi2022 · 2022-04-14T13:22:35Z

Sorry, I spelt it wrong. I used GMTSAR version 6.1. I would like to say that due to the software version, the gmtsar6 default generated file is netcdF4 and cannot be opened with kite. But there should be no problem using the kite to turn on the GMTSAR5 for data processing. In addition, when installing gmtsar5, I could not install it on ubuntu20 due to version migration, but I installed gmtsar5.8 on ubuntu16. (I hope my mistake will save others some trouble.)
Good luck !

Meysam-Amiri · 2023-07-24T14:22:39Z

I have a solution for importing GMTSAR data into Kite. Here's a step-by-step guide on how to do it:

Copy the following files from the merge directory into a separate directory:
- los_ll.grd
- supermaster.PRM
- S1_20211109_ALL_F3.LED
- dem.grd
Create a new file named "gmt.conf" and add the following lines to it:

#
# GMT 6.3.0 Defaults file
#
# COLOR Parameters
#
COLOR_MODEL                    = hsv
#
# FORMAT Parameters
#
FORMAT_GEO_OUT                 = D
#
# I/O Parameters
#
IO_NC4_CHUNK_SIZE              = classic
#
# PostScript Parameters
#
PS_MEDIA                       = a2

Note that you can Replace the "IO_NC4_CHUNK_SIZE" value in the existing "gmt.conf" file from the merge directory with the line:

IO_NC4_CHUNK_SIZE              = classic

By setting "IO_NC4_CHUNK_SIZE" to "classic," the grd files will be saved in Netcdf3 format.

Convert the "los_ll.grd" file from millimeters (mm) to meters (m):

gmt grdmath los_ll.grd 0.001 MUL = los_ll_m.grd

Extract the lon, lat, and elevation data from the "dem.grd" file and apply it to the converted "los_ll_m.grd" file:

gmt grd2xyz los_ll_m.grd | gmt grdtrack -Gdem.grd | awk '{print $1, $2, $4}' | SAT_look supermaster.PRM -bos > 20211109.los.enu

Save the "los_ll_m.grd" as a file np).

spool --load los_ll_m.grd
save los_ll_m.npz

After following these steps, you should have successfully imported the GMTSAR data into Kite, and the results will be saved in Netcdf3 format and a npz file.
If you would like to cut grd file for a small region:
gmt grdcut los_ll_m.grd -Rmin_lon/max_lon/min_lat/max_lat -Glos_ll_m_cut.grd

ditafaith · 2023-08-03T11:22:46Z

gmt grd2xyz los_ll_m.grd | gmt grdtrack -Gdem.grd | awk '{print $1, $2, $4}' | SAT_look supermaster.PRM -bos > 20211109.los.enu

Hi @Meysam-Amiri ,
I am using ALOS data, is there any difference command ?
what is SAT_look ? and what is the purpose to create gmt.conf file ?

Meysam-Amiri · 2023-08-03T12:07:19Z

gmt grd2xyz los_ll_m.grd | gmt grdtrack -Gdem.grd | awk '{print $1, $2, $4}' | SAT_look supermaster.PRM -bos > 20211109.los.enu

Hi @Meysam-Amiri , I am using ALOS data, is there any difference command ? what is SAT_look ? and what is the purpose to create gmt.conf file ?

Hi @ditafaith. I am not sure about ALOS data. But, I think the procedure would be the same.
SAT_look is one of the GMTSAR commands (https://github.com/gmtsar/gmtsar/blob/master/gmtsar/SAT_look.c).
The purpose of creating gmt.conf with new adding IO_NC4_CHUNK_SIZE = classic is to create required grd files in Netcdf 3 format which is readable by kite.

ditafaith · 2023-08-03T12:18:05Z

@Meysam-Amiri
I got the point, yes..it is running. However there is no image appears. It looks similar when I tried to run example from kite's website.

Meysam-Amiri · 2023-08-03T12:39:41Z

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

ditafaith · 2023-08-03T13:33:11Z

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

What do you mean import to kite ?
I just follow the guidance, directly open file from gmtsar.
https://pyrocko.org/kite/docs/current/quickstart.html

Meysam-Amiri · 2023-08-03T13:59:54Z

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

What do you mean import to kite ? I just follow the guidance, directly open file from gmtsar. https://pyrocko.org/kite/docs/current/quickstart.html

You must prepare some files before using spool --load command. Please read where it explained GMTSAR: https://pyrocko.org/kite/docs/current/reference/kite.scene.html#module-kite.scene_io
GMTSAR

Reading GMTSAR grid files.

Note Expects:
Displacement grid (NetCDF, *los_ll.grd) in meter (in case use “gmt grdmath los_cm_ll.grd 0.01 MUL = los_m_ll.grd’)

LOS binary data (see instruction, *.los.enu)

Calculate the corresponding unit look vectors with GMT5SAR SAT_look:

gmt grd2xyz los_ll.grd | gmt grdtrack -Gdem.grd |
awk {'print $1, $2, $4'} |
SAT_look 20050731.PRM -bos > 20050731.los.enu

ditafaith · 2023-08-03T14:11:43Z

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

What do you mean import to kite ? I just follow the guidance, directly open file from gmtsar. https://pyrocko.org/kite/docs/current/quickstart.html

You must prepare some files before using spool --load command. Please read where it explained GMTSAR: https://pyrocko.org/kite/docs/current/reference/kite.scene.html#module-kite.scene_io GMTSAR

Reading GMTSAR grid files.

Note Expects: Displacement grid (NetCDF, *los_ll.grd) in meter (in case use “gmt grdmath los_cm_ll.grd 0.01 MUL = los_m_ll.grd’)

LOS binary data (see instruction, *.los.enu)

Calculate the corresponding unit look vectors with GMT5SAR SAT_look:

gmt grd2xyz los_ll.grd | gmt grdtrack -Gdem.grd | awk {'print $1, $2, $4'} | SAT_look 20050731.PRM -bos > 20050731.los.enu

Yes, I did.
This is similar as you explained in this threads as well.
I changed to netcdf3 for los_ll files.
However, I can't read *.los.enu file due to binary file. Is it correct ?
There is no different size between original los file and the new los in netcdf3 version.

Meysam-Amiri · 2023-08-03T19:40:56Z

What is the output of gmt grdinfo los_ll*.grd? No, it does not change the size.

ditafaith · 2023-08-06T14:00:21Z

I'd be glad to getting insighful comments from @Meysam-Amiri and others regarding this case.

Meysam-Amiri · 2023-08-06T14:27:44Z

I can check it if you share your results with me. Here is my email address : ***@***.***

…

On Sun, Aug 6, 2023, 17:30 ditafaith ***@***.***> wrote: I'd be glad to getting insighful comments from @Meysam-Amiri <https://github.com/Meysam-Amiri> and other regarding this case. — Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AY6U2KYADEKBHXEVLEYPPQ3XT6PQBANCNFSM5S4DMRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

ditafaith · 2023-08-06T14:32:06Z

I can check it if you share your results with me. Here is my email address
: @.***

On Sun, Aug 6, 2023, 17:30 ditafaith @.***> wrote:

I'd be glad to getting insighful comments from @Meysam-Amiri
https://github.com/Meysam-Amiri and other regarding this case.

—
Reply to this email directly, view it on GitHub
#73 (comment), or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AY6U2KYADEKBHXEVLEYPPQ3XT6PQBANCNFSM5S4DMRGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

You can't post email address through github, since they automatically changed *.
Do you mean result netcdf3 from GMTSAR ?

Meysam-Amiri · 2023-08-06T14:43:11Z

[email protected]
Yes please send me grd file. Maybe I can ssh and produce grd files. Anyway, Let's be in contact and solve it through email.

Meysam-Amiri · 2023-08-06T21:24:37Z

You can also look at https://hive.pyrocko.org/pyrocko-support/channels/kite and ask there for help. Please DM me there to chat easily.

hsudha self-assigned this Jun 14, 2022

Yaoshangheng mentioned this issue Nov 26, 2022

How to stores the data in netcdf3 format gmtsar/gmtsar#578

Closed

CUMT530849827 mentioned this issue Nov 16, 2024

look vectors problem #128

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems importing GMTSAR data #73

Problems importing GMTSAR data #73

WangJunyi2022 commented Apr 8, 2022 •

edited by miili

Loading

hvasbath commented Apr 8, 2022

wasjabloch commented Apr 8, 2022 •

edited

Loading

WangJunyi2022 commented Apr 14, 2022

hvasbath commented Apr 14, 2022

WangJunyi2022 commented Apr 14, 2022

Meysam-Amiri commented Jul 24, 2023

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023 •

edited

Loading

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023

ditafaith commented Aug 6, 2023 •

edited

Loading

Meysam-Amiri commented Aug 6, 2023 via email

ditafaith commented Aug 6, 2023

Meysam-Amiri commented Aug 6, 2023

Meysam-Amiri commented Aug 6, 2023

Problems importing GMTSAR data #73

Problems importing GMTSAR data #73

Comments

WangJunyi2022 commented Apr 8, 2022 • edited by miili Loading

hvasbath commented Apr 8, 2022

wasjabloch commented Apr 8, 2022 • edited Loading

WangJunyi2022 commented Apr 14, 2022

hvasbath commented Apr 14, 2022

WangJunyi2022 commented Apr 14, 2022

Meysam-Amiri commented Jul 24, 2023

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023 • edited Loading

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023

ditafaith commented Aug 3, 2023

Meysam-Amiri commented Aug 3, 2023

ditafaith commented Aug 6, 2023 • edited Loading

Meysam-Amiri commented Aug 6, 2023 via email

ditafaith commented Aug 6, 2023

Meysam-Amiri commented Aug 6, 2023

Meysam-Amiri commented Aug 6, 2023

WangJunyi2022 commented Apr 8, 2022 •

edited by miili

Loading

wasjabloch commented Apr 8, 2022 •

edited

Loading

Meysam-Amiri commented Aug 3, 2023 •

edited

Loading

ditafaith commented Aug 6, 2023 •

edited

Loading