Slow torch_module_forward

[yohyamamoto]

I'm trying to use FTorch to load my trained network in a fortran code. I followed the installation guide and the example fortran code in the README file for using the model. So far I'm finding a very slow performance where the torch_module_forward call is taking significant time (much slower than calculating O(n**4) integrals). Is this normal? If not, is there a way to speed this up? Any suggestions would be appreciated.

I'm running this on my desktop (CPU machine).
cmake version 3.27.4
pytorch version 2.1.0+cpu
gcc version 13.2.1

My input array has the size of 80000x6 and output array is 80000x1
My model on pytorch looks like this:
Sequential(
(0): Linear(in_features=6, out_features=100, bias=True)
(1): ReLU()
(2): Linear(in_features=100, out_features=100, bias=True)
(3): ReLU()
(4): Linear(in_features=100, out_features=100, bias=True)
(5): ReLU()
(6): Linear(in_features=100, out_features=1, bias=True)
)

[jatkinson1000]

Hi @yohyamamoto thanks for getting in touch.
We typically see comparable performance between using FTorch calling TorchScript and running the net in Torch.

Are you saying that the inference call is slower compared to running inference in python, or that it is slower than a physics-based Fortran scheme that you are trying to emulate?
Have you tried timing both of these - we have been doing some benchmarking here (work in progress) which may be of interest?

Without seeing the code it is difficult to suggest what, if anything, the issue might be.
Could you provide a link to the code in a repository for us to look at?

[yohyamamoto]

Thank you for your comments and suggestions!

After running the subroutine call ten times in a loop, I see that there is drop in time. The drop is more significant in the system_clock timing.

• timing for looping 10 times:

no.  cpu_time                            system_clock (integer(8))
1    0.24745100000000003                 112747781
2    0.16257199999999994                  43982154
3    0.16426900000000011                  42551195
4    0.15381500000000004                  38858507
5    0.13767799999999997                  34746265
6    0.13468000000000013                  34460029
7    0.13756900000000005                  34468246
8    0.13485400000000025                  34576739
9    0.13754500000000025                  34737416
10   0.13749399999999978                  34557110

After more careful observation, I noticed that although the using system call the python code gives a short cpu time, the actual wall time is terrible. This is likely due to the system time that are not accounted by the cpu time. But this is okay since avoiding this overhead is the reason why we want to use FTorch. Overall, this FTorch timing is good for our case.

At the moment, when I call the same Ftorch interface subroutine within our quantum chemistry code, the timing somehow becomes one order longer in both cpu time and system clock time. Also, the timing doesn't go down after the first call either. I do not understand yet why that is.

• Time measured after interfacing the same subroutine to our quantum chemistry code:

cpu_time                          system_clock (integer(8))
1.3374880000000005                1222582940

Following your comment, I removed if (first) then and initialized the model at the beginning. I did not see effect on timing though.

[jatkinson1000]

OK, glad we are making progress!
The observation about wall time is interesting, thanks - cpu-time may just be timing the system call processes, but not the content that gets called.

As for the slow-down in 'real' code it's hard to say without seeing but a few thoughts:

• Are you running it on a local machine or on a cluster?
• If on a cluster is the saved net and FTorch on the compute storage rather than home/login storage?
• Are you instantiating the net just once at the start of the program, or every time you get to this section of the code (this is why we have the separate net_init and net_finalize routines called only once at the startup and teardown of the program - I can perhaps call and discuss with you in more detail if you wanted)? If the latter you will be getting the slowdown due to first iteration each time.
• Finally, is this an MPI code?

[ElliottKasoar]

Not the solution, but something else to be aware of is that cpu_time is not thread aware, so the timings may be less reliable than using something like omp_get_wtime.

(See @TomMelt's comment)

[jatkinson1000]

@yohyamamoto have you had any progress with this?
We have just merged a sizeable PR that updates the interface to simplify the process of creating Tensors from Fortran data.

[yohyamamoto]

@ElliottKasoar I haven't made much progress with the speed in the original post. (Tried different I/O, compilation flags, etc.)

For now, we are trying a different way to approach the problem. Instead of using the neural network to obtain energy density (array length of ~80000 at a time) from electron density, we are trying to get energy (integrated quantity, hence a short array length at a time) right now. The speed is not a problem for this case.

We are still fairly new to neural network and experimenting what is a good application of neural network in our Fortran code.
Also, thank you for the update to the interface. I will definitely take a look.

[jatkinson1000]

@yohyamamoto Did you make any progress with this?

Without more information I still have a suspicion that you need to instantiate the net only once rather than at each iteration.,
I would like to add an example for how to do this properly at some point.

As there has been no update in 6 months I am closing this, but please do re-open if you wish to follow up.