Inconsistent build path (cmake vs run_sample.py) on Windows

[AmolSakhale]

As a first step, I am trying to run the model using run_sample.py and generate the output files.
But I am getting this error. The error doesn't specify which file is missing.
Did anyone faced similar problem or can tell what I might be doing wrong?
ErrorOutput.txt

[weshinsley]

Nothing immediately obvious from that output - although personally, I have never had much luck with Anaconda.

I am tied up with other things right now but will come back to this - could you add some debug around run_sample.py:245 and see what "exe" is - I'm guessing that from Anaconda it just can't find the executable you build for some reason....

[zebmason]

FWIW it used to work in Anaconda after I submitted #254 to allow passing in of the location of the Windows executable as a command line argument. Obviously those files no longer exist.

[AmolSakhale]

Thank you for your reply.
I am actually using Microsoft Visual Studio.
I installed visual studio with required packages and all like CMake, C, C++ compilers.
Then cloned the repository.
And now simply trying to execute run_sample.py
The code was executed without any error until line 240, It means it could build the model, find required files as code above this didn't give an error.
Line 240 in run_sample.py is process = subprocess.run(cmd, check=True) where cmd is the command created for No intervention case.
and cmd is

Command line:
C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\build\Debug\src\CovidSim.exe
/c:8 /A:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\admin_units\United_Kingdom_admin.txt /PP:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\preUK_R0=2.0.txt /P:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\p_NoInt.txt /O:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_Kingdom_NoInt_R0=3.0 /D:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\wpop_eur.txt
/M:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_Kingdom_pop_density.bin /S:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\Network_United_Kingdom_T8_R3.0.bin
/R:1.5
98798150 729101 17389101 4797132

So, @weshinsley I won't be able to reach line:245 as the error is coming before that.

What IDE and what required packages are you using? I can install the suggested software.

[zebmason]

CI for Windows uses standalone CMake and CTest - see windows_distribution_check.yml

[zebmason]

Just run from an Anaconda prompt

cd C:\Projects\GitHub\covid-sim\out\build\x64-Debug\tests
python ..\..\..\..\tests\integration-test.py --input C:\Projects\GitHub\covid-sim\tests\uk-input --output C:\Projects\GitHub\covid-sim\out\build\x64-Debug\tests\uk-output-j1 --checksums C:\Projects\GitHub\covid-sim\tests\uk-input\results-j1.cksum --covidsim C:\Projects\GitHub\covid-sim\out\build\x64-Debug\src\CovidSim.exe --popfile C:\Projects\GitHub\covid-sim\data\populations\wpop_eur.txt.gz --r 1.1 -j 1

which was based on the contents of

C:\Projects\GitHub\covid-sim\out\build\x64-Debug\EBF.Tests.5248968360651308223.inttest-uk-based-j1.4220.json

[zebmason]

Sorry I've just realised that you are referring to a different Python file than I thought you were. This particular file hasn't been updated since 2020/05/13, is not tested by CI and it is possible that the inputs to CovidSim.exe have changed sufficiently in the meantime to break the script. In other words I'll let someone else help you with this as I don't know whether it is maintained.

[weshinsley]

I know you're using Visual Studio from the text - but I am wondering why I am seeing references to Anaconda3\lib. Perhaps that's a red herring anyway.

So, @weshinsley I won't be able to reach line:245 as the error is coming before that.

I am seeing on your second and third screenshots, an error in covidsim/data/run_sample.py on line 245, so I want to know what's going on around that line that would cause it to fail. However, it looks like (as zebmason said) you're running an old version - since mid-May, line 245 has been this:

cmd = [
            exe,
            "/c:{0}".format(args.threads),
            "/A:" + admin_file
            ]

whereas I think your error is just a File not Found coming from the subprocess.run which is now line 266.

Command line:
C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\build\Debug\src\CovidSim.exe

So does this file exist? I am surprised it is compiling in debug mode, or looking in the Debug folder - my cmake produces build\src\CovidSim.exe.

Moreover, it will be easier for us to debug if you pull master...

[AmolSakhale]

@weshinsley Thank you for your comments.
I could solve the problem.
I added a few prints statements above because of which line 240 became line 245.

Yes, problem was with the location where the CovidSim.exe is getting stored.

In the code the location is

if os.name == 'nt':
exe = os.path.join(build_dir, "Debug", "src", "CovidSim.exe")
else:
exe = os.path.join(build_dir, "src", "CovidSim")

whereas in my Windows 10 OS,
It was getting saved here
C:\Users\amols\source\repos\AmolSakhale\covid-sim\build\src\Debug\CovidSim.exe

I just made following change,

if os.name == 'nt':
exe = os.path.join(build_dir, "src", "Debug", "CovidSim.exe")
else:
exe = os.path.join(build_dir, "src", "CovidSim")

I think this change needs to be made in run_sample.py
Thank you for your help.

[weshinsley]

Ah, ok - so the executable in the top screenshot built to src\Debug\CovidSim.exe, but run_sample.py has the path differently...

[weshinsley]

We'll work out which of these to fix - thanks for reporting.

[AmolSakhale]

I encountered another error while running the CovidSim.exe with given parameters.
Can you please help me with this?

It said, [C:\Users\amols\source\repos\AmolSakhale\covid-sim\src\SetupModel.cpp line 1078] Unable to allocate cell storage

Log:

-- Building for: Visual Studio 16 2019
-- The C compiler identification is MSVC 19.26.28805.0
-- The CXX compiler identification is MSVC 19.26.28805.0
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.26.28801/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.26.28801/bin/Hostx64/x64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.26.28801/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.26.28801/bin/Hostx64/x64/cl.exe - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP_CXX: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0")
-- Found Python3: C:/Program Files/WindowsApps/PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0/python3.8.exe (found version "3.8.3") found components: Interpreter
-- Found Doxygen: C:/Program Files/doxygen/bin/doxygen.exe (found version "1.8.18 (a1b07ad0e92e4526c9ba1711d39f06b58c2a7459)") found components: doxygen missing components: dot
Doxygen build started
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/amols/source/repos/AmolSakhale/covid-sim/build
Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Checking Build System
Building Custom Rule C:/Users/amols/source/repos/AmolSakhale/covid-sim/src/CMakeLists.txt
CovidSim.cpp
BinIO.cpp
Rand.cpp
Error.cpp
Dist.cpp
Kernels.cpp
Bitmap.cpp
SetupModel.cpp
C:\Users\amols\source\repos\AmolSakhale\covid-sim\src\SetupModel.cpp(704): warning C4101: 'k': unreferenced local variable [C:\Users\amols\source\repos\AmolSakhale\covid-sim\build\src\CovidSim.vcxproj]
CalcInfSusc.cpp
Sweep.cpp
Update.cpp
Param.cpp
MicroCellPosition.cpp
Direction.cpp
Vector2.cpp
Generating Code...
CovidSim.vcxproj -> C:\Users\amols\source\repos\AmolSakhale\covid-sim\build\src\Debug\CovidSim.exe
Building Custom Rule C:/Users/amols/source/repos/AmolSakhale/covid-sim/CMakeLists.txt
No intervention: United_States NoInt 3.0
Command line: C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\build\src\Debug\CovidSim.exe /c:8 /A:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\admin_units\United_States_admin.txt /s:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\populations\USschools.txt /PP:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\preUS_R0=2.0.txt /P:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\p_NoInt.txt /O:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_NoInt_R0=3.0 /D:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\wpop_usacan.txt /M:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_pop_density.bin /S:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\Network_United_States_T8_R3.0.bin /R:1.5 98798150 729101 17389101 4797132
sizeof(int)=4 sizeof(long)=4 sizeof(float)=4 sizeof(double)=8 sizeof(unsigned short int)=2 sizeof(int *)=8
Param=C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\p_NoInt.txt
Out=C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_NoInt_R0=3.0
Dens=C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\wpop_usacan.txt
Bitmap Format = *.png
Using 8 threads
Thread 1 initialised
Thread 0 initialised
Thread 2 initialised
Thread 6 initialised
Thread 7 initialised
Thread 3 initialised
Thread 4 initialised
Thread 5 initialised
Update step = 0.250000
Sampling step = 1.000000
Updates per sample=4
TimeStepsPerDay=4.000000
Using 49 administrative units
Parameters read
Scanning population density file
Adjusted bounding box = (-124.725, 24.525)- (-66.825, 49.425)
Number of cells = 256304 (772 x 332)
Population size = 338798532
Number of microcells = 20760624
Bitmap width = 3476
Bitmap height = 2244
Coords xmcell=739.863 m ymcell = 926.107 m
Density file contains 5932201 datapoints.
5005243 valid microcells read from density file.
Binary density file should contain 5005243 microcells.
Saving population density file with NC=5005243...
Population files read.
Number of hosts assigned = 338798532
Number of cells with non-zero population = 110384
Number of microcells with non-zero population = 4478819
sizeof(Person)=128
[C:\Users\amols\source\repos\AmolSakhale\covid-sim\src\SetupModel.cpp line 1078] Unable to allocate cell storage

Command '['C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\build\src\Debug\CovidSim.exe', '/c:8', '/A:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\admin_units\United_States_admin.txt', '/s:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\populations\USschools.txt', '/PP:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\preUS_R0=2.0.txt', '/P:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\p_NoInt.txt', '/O:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_NoInt_R0=3.0', '/D:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\wpop_usacan.txt', '/M:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_pop_density.bin', '/S:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\Network_United_States_T8_R3.0.bin', '/R:1.5', '98798150', '729101', '17389101', '4797132']' returned non-zero exit status 1.
Stack trace:

File "C:\Users\amols\source\repos\AmolSakhale\covid-sim\data\run_sample.py", line 240, in
process = subprocess.run(cmd, check=True)
Loaded 'main'
Loaded 'runpy'
Traceback (most recent call last):
File "C:\Users\amols\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\amols\Anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "c:\program files (x86)\microsoft visual studio\2019\community\common7\ide\extensions\microsoft\python\core\debugpy_main.py", line 45, in
cli.main()
File "c:\program files (x86)\microsoft visual studio\2019\community\common7\ide\extensions\microsoft\python\core\debugpy/..\debugpy\server\cli.py", line 429, in main
run()
File "c:\program files (x86)\microsoft visual studio\2019\community\common7\ide\extensions\microsoft\python\core\debugpy/..\debugpy\server\cli.py", line 266, in run_file
runpy.run_path(options.target, run_name=compat.force_str("main"))
File "C:\Users\amols\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\amols\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\amols\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\run_sample.py", line 240, in
process = subprocess.run(cmd, check=True)
File "C:\Users\amols\Anaconda3\lib\subprocess.py", line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessErrorThe thread 'MainThread' (0x1) has exited with code 0 (0x0).
: Command '['C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\build\src\Debug\CovidSim.exe', '/c:8', '/A:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\admin_units\United_States_admin.txt', '/s:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\populations\USschools.txt', '/PP:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\preUS_R0=2.0.txt', '/P:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\data\param_files\p_NoInt.txt', '/O:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_NoInt_R0=3.0', '/D:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\wpop_usacan.txt', '/M:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\United_States_pop_density.bin', '/S:C:\Users\amols\Source\Repos\AmolSakhale\covid-sim\Network_United_States_T8_R3.0.bin', '/R:1.5', '98798150', '729101', '17389101', '4797132']' returned non-zero exit status 1.
The program 'python.exe' has exited with code 1 (0x1).

[weshinsley]

You may be running out of memory... how much RAM do you have?

But I'm still not sure you're on master? SetupModel.cpp:1078 is just a close brace.

[weshinsley]

It looks like you're trying to do a full US run - I forget exactly how much RAM that needs, but for us, that would be one for a cluster node (all of ours are >128Gb), not a desktop.

[AmolSakhale]

Thank you for your reply.
Yes, I was trying to run it on my local and have 16 Gb of RAM. Can you please me how much memory will it need to run for the full US? How much time does it take?
Can you please tell me how can I run it for the smaller part of the US? Some specific regions, let's say California state.
It would be big help.

Thank you for reminding me, I pulled the changes from the master. I was using 15 days old repository.

[weshinsley]

Have a look in the regression tests for an example of a single state (Wyoming) - see lines 28-32 of
https://github.com/mrc-ide/covid-sim/blob/master/tests/us-input/admin-params.txt, and compare with https://github.com/mrc-ide/covid-sim/blob/master/data/admin_units/United_States_admin.txt to choose your state.

Memory for full USA - quick guess, full UK takes 20Gb, so roughly scale up by 5 and I'd say >=100Gb.

[AmolSakhale]

Hello @weshinsley,
Thank you for your comments.
It had helped me a lot.
Is it okay to continue asking my doubts in this thread?

I have a few questions.

1. How can I generate output files without severity analysis?
   I have tried putting
   [OutputNonSeverity]
   1
   and
   [Do Severity Analysis]
   0

in covid-sim\tests\us-input\pre-param.txt file
But after running the integration-test.py, it gave an error after some time.
The error is as follows,

Can you please tell me what can be the problem?

2. As there is an age group wise number of infected people data in age.xls, similarly can I generate the age group wise number of susceptible people in this file? Is there any way to get such data?

[weshinsley]

Separate issue might be better really, as your error above is a separate bug, so we can deal with them one by one. We're still very busy, so not really much time capacity for this, but:-

1. Is there a pressing reason you want to disable severity? I'm not sure at the moment what changing those parameters would do; it shouldn't crash as above so ll put that bug on the list (in a separate issue), but I don't think that's a way that the code would usually be used.

2. It doesn't really work like that; people's individual susceptibility is calculated on the fly at the point when they might become infected, so there is no single group of "susceptible people" by age that you could output over time. It'll be a combination of the demographic properties of the population (in the admin file), initial age-specific immunity (if that's switched on - that'll be in the params file somewhere), and then anything in the simulation that might affect susceptibility of individuals. See CalcInfSusc.cpp.

[weshinsley]

The issue with the crash window is that you can't have DoSeverity switched off, but DoICUTriggers switched on... (can't have ICUs without severe cases)