Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG] run_regularly operations on the CPU #97

Merged
merged 7 commits into from
Nov 7, 2019
Merged

[MRG] run_regularly operations on the CPU #97

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d50b8cb
WIP: More general support for run_regularly operations
mstimberg Sep 2, 2019
d10cb85
Support run_regularly operations on synapses
mstimberg Oct 23, 2019
fa441a2
Support multiple run_regularly operations per NeuronGroup/Synapses
mstimberg Oct 24, 2019
e02a54e
Correctly order run_regularly and StateMonitor operations
mstimberg Oct 24, 2019
0f57fb7
Only push/pull states to/from GPU once per time step
mstimberg Oct 24, 2019
10fa44a
Update docs about `TimedArray`
mstimberg Oct 24, 2019
05b34bf
Only push/pull variables for run_regularly operations when needed
mstimberg Nov 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 116 additions & 88 deletions brian2genn/device.py
View file
Open in desktop

Large diffs are not rendered by default.

142 changes: 113 additions & 29 deletions brian2genn/templates/engine.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,46 +108,106 @@ void engine::run(double duration, //!< Duration of time to run the model for
double elapsed_realtime;

for (int i= 0; i < riT; i++) {
// report state
{% for sm in state_monitor_models %}
{% if sm.when == 'start' %}
{% for var in sm.variables %}
{% if sm.isSynaptic %}
{% if sm.connectivity == 'DENSE' %}
convert_dense_matrix_2_dynamic_arrays({{var}}{{sm.monitored}}, {{sm.srcN}}, {{sm.trgN}},brian::_dynamic_array_{{sm.monitored}}__synaptic_pre, brian::_dynamic_array_{{sm.monitored}}__synaptic_post, brian::_dynamic_array_{{sm.monitored}}_{{var}});
// The StateMonitor and run_regularly operations are ordered by their "order" value
{% for is_state_monitor, obj in run_reg_state_monitor_operations %}
{% if is_state_monitor %}
{% if obj.when == 'start' %}
{% for var in obj.variables %}
{% if obj.isSynaptic %}
{% if obj.connectivity == 'DENSE' %}
convert_dense_matrix_2_dynamic_arrays({{var}}{{obj.monitored}},
{{obj.srcN}}, {{obj.trgN}},
brian::_dynamic_array_{{obj.monitored}}__synaptic_pre,
brian::_dynamic_array_{{obj.monitored}}__synaptic_post,
brian::_dynamic_array_{{obj.monitored}}_{{var}});
{% else %}
convert_sparse_synapses_2_dynamic_arrays(C{{sm.monitored}}, {{var}}{{sm.monitored}}, {{sm.srcN}}, {{sm.trgN}}, brian::_dynamic_array_{{sm.monitored}}__synaptic_pre, brian::_dynamic_array_{{sm.monitored}}__synaptic_post, brian::_dynamic_array_{{sm.monitored}}_{{var}}, b2g::FULL_MONTY);
convert_sparse_synapses_2_dynamic_arrays(C{{obj.monitored}},
{{var}}{{obj.monitored}},
{{obj.srcN}}, {{obj.trgN}},
brian::_dynamic_array_{{obj.monitored}}__synaptic_pre,
brian::_dynamic_array_{{obj.monitored}}__synaptic_post,
brian::_dynamic_array_{{obj.monitored}}_{{var}},
b2g::FULL_MONTY);
{% endif %}
{% else %}
copy_genn_to_brian({{var}}{{sm.monitored}}, brian::_array_{{sm.monitored}}_{{var}}, {{sm.N}});
copy_genn_to_brian({{var}}{{obj.monitored}}, brian::_array_{{obj.monitored}}_{{var}}, {{obj.N}});
{% endif %}
{% endfor %}
_run_{{sm.name}}_codeobject();
_run_{{obj.name}}_codeobject();
{% endif %}
{% endfor %}
// Execute scalar code for run_regularly operations (if any)
{% for nm in neuron_models %}
{% if nm.run_regularly_object != None %}
if (i % {{nm.run_regularly_step}} == 0)
{% else %}
if (i % {{obj['step']}} == 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be sure - is the intended behaviour that run_regulary timing is based on the local iteration variable i? Rather than on the global integer time step iT? I can't remember how we determine riT and whether the loop would ever be executed multiple times?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We currently do not allow for multiple runs, so i and iT should always be identical. I think I prefer to use i here, since iT is a bit less straightforward (it gets updated as part of stepTimeGPU/stepTimeCPU, etc.).

{
{% for var in nm.run_regularly_read %}
{% if var == 't' %}
copy_genn_to_brian(&t, brian::_array_{{nm.clock.name}}_t, 1);
{% elif var == 'dt' %}
{# nothing to do #}
{% else %}
copy_genn_to_brian(&{{var}}{{nm.name}}, brian::_array_{{nm.name}}_{{var}}, 1);
{% endif %}
{% endfor %}
_run_{{nm.run_regularly_object.name}}();
{% for var in nm.run_regularly_write %}
copy_brian_to_genn(brian::_array_{{nm.name}}_{{var}}, &{{var}}{{nm.name}}, 1);
{% endfor %}
// Execute run_regularly operation: {{obj['name']}}
{% for var in obj['read'] %}
{% if var == 't' %}
copy_genn_to_brian(&t, brian::_array_{{obj['owner'].clock.name}}_t, 1);
{% elif var == 'dt' %}
{# nothing to do #}
{% else %}
{% if obj['isSynaptic'] %}
{% if obj['connectivity'] == 'DENSE' %}
convert_dense_matrix_2_dynamic_arrays({{var}}{{obj['owner'].name}},
{{obj['srcN']}}, {{obj['trgN']}},
brian::_dynamic_array_{{obj['owner'].name}}__synaptic_pre,
brian::_dynamic_array_{{obj['owner'].name}}__synaptic_post,
brian::_dynamic_array_{{obj['owner'].name}}_{{var}});
{% else %}
convert_sparse_synapses_2_dynamic_arrays(C{{obj['owner'].name}}, {{var}}{{obj['owner'].name}},
{{obj['srcN']}}, {{obj['trgN']}},
brian::_dynamic_array_{{obj['owner'].name}}__synaptic_pre,
brian::_dynamic_array_{{obj['owner'].name}}__synaptic_post,
brian::_dynamic_array_{{obj['owner'].name}}_{{var}}, b2g::FULL_MONTY);
{% endif %}
{% else %}
copy_genn_to_brian({{var}}{{obj['owner'].name}},
brian::_array_{{obj['owner'].name}}_{{var}},
{{obj['owner'].variables[var].size}});
{% endif %}
{% endif %}
{% endfor %}

_run_{{obj['codeobj'].name}}();

{% for var in obj['write'] %}
{% if obj['isSynaptic'] %}
{% if obj['connectivity'] == 'DENSE' %}
convert_dynamic_arrays_2_dense_matrix(brian::_dynamic_array_{{obj['owner'].name}}__synaptic_pre,
brian::_dynamic_array_{{obj['owner'].name}}__synaptic_post,
brian::_dynamic_array_{{obj['owner'].name}}_{{var}},
{{var}}{{obj['owner'].name}},
{{obj['srcN']}}, {{obj['trgN']}});
{% else %}
convert_dynamic_arrays_2_sparse_synapses(brian::_dynamic_array_{{obj['owner'].name}}__synaptic_pre,
brian::_dynamic_array_{{obj['owner'].name}}__synaptic_post,
brian::_dynamic_array_{{obj['owner'].name}}_{{var}},
{{var}}{{obj['owner'].name}},
{{obj['srcN']}}, {{obj['trgN']}},
_{{obj['owner'].name}}_bypre);
{% endif %}
{% else %}
copy_brian_to_genn(brian::_array_{{obj['owner'].name}}_{{var}},
{{var}}{{obj['owner'].name}},
{{obj['owner'].variables[var].size}});
{% endif %}
{% endfor %}
}
{% endif %}
{% endfor %}
#ifndef CPU_ONLY
if (which == GPU) {
{% set states_pushed = [] %}
{% for run_reg in run_regularly_operations %}
if (i % {{run_reg['step']}} == 0) // only push state if we executed the operation
{
{% for var in run_reg['write'] %}
{% if not run_reg['owner'].variables[var].owner.name in states_pushed %}
push{{run_reg['owner'].variables[var].owner.name}}StateToDevice();
{% if states_pushed.append(run_reg['owner'].variables[var].owner.name) %}{% endif %}
{% endif %}
{% endfor %}
}
{% endfor %}
stepTimeGPU();
// The stepTimeGPU function already updated everything for the next time step
iT--;
Expand All @@ -156,24 +216,48 @@ void engine::run(double duration, //!< Duration of time to run the model for
_run_{{spkGen.name}}_codeobject();
push{{spkGen.name}}SpikesToDevice();
{% endfor %}
{% set spikes_pulled = [] %}
{% for spkMon in spike_monitor_models %}
{% if (spkMon.notSpikeGeneratorGroup) %}
{% if not spkMon.neuronGroup in spikes_pulled %}
pull{{spkMon.neuronGroup}}SpikesFromDevice();
{% if spikes_pulled.append(spkMon.neuronGroup) %}{% endif %}
{% endif %}
{% endif %}
{% endfor %}
{% for rateMon in rate_monitor_models %}
{% if (rateMon.notSpikeGeneratorGroup) %}
{% if not rateMon.neuronGroup in spikes_pulled %}
pull{{rateMon.neuronGroup}}SpikesFromDevice();
{% if spikes_pulled.append(rateMon.neuronGroup) %}{% endif %}
{% endif %}
{% endif %}
{% endfor %}
{% set states_pulled = [] %}
{% for sm in state_monitor_models %}
{% if not sm.monitored in states_pulled %}
pull{{sm.monitored}}StateFromDevice();
{% if states_pulled.append(sm.monitored) %}{% endif %}
{% endif %}
{% endfor %}
{% for run_reg in run_regularly_operations %}
if ((i + 1) % {{run_reg['step']}} == 0) // only pull state if next time step executes operation
{
{% for var in run_reg['read'] %}
{% if not var in ['t', 'dt'] %}
{% if not run_reg['owner'].variables[var].owner.name in states_pulled %}
pull{{run_reg['owner'].variables[var].owner.name}}StateFromDevice();
{% if states_pulled.append(run_reg['owner'].variables[var].owner.name) %}{% endif %}
{% endif %}
{% endif %}
{% endfor %}
}
{% endfor %}
}
#endif
if (which == CPU) {
stepTimeCPU();
// The stepTimeGPU function already updated everything for the next time step
// The stepTimeCPU function already updated everything for the next time step
iT--;
t = iT*DT;
{% for spkGen in spikegenerator_models %}
Expand Down
10 changes: 7 additions & 3 deletions brian2genn/templates/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ int main(int argc, char *argv[])
create_hidden_weightmatrix(brian::_dynamic_array_{{synapses.name}}__synaptic_pre, brian::_dynamic_array_{{synapses.name}}__synaptic_post, _hidden_weightmatrix{{synapses.name}},{{synapses.srcN}}, {{synapses.trgN}});
{% else %} {# for sparse matrix representations #}
allocate{{synapses.name}}(brian::_dynamic_array_{{synapses.name}}__synaptic_pre.size());
vector<vector<int32_t> > _{{synapses.name}}_bypre;
initialize_sparse_synapses(brian::_dynamic_array_{{synapses.name}}__synaptic_pre, brian::_dynamic_array_{{synapses.name}}__synaptic_post,
C{{synapses.name}}, {{synapses.srcN}}, {{synapses.trgN}}, _{{synapses.name}}_bypre);
{% for var in synapses.variables %}
Expand Down Expand Up @@ -225,6 +224,7 @@ using namespace std;

#include "utils.h" // for CHECK_CUDA_ERRORS
#include "stringUtils.h"
#include <vector>

#ifndef CPU_ONLY
#include <cuda_runtime.h>
Expand All @@ -248,6 +248,10 @@ CStopWatch timer;

//----------------------------------------------------------------------
// other stuff:


// global variables for pre-calculated list of the postsynaptic targets ordered by presynaptic sources
{% for synapses in synapse_models %}
{% if synapses.connectivity != 'DENSE' %}
vector<vector<int32_t> > _{{synapses.name}}_bypre;
{% endif %}
{% endfor %}
{% endmacro %}
8 changes: 0 additions & 8 deletions brian2genn/templates/neuron_code.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,6 @@
// Update "constant over dt" subexpressions (if any)
{{(scalar_code['subexpression_update'] + vector_code['subexpression_update'])|autoindent}}

{% if has_run_regularly %}
// Run regular operations on a slower clock
int _timesteps = (int)(t/dt + 0.5);
if (_timesteps % (int)_run_regularly_steps == 0) { {# we need a type cast because GeNN parameters are double values #}
{{vector_code['run_regularly']|autoindent}} {# Note that the scalar code (if any) is in a separate code object #}
}
{% endif %}

// PoissonInputs targetting this group (if any)
{{(scalar_code['poisson_input'] + vector_code['poisson_input'])|autoindent}}

Expand Down
6 changes: 0 additions & 6 deletions brian2genn/templates/run_regularly_scalar_code.cpp
View file
Open in desktop

This file was deleted.

5 changes: 3 additions & 2 deletions docs_sphinx/introduction/exclusions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,9 @@ Timed arrays
Timed arrays post a problem in the Brian2GeNN interface because they
necessitate communication from the timed array to the target group at
runtime that would result in host to GPU copies in the final CUDA/C++
code. This could lead to large inefficiences and for the moment we
have therefore decided to not support this feature.
code. This could lead to large inefficiences, the use of ``TimedArray`` is therefore
currently restricted to code in ``run_regularly`` operations that will be executed on
the CPU.

Multiple clocks
---------------
Expand Down