rss.xml

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="assets/xml/rss.xsl" media="all"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>GeNN (Software Developer Blog)</title><link>http://genn-team.github.io/</link><description>Welcome to GeNN</description><atom:link href="http://genn-team.github.io/rss.xml" rel="self" type="application/rss+xml"></atom:link><language>en</language><copyright>Contents © 2024 &lt;a href="mailto:t.nowotny@sussex.ac.uk"&gt;GeNN Team&lt;/a&gt; </copyright><lastBuildDate>Thu, 07 Nov 2024 17:17:52 GMT</lastBuildDate><generator>Nikola (getnikola.com)</generator><docs>http://blogs.law.harvard.edu/tech/rss</docs><item><title>Software Developer Blog: Running away</title><link>http://genn-team.github.io/posts/running-away.html</link><dc:creator>GeNN Team</dc:creator><description>&lt;p&gt;After spending a long time updating GeNN's code generator to generate more efficient CUDA kernels which have the side benefit of compiling much more quickly, there remained something of a dirty secret.
The &lt;code&gt;runner.cc&lt;/code&gt; file which contains the helper functions generated by GeNN for allocating memory and copying variables between GPU and CPU could still easily grow to the point that compilation would take an extremely long time and consume all available memory.
For our &lt;a href="https://github.com/neworderofjamie/multi-area-model/"&gt;multi-area model implementation&lt;/a&gt;, I added various options which turn off the generation of empty functions and, as everything in this model was generated on the GPU anyway, I also turned off the generation of host copies of almost all variables.
This resulted in a paltry 40 mbyte &lt;code&gt;runner.cc&lt;/code&gt; which compiled in a couple of minutes which, for a model this size, is just about acceptable.
However, as users have started making bigger models and not always wanting to generate everything on the GPU, this issue has kept reappearing.&lt;/p&gt;
&lt;h2&gt;Jinjaly investigating&lt;/h2&gt;
&lt;p&gt;To investigate this in a slightly simpler way than just building larger and larger GeNN models until things break, I used &lt;a href="https://jinja.palletsprojects.com"&gt;Jinja&lt;/a&gt; to build a template that could generate fake &lt;code&gt;runner.cc&lt;/code&gt; files containing varying number of arrays, representing the state variables in a real model. 
The heart of this template looked something like this:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="c1"&gt;// Push and pull functions&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;arrays&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;push&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="n"&gt;ToDevice&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;CHECK_CUDA_ERRORS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cudaMemcpy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d_&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                 &lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sizeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                 &lt;/span&gt;&lt;span class="n"&gt;cudaMemcpyHostToDevice&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pull&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="n"&gt;FromDevice&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;CHECK_CUDA_ERRORS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cudaMemcpy&lt;/span&gt;&lt;span class="p"&gt;({{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;d_&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                 &lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sizeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                 &lt;/span&gt;&lt;span class="n"&gt;cudaMemcpyDeviceToHost&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;endfor&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;allocateMem&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;CHECK_CUDA_ERRORS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cudaSetDevice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;arrays&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;CHECK_CUDA_ERRORS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cudaHostAlloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                    &lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sizeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                    &lt;/span&gt;&lt;span class="n"&gt;cudaHostAllocPortable&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;CHECK_CUDA_ERRORS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cudaMalloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;d_&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="w"&gt;                                 &lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sizeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;endfor&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;this template (saved in &lt;code&gt;runner.cc.template&lt;/code&gt;) could then be used to generate C++ and print it to stdout like:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;jinja2&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Template&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"runner.cc.template"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"r"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;template&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;arrays&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"array_&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"size"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; 
          &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;num_arrays&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;


&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;template&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;arrays&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;arrays&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On Linux, the C++ could then be built using the same command line used by GeNN itself (some options omitted for brevity) and timed using the &lt;code&gt;/usr/bin/time&lt;/code&gt; (I recently discovered that &lt;code&gt;command time&lt;/code&gt; can be used as an alternative way of disambiguating this from the bash builtin &lt;code&gt;time&lt;/code&gt;):&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;/usr/bin/time&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;nvcc&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;cu&lt;span class="w"&gt; &lt;/span&gt;-arch&lt;span class="w"&gt; &lt;/span&gt;sm_86&lt;span class="w"&gt; &lt;/span&gt;-std&lt;span class="o"&gt;=&lt;/span&gt;c++11&lt;span class="w"&gt; &lt;/span&gt;test.cc
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Initial experiments showed that while both the wall clock time and maximum resident set size (roughly analagous to peak physical memory usage) grow approximately linearly (much to my relief after reading way too many &lt;a href="https://randomascii.wordpress.com/category/quadratic/"&gt;horror stories&lt;/a&gt; on Bruce Dawson's &lt;a href="https://randomascii.wordpress.com"&gt;excellent blog&lt;/a&gt;) with the number of arrays, it still grew extremely rapidly:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Relationship between runner size; and compile time and memory usage" src="http://genn-team.github.io/images/blog_running_away/fig_1_linearity.png"&gt;&lt;/p&gt;
&lt;p&gt;Therefore, a model with 10000 arrays will take over 4 minutes and around 8 gbyte of memory to compile — neither of which are really acceptable.
To put this in perspective, if you split a model up into about 100 populations and connect most of the permutations together (this is an all-too-reasonable assumption in many areas of the mammalian brain), you could easily reach this many variables.&lt;/p&gt;
&lt;p&gt;So....what is NVCC doing with all this time and memory?
&lt;code&gt;runner.cc&lt;/code&gt; only contains host code (NVCC is just used to ensure the same compiler/options across execution units and to deal with setting up the CUDA linker/include paths) but, when you pass a 5 mbyte &lt;code&gt;runner.cc&lt;/code&gt; file to NVCC, the file that is passed on to the host compiler (GCC) has grown to 15 mbyte!
However, this turned out to be simply because NVCC is in charge of running the preprocessor so that 10 mbyte is 'just' the result of expanding macros and including C++ standard library header files!&lt;/p&gt;
&lt;h2&gt;Profiling&lt;/h2&gt;
&lt;p&gt;Around this point, I remembered reading a blog post about &lt;a href="https://aras-p.info/blog/2019/01/12/Investigating-compile-times-and-Clang-ftime-report/"&gt;profiling compiler times&lt;/a&gt; on yet another &lt;a href="https://aras-p.info/blog/"&gt;excellent blog&lt;/a&gt; and turned on the &lt;code&gt;-ftime-report&lt;/code&gt; GCC option.
As the blog promised, this generates a gargantuan report which starts by splitting the compilation time of this 10000 array model into 'phases':&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;phase setup             :  ...   0.00 ( 0%) wall    1384 kB ( 0%)
phase parsing           :  ...  20.81 ( 9%) wall 1794944 kB (25%)
phase lang. deferred    :  ...   0.02 ( 0%) wall    2426 kB ( 0%)
phase opt and generate  :  ... 214.14 (91%) wall 5412439 kB (75%)
phase finalize          :  ...   0.54 ( 0%) wall       0 kB ( 0%)
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Somewhat surprisingly (as, after all, we're throwing a massive source file at GCC), the vast majority of time is spent in "opt and generate" (code-generation and optimisation) rather than in parsing.
Looking a little further down, where the report contains a seeminly unsorted list of processes within phases, the only other 'hot' line is:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;expand vars             :  ... 101.93 (43%) wall   50597 kB ( 1%)
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;but, as a non-GCC developer, this doesn't help me a great deal....back to pursuing random hunches!&lt;/p&gt;
&lt;h2&gt;Smoking gun&lt;/h2&gt;
&lt;p&gt;As there's no virtual functions in this code, my C++ prejudices suggest that only exceptions could &lt;em&gt;possibly&lt;/em&gt; be to blame and, as each of those &lt;code&gt;CHECK_CUDA_ERRORS&lt;/code&gt; macros hides a &lt;code&gt;throw std::runtime_error&lt;/code&gt;, maybe that's not unreasonable.
Generating all that zero-cost abstraction must involve &lt;em&gt;expanding&lt;/em&gt; a lot of &lt;em&gt;variables&lt;/em&gt;....right!?
How about if we replace our current implemementation of &lt;code&gt;CHECK_CUDA_ERRORS&lt;/code&gt;:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="cp"&gt;#define CHECK_CUDA_ERRORS(call) {\&lt;/span&gt;
&lt;span class="cp"&gt;    cudaError_t error = call;\&lt;/span&gt;
&lt;span class="cp"&gt;    if(error != cudaSuccess) {\&lt;/span&gt;
&lt;span class="cp"&gt;        throw std::runtime_error(__FILE__": " \&lt;/span&gt;
&lt;span class="cp"&gt;                                 + std::to_string(__LINE__) \&lt;/span&gt;
&lt;span class="cp"&gt;                                 + ": cuda error " \&lt;/span&gt;
&lt;span class="cp"&gt;                                 + std::to_string(error) \&lt;/span&gt;
&lt;span class="cp"&gt;                                 + ": " + cudaGetErrorString(error));\&lt;/span&gt;
&lt;span class="cp"&gt;    }\&lt;/span&gt;
&lt;span class="cp"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;with:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="cp"&gt;#define CHECK_CUDA_ERRORS(call) {\&lt;/span&gt;
&lt;span class="cp"&gt;    cudaError_t error = call;\&lt;/span&gt;
&lt;span class="cp"&gt;    assert(error == cudaSuccess); \&lt;/span&gt;
&lt;span class="cp"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;or even:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="cp"&gt;#define CHECK_CUDA_ERRORS(call) {\&lt;/span&gt;
&lt;span class="cp"&gt;    cudaError_t error = call;\&lt;/span&gt;
&lt;span class="cp"&gt;    if(error != cudaSuccess) {\&lt;/span&gt;
&lt;span class="cp"&gt;        std::abort();\&lt;/span&gt;
&lt;span class="cp"&gt;    }\&lt;/span&gt;
&lt;span class="cp"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Some template-meddling and sweeping later we can produce:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Relationship between number of arrays and error handling approach; and compile time and memory usage" src="http://genn-team.github.io/images/blog_running_away/fig_2_alternate_error.png"&gt;&lt;/p&gt;
&lt;p&gt;Seems like this actually works! Our 10000 array model now only takes 30 seconds and less than 1 gbyte of memory to compile which is much more reasonable!&lt;/p&gt;
&lt;p&gt;However, are the exceptions &lt;em&gt;really&lt;/em&gt; to blame? 
Compile times seem much better when using the single-threaded CPU backend and that allocates memory with &lt;code&gt;new[]&lt;/code&gt; (obviously, modern C++ rules don't apply in generated code...) which throws &lt;code&gt;std::bad_alloc&lt;/code&gt; to signal failure.
Admittedly, because there's no need to copy data when everthing's on the CPU, this backend generates empty 'push' and 'pull' functions so there's less code to compile overall but, if generating exception handling code was the problem, you would expect issues here too.
Maybe expanding all that message-generating code is the real issue...
How about we hack the following additional variants into the template:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="cp"&gt;#define CHECK_CUDA_ERRORS(call) {\&lt;/span&gt;
&lt;span class="cp"&gt;    cudaError_t error = call;\&lt;/span&gt;
&lt;span class="cp"&gt;    if(error != cudaSuccess) {\&lt;/span&gt;
&lt;span class="cp"&gt;        throw std::runtime_error();\&lt;/span&gt;
&lt;span class="cp"&gt;    }\&lt;/span&gt;
&lt;span class="cp"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;and&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="cp"&gt;#define CHECK_CUDA_ERRORS(call) {\&lt;/span&gt;
&lt;span class="cp"&gt;    cudaError_t error = call;\&lt;/span&gt;
&lt;span class="cp"&gt;    if(error != cudaSuccess) {\&lt;/span&gt;
&lt;span class="cp"&gt;        std::cerr &amp;lt;&amp;lt; __FILE__ &amp;lt;&amp;lt; ": " &amp;lt;&amp;lt; __LINE__;\&lt;/span&gt;
&lt;span class="cp"&gt;        std::cerr &amp;lt;&amp;lt; ": cuda error " &amp;lt;&amp;lt; error &amp;lt;&amp;lt; ": ";\&lt;/span&gt;
&lt;span class="cp"&gt;        std::cerr &amp;lt;&amp;lt; cudaGetErrorString(error) &amp;lt;&amp;lt; std::endl;\&lt;/span&gt;
&lt;span class="cp"&gt;        std::abort();\&lt;/span&gt;
&lt;span class="cp"&gt;    }\&lt;/span&gt;
&lt;span class="cp"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;and sweep:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Relationship between number of arrays and error handling approach; and compile time and memory usage" src="http://genn-team.github.io/images/blog_running_away/fig_3_more_alternate_error.png"&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;std::abort&lt;/code&gt; is definitely easier on the compiler than throwing exceptions but, compiling the message generation code also seems to make a large difference.&lt;/p&gt;
&lt;h2&gt;MSVC&lt;/h2&gt;
&lt;p&gt;The other compiler I often use with GeNN is Microsoft Visual C++.
I can't quite face repeating this whole process again but, initial tests suggest that this optimisation is even more valuable here.
Using the simplest &lt;code&gt;std::abort&lt;/code&gt; raising &lt;code&gt;CHECK_CUDA_ERRORS&lt;/code&gt; macro, the 10000 array model can be compiled in around 19 seconds whereas, using the original exception-throwing macro...I have given up waiting after around 1 hour!&lt;/p&gt;
&lt;h2&gt;Implementing a workaround&lt;/h2&gt;
&lt;p&gt;GeNN is approaching the end of the 4.X release cycle so, for now, I have added a simple but slightly hacky workaround for these issues by adding a &lt;code&gt;generateSimpleErrorHandling&lt;/code&gt; flag to GeNN's CUDA backend to switch from generating code with the previous full-fat &lt;code&gt;CHECK_CUDA_ERRORS&lt;/code&gt; macro to the simplest version which simply calls &lt;code&gt;std::abort&lt;/code&gt; without generating a message. This can be turned on from C++ like:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;modelDefinition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;NNmodel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;GENN_PREFERENCES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;generateSimpleErrorHandling&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;or from Python like:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;GeNNModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"float"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"my_model"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generateSimpleErrorHandling&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;h2&gt;Real models&lt;/h2&gt;
&lt;p&gt;The largest model we currently have to play with with is the &lt;a href="https://github.com/neworderofjamie/multi-area-model/"&gt;multi-area cortical model&lt;/a&gt;.
Although it has 64516 synapse groups, due to its use of procedural connectivity (where all synaptic connectivity, weights and delays are generated on the fly), it doesn't actually have any per-synapse group variables with push and pull functions.
Nonetheless, using the new simple error handling reduces the compilation time of the &lt;code&gt;runner.cc&lt;/code&gt; from 155 to 129 seconds.
Finally, although it is not possible to &lt;em&gt;run&lt;/em&gt; the model in this way as no single GPU has enough memory, we can generate a &lt;code&gt;runner.cc&lt;/code&gt; from this model with standard, in-memory sparse connectivity and push and pull functions for each variable.
This results in a nightmarish, 114 mbyte &lt;code&gt;runner.cc&lt;/code&gt; which, using the original &lt;code&gt;CHECK_CUDA_ERRORS&lt;/code&gt; macro, would definitely be impossible to compile on any reasonable machine.
However, using the new simplified macro, the runner can be compiled in just over 20 minutes and requires just over 14 gbyte of memory — still pretty unusable but definitely progress!&lt;/p&gt;
&lt;h2&gt;Long-term solutions&lt;/h2&gt;
&lt;p&gt;The majority of the time, the errors which the &lt;code&gt;CHECK_CUDA_ERRORS&lt;/code&gt; macro is aiming to catch are out of memory errors in the &lt;code&gt;allocateMem&lt;/code&gt; function and errors that occured during (asynchronous) kernel launches that are only caught at the next push or pull call (which are typically the main synchronisation points) so perhaps, in future, we could adopt a more targetted error-handling approach which provides a balance between sufficient debugging information and compilation time.&lt;/p&gt;
&lt;p&gt;However, while the error handling changes discussed here allow the current approach to generating &lt;code&gt;runner.cc&lt;/code&gt; files to scale a bit further, the code we are generating is still pretty pathological, least of all because &lt;a href="https://github.com/genn-team/genn/issues/408"&gt;the Windows PE executable format has a limit of 65535 symbol limit&lt;/a&gt; which you can hit quite easily with a large model.
Early this year, I made an attempt at re-writing the code generator to apply the same merging strategy GeNN uses elsewhere to &lt;code&gt;runner.cc&lt;/code&gt;. 
This means that all the variables associated with neuron and synapse populations with the same types of state variable can be allocated using one piece of shared generated code.
While this works, it adds yet more complexity to GeNN and fundamentally breaks the 'classic' way of using GeNN from C++, where you link some C++ simulation code against your generated code and can access state variables directly by name.
However, based on this investigation, maybe that project needs resurrecting!&lt;/p&gt;
&lt;p&gt;All the code I've developed to explore this problem is available from &lt;a href="https://github.com/neworderofjamie/nvcc_breaker"&gt;my Github&lt;/a&gt;.&lt;/p&gt;</description><guid>http://genn-team.github.io/posts/running-away.html</guid><pubDate>Tue, 01 Nov 2022 13:35:07 GMT</pubDate></item><item><title>Software Developer Blog: How to do convolutions with doubly blocked Toeplitz matrices</title><link>http://genn-team.github.io/posts/sw_blog_toeplitz.html</link><dc:creator>GeNN Team</dc:creator><description>&lt;h2&gt;How to do convolutions with doubly blocked Toeplitz matrices&lt;/h2&gt;
&lt;p&gt;A few weeks ago, Jamie (@neworderofjamie) asked me on the chat whether I knew what doubly blocked Toeplitz matrices are and how they implement convolutions. I had no clue. Since then we have implemented convolutions using doubly blocked Toeplitz matrices in GeNN and found them to be extremely useful and efficient.
1
In this software blog I will give a brief overview on the why and how convolutions relate to doubly blocked Toeplitz matrices. My blog is based on Ali Salehi's tutorial &lt;a href="https://raw.githubusercontent.com/alisaaalehi/convolution_as_multiplication/master/ConvAsMulExplained.pdf"&gt;Convolution as Matrix Multiplication&lt;/a&gt;  but updated to use machine-learning rather than signal-processing conventions and I am trying to avoid using too many unusual ways of re-arranging rows and columns.&lt;/p&gt;
&lt;h3&gt;The why&lt;/h3&gt;
&lt;p&gt;Let us consider the convolution of a \(2\times 2\) kernel with a \(3\times 3\) layer. We denote the kernel as
\[
K= \left(\matrix{
k_{11} &amp;amp; k_{12} \cr
k_{21} &amp;amp; k_{22}}\right)
\]
and the layer as
\[
I= \left(\matrix{
i_{11} &amp;amp; i_{12} &amp;amp; i_{13} \cr
i_{21} &amp;amp; i_{22} &amp;amp; i_{23} \cr
i_{31} &amp;amp; i_{32} &amp;amp; i_{33}
} \right).
\]
Then the convolution in the machine learning use of the term is calculating the cross-correlation of the kernel "moving across" the layer as illustrated below. The layer \(I\) is in blue, the kernel \(K\) in grey and the result \(R\) in green.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style="text-align: center;"&gt;&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="http://genn-team.github.io/posts/sw_blog_toeplitz.html#fn:1"&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_00.png"&gt;&lt;/th&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_01.png"&gt;&lt;/th&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_02.png"&gt;&lt;/th&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_03.png"&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="text-align: center;"&gt;\(r_{11}\)&lt;/td&gt;
&lt;td style="text-align: center;"&gt;\(r_{12}\)&lt;/td&gt;
&lt;td style="text-align: center;"&gt;\(r_{13}\)&lt;/td&gt;
&lt;td style="text-align: center;"&gt;\(3_{14}\)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For the first non-zero entry at \((1,1)\) of the result matrix \(R\), we therefore have \(r_{11} = k_{22} i_{11}\).
Then the kernel moves one over and \(r_{12} = k_{21}i_{11} + k_{22} i_{12}\). Then, \(r_{13} = k_{21}i_{12} + k_{22} i_{13}\) and \(r_{14} = k_{21}i_{13} \).&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_04.png"&gt;&lt;/th&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_05.png"&gt;&lt;/th&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_06.png"&gt;&lt;/th&gt;
&lt;th style="text-align: center;"&gt;&lt;img alt="Illustration of convolution step" src="http://genn-team.github.io/images/blog_07.png"&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="text-align: center;"&gt;\(r_{21}\)&lt;/td&gt;
&lt;td style="text-align: center;"&gt;\(r_{22}\)&lt;/td&gt;
&lt;td style="text-align: center;"&gt;\(r_{23}\)&lt;/td&gt;
&lt;td style="text-align: center;"&gt;\(r_{24}\)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;So, for the second row,
\(r_{21} = k_{12} i_{11} + k_{22} i_{21} \), move one over,
\(r_{22} = k_{11} i_{11} + k_{12} i_{12} + k_{21} i_{21} + k_{22} i_{22} \), one more to the right,
\(r_{23} = k_{11}i_{12} + k_{12} i_{13} + k_{21} i_{22} + k_{22} i_{23} \), and finally
\(r_{24} = k_{11}i_{13} + k_{21} i_{23} \).&lt;/p&gt;
&lt;p&gt;It works similar for the remaining two rows.&lt;/p&gt;
&lt;p&gt;If we unroll the layer \(I\) row-wise into a column vector \(I_\text{col}\),
\[
    I_\text{col} = 
    \left(
    \matrix{ 
        i_{11} \cr
        i_{12} \cr
        i_{13} \cr
        i_{21} \cr
        i_{22} \cr
        i_{23} \cr
        i_{31} \cr
        i_{32} \cr
        i_{33}
        }
    \right),
\]
then we can express this as a matrix-vector multiplication of a matrix formed from the entries of the kernel \(K\) and the vector\(I_\text{col}\), 
\[
\left(\matrix{
k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
k_{21} &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; k_{21} &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; k_{21} &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
k_{12} &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
k_{11} &amp;amp; k_{12} &amp;amp; 0 &amp;amp; k_{21} &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; k_{11} &amp;amp; k_{12} &amp;amp; 0 &amp;amp; k_{21} &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; k_{11} &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{21} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{12} &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{22} &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{11} &amp;amp; k_{12} &amp;amp; 0 &amp;amp; k_{21} &amp;amp; k_{22} &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{11} &amp;amp; k_{12} &amp;amp; 0 &amp;amp; k_{21} &amp;amp; k_{22} \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{11} &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{21} \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{12} &amp;amp; 0 &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{11} &amp;amp; k_{12} &amp;amp; 0 \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{11} &amp;amp; k_{12} \cr
0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; k_{11} }\right)
\cdot
\left(\matrix{
i_{11} \cr
i_{12} \cr
i_{13} \cr
i_{21} \cr
i_{22} \cr
i_{23} \cr
i_{31} \cr
i_{32} \cr
i_{33}} 
\right)
\]&lt;/p&gt;
&lt;p&gt;Now one can already see that the matrix formed from the kernel entries has a very peculiar shape - the shape of a doubly blocked Toeplitz matrix&lt;/p&gt;
&lt;h3&gt;Doubly blocked Toeplitz matrix&lt;/h3&gt;
&lt;p&gt;A Toeplitz matrix is a matrix where the values along all diagonals are constant, i.e.&lt;/p&gt;
&lt;p&gt;\[
\left(
    \matrix{ 
        a_{0} &amp;amp; a_{-1} &amp;amp; a_{-2} &amp;amp; \cdots  &amp;amp; \cdots &amp;amp; \cdots &amp;amp; a_{-(N-1)} \cr
        a_{1} &amp;amp; a_{0} &amp;amp; a_{-1} &amp;amp; a_{-2} &amp;amp;  &amp;amp; &amp;amp; \vdots \cr
        a_{2} &amp;amp; a_{1} &amp;amp; a_{0} &amp;amp; a_{-1} &amp;amp;  &amp;amp; &amp;amp; \vdots \cr
        \vdots &amp;amp; \ddots &amp;amp; \ddots &amp;amp; \ddots &amp;amp; \ddots &amp;amp; \ddots &amp;amp; &amp;amp; \vdots \cr
        \vdots &amp;amp; &amp;amp; &amp;amp; \ddots  &amp;amp; a_{0} &amp;amp; a_{-1} &amp;amp; a_{-2} \cr
        \vdots &amp;amp; &amp;amp; &amp;amp;  &amp;amp; a_{1} &amp;amp; a_{0} &amp;amp; a_{-1} \cr
        a_{M-1} &amp;amp; \cdots  &amp;amp; \cdots &amp;amp; \cdots &amp;amp; a_{2} &amp;amp; a_{1} &amp;amp; a_{0} }
    \right) .
\]&lt;/p&gt;
&lt;p&gt;Furthermore, if we build a matrix \(A\) out of Toeplitz sub-matrices \(A_{k}\) &lt;em&gt;and&lt;/em&gt; the structure of \(A\) with respect to these submatrices is also Toeplitz:&lt;/p&gt;
&lt;p&gt;\[
    A = \left(
    \matrix{ 
        A_{0} &amp;amp; A_{-1} &amp;amp; \cdots &amp;amp; A_{-(L-1)} \cr
        A_{1} &amp;amp; A_{0} &amp;amp; \cdots &amp;amp; A_{-(L-2)} \cr
        \vdots &amp;amp; \vdots &amp;amp; \ddots &amp;amp; \vdots \cr
        A_{K} &amp;amp; A_{K-1} &amp;amp; \cdots &amp;amp; A_{0}}
    \right),
\]&lt;/p&gt;
&lt;p&gt;then, this matrix is called a doubly-blocked Toeplitz matrix. A standard way to generate a Toeplitz matrix from a vector \(v\) is to use \(v\) as the first column vector, then make one cyclic permutation and use it as the second column vector and so on.&lt;/p&gt;
&lt;h3&gt;The method&lt;/h3&gt;
&lt;p&gt;As we have seen on the example above, 2D convolution operations can be expressed as multiplication by a doubly-blocked Toeplitz matrix. As a general method, applied to the example above,
to convolve \(K\) with \(I\), we first flip \(K\) across the horizontal and vertical axis and pad it to the output size \((I_\text{height} + K_\text{height} - 1) \times (I_\text{width} + K_\text{width} - 1)\) of the convolution.
For instance, here, the \(3 \times 3\) layer \(I\) covolved by \(K\) above, leads to output size \(4 \times 4\).
Depending on the padding mode used by the convolution, typically, only part of this output is actually required.
The flipped and padded kernel \(K\) from above is
\[
    K_\text{pad}=
    \left(
    \matrix{ 
        k_{22} &amp;amp; k_{21} &amp;amp; 0 &amp;amp; 0 \cr
        k_{12} &amp;amp; k_{11} &amp;amp; 0 &amp;amp; 0 \cr
        0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 \cr
        0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 }
    \right)
\]&lt;/p&gt;
&lt;p&gt;We then convert each &lt;em&gt;row vector&lt;/em&gt; of this matrix into Toeplitz matrices \(F_i\) as described above:
\[
    F_0=
    \left(
    \matrix{ 
        k_{22} &amp;amp; 0 &amp;amp; 0 \cr
        k_{21} &amp;amp; k_{22} &amp;amp; 0 \cr
        0 &amp;amp; k_{21} &amp;amp; k_{22} \cr
        0 &amp;amp; 0 &amp;amp; k_{21}}
    \right)
    \quad
    F_1=
    \left(
    \matrix{ 
        k_{12} &amp;amp; 0 &amp;amp;  0 \cr
        k_{11} &amp;amp; k_{12} &amp;amp; 0 \cr
        0 &amp;amp;  k_{11} &amp;amp; k_{12} \cr
        0 &amp;amp;  0 &amp;amp;  k_{11}}
    \right)
    \]
    \[
    F_2=
    \left(
    \matrix{ 
        0 &amp;amp; 0  &amp;amp; 0 \cr
        0 &amp;amp; 0 &amp;amp; 0 \cr
        0  &amp;amp; 0 &amp;amp; 0 \cr
        0  &amp;amp; 0  &amp;amp; 0}
    \right)
    \quad
    F_3=
    \left(
    \matrix{ 
        0 &amp;amp; 0  &amp;amp; 0 \cr
        0 &amp;amp; 0 &amp;amp; 0 \cr
        0  &amp;amp; 0 &amp;amp; 0 \cr
        0  &amp;amp; 0  &amp;amp; 0}
    \right)
\]
and, finally, assemble these into a doubly blocked Toeplitz matrix \(F\):&lt;/p&gt;
&lt;p&gt;\[
    F=
    \left(
    \matrix{ 
        F_0 &amp;amp; F_3 &amp;amp; F_2 \cr
        F_1 &amp;amp; F_0 &amp;amp; F_3 \cr
        F_2 &amp;amp; F_1 &amp;amp; F_0 \cr
        F_3 &amp;amp; F_2 &amp;amp; F_1
    }
    \right)
\]&lt;/p&gt;
&lt;p&gt;The convolution of \(K\) with \(I\)
is then given by multiplying F from the left onto \(I_\text{col}\) as defined above,
\[
  R_{\text{col}} = F \cdot I  \quad 
  \Leftrightarrow \quad R_{\text{col},j}= \sum_i F_{ji}I_i 
  \]&lt;/p&gt;
&lt;p&gt;Finally, \(R_{\text{col}}\) can be reinterpreted as the output matrix \(R\) by arranging its entries row-wise in a \(4\times 4\) matrix.&lt;/p&gt;
&lt;p&gt;There we have it - convolution (in the machine learning sense, i.e. corss-correlation) of a kernel \(K\) with a layer \(I\) expressed as the product of a doubly blocked Toeplitz matrix derived from \(K\) with the column vector of the row-wise unrolled entries from \(I\).&lt;/p&gt;
&lt;p&gt;The following python function is a simple implementation of this method&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;np&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;scipy.linalg&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;toeplitz&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;convolution&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# flip the kernel&lt;/span&gt;
    &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fliplr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flipud&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="c1"&gt;# calculate sizes&lt;/span&gt;
    &lt;span class="n"&gt;K_row_num&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K_col_num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;
    &lt;span class="n"&gt;I_row_num&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;I_col_num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;
    &lt;span class="n"&gt;R_row_num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;K_row_num&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;I_row_num&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="n"&gt;R_col_num&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;K_col_num&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;I_col_num&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="c1"&gt;# pad the kernel&lt;/span&gt;
    &lt;span class="n"&gt;K_pad&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;R_row_num&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;K_row_num&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                      &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;R_col_num&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;K_col_num&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; 
                  &lt;span class="s1"&gt;'constant'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;constant_values&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"padded kernel= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K_pad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Assemble the list of Toeplitz matrices F_i&lt;/span&gt;
    &lt;span class="n"&gt;toeplitz_list&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;R_row_num&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;K_pad&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,:]&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;r_&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;zeros&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;I_col_num&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
        &lt;span class="n"&gt;toeplitz_list&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;toeplitz&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Toeplitz list= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;toeplitz_list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# make a matrix with the indices of the block F_i &lt;/span&gt;
    &lt;span class="c1"&gt;# of the doubly blocked Toeplitz matrix&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;R_row_num&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;r_&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
    &lt;span class="n"&gt;doubly_indices&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;toeplitz&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"doubly_indices= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;doubly_indices&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# assemble the doubly blocked toeplitz matrix&lt;/span&gt;
    &lt;span class="n"&gt;toeplitz_m&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;R_row_num&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;I_row_num&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;toeplitz_list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;doubly_indices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hstack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;toeplitz_m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;toeplitz_m&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;vstack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;toeplitz_m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Toeplitz matrix= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;toeplitz_m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# make layer into column vector&lt;/span&gt;
    &lt;span class="n"&gt;I_col&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flatten&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"I_col= "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;I_col&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;R&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;matmul&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;toeplitz_m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;I_col&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'R as vector= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reshape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;R_row_num&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;R_col_num&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; 
        &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'R as matrix= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;R&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To test, one can, for instance, use&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="c1"&gt;# kernel&lt;/span&gt;
&lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;([[&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;],[&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
&lt;span class="c1"&gt;# layer&lt;/span&gt;
&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;([[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;],[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
&lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;convolution&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The output would then be&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;padded kernel= 
 [[40 30  0  0]
 [20 10  0  0]
 [ 0  0  0  0]]
Toeplitz list= 
 [array([[40.,  0.,  0.],
       [30., 40.,  0.],
       [ 0., 30., 40.],
       [ 0.,  0., 30.]]), array([[20.,  0.,  0.],
       [10., 20.,  0.],
       [ 0., 10., 20.],
       [ 0.,  0., 10.]]), array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])]
doubly_indices= 
 [[0 2]
 [1 0]
 [2 1]]
Toeplitz matrix= 
 [[40.  0.  0.  0.  0.  0.]
 [30. 40.  0.  0.  0.  0.]
 [ 0. 30. 40.  0.  0.  0.]
 [ 0.  0. 30.  0.  0.  0.]
 [20.  0.  0. 40.  0.  0.]
 [10. 20.  0. 30. 40.  0.]
 [ 0. 10. 20.  0. 30. 40.]
 [ 0.  0. 10.  0.  0. 30.]
 [ 0.  0.  0. 20.  0.  0.]
 [ 0.  0.  0. 10. 20.  0.]
 [ 0.  0.  0.  0. 10. 20.]
 [ 0.  0.  0.  0.  0. 10.]]
I_col=  [1 2 3 4 5 6]
R as vector= 
 [ 40. 110. 180.  90. 180. 370. 470. 210.  80. 140. 170.  60.]
R as matrix= 
 [[ 40. 110. 180.  90.]
 [180. 370. 470. 210.]
 [ 80. 140. 170.  60.]]
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note, that this example is inspired by &lt;a href="https://raw.githubusercontent.com/alisaaalehi/convolution_as_multiplication/master/ConvAsMulExplained.pdf"&gt;Salehi's tutorial&lt;/a&gt; but because we are calculating the machine learning covolution (cross-correlation) and Salehi the mathematical convolution as used in signal processing, the results are not the same. To generate identical results one can use the doubly flipped kernel,&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="c1"&gt;# kernel&lt;/span&gt;
&lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;([[&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;],[&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
&lt;span class="c1"&gt;# layer&lt;/span&gt;
&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;([[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;],[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
&lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;convolution&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"R= &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;and obtain&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code literal-block"&gt; &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; 
 &lt;span class="p"&gt;[[&lt;/span&gt; &lt;span class="mf"&gt;10.&lt;/span&gt;  &lt;span class="mf"&gt;40.&lt;/span&gt;  &lt;span class="mf"&gt;70.&lt;/span&gt;  &lt;span class="mf"&gt;60.&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
 &lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="mf"&gt;70.&lt;/span&gt; &lt;span class="mf"&gt;230.&lt;/span&gt; &lt;span class="mf"&gt;330.&lt;/span&gt; &lt;span class="mf"&gt;240.&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
 &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mf"&gt;120.&lt;/span&gt; &lt;span class="mf"&gt;310.&lt;/span&gt; &lt;span class="mf"&gt;380.&lt;/span&gt; &lt;span class="mf"&gt;240.&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;which exactly is Salehi's result.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;Convolution images created with software from:
Vincent Dumoulin and Francesco Visin, A guide to convolution arithmetic for deep learning (2016) ArXiv e-prints 1603.07285; &lt;a href="https://github.com/vdumoulin/conv_arithmetic"&gt;Software on github&lt;/a&gt; &lt;a class="footnote-backref" href="http://genn-team.github.io/posts/sw_blog_toeplitz.html#fnref:1" title="Jump back to footnote 1 in the text"&gt;↩&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description><category>convolution</category><category>math</category><category>toeplitz</category><guid>http://genn-team.github.io/posts/sw_blog_toeplitz.html</guid><pubDate>Tue, 21 Dec 2021 14:39:44 GMT</pubDate></item></channel></rss>