Feature/3299 chainset #3313

mitzimorris · 2024-10-11T20:55:11Z

Submission Checklist

Run unit tests: ./runTests.py src/test/unit
Run cpplint: make cpplint
Declare copyright holder and open-source license: see below

Summary

This is a do-over of PR #3310. The overall goal is to make the improved R-hat diagnostics (#3266), and rank-normalized ESS bulk and tail, (#3312) available to CmdStan, following the definitions in https://arxiv.org/abs/1903.08008.

This PR adds a new class stan::mcmc::chainset. A chainset object requires that all chains are the same size on construction, unlike the chains<> object. This simplifies computation of split-Rhat and split-ESS diagnostics. The chainset object provides rank-normalized split Rhat, rank-normalized split ESS which returns bulk and tail ESS, and functions mcse and mcse_sd which calculate the mean Monte Carlo standard error and its variance.

How to Verify

Unit tests for the chainset test performance against Stan CSV outputs. The CmdStan branch https://github.com/stan-dev/cmdstan/tree/feature/1263-new-rhat-summary further checks that chainset provides the necessary functions to output the same set of summary statistics as the R interfaces.

Side Effects

N/A

Documentation

Documentation will be added to the Stan docset in a separate PR.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

into feature/3299-chainset

stan-buildbot · 2024-10-12T02:26:04Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.35	0.34	1.01	0.77% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.07	6.59% faster
gp_regr/gen_gp_data.stan	0.03	0.03	1.01	0.99% faster
gp_regr/gp_regr.stan	0.09	0.09	1.0	-0.24% slower
sir/sir.stan	70.35	71.01	0.99	-0.94% slower
irt_2pl/irt_2pl.stan	4.12	4.33	0.95	-5.01% slower
eight_schools/eight_schools.stan	0.06	0.05	1.04	4.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.24	1.04	4.17% faster
pkpd/one_comp_mm_elim_abs.stan	19.49	18.79	1.04	3.59% faster
garch/garch.stan	0.44	0.4	1.09	7.89% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.7	2.58	1.05	4.51% faster
arK/arK.stan	1.78	1.71	1.05	4.36% faster
gp_pois_regr/gp_pois_regr.stan	2.83	2.69	1.05	4.99% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.9	8.35	1.07	6.18% faster
performance.compilation	183.57	180.29	1.02	1.79% faster
Mean result: 1.0311739250234235

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

WardBrian

I see this is still marked as draft, but most of my concerns are in the tests, so I hope this is still helpful at this stage

src/stan/mcmc/chainset.hpp

src/test/unit/mcmc/chainset_test.cpp

into feature/3299-chainset

WardBrian · 2024-10-16T19:02:38Z

made test more strict, but differences between R and C++ are to be expected.

I am not convinced that they are expected (I recently wrote a javascript version of ESS and RHat that matches stan's current C++ out to ~10 digits). I would want to see some explanation of exactly where and why they do to this degree, if we're claiming to implement the same things.

But, if these differences can be justified, then we shouldn't be testing against those values at all. The presence of those values in the tests visually implies we do match the R values (when we don't), and the resulting tests have very little power to catch issues. For example:

the MCSE calculation needs the sample standard deviation. If you replace it with the population standard deviation (by deleting the -1 in the denominator), the tests all pass.
if you replace it with something even more outlandish, like adding twenty to the denominator instead of subtracting one, the tests also still pass

I pick this example not just to be a pain, but because replacing the in-line standard deviation calculation in mcse_mean function is something a future code editor may be likely to do (if we don't already change it during review for this!), and accidentally picking the wrong library function is not hard to imagine, so failing the tests if that mistake gets made would be very helpful. Similar arguments can be made for basically all the other tests having tighter tolerances

into feature/3299-chainset

mitzimorris · 2024-10-16T19:16:50Z

But, if these differences can be justified, then we shouldn't be testing against those values at all.

I would be happy to find another way to test these. what would you suggest?

WardBrian · 2024-10-16T19:21:14Z

What previous implementations of diagnostics seem to do is just run the code once to get values and then "freeze" those in the test. I think #3312 deleted some tests in that style.

Doing this requires us to be pretty certain the implementation is correct, since we're making it the gold standard for all future versions of the code, which is why it would be important to understand what parts of the calculation are different than R

mitzimorris · 2024-10-16T19:32:49Z

which is why it would be important to understand what parts of the calculation are different than R

perhaps we should create an artificial dataset to convince ourselves of the goodness of ESS and Rhat. what did you use for Javascript?

WardBrian · 2024-10-16T19:35:08Z

For testing I ended up using the same blocker1.csv and blocker2.csv as the stan tests, but for initially getting them to agree I was just using an arbitrary CSV file, I think from the Bernoulli model?

stan-buildbot · 2024-10-16T21:23:16Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.35	0.34	1.04	3.66% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.06	5.68% faster
gp_regr/gen_gp_data.stan	0.03	0.03	1.1	9.3% faster
gp_regr/gp_regr.stan	0.09	0.09	1.06	5.98% faster
sir/sir.stan	70.01	71.77	0.98	-2.52% slower
irt_2pl/irt_2pl.stan	4.15	4.49	0.93	-8.1% slower
eight_schools/eight_schools.stan	0.06	0.06	1.0	-0.25% slower
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.26	0.95	-5.63% slower
pkpd/one_comp_mm_elim_abs.stan	19.4	19.46	1.0	-0.28% slower
garch/garch.stan	0.44	0.44	1.0	-0.08% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.72	2.71	1.01	0.51% faster
arK/arK.stan	1.8	1.8	1.0	0.13% faster
gp_pois_regr/gp_pois_regr.stan	2.88	3.07	0.94	-6.68% slower
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.83	8.62	1.02	2.31% faster
performance.compilation	182.25	178.41	1.02	2.11% faster
Mean result: 1.0063032409056276

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

into feature/3299-chainset

stan-buildbot · 2024-10-21T23:41:48Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.37	0.35	1.04	3.89% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.04	3.64% faster
gp_regr/gen_gp_data.stan	0.03	0.03	1.19	15.76% faster
gp_regr/gp_regr.stan	0.1	0.09	1.03	3.31% faster
sir/sir.stan	69.95	69.81	1.0	0.2% faster
irt_2pl/irt_2pl.stan	4.07	4.02	1.01	1.19% faster
eight_schools/eight_schools.stan	0.06	0.06	1.02	2.21% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.25	0.99	-1.01% slower
pkpd/one_comp_mm_elim_abs.stan	19.15	18.84	1.02	1.63% faster
garch/garch.stan	0.45	0.41	1.1	8.81% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.78	2.6	1.07	6.59% faster
arK/arK.stan	1.81	2.58	0.7	-42.36% slower
gp_pois_regr/gp_pois_regr.stan	2.83	2.71	1.04	4.07% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.85	8.43	1.05	4.73% faster
performance.compilation	178.71	183.63	0.97	-2.75% slower
Mean result: 1.0185136446700325

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

into feature/3299-chainset

stan-buildbot · 2024-10-22T03:24:12Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.36	0.36	0.99	-0.67% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	0.94	-6.54% slower
gp_regr/gen_gp_data.stan	0.03	0.03	1.0	0.42% faster
gp_regr/gp_regr.stan	0.1	0.09	1.03	3.21% faster
sir/sir.stan	70.33	72.61	0.97	-3.24% slower
irt_2pl/irt_2pl.stan	4.17	4.28	0.97	-2.74% slower
eight_schools/eight_schools.stan	0.06	0.06	0.98	-2.4% slower
pkpd/sim_one_comp_mm_elim_abs.stan	0.26	0.25	1.06	5.97% faster
pkpd/one_comp_mm_elim_abs.stan	19.49	19.02	1.02	2.42% faster
garch/garch.stan	0.44	0.41	1.07	6.21% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.7	2.57	1.05	4.66% faster
arK/arK.stan	1.79	1.7	1.05	4.93% faster
gp_pois_regr/gp_pois_regr.stan	2.82	2.67	1.06	5.33% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.78	8.44	1.04	3.83% faster
performance.compilation	178.74	177.93	1.0	0.45% faster
Mean result: 1.0162582955033252

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

into feature/3299-chainset

stan-buildbot · 2024-10-22T15:14:20Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.43	0.38	1.13	11.79% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.11	9.84% faster
gp_regr/gen_gp_data.stan	0.03	0.03	1.06	5.72% faster
gp_regr/gp_regr.stan	0.11	0.1	1.15	12.69% faster
sir/sir.stan	80.7	74.12	1.09	8.16% faster
irt_2pl/irt_2pl.stan	5.11	4.5	1.13	11.89% faster
eight_schools/eight_schools.stan	0.06	0.06	1.07	6.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.27	0.26	1.03	2.9% faster
pkpd/one_comp_mm_elim_abs.stan	22.08	20.37	1.08	7.74% faster
garch/garch.stan	0.65	0.46	1.42	29.71% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.99	2.81	1.06	6.04% faster
arK/arK.stan	2.0	1.85	1.08	7.82% faster
gp_pois_regr/gp_pois_regr.stan	3.44	2.94	1.17	14.51% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	9.72	9.39	1.04	3.45% faster
performance.compilation	199.44	204.54	0.98	-2.55% slower
Mean result: 1.1069147395963945

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 3375.014
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

into feature/3299-chainset

mitzimorris · 2024-10-22T17:04:26Z

The unit tests now test basic ESS and Rhat against the values computed by CmdStan 2.35.0's bin/stansummary and the saved summary CSV file has been checked into the folder src/test/unit/analyze/mcmc/test_csv_files.

I have added unit tests for rank-normalization and splitting the chains - the latter did uncover a bug where splitting a chain with an odd number of draws would omit the last draw, not the (N+1)/2 draw; fixed - but this bug would not have been tickled by any of the existing tests since all CSV test sets had and even number of draws.

WardBrian

A few things I'd like to see before merge but then I think this can fly. Thank you for the diligence!

src/stan/analyze/mcmc/compute_effective_sample_size.hpp

src/stan/analyze/mcmc/compute_potential_scale_reduction.hpp

src/stan/analyze/mcmc/mcse.hpp

src/stan/mcmc/chainset.hpp

stan-buildbot · 2024-10-22T20:56:00Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.37	0.38	0.98	-1.81% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	0.71	-41.17% slower
gp_regr/gen_gp_data.stan	0.03	0.03	1.1	8.85% faster
gp_regr/gp_regr.stan	0.1	0.1	0.94	-5.86% slower
sir/sir.stan	76.62	74.8	1.02	2.37% faster
irt_2pl/irt_2pl.stan	4.31	4.17	1.03	3.27% faster
eight_schools/eight_schools.stan	0.06	0.06	0.97	-2.85% slower
pkpd/sim_one_comp_mm_elim_abs.stan	0.26	0.26	0.98	-1.88% slower
pkpd/one_comp_mm_elim_abs.stan	19.74	20.48	0.96	-3.78% slower
garch/garch.stan	0.46	0.44	1.05	5.17% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.82	2.76	1.02	1.96% faster
arK/arK.stan	1.8	1.84	0.98	-1.86% slower
gp_pois_regr/gp_pois_regr.stan	2.84	2.89	0.98	-1.7% slower
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	9.06	9.04	1.0	0.13% faster
performance.compilation	187.23	194.85	0.96	-4.07% slower
Mean result: 0.9806370865532955

Jenkins Console Log
Blue Ocean
Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

mitzimorris added 2 commits October 11, 2024 14:24

Merge branch 'develop' of https://github.com/stan-dev/stan into develop

f627e7f

adding chainset and unit tests

7946efe

mitzimorris marked this pull request as draft October 11, 2024 20:55

mitzimorris and others added 8 commits October 11, 2024 17:15

basic_ess needed for mcse

35bdb07

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

12811f2

added ess_basic, unit tests

7b4843f

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

4b65534

into feature/3299-chainset

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

da1b6c5

mcse, mcse_sd match posterior implementation

6cba5df

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

4b4b8b0

into feature/3299-chainset

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

345fcea

mitzimorris requested a review from WardBrian October 12, 2024 19:13

WardBrian requested changes Oct 15, 2024

View reviewed changes

mitzimorris marked this pull request as ready for review October 15, 2024 16:02

mitzimorris and others added 3 commits October 15, 2024 12:15

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

427cf70

into feature/3299-chainset

changes per code review

79c675a

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

ead22cc

mitzimorris and others added 3 commits October 16, 2024 15:10

changes per code review

3534106

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

e38ddfc

into feature/3299-chainset

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

210ac05

mitzimorris and others added 3 commits October 17, 2024 19:48

ess use stan::analyze::autocovariance, not stan::math

ada8407

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

2a21f04

into feature/3299-chainset

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

9fed191

mitzimorris and others added 5 commits October 21, 2024 16:23

changes per code review

c38b15f

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

199155d

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

32b7fec

into feature/3299-chainset

changes per code review

b4febe8

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

b4bca53

mitzimorris and others added 6 commits October 21, 2024 20:06

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

608217c

into feature/3299-chainset

increase test precision, fix split chain logic

bf0d581

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

c411907

more unit tests

17009f8

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

2950a00

into feature/3299-chainset

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

1195c35

mitzimorris and others added 3 commits October 22, 2024 08:53

unit tests against cmdstan 2.35.0

edbc96e

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

3099088

into feature/3299-chainset

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

c3ae868

mitzimorris added 2 commits October 22, 2024 12:15

Merge branch 'feature/3299-chainset' of https://github.com/stan-dev/stan

1ccb254

into feature/3299-chainset

checkpointing

ad0bd35

mitzimorris requested a review from SteveBronder October 22, 2024 17:04

WardBrian approved these changes Oct 22, 2024

View reviewed changes

mitzimorris and others added 2 commits October 22, 2024 14:36

changes per code review

2ee4dc1

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

68a3cfb

mitzimorris removed the request for review from SteveBronder October 22, 2024 18:37

mitzimorris merged commit 9048555 into develop Oct 22, 2024
3 checks passed

mitzimorris mentioned this pull request Oct 24, 2024

Feature/1263 update stansummary to report rank-normalized ESS tail, ESS bulk, max abs deviation(MAD), and Rhat stan-dev/cmdstan#1290

Open

2 tasks

WardBrian mentioned this pull request Nov 7, 2024

use improved Rhat to implement ESS-bulk and ESS-tail #3299

Closed

WardBrian deleted the feature/3299-chainset branch November 14, 2024 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/3299 chainset #3313

Feature/3299 chainset #3313

mitzimorris commented Oct 11, 2024 •

edited

Loading

stan-buildbot commented Oct 12, 2024

WardBrian left a comment

WardBrian commented Oct 16, 2024 •

edited

Loading

mitzimorris commented Oct 16, 2024

WardBrian commented Oct 16, 2024

mitzimorris commented Oct 16, 2024

WardBrian commented Oct 16, 2024

stan-buildbot commented Oct 16, 2024

stan-buildbot commented Oct 21, 2024

stan-buildbot commented Oct 22, 2024

stan-buildbot commented Oct 22, 2024

mitzimorris commented Oct 22, 2024

WardBrian left a comment

stan-buildbot commented Oct 22, 2024

Feature/3299 chainset #3313

Feature/3299 chainset #3313

Conversation

mitzimorris commented Oct 11, 2024 • edited Loading

Submission Checklist

Summary

How to Verify

Side Effects

Documentation

Copyright and Licensing

stan-buildbot commented Oct 12, 2024

WardBrian left a comment

Choose a reason for hiding this comment

WardBrian commented Oct 16, 2024 • edited Loading

mitzimorris commented Oct 16, 2024

WardBrian commented Oct 16, 2024

mitzimorris commented Oct 16, 2024

WardBrian commented Oct 16, 2024

stan-buildbot commented Oct 16, 2024

stan-buildbot commented Oct 21, 2024

stan-buildbot commented Oct 22, 2024

stan-buildbot commented Oct 22, 2024

mitzimorris commented Oct 22, 2024

WardBrian left a comment

Choose a reason for hiding this comment

stan-buildbot commented Oct 22, 2024

mitzimorris commented Oct 11, 2024 •

edited

Loading

WardBrian commented Oct 16, 2024 •

edited

Loading