agd throws panic: counter cannot decrease in value

[michaelfig]

Describe the bug

If agd is configured with telemetry enabled, and runs a fairly typical workload (e.g. with a chainStorage set or delete that decrease the size of the stored key), it panics the Golang side, causing an exception on the JS side.

To Reproduce

Steps to reproduce the behavior:

1. git checkout mfig-devnet-config-steps (the PR fix(vm-config): vats/init-core.js relies on econCommittee #11056 branch)
2. cd agoric-sdk/packages/cosmic-swingset
3. make scenario2-setup scenario2-run-chain to start a new chain with Golang telemetry enabled
4. Watch the output until it starts to produce blocks past height 17
5. In another terminal, run make fund-provision-pool (which triggers the SwingSet provisionPool vat to interact with the Cosmos bridge)
6. Run make provision-acct ACCT_ADDR=$(cat t1/8000/ag-cosmos-helper-address)
7. See panic: counter cannot decrease in value errors logged by the running chain, as below

2025-02-27T03:09:04.893Z SwingSet: ls: v10: Error#1: panic: counter cannot decrease in value

2025-02-27T03:09:00.020Z block-manager: block 27 commit
2025-02-27T03:09:04.782Z block-manager: block 28 begin
2025-02-27T03:09:04.794Z SwingSet: vat: v32: ----- ProvPool.4  26 PLEASE_PROVISION { address: 'agoric16zl6s220j6mxa9saeyhymvaatagzvmaxpmcykt', autoProvision: false, blockHeight: 27, blockTime: 1740625734, nickname: 't1/8000', powerFlags: [ 'SMART_WALLET' ], submitter: 'agoric1h8sekr3hetafcl7aaq8m0w4mmxrnk7u9x83s7u', type: 'PLEASE_PROVISION' }
2025-02-27T03:09:04.817Z SwingSet: vat: v32: ----- ProvPool.4  27 provisionPool balance update { brand: Object [Alleged: IST brand] {}, value: 999_500_000n }
2025-02-27T03:09:04.881Z SwingSet: vat: v32: ----- ProvPool.4  28 provisionPool sent { brand: Object [Alleged: IST brand] {}, value: 250_000n }
2025-02-27T03:09:04.893Z SwingSet: ls: v10: Logging sent error stack (Error#1)
2025-02-27T03:09:04.893Z SwingSet: ls: v10: Error#1: panic: counter cannot decrease in value
2025-02-27T03:09:04.893Z SwingSet: ls: v10: Error: panic: counter cannot decrease in value
 at apply ()
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:60)
 at outbound (.../vats/src/bridge.js:162)
 at outbound (.../vats/src/bridge.js:157)
 at apply ()
 at In "outbound" method of (BridgeManagerKit privateOutbounder) (/bundled-source/.../node_modules/@endo/exo/src/exo-tools.js:171)
 at apply ()
 at localApplyMethod (/bundled-source/.../node_modules/@endo/eventual-send/src/local.js:126)
 at apply ()
 at dispatchToHandler (/bundled-source/.../node_modules/@endo/eventual-send/src/handled-promise.js:159)
 at win (/bundled-source/.../node_modules/@endo/eventual-send/src/handled-promise.js:511)
 at ()

2025-02-27T03:09:04.894Z SwingSet: ls: v10: Error#1 ERROR_NOTE: Sent as error:liveSlots:v10#70001
2025-02-27T03:09:04.895Z SwingSet: vat: v32: ----- ProvPool.4  29 re-provisioned agoric16zl6s220j6mxa9saeyhymvaatagzvmaxpmcykt
2025-02-27T03:09:04.896Z SwingSet: xsnap: v32: UnhandledPromiseRejectionWarning: (RemoteError(error:liveSlots:v10#70001)#1)
2025-02-27T03:09:04.896Z SwingSet: xsnap: v32: RemoteError(error:liveSlots:v10#70001)#1: panic: counter cannot decrease in value
2025-02-27T03:09:04.896Z SwingSet: xsnap: v32: Error: panic: counter cannot decrease in value
 at apply ()
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:60)
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:352)
 at decodeErrorCommon (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:309)
 at decodeFromSmallcaps (/bundled-source/.../node_modules/@endo/marshal/src/encodeToSmallcaps.js:437)
 at fromCapData (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:398)
 at notifyOnePromise (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1043)
 at notify (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1067)
 at dispatchToUserspace (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1336)
 at runWithoutMetering (/bundled-source/.../packages/swingset-xsnap-supervisor/lib/supervisor-subprocess-xsnap.js:59)
 at ()

2025-02-27T03:09:04.906Z block-manager: block 28 commit
2025-02-27T03:09:09.879Z block-manager: block 29 begin

Expected behavior

No panics.

Platform Environment

• what OS are you using? macOS 14.7.4 what version of Node.js? v20.7.0
• is there anything special/unusual about your platform? no
• what version of the Agoric-SDK are you using? PR fix(vm-config): vats/init-core.js relies on econCommittee #11056 , currently git describe --tags --always yields fast-usdc-beta-1-137-gef30b2a883

Additional context

#10997 introduced an idiom that looks like:

			sizeDelta := float32(-len(encodedKey) - len(oldRawValue))
			telemetry.IncrCounterWithLabels(MetricKeysSizeDelta, sizeDelta, NewMetricsLabels(k))

sizeDelta may very well be negative, which is invalid for telemetry.IncrCounter*. For the metric to go up and down over time, use telemetry.SetGauge* instead. More detail than only the current number may need to be implemented with multiple Counters and/or Gauges using different metric keys.