From 092a67cbc07e5e996bf35cb1b4d438077030c1af Mon Sep 17 00:00:00 2001 From: hanako mumei <81144685+2501babe@users.noreply.github.com> Date: Sun, 20 Oct 2024 21:48:54 -0700 Subject: [PATCH 01/11] SIMD-0186: Transaction Data Size Specification --- ...186-transaction-data-size-specification.md | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 proposals/0186-transaction-data-size-specification.md diff --git a/proposals/0186-transaction-data-size-specification.md b/proposals/0186-transaction-data-size-specification.md new file mode 100644 index 000000000..7d0e23d6f --- /dev/null +++ b/proposals/0186-transaction-data-size-specification.md @@ -0,0 +1,143 @@ +--- +simd: '0186' +title: Transaction Data Size Specification +authors: + - Hanako Mumei +category: Standard +type: Core +status: Review +created: 2024-10-20 +feature: (fill in with feature tracking issues once accepted) +--- + +## Summary + +Before a transaction can be executed, every account it may read from or write to +must be loaded, including any programs it may call. The amount of data a +transaction is allowed to load is capped, and if it exceeds that limit, loading +is aborted. This functionality is already implemented in the validator. The +purpose of this SIMD is to explicitly define how transaction size is calculated. + +## Motivation + +Transaction data size accounting is currently unspecified, and the +implementation-defined algorithm used in the Agave client exhibits some +surprising behaviors: + +* BPF loaders required by top-level programs are counted against transaction +data size. BPF loaders required by CPI programs are not. If a required BPF +loader is also included in the accounts list, it is counted twice. +* The size of a program owned by LoaderV3 may or may not include the size of its +programdata depending on how the program account is used on the transaction. +Programdata is also itself counted if included in the transaction accounts list. +This means programdata may be counted zero, one, or two times per transaction. + +All validator clients must arrive at precisely the same transaction data size +for all transactions because a difference of one byte can determine whether a +transaction is executed or failed, and thus affects consensus. Also, we want the +calculated transaction data size to correspond well with the actual amount of +data the transaction requests. + +Therefore, this SIMD seeks to specify an algorithm that is straightforward to +implement in a client-agnostic way, while also accurately accounting for the +total data required by the transaction. + +## New Terminology + +One term is defined within the scope of this SIMD: + +* Valid program: a program that has been loaded, or a builtin. This definition +excludes programs that have failed verification, or LoaderV3 programs that have +been closed or have delayed visibility due to being deployed or modified in the +current slot. + +These terms are not new, however we define them for clarity: + +* Top-level program: the program corresponding to the program id on a given +instruction. +* Instruction account: an account passed to an instruction, which allows its +program to view the actual bytes of the account. CPI can only happen through +programs provided as instruction accounts. +* Transaction accounts list: all accounts for the transaction, which includes +top-level programs, the fee-payer, all instruction accounts, and any extra +accounts added to the list but not used for any purpose. + +## Detailed Design + +The proposed algorithm is as follows: + +1. Every account explicitly included on the transaction accounts list is counted +once and only once. +2. A valid program owned by LoaderV3 also includes the size of its programdata. +3. Other than point 2, no accounts are implicitly added to the total data size. + +Transactions may include a +`ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define +a data size limit for the transaction. Otherwise, the default limit is 64MiB +(`64 * 1024 * 1024` bytes). + +If a transaction exceeds its data size limit, account loading is aborted and the +transaction is failed. Fees will be charged once +`enable_transaction_loading_failure_fees` is enabled. + +Adding required loaders to transaction data size is abolished. They are treated +the same as any other account: counted if on the transaction accounts list, not +counted otherwise. + +Read-only and writable accounts are treated the same. In the future, when direct +mapping is enabled, this SIMD may be amended to count them differently. + +As a consequence of 1 and 2, for LoaderV3 programs, programdata is counted twice +if a transaction includes both programdata and the program account itself in the +accounts list, unless the program is not valid for execution. This is partly +done for ease of implementation: we always want to count programdata when the +program is included, and there is no reason for any transaction to include both +accounts except during initial deployment, in which case the program is not yet +valid. + +We include programdata size in program size for LoaderV3 programs because in +nearly all cases a transaction will include the program account (the only way to +invoke the program) and will not include the programdata account because +including it serves no purpose. Including the program account forces an +unconditional load of the programdata account because it is required to compile +the program for execution. Therefore we always count it, even when the program +is an instruction account. + +There is no special handling for programs owned by the native loader, LoaderV1, +or LoaderV2. + +Account size for programs owned by LoaderV4 is left undefined. This SIMD should +be amended to define the required semantics before LoaderV4 is enabled on any +network. + +## Alternatives Considered + +* Transaction data size accounting is already enabled, so the null option is to +enshrine the current Agave behavior in the protocol. This is undesirable because +the current behavior is highly idiosyncratic, and LoaderV3 program sizes are +routinely undercounted. +* Builtin programs are backed by accounts that only contain the program name as +a string, typically making them 15-40 bytes. We could make them free when not +instruction accounts, since they're part of the validator. However this +adds complexity for no real benefit. + +## Impact + +The primary impact is this SIMD makes correctly implementing transaction data +size accounting much easier for other validator clients. + +It makes transactions which include program accounts for CPI somewhat larger, +but given the generous 64MiB limit, it is unlikely that any existing users will +be affected. + +## Security Considerations + +Security impact is minimal because this SIMD merely simplifies an existing +feature. + +This SIMD requires a feature gate. + +## Backwards Compatibility + +Transactions that call LoaderV3 programs via CPI and are extremely close to the +64MiB limit may now exceed it. From 04e283b258aa3e049e728189024a37bbf8151467 Mon Sep 17 00:00:00 2001 From: hanako mumei <81144685+2501babe@users.noreply.github.com> Date: Mon, 21 Oct 2024 07:48:25 -0700 Subject: [PATCH 02/11] edits for clarity --- proposals/0186-transaction-data-size-specification.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/proposals/0186-transaction-data-size-specification.md b/proposals/0186-transaction-data-size-specification.md index 7d0e23d6f..b01b483ab 100644 --- a/proposals/0186-transaction-data-size-specification.md +++ b/proposals/0186-transaction-data-size-specification.md @@ -16,7 +16,8 @@ Before a transaction can be executed, every account it may read from or write to must be loaded, including any programs it may call. The amount of data a transaction is allowed to load is capped, and if it exceeds that limit, loading is aborted. This functionality is already implemented in the validator. The -purpose of this SIMD is to explicitly define how transaction size is calculated. +purpose of this SIMD is to explicitly define how loaded transaction data size is +calculated. ## Motivation @@ -139,5 +140,5 @@ This SIMD requires a feature gate. ## Backwards Compatibility -Transactions that call LoaderV3 programs via CPI and are extremely close to the -64MiB limit may now exceed it. +Transactions that currently have a total transaction data size close to the +64MiB limit, which call LoaderV3 programs via CPI, may now exceed it and fail. From c694155a8254ee665c19c6bf1d878f2d5d5615ba Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Wed, 30 Oct 2024 03:17:34 -0700 Subject: [PATCH 03/11] new language for another edge case --- proposals/0186-transaction-data-size-specification.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/proposals/0186-transaction-data-size-specification.md b/proposals/0186-transaction-data-size-specification.md index b01b483ab..677665df4 100644 --- a/proposals/0186-transaction-data-size-specification.md +++ b/proposals/0186-transaction-data-size-specification.md @@ -32,6 +32,10 @@ loader is also included in the accounts list, it is counted twice. programdata depending on how the program account is used on the transaction. Programdata is also itself counted if included in the transaction accounts list. This means programdata may be counted zero, one, or two times per transaction. +* Due to certain quirks of implementation, accounts owned by loaders which do +not contain valid programs for execution may or may not be counted against the +transaction data size total depending on how it is used on the transaction. This +includes, but is not limited to, LoaderV3 buffer accounts. All validator clients must arrive at precisely the same transaction data size for all transactions because a difference of one byte can determine whether a @@ -107,6 +111,10 @@ is an instruction account. There is no special handling for programs owned by the native loader, LoaderV1, or LoaderV2. +There is no special handling for non-executable accounts owned by the loaders. +These are to be included in the transaction data size total under all +circumstances. + Account size for programs owned by LoaderV4 is left undefined. This SIMD should be amended to define the required semantics before LoaderV4 is enabled on any network. From 078a6f9749f4919aa9937d91954834bfe048e9e6 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Mon, 4 Nov 2024 03:37:52 -0800 Subject: [PATCH 04/11] update to new algorithm --- ...ded-transaction-data-size-specification.md | 163 ++++++++++++++++++ ...186-transaction-data-size-specification.md | 152 ---------------- 2 files changed, 163 insertions(+), 152 deletions(-) create mode 100644 proposals/0186-loaded-transaction-data-size-specification.md delete mode 100644 proposals/0186-transaction-data-size-specification.md diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md new file mode 100644 index 000000000..df8800096 --- /dev/null +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -0,0 +1,163 @@ +--- +simd: '0186' +title: Loaded Transaction Data Size Specification +authors: + - Hanako Mumei +category: Standard +type: Core +status: Review +created: 2024-10-20 +feature: (fill in with feature tracking issues once accepted) +--- + +## Summary + +Before a transaction can be executed, every account it may read from or write to +must be loaded, including any programs it may call. The amount of data a +transaction is allowed to load is capped, and if it exceeds that limit, loading +is aborted. This functionality is already implemented in the validator. The +purpose of this SIMD is to explicitly define how loaded transaction data size is +calculated. + +## Motivation + +Transaction data size accounting is currently unspecified, and the +implementation-defined algorithm used in the Agave client exhibits some +surprising behaviors: + +* BPF loaders required by instructions' program IDs are counted against +transaction data size. BPF loaders required by CPI programs are not. If a +required BPF loader is also included in the accounts list, it is counted twice. +* The size of a program owned by LoaderV3 may or may not include the size of its +programdata depending on how the program account is used on the transaction. +Programdata is also itself counted if included in the transaction accounts list. +This means programdata may be counted zero, one, or two times per transaction. +* Due to certain quirks of implementation, loader-owned accounts which do not +contain valid programs for execution may or may not be counted against the +transaction data size total depending on how they are used on the transaction. +This includes, but is not limited to, LoaderV3 buffer accounts, and accounts +which fail ELF validation. +* Accounts can be included on a transaction account list without being an +instruction account, fee-payer, or program ID. These accounts are presently +loaded and counted against transaction data size, although they can never be +used for any purpose by the transaction. + +All validator clients must arrive at precisely the same transaction data size +for all transactions because a difference of one byte can determine whether a +transaction is executed or failed, and thus affects consensus. Also, we want the +calculated transaction data size to correspond well with the actual amount of +data the transaction requests. + +Therefore, this SIMD seeks to specify an algorithm that is straightforward to +implement in a client-agnostic way, while also accurately accounting for all +account data required by the transaction. + +## New Terminology + +No new terms are introduced by this SIMD, however we define these for clarity: + +* Instruction account: an account passed to an instruction in its accounts +array, which allows the program to view the actual bytes contained in the +account. CPI can only happen through programs provided as instruction accounts. +* Transaction accounts list: all accounts for the transaction, which includes +instruction accounts, the fee-payer, program IDs, and any extra accounts added +to the list but not used for any purpose. +* LoaderV3 program account: an account owned by +`BPFLoaderUpgradeab1e11111111111111111111111` which contains in its account data +the first four bytes `02 00 00 00` followed by a pubkey which points to an +account which is defined as the program's programdata account. + +For the purposes of this SIMD, we make no assumptions about the contents of the +programdata account. + +## Detailed Design + +The proposed algorithm is as follows: + +1. Given a transaction, take the unique set of account keys which are used as: + * An instruction account. + * A program ID for an instruction. + * The fee-payer. +2. Each account's size is determined solely by the byte length of its data prior +to transaction execution. +3. For any `LoaderV3` program account, add the size of the programdata account +it references, if it exists. +4. The total transaction size is the sum of these sizes. + +Transactions may include a +`ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define +a data size limit for the transaction. Otherwise, the default limit is 64MiB +(`64 * 1024 * 1024` bytes). + +If a transaction exceeds its data size limit, the transaction is failed. Fees +will be charged once `enable_transaction_loading_failure_fees` is enabled. + +Adding required loaders to transaction data size is abolished. They are treated +the same as any other account: counted if used in a manner described by 1, not +counted otherwise. + +No account that falls outside of the three categories listed by 1 is counted +against transaction data size. Validator clients are free to decline to load +them. + +Read-only and writable accounts are treated the same. In the future, when direct +mapping is enabled, this SIMD may be amended to count them differently. + +As a consequence of 1 and 3, for LoaderV3 programs, programdata is counted twice +if a transaction explicitly references the program account and its programdata +account. This is done partly for simplicity, and partly to account for the cost +of maintaining the compiled program in addition to the actual bytes of +the programdata account. + +We include programdata size in account size for LoaderV3 programs because using +the program account on a transaction forces an unconditional load of programdata +to compile the program for execution. We always count it, even when the program +is an instruction account, because the program must be available for CPI. + +There is no special handling for any account owned by the native loader, +LoaderV1, or LoaderV2. + +Account size for programs owned by LoaderV4 is left undefined. This SIMD should +be amended to define the required semantics before LoaderV4 is enabled on any +network. + +## Alternatives Considered + +* Transaction data size accounting is already enabled, so the null option is to +enshrine the current Agave behavior in the protocol. This is undesirable because +the current behavior is highly idiosyncratic, and LoaderV3 program sizes are +routinely undercounted. +* Builtin programs are backed by accounts that only contain the program name as +a string, typically making them 15-40 bytes. We could impose a larger fixed cost +for these. However, they must be made available for all programs anyway, and +most of them are likely to be ported to BPF eventually, so this adds complexity +for no real benefit. +* Several slightly different algorithms were considered for handling LoaderV3 +programs in particular, for instance only counting programs that are valid for +execution in the current slot. However, this would implicitly couple transaction +data size with the results of ELF validation, which is highly undesirable. +* We considered loading and counting sizes for accounts on the transaction +account list which are not used for any purpose. This is the current behavior, +but there is no reason to load such accounts at all. + +## Impact + +The primary impact is this SIMD makes correctly implementing transaction data +size accounting much easier for other validator clients. + +It makes the calculated size of transactions which include program accounts for +CPI somewhat larger, but given the generous 64MiB limit, it is unlikely that any +existing users will be affected. Based on an investigation of a 30-day window, +transactions larger than 30MiB are virtually never seen. + +## Security Considerations + +Security impact is minimal because this SIMD merely simplifies an existing +feature. Care must be taken to implement the rules exactly. + +This SIMD requires a feature gate. + +## Backwards Compatibility + +Transactions that currently have a total transaction data size close to the +64MiB limit, which call LoaderV3 programs via CPI, may now exceed it and fail. diff --git a/proposals/0186-transaction-data-size-specification.md b/proposals/0186-transaction-data-size-specification.md deleted file mode 100644 index 677665df4..000000000 --- a/proposals/0186-transaction-data-size-specification.md +++ /dev/null @@ -1,152 +0,0 @@ ---- -simd: '0186' -title: Transaction Data Size Specification -authors: - - Hanako Mumei -category: Standard -type: Core -status: Review -created: 2024-10-20 -feature: (fill in with feature tracking issues once accepted) ---- - -## Summary - -Before a transaction can be executed, every account it may read from or write to -must be loaded, including any programs it may call. The amount of data a -transaction is allowed to load is capped, and if it exceeds that limit, loading -is aborted. This functionality is already implemented in the validator. The -purpose of this SIMD is to explicitly define how loaded transaction data size is -calculated. - -## Motivation - -Transaction data size accounting is currently unspecified, and the -implementation-defined algorithm used in the Agave client exhibits some -surprising behaviors: - -* BPF loaders required by top-level programs are counted against transaction -data size. BPF loaders required by CPI programs are not. If a required BPF -loader is also included in the accounts list, it is counted twice. -* The size of a program owned by LoaderV3 may or may not include the size of its -programdata depending on how the program account is used on the transaction. -Programdata is also itself counted if included in the transaction accounts list. -This means programdata may be counted zero, one, or two times per transaction. -* Due to certain quirks of implementation, accounts owned by loaders which do -not contain valid programs for execution may or may not be counted against the -transaction data size total depending on how it is used on the transaction. This -includes, but is not limited to, LoaderV3 buffer accounts. - -All validator clients must arrive at precisely the same transaction data size -for all transactions because a difference of one byte can determine whether a -transaction is executed or failed, and thus affects consensus. Also, we want the -calculated transaction data size to correspond well with the actual amount of -data the transaction requests. - -Therefore, this SIMD seeks to specify an algorithm that is straightforward to -implement in a client-agnostic way, while also accurately accounting for the -total data required by the transaction. - -## New Terminology - -One term is defined within the scope of this SIMD: - -* Valid program: a program that has been loaded, or a builtin. This definition -excludes programs that have failed verification, or LoaderV3 programs that have -been closed or have delayed visibility due to being deployed or modified in the -current slot. - -These terms are not new, however we define them for clarity: - -* Top-level program: the program corresponding to the program id on a given -instruction. -* Instruction account: an account passed to an instruction, which allows its -program to view the actual bytes of the account. CPI can only happen through -programs provided as instruction accounts. -* Transaction accounts list: all accounts for the transaction, which includes -top-level programs, the fee-payer, all instruction accounts, and any extra -accounts added to the list but not used for any purpose. - -## Detailed Design - -The proposed algorithm is as follows: - -1. Every account explicitly included on the transaction accounts list is counted -once and only once. -2. A valid program owned by LoaderV3 also includes the size of its programdata. -3. Other than point 2, no accounts are implicitly added to the total data size. - -Transactions may include a -`ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define -a data size limit for the transaction. Otherwise, the default limit is 64MiB -(`64 * 1024 * 1024` bytes). - -If a transaction exceeds its data size limit, account loading is aborted and the -transaction is failed. Fees will be charged once -`enable_transaction_loading_failure_fees` is enabled. - -Adding required loaders to transaction data size is abolished. They are treated -the same as any other account: counted if on the transaction accounts list, not -counted otherwise. - -Read-only and writable accounts are treated the same. In the future, when direct -mapping is enabled, this SIMD may be amended to count them differently. - -As a consequence of 1 and 2, for LoaderV3 programs, programdata is counted twice -if a transaction includes both programdata and the program account itself in the -accounts list, unless the program is not valid for execution. This is partly -done for ease of implementation: we always want to count programdata when the -program is included, and there is no reason for any transaction to include both -accounts except during initial deployment, in which case the program is not yet -valid. - -We include programdata size in program size for LoaderV3 programs because in -nearly all cases a transaction will include the program account (the only way to -invoke the program) and will not include the programdata account because -including it serves no purpose. Including the program account forces an -unconditional load of the programdata account because it is required to compile -the program for execution. Therefore we always count it, even when the program -is an instruction account. - -There is no special handling for programs owned by the native loader, LoaderV1, -or LoaderV2. - -There is no special handling for non-executable accounts owned by the loaders. -These are to be included in the transaction data size total under all -circumstances. - -Account size for programs owned by LoaderV4 is left undefined. This SIMD should -be amended to define the required semantics before LoaderV4 is enabled on any -network. - -## Alternatives Considered - -* Transaction data size accounting is already enabled, so the null option is to -enshrine the current Agave behavior in the protocol. This is undesirable because -the current behavior is highly idiosyncratic, and LoaderV3 program sizes are -routinely undercounted. -* Builtin programs are backed by accounts that only contain the program name as -a string, typically making them 15-40 bytes. We could make them free when not -instruction accounts, since they're part of the validator. However this -adds complexity for no real benefit. - -## Impact - -The primary impact is this SIMD makes correctly implementing transaction data -size accounting much easier for other validator clients. - -It makes transactions which include program accounts for CPI somewhat larger, -but given the generous 64MiB limit, it is unlikely that any existing users will -be affected. - -## Security Considerations - -Security impact is minimal because this SIMD merely simplifies an existing -feature. - -This SIMD requires a feature gate. - -## Backwards Compatibility - -Transactions that currently have a total transaction data size close to the -64MiB limit, which call LoaderV3 programs via CPI, may now exceed it and fail. From 8c0603b548bd5e89815cd79fd4ff004677c4f616 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Tue, 5 Nov 2024 08:18:51 -0800 Subject: [PATCH 05/11] minor edits from reviews --- ...aded-transaction-data-size-specification.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index df8800096..f9355b02a 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -15,9 +15,10 @@ feature: (fill in with feature tracking issues once accepted) Before a transaction can be executed, every account it may read from or write to must be loaded, including any programs it may call. The amount of data a transaction is allowed to load is capped, and if it exceeds that limit, loading -is aborted. This functionality is already implemented in the validator. The -purpose of this SIMD is to explicitly define how loaded transaction data size is -calculated. +is aborted. This functionality is already implemented in the validator. + +This SIMD defines a new algorithm for calculating the consensus-enforced total +size of loaded transaction data during transaction processing. ## Motivation @@ -79,15 +80,16 @@ The proposed algorithm is as follows: * A program ID for an instruction. * The fee-payer. 2. Each account's size is determined solely by the byte length of its data prior -to transaction execution. -3. For any `LoaderV3` program account, add the size of the programdata account -it references, if it exists. +to transaction execution, irrespective of it is used on the transaction. +3. For any loaded account identified as a `LoaderV3` program account, add the +size of the programdata account it references to its own size, irrespective of +how the program account is used on the transaction. 4. The total transaction size is the sum of these sizes. Transactions may include a `ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define -a data size limit for the transaction. Otherwise, the default limit is 64MiB -(`64 * 1024 * 1024` bytes). +a lower data size limit for the transaction. Otherwise, the default limit is +64MiB (`64 * 1024 * 1024` bytes). If a transaction exceeds its data size limit, the transaction is failed. Fees will be charged once `enable_transaction_loading_failure_fees` is enabled. From a015ab77624cde74db78f1ef38511e3be21648c1 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Tue, 5 Nov 2024 08:37:35 -0800 Subject: [PATCH 06/11] minor copyedits --- .../0186-loaded-transaction-data-size-specification.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index f9355b02a..b0c21f077 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -80,11 +80,11 @@ The proposed algorithm is as follows: * A program ID for an instruction. * The fee-payer. 2. Each account's size is determined solely by the byte length of its data prior -to transaction execution, irrespective of it is used on the transaction. +to transaction execution, irrespective of how it is used on the transaction. 3. For any loaded account identified as a `LoaderV3` program account, add the size of the programdata account it references to its own size, irrespective of -how the program account is used on the transaction. -4. The total transaction size is the sum of these sizes. +how the program account is used in the transaction. +4. The total transaction loaded account data size is the sum of these sizes. Transactions may include a `ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define From 64db94fa3e8988d0f6cf00d7ba07f4dc296ed6f9 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Wed, 6 Nov 2024 14:30:54 -0800 Subject: [PATCH 07/11] add invisible accounts back --- ...ded-transaction-data-size-specification.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index b0c21f077..5c2c03326 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -75,16 +75,15 @@ programdata account. The proposed algorithm is as follows: -1. Given a transaction, take the unique set of account keys which are used as: - * An instruction account. - * A program ID for an instruction. - * The fee-payer. -2. Each account's size is determined solely by the byte length of its data prior -to transaction execution, irrespective of how it is used on the transaction. -3. For any loaded account identified as a `LoaderV3` program account, add the -size of the programdata account it references to its own size, irrespective of -how the program account is used in the transaction. -4. The total transaction loaded account data size is the sum of these sizes. +1. The set of accounts that determine loaded transaction data size is defined as +the unique intersection of: + * The set of account keys explicitly specified on the transaction, +irrespective of how they are used. + * The set of programdata accounts referenced by the LoaderV3 program +accounts specified on the transaction. +2. Each account's size is defined as the byte length of its data prior to +transaction execution plus 64 bytes to account for metadata. +3. The total transaction loaded account data size is the sum of these sizes. Transactions may include a `ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define From ec590039668ea24d182b1acf8c111f943bc688c3 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Tue, 12 Nov 2024 11:18:39 -0800 Subject: [PATCH 08/11] minor edits --- ...6-loaded-transaction-data-size-specification.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index 5c2c03326..16924b0b2 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -104,12 +104,6 @@ them. Read-only and writable accounts are treated the same. In the future, when direct mapping is enabled, this SIMD may be amended to count them differently. -As a consequence of 1 and 3, for LoaderV3 programs, programdata is counted twice -if a transaction explicitly references the program account and its programdata -account. This is done partly for simplicity, and partly to account for the cost -of maintaining the compiled program in addition to the actual bytes of -the programdata account. - We include programdata size in account size for LoaderV3 programs because using the program account on a transaction forces an unconditional load of programdata to compile the program for execution. We always count it, even when the program @@ -137,9 +131,11 @@ for no real benefit. programs in particular, for instance only counting programs that are valid for execution in the current slot. However, this would implicitly couple transaction data size with the results of ELF validation, which is highly undesirable. -* We considered loading and counting sizes for accounts on the transaction -account list which are not used for any purpose. This is the current behavior, -but there is no reason to load such accounts at all. +* We considered skipping loading of accounts included in the transaction +accounts list but not used as an instruction account, program ID, or fee-payer. +However, [SIMD-0163]( +https://github.com/solana-foundation/solana-improvement-documents/pull/163) +intends to leverage these accounts to make CPI more CU-efficient. ## Impact From 173f092932ae7cbc441a61e53a2b2ffc3ccfddb8 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Tue, 12 Nov 2024 14:14:02 -0800 Subject: [PATCH 09/11] add lookup table cost --- proposals/0186-loaded-transaction-data-size-specification.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index 16924b0b2..c82478ec0 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -83,7 +83,10 @@ irrespective of how they are used. accounts specified on the transaction. 2. Each account's size is defined as the byte length of its data prior to transaction execution plus 64 bytes to account for metadata. -3. The total transaction loaded account data size is the sum of these sizes. +3. There is an additional flat 8248 byte cost for transactions that use an +address lookup table, accounting for the 8192 bytes for the maximum size of such +a table plus 56 bytes for metadata. +4. The total transaction loaded account data size is the sum of these sizes. Transactions may include a `ComputeBudgetInstruction::SetLoadedAccountsDataSizeLimit` instruction to define From 194c751adfc1ed6789ea67147a7e3b9473b19a4c Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Fri, 15 Nov 2024 02:41:44 -0800 Subject: [PATCH 10/11] trivial copyedits since we are enshrining a use for invisible accounts in another simd --- ...aded-transaction-data-size-specification.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index c82478ec0..47b8b75f4 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -38,10 +38,6 @@ contain valid programs for execution may or may not be counted against the transaction data size total depending on how they are used on the transaction. This includes, but is not limited to, LoaderV3 buffer accounts, and accounts which fail ELF validation. -* Accounts can be included on a transaction account list without being an -instruction account, fee-payer, or program ID. These accounts are presently -loaded and counted against transaction data size, although they can never be -used for any purpose by the transaction. All validator clients must arrive at precisely the same transaction data size for all transactions because a difference of one byte can determine whether a @@ -62,7 +58,7 @@ array, which allows the program to view the actual bytes contained in the account. CPI can only happen through programs provided as instruction accounts. * Transaction accounts list: all accounts for the transaction, which includes instruction accounts, the fee-payer, program IDs, and any extra accounts added -to the list but not used for any purpose. +to the list but not explicitly available to programs. * LoaderV3 program account: an account owned by `BPFLoaderUpgradeab1e11111111111111111111111` which contains in its account data the first four bytes `02 00 00 00` followed by a pubkey which points to an @@ -100,17 +96,13 @@ Adding required loaders to transaction data size is abolished. They are treated the same as any other account: counted if used in a manner described by 1, not counted otherwise. -No account that falls outside of the three categories listed by 1 is counted -against transaction data size. Validator clients are free to decline to load -them. - Read-only and writable accounts are treated the same. In the future, when direct mapping is enabled, this SIMD may be amended to count them differently. -We include programdata size in account size for LoaderV3 programs because using -the program account on a transaction forces an unconditional load of programdata -to compile the program for execution. We always count it, even when the program -is an instruction account, because the program must be available for CPI. +We include programdata size for LoaderV3 programs because using the program +account on a transaction forces an unconditional load of programdata to compile +the program for execution. We always count it, even when the program account is +not a transaction program ID, because the program must be available for CPI. There is no special handling for any account owned by the native loader, LoaderV1, or LoaderV2. From 59bf637779f4f906b53d488497f297584b67d223 Mon Sep 17 00:00:00 2001 From: hana <81144685+2501babe@users.noreply.github.com> Date: Sat, 16 Nov 2024 01:41:31 -0800 Subject: [PATCH 11/11] clarify wording --- .../0186-loaded-transaction-data-size-specification.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/0186-loaded-transaction-data-size-specification.md b/proposals/0186-loaded-transaction-data-size-specification.md index 47b8b75f4..7facf6bf0 100644 --- a/proposals/0186-loaded-transaction-data-size-specification.md +++ b/proposals/0186-loaded-transaction-data-size-specification.md @@ -89,8 +89,11 @@ Transactions may include a a lower data size limit for the transaction. Otherwise, the default limit is 64MiB (`64 * 1024 * 1024` bytes). -If a transaction exceeds its data size limit, the transaction is failed. Fees -will be charged once `enable_transaction_loading_failure_fees` is enabled. +If a transaction exceeds its data size limit, a loading failure occurs. This +SIMD does not change any aspect of how such a failure is handled. At time of +writing, such a transaction would be excluded from the ledger. When +`enable_transaction_loading_failure_fees` is enabled, it will be written to the +ledger and charged fees as a processed, failed transaction. Adding required loaders to transaction data size is abolished. They are treated the same as any other account: counted if used in a manner described by 1, not