Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Athena: Initial POC integration #6380

Open
wants to merge 73 commits into
base: athena-poc
Choose a base branch
from
Open

Athena: Initial POC integration #6380

wants to merge 73 commits into from

Conversation

lrettig
Copy link
Member

@lrettig lrettig commented Oct 8, 2024

Motivation

Replace existing genvm functionality with athena

Description

  • clean up and refactor vm code
  • basic athena (Rust) integration
  • replace existing vm template-related code with calls into athena

Test Plan

Includes almost all existing production tests. Disabled a few tests that aren't compatible with the Athena VM, and also added a few Athena-specific tests.

TODO

  • Explain motivation or link existing issue(s)
  • Test changes and document test plan
  • Update documentation as needed

Misc

  • some of the systests rely on using vault, vesting, and multisig accounts. Athena currently doesn't support any of these. These systests were disabled or removed.
  • the genvm code remains but it's totally unused outside of genvm tests.
  • both old and new APIs have been updated to work with the Athena VM. Their existing tests are passing, but I haven't done extensive testing. The v2alpha API does differentiate between spawn and spend transactions, but some data (recipient, amount) is currently missing for spend transactions.
  • this PR also adds an Athena preset config, including genesis config, that can be used for running an Athena devnet/testnet. It's copied from the existing testnet config and just changes the genesis time and adds the single sig wallet template account, which needs to be part of the genesis accounts.
  • obviously, the plan is to eventually redo this integration from scratch. But this POC integration tries to make as few changes to existing code as possible. One consequence is that the VM code is messy. It probably doesn't make sense to split the wallet template and handler in Athena, for instance. Also, the host and context can probably be unified in a future, Athena-native implementation. The goal here is expediency, not pretty, maintainable code.
  • no other changes have been made to the node itself. It should be possible to run as on mainnet/testnet.
  • the account structure also remains mostly the same. The main difference is the addition of State and Storage blobs to the account. Storage is implemented very naively, as an array rather than as a hash table, for simplicity of serialization.
  • the encoded Athena payload is encoded again in the transaction, for now. This has to do with the way go-spacemesh encodes and decodes transactions. It's only capable of handling SCALE-encoded data types, so it was simplest to just SCALE-encode everything, even the opaque payload. This is much simpler but adds one byte to the tx length, which is an acceptable tradeoff for now.
  • this introduces new HRPs ath and atest for Athena transactions, and it changes the tx version byte to 1.

@lrettig lrettig added the area/vm label Oct 8, 2024
common/types/transaction_header.go Outdated Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
Copy link

codecov bot commented Nov 5, 2024

Codecov Report

Attention: Patch coverage is 70.44335% with 300 lines in your changes missing coverage. Please review.

Project coverage is 79.4%. Comparing base (81da1cd) to head (69a90c5).
Report is 1 commits behind head on athena-poc.

Files with missing lines Patch % Lines
vm/host/host.go 49.5% 99 Missing and 9 partials ⚠️
vm/templates/wallet/wallet.go 63.1% 35 Missing and 18 partials ⚠️
vm/core/context.go 75.8% 25 Missing and 12 partials ⚠️
vm/vm.go 78.0% 20 Missing and 11 partials ⚠️
vm/templates/wallet/handler.go 62.9% 13 Missing and 7 partials ⚠️
api/grpcserver/v2alpha1/transaction.go 64.8% 8 Missing and 5 partials ⚠️
vm/sdk/wallet/tx.go 76.1% 6 Missing and 4 partials ⚠️
vm/core/staged_cache.go 0.0% 7 Missing ⚠️
vm/templates/wallet/gas.go 30.0% 7 Missing ⚠️
vm/core/types.go 70.0% 4 Missing and 2 partials ⚠️
... and 2 more
Additional details and impacted files
@@             Coverage Diff              @@
##           athena-poc   #6380     +/-   ##
============================================
- Coverage        80.3%   79.4%   -0.9%     
============================================
  Files             328     330      +2     
  Lines           35765   42715   +6950     
============================================
+ Hits            28752   33957   +5205     
- Misses           5153    6812   +1659     
- Partials         1860    1946     +86     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lrettig lrettig marked this pull request as ready for review November 5, 2024 02:36
@lrettig
Copy link
Member Author

lrettig commented Nov 5, 2024

@poszu this is ready for review. All tests are passing on ubuntu; mac tests are failing due to athenavm/athena#215 which is unrelated to the changes here. Sorry this is so big, but this is a minimum viable integration!

@lrettig lrettig requested a review from poszu November 5, 2024 18:11
Copy link
Contributor

@poszu poszu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posting an in-progress review - I will continue tommorow.

@@ -126,7 +126,7 @@ jobs:
os:
- ubuntu-22.04
- ubuntu-latest-arm-8-cores
- macos-13
# - macos-13
Copy link
Contributor

@poszu poszu Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to leave a note about why it's disabled and when it should be re-enabled:

Suggested change
# - macos-13
# Athena doesn't support Intel Mac
# - macos-13

Comment on lines +19 to +20
# - os: macos-13
# outname_sufix: "mac-amd64"
Copy link
Contributor

@poszu poszu Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

Suggested change
# - os: macos-13
# outname_sufix: "mac-amd64"
# Athena doesn't support Intel Mac
# - os: macos-13
# outname_sufix: "mac-amd64"

@@ -1,10 +1,11 @@
package core

import (
"github.com/spacemeshos/go-scale"
"github.com/ChainSafe/gossamer/pkg/scale"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the go-scale?

Comment on lines +35 to +37
if err != nil {
return Address{}, err
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to panic in this situation, encoding should never fail. A lot of code would become simpler without this unnecessary error handling.

Comment on lines +160 to +170
func (t *Payload) EncodeScale(enc *scale.Encoder) (total int, err error) {
{
n, err := scale.EncodeByteSlice(enc, *t)
if err != nil {
return total, err
}
total += n
}

return total, nil
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func (t *Payload) EncodeScale(enc *scale.Encoder) (total int, err error) {
{
n, err := scale.EncodeByteSlice(enc, *t)
if err != nil {
return total, err
}
total += n
}
return total, nil
}
func (t *Payload) EncodeScale(enc *scale.Encoder) (int, error) {
return scale.EncodeByteSlice(enc, *t)
}

Comment on lines +22 to +23
logger, err := zap.NewDevelopment()
require.NoError(t, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider:

Suggested change
logger, err := zap.NewDevelopment()
require.NoError(t, err)
logger := zaptest.NewLogger(t)

Comment on lines +83 to +98
func TestEmptyCode(t *testing.T) {
host, _ := getHost(t)
defer host.Destroy()

_, _, err := host.Execute(
10,
10,
types.Address{1, 2, 3, 4},
types.Address{1, 2, 3, 4},
nil,
0,
[]byte{},
)

require.ErrorContains(t, err, "athcon execute: no input code")
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not testing any functionality implemented in go-spacemesh but in the Athena bindings. I'd remove it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but that's testing the contract between node and athena lib. imo it's fine.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A contract that it is impossible to execute empty code? This test is already present in the athcon library: https://github.com/athenavm/athena/blob/3f397a807111db7fc9a1ea9efd3be93bd3007730/ffi/athcon/bindings/go/athcon_test.go#L55-L66.

I mean, the check if code is empty is in the athcon library (not anywhere in the go-spacemesh) and it already has a test. It's a requirement imposed by the athcon's vm.Execute() method that the code cannot be empty. The user of the library doesn't need to verify if the library tests its inputs IMO.

Comment on lines +100 to +126
func TestSetGetStorage(t *testing.T) {
host, cache := getHost(t)
defer host.Destroy()

storageKey := athcon.Bytes32{0xc0, 0xff, 0xee}
storageValue := athcon.Bytes32{0xde, 0xad, 0xbe, 0xef}

address := types.Address{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}
account := types.Account{
Address: address,
Balance: 10000,
Storage: []types.StorageItem{
{Key: storageKey, Value: storageValue},
},
}
err := cache.Update(account)
require.NoError(t, err)

_, gasLeft, err := host.Execute(
account.Layer,
100000,
account.Address,
account.Address,
nil,
0,
hostprogram.PROGRAM,
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, the host functionalities could be tested without running actual programs in the VM. The host has an API exposed to VM which abstracts things nicely and the host features could be tested by exercising this API.

In this test, it's not even possible to tell what the hostprogram.PROGRAM will do because there is no source code for it (same for the get balance test above).

req := smocks.NewMockValidationRequest(ctrl)
req.EXPECT().Parse().Times(1).Return(tx.TxHeader, nil)
req := smocks.NewMockValidationRequestNew(ctrl)
req.EXPECT().Cache().Times(1).Return(nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Times(1) and Return(nil) are defaults, so it's equivalent to:

Suggested change
req.EXPECT().Cache().Times(1).Return(nil)
req.EXPECT().Cache()

IMHO less is more and it's more readable (less cluttered) this way.

@@ -79,7 +82,7 @@ type VM struct {
}

// Validation initializes validation request.
func (v *VM) Validation(raw types.RawTx) system.ValidationRequest {
func (v *VM) Validation(raw types.RawTx) system.ValidationRequestNew {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it return an interface instead of the Request struct that implements it?

Suggested change
func (v *VM) Validation(raw types.RawTx) system.ValidationRequestNew {
func (v *VM) Validation(raw types.RawTx) *Request {

It's against the common rule "accept interfaces, return concrete implementations". Is there a need for this?

Comment on lines +403 to +407
tx2, err := wallet.Spend(signer.PrivateKey(), recipient, 1, nonce, sdk.WithGasPrice(0))
if err != nil {
panic(err)
}
tx.RawTx = types.NewRawTx(tx2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As NewTx is a test-only function, it could take tb testing.TB and then this could be simplified to:

Suggested change
tx2, err := wallet.Spend(signer.PrivateKey(), recipient, 1, nonce, sdk.WithGasPrice(0))
if err != nil {
panic(err)
}
tx.RawTx = types.NewRawTx(tx2)
tx2, err := wallet.Spend(signer.PrivateKey(), recipient, 1, nonce, sdk.WithGasPrice(0))
require.NoError(tb, err)
tx.RawTx = types.NewRawTx(tx2)

t.Method = uint32(tx.Method)
// t.Method = uint32(tx.Method)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the method?

Comment on lines +442 to +446
if spawnSelector, err := athcon.FromString("athexp_spawn"); err != nil {
return res, txType, fmt.Errorf("%w: failed to create spawn selector: %w", core.ErrInternal, err)
} else if spendSelector, err := athcon.FromString("athexp_spend"); err != nil {
return res, txType, fmt.Errorf("%w: failed to create spend selector: %w", core.ErrInternal, err)
} else if *method == spawnSelector {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need for else branches, as the previous branch returns on error. Consider:

Suggested change
if spawnSelector, err := athcon.FromString("athexp_spawn"); err != nil {
return res, txType, fmt.Errorf("%w: failed to create spawn selector: %w", core.ErrInternal, err)
} else if spendSelector, err := athcon.FromString("athexp_spend"); err != nil {
return res, txType, fmt.Errorf("%w: failed to create spend selector: %w", core.ErrInternal, err)
} else if *method == spawnSelector {
spawnSelector, err := athcon.FromString("athexp_spawn")
if err != nil {
return res, txType, fmt.Errorf("%w: failed to create spawn selector: %w", core.ErrInternal, err)
}
spendSelector, err := athcon.FromString("athexp_spend")
if err != nil {
return res, txType, fmt.Errorf("%w: failed to create spend selector: %w", core.ErrInternal, err)
}
switch *method {
case == spawnSelector:
// body
case == spendSelector:
// body

Comment on lines +15 to +16
// TODO(lane): Method is unused by the Athena VM, and should be removed.
Method uint8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why keep it?

type ValidationRequestNew interface {
Parse(core.AccountLoader) (*types.TxHeader, error)
Verify() bool
Cache() *core.StagedCache
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the need for Cache(). It's only used as a parameter for Parse() as far as I can tell. Why have it?

Comment on lines +101 to +102
// sig := ed25519.Sign(ed25519.PrivateKey(pk), core.SigningBody(options.GenesisID[:], tx))
sig := ed25519.Sign(ed25519.PrivateKey(pk), tx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this?

Comment on lines +65 to +66
// tx := encode(&sdk.TxVersion, &principal, &meta)
// tx = append(tx, payload...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// tx := encode(&sdk.TxVersion, &principal, &meta)
// tx = append(tx, payload...)

Comment on lines +68 to +69
// sig := ed25519.Sign(ed25519.PrivateKey(pk), core.SigningBody(options.GenesisID[:], tx))
sig := ed25519.Sign(ed25519.PrivateKey(pk), tx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

Comment on lines +98 to +99
// tx := encode(&sdk.TxVersion, &principal, &meta)
// tx = append(tx, payload...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// tx := encode(&sdk.TxVersion, &principal, &meta)
// tx = append(tx, payload...)

Comment on lines +571 to 581
} else {
// case 3: principal account exists but is not spawned
// go ahead and assume it's a self-spawn for a Wallet template
ctx.PrincipalHandler = reg.Get(wallet.TemplateAddress)
if ctx.PrincipalHandler == nil {
return nil, nil, fmt.Errorf("%w: wallet template missing", core.ErrInternal)
}
ctx.Header.TemplateAddress = wallet.TemplateAddress
ctx.SpawnTx = true
ctx.Header.MaxGas = core.ATHENA_GAS_SPAWN + core.ATHENA_GAS_VERIFY
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will it support spawning using other templates? Shouldn't the spawn TX also carry the template address?

func ComputePrincipalFromPubkey(template types.Address, pubkey signing.PublicKey) (Address, error) {
// construct and encode the blob, which is a SCALE-encoded Athena wallet template instance
blob, err := scale.Marshal(struct {
Nonce, Balance uint64
Copy link
Member

@brusherru brusherru Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need Balance here?
IMO it looks weird to add balance as a "hidden spawn argument" for the Wallet account type, it's not a vault or some token smart-contract (total issuance)...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, also I guess such Nonce field should be a part of payload of spawn tx, and it's the different nonce from the "tx counter"

@brusherru
Copy link
Member

Also I suggest to use another TemplateAddress for Athena's Wallet to avoid collisions with the "genvm" template addresses, just to keep things simpler.

Comment on lines +18 to +27
// ComputePrincipal address as the last 24 bytes of Hash(template || blob).
// See https://github.com/spacemeshos/go-spacemesh/issues/6420 for more details.
func ComputePrincipalFromBlob(template types.Address, blob []byte) Address {
hasher := hash.GetHasher()
defer hash.PutHasher(hasher)
encoder := scale.NewEncoder(hasher)
template.EncodeScale(encoder)
args.EncodeScale(encoder)
hasher.Write(template[:])
hasher.Write(blob)
sum := hasher.Sum(nil)
rst := types.GenerateAddress(sum[12:])
return rst
return types.GenerateAddress(sum)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure if using types.GenerateAddress is OK here, see its implementation:

func GenerateAddress(publicKey []byte) Address {
var addr Address
if len(publicKey) > len(addr)-AddressReservedSpace {
publicKey = publicKey[len(publicKey)-AddressLength+AddressReservedSpace:]
}
copy(addr[AddressReservedSpace:], publicKey[:])
return addr
}

It does more than taking the upper 24B - it also reserves the lower 4B of the returned address (they are zeros)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it built from the example wallet in the athena repo? Why not have the sources of the wallet in go-sm?

Comment on lines +85 to +88
vmhost, err := vmhost.NewHost(host)
if err != nil {
return 0, fmt.Errorf("loading Athena VM: %w", err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and in few other places the VM is not destroyed (missing call to Destroy)

Suggested change
vmhost, err := vmhost.NewHost(host)
if err != nil {
return 0, fmt.Errorf("loading Athena VM: %w", err)
}
vmhost, err := vmhost.NewHost(host)
if err != nil {
return 0, fmt.Errorf("loading Athena VM: %w", err)
}
defer vmhost.Destroy()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants