Cumulative PR for Wallet & Metrics Refactoring

[aspect]

Please note this PR breaks previously created wallet data files. The wallet data storage has been completely redesigned to provide easier management of data structure versioning. Following this PR data files will be always backward-compatible.

Consensus & RPC service layers:

• Debug formatting of various consensus data structures (SPK, Address etc.) have been changed from default implementations to render in hex/human-readable.
• Refactored signed_with_multiple_v2() to return enum Signed comprised of two states: Signed::Fully and Signed::Partially to be able to distinguish transaction signed states. Signed also implements fully_signed() -> Result<()> that will return a transaction or an error if the transaction is not fully signed.
• Updated GetMetrics RPC call to return various additional metrics including bandwidth monitoring.
• Add a few auxiliary functions to consensus_manager to return lengths instead of collection clones such as async_get_tips_len() and async_get_virtual_parents_len(), these are used by the GetMetrics RPC call when the metrics poll occurs.

/cli:

• AddPeer has been added to the RPC command handler

/kaspad:

• Modified kaspa_daemon::Args to accept an argument iterator, this allows Args to be used externally to perform argument validation as well as to supply kaspad externally generated arguments.

/metrics/kaspa-metrics-core

• Migrated metrics from the /kos crate into its own standalone /metrics/kaspa-metrics-core crate.
• Consolidated all metrics into a series of data structures designed for client-side use.
• Exposed additional metrics like mempool transaction count and gRPC/p2p bandwidth metering introduced in gRPC bandwidth tracking #346 .
• Exposed a helper task designed to bind to an existing RpcApi interface for polling metrics and delivering it to a supplied closure.

Wallet framework:

• Restructure modules to improve module layout
• Introduced WalletApi trait that encapsulates all wallet functions in a similar style to RpcApi, allowing the wallet to be "controlled by wire" (i.e. via serializable data structures over RPC). The WalletApi follows the same style and code notations as the RpcApi (i.e. _call fn suffixes and serializable *Request/*Response structs). The original development effort was aiming to avoid RPC-style communication with the wallet framework. However, while testing the framework in the browser extension environment we ran into problems and a requirement for the wallet framework to run "headless" in the background task. This refactoring enables such functionality, while also bringing all APIs into a single trait, making it easy to develop any kind of RPC bindings as well as WASM bindings (since all data structures are serializable).
• Completely restructure how data storage functions in the wallet framework. Previous implementation relied on Serde JSON which exposed a variety of problems related to long-term backward compatibility and ease of use. The new data storage subsystem allows each wallet account to manage its own data storage and offers easy-to-use data version control. The new storage uses manually constructed binary serialization functions (using Borsh serializer manually to store binary data) @biryukovmaxim .
• Accounts are now standalone data constructs managed by and created by account factories. This allows us to easily create different account types that have custom behavior or a 3rd-party developer to create a custom account externally without changes to the framework (this structure is provisional and may require further optimizations of the Account trait). In essence, Accounts are now similar to plugins allowing the creation of custom accounts that may opt to handle custom scripts (or other custom interactions such as hardware interfaces) @biryukovmaxim .
• AccessContextT trait has been removed from the storage subsystem. Storage subsystem now accepts secrets directly. This construct was created as a "layer responsibility isolation" effort but was making APIs and secret retention more complex than they should be.
• All transaction events have been refactored where the events are now strictly focused on transaction maturity status (and certain outlier conditions) and the TransactionRecord carries the transaction status.
• Implemented support for inter-account zero-conf transfers. The transfers function is based on the tx acceptance and can be executed via a new Account::transfer() method by supplying the target account id. Transfers require a destination account ID and both from and to accounts will receive TransferIncoming and TransferOutgoing notifications containing the corresponding TransactionRecord.
• UtxoContext and UtxoProcessor have been refactored to track a new OutgoingTransaction that wraps PendingTransaction (and contains additional context information such as submission time and acceptance time). The OutgoingTransaction is not tracked by UtxoProcessor and plays the main role in handling transfers as well as monitoring for Change outputs.
• TransactionRecord processing has been modified to resolve transaction Unixtime timestamp automatically using methods introduced in Implement DAA score timestamp estimation #268. Newly discovered transactions cascade to UtxoProcessor and are then handled internally or propagated to the wallet that checks transaction storage to see if transactions are unknown, resolves unix time from the DAA score, and cascades them to the wallet multiplexer channel clients via the new Discovery event @coderofstuff .
• Introduced Stasis transaction processing queue. The previous implementation places new coinbase transactions into the Pending state which later propagates into Reorg or Maturity states resulting in additional Reorg notifications. The new implementation places new coinbase transactions into a Stasis state for a period of 50 DAA. Upon reorg of a transaction that is currently in the Stasis state, no events are emitted. After 50 DAA expiration, the transaction is placed into a Pending state at which point the Pending event occurs, followed by the Maturity event once the transaction age reaches 100 DAA. This was implemented to simplify coinbase transaction processing for wallets as during the initial few DAA of the coinbase transaction lifetime it can frequently become subject to a reorg.
• Improved internal wallet data structures to support different types/algorithms of wallet data encryption (provisional).
• Implemented provisional support for TransactionRecord storage encryption (allows transaction data to be encrypted as well - for now provisional only, additional work is required to bring this feature online).
• Fix a deadlock (recursive mutex) while storing multiple accounts via the storage API.
• Fix a bug (trait mishandling) resulting in the dyn Account always returning account_index() as 0, preventing any additional BIP44 accounts from being signed correctly.
• Refactor wallet open() and reload()processes. Opening a wallet now loads the storage dataset and related cache, following which the developer is required to explicitly activate account processing (a.k.a. activate accounts). (Due to event-driven architecture, starting account processing during the open() call creates event sequence difficulties where after opening the client may need to load existing transactions before any other processing should occur).
• Add the ability to custom-configure wallet storage folders (via static mut, before using the framework).
• Fix a rare panic during the transaction submission process occurring in an async context when the node disconnects during a tx submission setting DAA to None while async functions that require DAA would continue their processing. The panic is now converted to a cascading error.
• Introduced transaction submission reversal process, where the failure to submit a transaction to the node results in proper "unwinding" of states (see next item).
• Introduced an async locking mechanism (using async_std::Mutex) that blocks UtxoChanged notification processing during the transaction submission process. Transaction submission is a multi-stage process where 1) OutgoingTransaction (PendingTransaction) is registered, 2) the transaction is submitted 3) submission can result in an error in which case 4) registration of the OutgoingTransaction has to be canceled/reversed. Since these processes are async, there is a danger that acceptance and the corresponding changes come in while this process is taking place.
• Various documentation updates.

WASM

• PR [WIP] Implementation of WASM Bindings for TypeScript #349 (GeneratorArguments) has been folded into this PR
• Fix a deadlock (recursive TryFrom) when calling UtxoContext.trackAddresses() from WASM
• Fix incorrect property assignment in the Transaction.gas setter.
• Rename (name more explicitly) some Address functions on the Rust side as they were making the Rust the API confusing (JS needs Strings while on Rust you expect to get native types).
• Most Wallet APIs have been disabled/removed (some left as comments for reference) in the preparation for exposing WalletApi trait to WASM.

workflow-rs

• Fix a rare wRPC server panic occurring when a client was connecting and immediately disconnecting from the WebSocket server. Due to async handling, the disconnection could occur during the WebSocket initialization phase. workflow-rs/workflow-rs@9d98829
• Add platform detection for Electron client/server environments @starkbamse . workflow-rs/workflow-rs@f565f46

[coderofstuff]

Some minor comments.

[coderofstuff]

Is this intentional that wasm is commented out?

[aspect]

Adding a //TODO note. This is a placeholder with some code copied over from RPC meant to automate bindings for WalletApi. I started working on bindings but quickly concluded that the WalletApi needed to be stabilized before the bindings were made.

Also, there have been efforts made by @starkbamse and other contributors with regard to improving types in RPC bindings (which also apply here) and I had to freeze the effort till January in order to get a concise view of all issues in front of us so that all of this can be addressed and implemented as a single dedicated effort/pass.

[coderofstuff]

Optional: This condition should never happen in a legitimate KAS value. So it might be worth changing this to an Err since this would mean you were passed a string that had 9+ decimals - which could mean something is off elsewhere in the code that made that happen.

[aspect]

Hm.. It's a valid point... The function is constructed and is used mainly for user input and I am not sure of all the pros or cons of both cases. Like if you copy/paste a value from somewhere, it can have a very long fractional value... and I am not sure whether such a value should be rejected or truncated. It kind of falls into "what is best/friendliest for the user" domain... technically I think we should perhaps do something like ceil(v*1e8)/1e8 and then truncate... this way the user can input anything but will get back a rounded value... but perhaps a separate function should be made for that.

Let me add a TODO to this so that we can take a look at this a bit later on.