Hedera transaction count might include consensus messages (HCS)

[tomerweller]

Based on comparison with this: https://app.metrika.co/hedera/dashboard/network-overview?tr=1d

(Bias alert: I'm with the Stellar development foundation)

[brson]

I'll give it a look this weekend.

[brson]

I see that the transactions are dominated by HCS, the Hedera Consensus Service, which I understand went live in the recent past.

It's not clear to me what the nature of these transactions are. I believe the HCS is a general purpose consensus service that any party can use. So if somebody is throwing a lot of external transactions at the HCS, possibly just for testing or appearances, that seems fine.

What I would be willing to filter out, based on the precedent with Solana, is internal consensus transactions without which Hedera would not work. In other words if the Hedera network relies on HCS for its own consensus, then those HCS transactions I would consider to be necessary overhead, the transaction space unavailable to users, and filter it out like Solana's consensus transactions.

I see that Hedera has one HCS "topic" that is much larger than the others, 1693742, but yet I don't understand what this topic is.

I'm happy to have any pointers to understand what this topic is being used for.

[brson]

cc @a-ridley perhaps you know how the HCS is currently being used. I am trying to understand if some subset of the HCS transactions are intrinsic to the operation of hedera itself the way solana consensus transactions are.

[brson]

Also, fwiw I don't technically know how obtain Hedera transactions from blocks yet. The hedera block RPC API doesn't seem to provide that information.

[nubeasado]

@brson @tomerweller Hi!
HCS transactions have absolutely no involvement in the Hedera network reaching consensus. The Hedera Consensus Service is a network service offered on Hedera, similarly to smart contracts, and the Hedera Token Service.

From Hedera's website:

Hedera Consensus Service (HCS) is a purpose-built tool for creating decentralized, auditable logs of immutable and timestamped events for web2 and web3 applications. Messages are submitted to the Hedera network for consensus, given a trusted timestamp, and fairly ordered. HCS is used by applications in production to track provenance across supply chains, log asset transfers between blockchain networks, count votes in a DAO, monitor IoT devices, and more.

https://hedera.com/consensus-service

perhaps you know how the HCS is currently being used

Atma.io (Avery Dennison) is using HCS to 'provide support for sending product lifecycle events such as creation, shipping, and sale to Hedera. Recording these events with Hedera Consensus Service establishes an immutable and verifiable history for every product.' source

I see that Hedera has one HCS "topic" that is much larger than the others, 1693742, but yet I don't understand what this topic is.

That is Atma.io iirc

ALT/AVE uses HCS for document sharing. 'docStribute uses Hedera Consensus Service to record and track immutable document sharing. By sending a hash of the document to the ledger, both ends of the communication know the information being shared has not been tampered with. Companies are thus able to do more digital exchanges in an effort to increase efficiency, reduce paper waste, and lower their carbon footprint.'

Hedera Consensus Service is also utilized in the new AI8112 Digital Coupon Standard by The Coupon Bureau.

You or I can also create Consensus Messages, it's actually fairly easy.
The code would look something like:

//Create the transaction
TopicMessageSubmitTransaction transaction = new TopicMessageSubmitTransaction()
    .setTopicId(newTopicId)
    .setMessage("hello, HCS! ");

//Sign with the client operator key and submit transaction to a Hedera network, get transaction ID
TransactionResponse txResponse = transaction.execute(client);

//Request the receipt of the transaction
TransactionReceipt receipt = txResponse.getReceipt(client);

//Get the transaction consensus status
Status transactionStatus = receipt.status;

System.out.println("The transaction consensus status is " +transactionStatus);

https://docs.hedera.com/hedera/docs/sdks/consensus/submit-a-message

[walter-hernandez]

What helps to explain the Hedera Consensus Service is to relate it to a cloud service provider like AWS, Azure, GCP etc. providing a NoSQL database to which you pay for each writing that you do to it. Coming back to Hedera and taking the previous explanation, a user or institution like Atma.io, pays Hedera for the orderly recording of their logs of their supply chain of different products. That recording happens in the following flow:

1. The Hedera network receives the logs of information from outside the network by Atma.io posting to Hedera by using a HCS "topic", which you can compare to a Queueing system like Apache Kafka or something like AWS SQS.
2. The network uses its Gossip about Gossip algorithm to order the transactions received and timestamp them before recording them in its ledger.

So, you can say that Hedera Consensus Service is pretty much a decentralized notary service recording everything on a decentralized ledger in an orderly fashion by using Gossip about Gossip, its consensus algorithm.

@brson @tomerweller if you are curious about how the Gossip about Gossip algorithm works to decide the order and timestamp of transactions, check this https://help.hedera.com/hc/en-us/articles/360000826697-What-is-gossip-about-gossip-and-virtual-voting- and I recommend you this youtube video of Leemon Baird explaining it in more detail https://www.youtube.com/watch?v=wgwYU1Zr9Tg . Now, I can try to briefly explain the Gossip about Gossip algorithm and virtual voting here, but I do recommend you to check the links shared for you to get a more in-depth explanation:

1. Each node keeps a hashgraph of all the transactions. They gossip to other nodes in the network when they receive a transaction while keeping track of who they hear a transaction from (they do this to around two levels deep)
2. Since each nodes know what the other nodes in the network know and who they hear/receive their transactions from, therefore, each node also knows how other nodes would vote on consensus, hence virtual voting, which requires no "broadcast of virtual votes". So, based on the virtual votes, the algorithm decides at each round what transaction should be added next to the hashgraph and timestamps the transaction.
3. Bockchain based DLT's may require their validator/miner nodes to timestamp a block added to the Blockchain. By contrast, Because Hedera's algorithm timestamps the transactions in itself based on the virtual votes of all the nodes without requiring validator/miner nodes to do the timestamping. Hedera's virtual votes would be akin to Solana's recording of the consensus transactions between their nodes. Hedera does not record the virtual votes between its nodes and if it did so, there would be an astonishing amount of transactions per second in the network.

The previous explanations are oversimplifying Hedera's Consensus Service, Gossip about Gossip algorithm, and virtual votes. I do recommend you to check the links that I shared you to get more details. However, I hope my explanations helps to define how Hedera's Consensus Service differs from, for example, Solana's recording of consensus between its nodes.

Finally, I have to clarify that in other chains to implement a use case to track supply chain, an institution may have to use an smart contract to keep track of events by writing to their state while Hedera implemented the recording of information natively to the network. The use of the terms "Consensus Service" is what is generating confusion for someone coming from a more Blockchain type of network

[a-ridley]

@brson
To answer your question directly, HCS transactions are valid user-submitted transactions to publish a message to a given topic. You can begin to understand HCS by relating it to being like an on-chain message bus. Users can store the data on chain. Although the word Consensus is in the name, it does not contribute to Hedera's consensus algorithm.
Hedera would function just fine if HCS transactions were at 0. They are not required to operate the network.

If you want to learn how to work with HCS check out this learning guide.

The Hedera consensus algorithm uses gossip about gossip and virtual voting to reach a consensus. You can learn more here: https://docs.hedera.com/hedera/core-concepts/hashgraph-consensus-algorithms

As for the large topic you pointed out, check out these use cases that are contributing to it:
https://www.youtube.com/watch?v=kGjV6Z1DEIQ
https://hedera.com/users/atma-io
https://rfid.averydennison.com/en/home/products-solutions/iot/faq-atma-io-hedera.html

[bgc100]

2. Then, since each nodes know what the other nodes in the network know and who they hear/receive their transactions from, the nodes can broadcast virtual votes that the algorithm takes to decide at each round what transaction should be added next to the hashgraph and timestamps the transaction.

To clarify, because each node knows what all other nodes know, each node also knows how other nodes would vote on consensus, hence virtual voting, which requires no "broadcast of virtual votes." There is some confusion as to whether Hedera's recent 50 million daily transactions include internal node communications. It does not, as there is no communication among nodes to reach a consensus. This makes Hedera aBFT, extremely fast finality, and gives it incredible throughput.

[walter-hernandez]

2. Then, since each nodes know what the other nodes in the network know and who they hear/receive their transactions from, the nodes can broadcast virtual votes that the algorithm takes to decide at each round what transaction should be added next to the hashgraph and timestamps the transaction.

To clarify, because each node knows what all other nodes know, each node also knows how other nodes would vote on consensus, hence virtual voting, which requires no "broadcast of virtual votes." There is some confusion as to whether Hedera's recent 50 million daily transactions include internal node communications. It does not, as there is no communication among nodes to reach a consensus. This makes Hedera aBFT, extremely fast finality, and gives it incredible throughput.

My bad. Thank you for making the correction! I updated my comment to reflect this

[a-ridley]

Also, fwiw I don't technically know how obtain Hedera transactions from blocks yet. The hedera block RPC API doesn't seem to provide that information.

I'll look into using the JSON RPC Relay api to get hedera blocks and get back to you. 👍
In the meantime, you can use the mirror nodes to retrieve blocks. The block interval is every 2 seconds.
https://mainnet-public.mirrornode.hedera.com/api/v1/docs/#/

[walter-hernandez]

Also, fwiw I don't technically know how obtain Hedera transactions from blocks yet. The hedera block RPC API doesn't seem to provide that information.

I'll look into using the JSON RPC Relay api to get hedera blocks and get back to you. +1 In the meantime, you can use the mirror nodes to retrieve blocks. The block interval is every 2 seconds. https://mainnet-public.mirrornode.hedera.com/api/v1/docs/#/

Check the implementation of Hedera's JSON RPC Relay API that Swirld Labs did: https://swirldslabs.com/hashio/ using https://github.com/hashgraph/hedera-json-rpc-relay

[a-ridley]

Also, fwiw I don't technically know how obtain Hedera transactions from blocks yet. The hedera block RPC API doesn't seem to provide that information.

I'll look into using the JSON RPC Relay api to get hedera blocks and get back to you. +1 In the meantime, you can use the mirror nodes to retrieve blocks. The block interval is every 2 seconds. https://mainnet-public.mirrornode.hedera.com/api/v1/docs/#/

Check the implementation of Hedera's JSON RPC Relay API that Swirld Labs did: https://swirldslabs.com/hashio/ using https://github.com/hashgraph/hedera-json-rpc-relay

@walter-hernandez Thanks for linking Hashio and the JSON RPC Relay repo. :) In addition, there is a JSON RPC Specification where you can see all the APIs. You can call the RPC eth_blockNumber which will return the number of the most recent block. Theres also eth_getBlockTransactionCountByHash to pull block info.