How NOT To Fracture A Layer 1 Chain!

As of today, there are over 24 layer-1 blockchain projects just within the first 100 listings in coinmarketcap.com. Guess, what? Each of them claims to be the “next-big-thing”, still they are just a broken iteration of one other.

Apologies. I should rephrase “broken” with “somewhat broken”.

Why?

One of the biggest issues is not being able to scale without trading off either security or decentralization.

A ton of research has been harvested and processed into this field without much being accomplished.

I am sure you are wondering, but that “credits blockchain” have claimed that they process over 1 mil tps?. what about {insert_shitcoin} which achieves 70,000 tps via sharding?

Lets me tell you a way with which you can also come up with these flabbergasting numbers.

You can run the chain just on your private computer. Turn off any latency simulations, persistent storage, and just store your data in the fastest hashmap you can find. Even better, just run all your experiments in memory and toss LevelDB in the trash. BOOM 1MIL TPS!!
You can bundle the transactions together into big bunches and then make only one decision on the terabyte block and then you count the number of transactions in that block~well this will give you a big number..hahaha.
Making transactions as small as possible, down to a single byte. Then cramming mil of txs into a single block of a few megabytes. And pass around only a block hash after assuming all those transactions are valid.
Sharding your system into millions of partitions, and making sure no transactions cross any of these partitions. Assuming no invalid state transaction, no data availability issue… effectively, just assume your system is running multiple parallel networks, and then aggregate all the TPS numbers together. Easily millions.
No signature verification….etc.

Okay seriously, whats are the main hindrances?

Lets again talk about Ethereum for a second, now if you remove all of the consensus parts and then if you try to run sync ups, as a state and execute transactions in the EVM. The limiting factor ends up not being processing but I/O, just because of how we organize the state in the blockchain. The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk because there’s almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way. Not only is storing the data very sub-optimal, but due to the 200 modification / second and pruning of past data, we cannot even download it is a properly pre-processed way to make it import faster without the underlying database shuffling it around too much. The end result is that even fast sync nowadays incurs a huge disk IO cost, which is too much for a mechanical hard drive. Solana tries hardware scaling but that just not feasible at the end of the day. State sync is mostly limited by disk IO, not bandwidth.

In the current system, we are bound by how fast we can read data from disk, even if we can solve that problem, there is only that much computational power that we have on one machine.

Many developers even argue on the fact that the main culprit is how quickly storage requirements and sync times grow for a node on a network that’s doing 7000 TPS. And that is the ultimate constraint on scalability.

Also, you can claim that one of the main reasons for BTC to be slow is because of the Nakamoto consensus. Synchronous protocols have a hard limitation on the data processed per unit time. In a synchronous network, the minimal viable quorum must necessarily be as fast the lowest/delta bound to spread information in the network. One of the main bottlenecks currently is broadcasting data. A synchronous network puts an artificial throttle on the speed at which the data can be processed.

Whereas, in an asynchronous network the data can be spread as quickly as the fastest minimally viable quorum subset that can package data and distribute it.

Also, let’s see how to not design a sharding system which has ~

“The Fisherman” to solve its data availability issue as Polkadot does, which in return can slow the chain.
Something that does not rotate its validators and has to TRUST its validators, like Cosmos.
Zilliqa does sharded validation but not data which does not even shards state they shard processing meaning that every node will still have to store the entire state which is a huge limitation.
Faulty load balancing design between shards like ETH 2.0.
The way inter-shard consensus sometime slow down confirmation times for transactions that took place between two shards in Near protocol.

As a matter of fact, let me give credit to Near Protocol, ETH 2.0 and Polkadot as they are the only ones who publicly even recognize the data availability issue, others don’t even talk about it. So,

How do we take care of the data storage problem? (sync)
How do we do proper load balancing?
How to solve data availability issues?

Let's try to solve all these fundamental problems but in general, we can come to a quick conclusion that whatever the case is, an asynchronous network with sharding || parallel processing is the only way towards scalability.

(A.) WHAT If WE HAVE TWO LEDGERS, INSTEAD OF ONE?

1. Supposedly, only one ledger, just stores the transactions? Let us make that ledger a block-lattice.

One of the main advantages of using a DAG chain is this won’t be vulnerable to quantum attacks. This architecture also makes it possible to have a scalable consensus.

This is the actual ledger consisting of settled transactions is structured as a block-lattice where there are stored independent individual user account chains. The blocks from different account-chains acknowledge each other and collectively form a mesh-like structure.

Blockchains are usually made to go on for many years, over the years the size of the chain grows, so what usually happens is not many computers are able to run the full node and storage isn’t cheap. Sure, you technically can buy those big external drives on Amazon, but if you want to run a node on an existing computer or VPS you’re stuck with a couple hundred GB or less. For example, the Ethereum state is already ~60 GB. Who will run a full node with 60gb of disk space only to find out that it is unusable in less than a year?

(Do an experiment and try running eos full node! Haha..)

Therefore, this is a major hindrance to decentralization. Increased ledger size is a significant problem for all DLTs and this issue is tackled by ledger pruning. The entire history is federated to historical nodes while other nodes only maintain a subset of historical data. Generally, a tradeoff between disk space usage and historical data accessibility is being made. Bitcoin does this via UTXO’s. You only need to focus on the current UTXO set..but even then sometimes you need to download the full state. And what Ethereum does is “account-based”, so what happens is whenever you remove an account you get a replay attack, so this is never possible.

That’s the story of the transactional ledger, well what about the consensus ledger that the full nodes will store?

2. Okay so, what if we have a separate ledger for the consensus? And let this ledger be a “meta-dag”.

Although one major drawback to DAGs is the difficulty in calculating the probability of transaction reversal due to non-linearity. But, we can design a novel variation of utilizing DAG in the consensus protocol which mitigates these.

Normally in systems, there’s a marked distinction between the consensus and the underlying machine state. What happens is in most of the system (For example, Hotstuff consensus, on which Mark Zuckerberg and his 28 friends are building Libra or Tendermint) is that each of the nodes has to wait until all of the nodes have updated their state and then only can it come to form a consensus.

But what if we have a BFT consensus algorithm that’s leaderless?

Okay, it feels like we are sliding into the part where we ask ourselves what role does the underlying consensus algorithm play in making the system completely secure, scalable, and decentralized!

So in a nutshell, the full node will just have to keep both the transactional ledger, i.e. the block lattice and the transactions in the epoch to come into consensus in the form of a dag.

We can design a consensus so that once a network transaction occurs, within seconds all nodes in the network will know where that transaction should be placed in a history of transactions with 100% certainty. At that point, the network can just incorporate the effects of the transaction and. So, in a minimal cryptocurrency system, each node would only need to store the current balance of each network account that is not empty. The nodes would not need to remember the full history of the transactions that resulted in those balances all the way back to “genesis”.

One of the major advantages of having a separated ledger architecture is that we achieve a better complexity and faster processing times when a user wants to query nodes for transactional data. Also, we have solved the space constraint issue that every other architecture faces. Still, the consensus algorithm is an important aspect for solving this so let's dive into that.

(B.) What BFT consensus would we use, and how, so it would make sense of the ledger architecture?

As I interjected you earlier, what if we make can reach a consensus without forming a committee!!! Awesome, right? Leaderless passive consensus negates the need for complex vote based rounds and message passing which can be risky liveness-wise for many automotive-based blockchain use cases due to the reliance on a round leader. We are building a novel consensus algorithm that's also deterministic in nature, which doesn't haven't fatal flaws like Avalanche consensus, which is another leaderless consensus that's probabilistic in nature, as the avalanche consensus is only able to guarantee a partial ordering of transactions. The avalanche paper doesn't describe how the parent transactions should be chosen, resulting in potentially controversial transactions that have never been voted on or approved.

So how would such consensus be designed for our ledger?

Imagine there to be 2 main nodes, and lets call them ‘sentinels’ and ‘pillars’-they keep the transactional and consensus ledger at all times.

Let’s build the first stage-

When a client seeks to bring a change in state, the message is dissected to a number of the sentinel node in the network and in return, the sentinel provides a receipt of the transaction by issuing a small Proof-Of-Work computation & some additional data; then it relays the message to randomly chosen sentinel nodes…and then that sentinel again computes a POW of its own, this process is repeated..until a “POW-LINK” is formed. Until a minimum number of three hops is taken and the network decided weight is met of the “POW-LINK”.

This is just one instance of the follow-through in the network.

Few advantages of this novel concept of POW-LINK are as follows~

It acts as a Sybil resistant layer in the network.
It helps us come to a solution in case of a double-spending attack.
Most importantly it helps collective participation in the network.
It acts as a parameter for virtual voting function in the consensus argument.

Moving on to the second stage-

After that, the message is sent over to a random pillar node, on reaching the POWLINK weight. Now the pillar node starts solving a POW of its own to keep the ‘epoch’ time-bound. During this epoch, the pillar node receives messages from other sentinel nodes. Every pillar node in the network starts solving a POW, and by doing so it gets rewarded for solving the puzzle in that epoch. The proof-of-work round is complete for a particular pillar when it has sent over the broadcast to the network. When supermajority of pillar nodes in the network has solved the POW and broadcasted the transactions that in received in the meanwhile, to all the pillar nodes, an epoch is complete.

Figure - A glimpse of what happens after each broadcast.

After the epoch when the pillars get the transactions it applies virtual voting for consensus. The voting can take place without any extra message propagation among the peers, the transactions can be ordered based on predefined guidelines.

click here to view our consensus infograph.

What happens usually in a normal Byzantine Fault Tolerant Consensus algorithm is that it scales with complexity, cause you need to have an answer from everyone. As the number of participants grows, the number of messages in the network grows exponentially, but that’s not the case for us.

Well now we have a consensus algorithm to follow which is

Efficient.
It has low latency and scales well.
It has maximum concurrency, i.e. the entire system creates a dag underneath at the very lowest layer, which means that it can concurrently work on different transactions at the same time achieving very high throughput.
Incorporates mechanism which is robust and can handle imprecision in memberships.
Unlike consensus like ‘hotstuff’ who has linear complexity in deciding what number of messages you want to have to asymptotically send to your peers; this consensus design helps us pipeline as many decisions as we want to particular predefined measurements; which indeed makes the bandwidth evenly distributed and very low.

Having a dual ledger and such a consensus algorithm also makes us ACID compliant. ACID (Atomicity, Consistency, Isolation, Durability) is a database term and applies to our network when it is used as a distributed database. A network of nodes uses it to reach a consensus on the order in which transactions occurred. If the local database of the node has all the standard properties of a database (ACID), then the network as a whole can be said to have a single, distributed database with those same properties. In the blockchain, there is never a moment when you know that consensus has been reached, so it would not be ACID compliant.

(C.) What's the implication of incorporating a POW+POS in the network?

Whats does the symbol on the top of the head say? Eh, more on that later.

Classical BFT protocols assume two kinds of players: cooperative and byzantine. This assumption works well in centralized settings where nodes are controlled by the same entity or federation. However, decentralized networks that rely on volunteer nodes need to provide incentives for participation.

Our game plan introduces a new kind of player~a rational player that assesses its expected utility in terms of its native coins. It argues that whoever solves the puzzle before the supermajority of nodes complete theirs, gets the reward and by doing so it creates competition. This ensures no one has an incentive to delay messages. It includes incentives for information propagation and presents a game-theoretic analysis that a miner’s best strategy is to propagate the PoW puzzle and charge a small fee.

And the way we use POS in our architecture acts as a cherry on top of the cake as we replace regular quorums with POS. There’s one big fundamental safety concern in existing big POS blockchains like algorand, dfinity, ouroboros cause they have some sort of notion of the committee in their consensus protocol.

A more equivocal and political way to say that security flaw would be, if you have had a country of a million people would you trust a government of just 1000 people to monitor and administer all the actions of those million people? Would you consider that safe? And surprisingly all the modern POS chains are just doing that.

But we can make that POS be leaderless where everyone has a say in it.

Let’s say. At the start of each epoch, all nodes determine the stake weight of all the nodes in the network, for building a vote weighing function. The pillar nodes can randomly vote on transactions; only if a few transactions have been kept to a simple majority and not all pillar nodes have them, even after few epochs.

As you know, a design that features a hybrid pos+pow system in the architecture has some screaming benefits:-

Greater protection against majority attacks by requiring PoW miners and PoS validators to depend on each other.
The hybrid system’s PoS mechanism requires wallet software to run constantly so that the stakeholder’s chance of voting isn’t missed, potentially greater network stability. Further, since rewards are distributed across PoW miners and PoS stakeholders, hybrid PoW mining is typically less profitable than pure PoW mining. Therefore, participating nodes tend to invest less in gaining hash power, thereby lowering the barrier to entry for new PoW miners. Both of these factors help to encourage greater network participation.
Also due to the potentially reduced total investment in hash power, the network’s energy consumption may be relatively lower than pure PoW.
The adoption of Hybrid PoW+PoS guarantees the price of “the network asset” from the PoW side, due to the relatively stable cost of mining. This is because miners will not sell their assets earned at a lower price than what it cost them to mine. Additionally, the increasing cost of computing power will support the value of the network, also alleviating the centralization problem of a purely PoS mechanism.

The way we are integrate our tweaked implementation of POS and POW, not only it helps us to get the best features of these consensus algorithms but also deport the bad ones. As a result it boosts enhancement of overall network throughput and provides faster finality time.

(D.) How are smart contracts integrated into this architecture?

It can be tough to achieve support for smart contracts in DAG but when we combine smart contracts with DAGs, we can circumvent the scalability problems of existing smart contract platforms, albeit on the cost of reduced interoperability. Cross-contract calls in Ethereum are represented as function calls or internal transactions. This set of calls are either completed at the same time or all fail. Obviously, this kind of atomic ACID semantic could become a performance bottleneck in the system. To tackle the issue, we can adopt an asynchronous, message-driven architecture. Our smart contracts can communicate with each other via sending messages instead of sharing states.

Also, we can design a system in such a manner so that we can use the dag as storage and also be able to perform and handle complex transactions in the network without incurring the high cost/storage/sync problems.

How would we be doing that?

What if separate the smart contract layer from the computation layer and design it in this fashion…

This is the design, we will built our smart contract and computationl upon.

okay let’s start with the most obvious question.

What are unikernels and what advantages does it bring to the table?

Unikernels are specialized, single-address-space machine images constructed by using library operating systems. By using a unikernel, the memory footprint of Virtual Machines (VMs) is decreased, which reduces the pressure on the cache system and improves the overall performance. They are single-purpose systems. Typically a Linux system will have a hundred or so processes running when you boot it up without you installing a single thing. Unikernels only have one process running: the application designed to run it. Unikernels embrace what’s called a single address space model. This allows them to run at screaming speeds. Unikernels appear to have a niche in creating specialized software appliances. Unikernels fits into microservice architectures, and capable of lowering computational costs in terms of time and space per service.

Okay, so what are zApps?

Apps deployed on this architecture can be known as zApps. We can host our app in unikernels, which is proved to be a better contestant then containers, and is especially useful for building IoT services. Now we have a different smart contract and the application stack but they are connected we can perform network-intensive & complex tasks that other platforms cant run nor support. Usually, it is not quite feasible to do complex tasks on the dag chain directly as the nodes won’t be incentivized to perform, but this was we solve that without harming the network throughput.

zApps that are hosted by unikernels which are indeed operated by special sentinel nodes that will have specific requirements (e.g. minimum resources in terms of connectivity, hardware specifications, collateral, etc.). This layer is like a decentralized infrastructure-as-a-service where users can have access to an instant computing infrastructure, managed and provisioned within the whole network.

The main purpose of zApps is to create an open ecosystem that will enable every developer to build from scratch or convert an existing application, whether it’s the web, mobile, or desktop to our Network.

What do these Smart Contracts do?

Users can use smart contracts to send messages for managing and configuring applications lifecycle and deployment. It also acts as a 2-way channel for the sentinel nodes running the unikernel to send the transactions for consensus and embed it into the ledger and also to check if the node has updated. We incorporate an asynchronous mode of communication between nodes that incorporate a message-driven architecture, and the contracts do not share states but only communicate by sending messages to each other.

What impact does all of these have?

1. Black box data sharing and processing

Black box data sharing and processing refers to a general class of applications that allow different entities to share data or process data from another organization. Our network incorporates this by having the zApps choose their special sentinel nodes that drive messages from and to the unikernels. All the transactions and load of the application is passed through these nodes onto the smart contract and to the ledger. One of the biggest benefits of this model is efficiency, allowing complex algorithms to be run on local data without requiring every compute node to run the same instructions. This allows for more private systems and potentially making regulatory compliance more simple. And by deploying this feature we have a proper load balancing architecture.

2. Ownership of account chain

State consistency is important, but slow propagated state updates result in incorrect permissions. But our architecture enables light clients to quickly gain access to updated information and therefore maintain state. Each account chain is updated and now interact with ecosystem players near real-time.

3. Functional Sharding

One thing our proposed architecture does is “functional sharding”, in contrast with the standard “random sharding” idea that you see people try to carry over from the database world. Random sharding splits the chain by something like the contract address, which sounds ok at first. But sharding like that creates an incentive to grind through hashes until your contact ends up on the same shard as the DAI contract. In the limit, it’d accomplish nothing other than complicating and slowing down the protocol as everyone ends up on the same shard. But our network supports a form of functional sharding. Gaming contracts have no reason to be on the same part of the ledger as, say, financial contracts. A node in one of those two sub-networks that hosts the unikernel transactions has no need to process any of the other sub-networks transactions. This is the reason our network will have unimaginable fast sync.

Conclusion

Let's take a step back and talk about the problems that existing solutions face

Data storage.
Disk I/O.
Consensus bottleneck.

But we conclusively proved through steps A ->B->C ->D that we can design a system that is decentralized, scalable and secure. The network does successfully mitigate all the issues and successfully builds a network for the future.

The network created as a result has the following properties~

High Throughput
Low Latency
High Scalability
Low hardware resource consumption
Low transactional data size
Low transactional fee

Putting all these together in the network will be quite the engineering feat.

But the good news is that there are already people working on it. They are Zenon Network. All these ideas and novel techniques stated in this article are their proposals for building a new decentralized world.

Its zenon.network

The name of the network that they are creating is Network Of Momentum. They are 70 percent on the way to launch their testnet. Exciting time ahead.