Original Author: Colby Serpa
Original Compilation: DAOrayaki
Nostr 2.0 may be able to build on top of Bitcoin as a Layer 2, providing secure off-chain data storage, just as the Lightning Network provides instant off-chain payments as a Layer 2.
This article will explain how the Nostr relay synchronizes its data while maintaining its lightweight nature, allowing users to selectively delete data, which is not available in Layer 1 blockchains. At the same time, compared to storing large amounts of data in the Bitcoin blockchain, because of the limited capacity and speed of Bitcoin blocks, it may be cheaper to store data using Nostr relays.
The following simple computer science design improves the distribution properties of Nostr networks under the standardized CAP theorem criterion. CAP stands for Consistency, Availability and Partition tolerance
image description
Relay lacks consistency (C in CAP theorem)
Consistency means that the databases synchronized on each computer are the same. Nostr relays cannot do trust-minimized synchronization in a way similar to how blockchains synchronize their data block by block. Unlike Bitcoin full nodes, the database stored by the Nostr relay is usually incomplete. Aside from blindly requesting all posts signed by a specific user, Nostr relay cannot discover missing data.
Consistency/sync issues with Nostr:
If two users upload their respective posts to different Nostr relays, those two users may not be able to see each others posts because Nostr is not like a blockchain. In the blockchain, every time there is a new record, all full nodes will synchronize the blockchain. All full nodes will add this data as a block to their blockchain at the same time. Every full node on the Bitcoin blockchain owns the exact same blockchain.
If we want Nostr users to always be able to see each others posts, then all Nostr relays need a way to identify missing data in user profiles so they can request the missing pieces from other Nostr relays or users.
Sync Nostr relays with weekly on-chain Merkle root and full tree hashes
About once a week, users can organize all their posts into a Merkle tree.
Each leaf in the Merkle tree contains a hash of a post, just like each leaf in Bitcoin contains a hash of a transaction.
Once the user has organized the entire configuration file into a Merkle tree, they will publish the Merkle root in OP_RETURN on-chain, below a normal Bitcoin transaction. This is why Nostr 2.0 does not require a hard fork of the blockchain to work. OP_RETURN is a block below a Bitcoin transaction that allows a small note to be appended before the sender signs it.
Additionally, the user will hash the entire tree and upload it to the chain (in OP_RETURN ) along with the Merkle root. The Merkle root is just the hash of the top branch, not the entire tree. The hash of the entire tree is essential for users and relayers to be able to detect if profile data is missing.
To get a hash of the entire tree, place the Merkle root at the root of the text file. Then, put the Merkle branch on the line below the root. Then, place Merkle leaves on the row below the branch. Once the tree is arranged as described, hash it all at once. Below is an example of a whole tree hash of what a whole tree hash would look like for the Merkle tree shown above. Whole tree hashing (hashing all Merkle tree data at once)
Merkle root and whole tree hashes provide two key functions:
The Merkle root allows users and relayers to download a portion of a configuration file at a time, like being able to download a transaction without having to download the entire block.
Whole tree hashing lets users and relayers know if their stored configuration files are incomplete. Unlike a Merkle root, the entire tree hash will only match if you have all the bits in the Merkle tree.
This cheap method can be used to update the entire configuration file on a weekly or user-defined frequency. Nostr will still work as it does now, but users can occasionally pay some sats to sync their data between Nostr relays if they want their posts to be seen by all users.
Users and relayers can download posts for one branch at a time. After each branch, they hash that branch with another branch closest to the Merkle root to check if it matches the Merkle root on the chain (similar to SPV). If those branches hash together to match the Merkle root, then theyll know that branch is part of the user profile, even if they dont have the full user profile yet. Users can download different branches of the same configuration file from many different Nostr trunks, while verifying the validity of each branch and ensuring that the downloaded configuration file is complete.
first level title
Why cant a Merkle root function like a whole tree hash?
If Nostr relays only depended on the Merkle root, they would have no way of knowing when the Merkle tree is complete, since every pair of branches closest to the Merkle root would hash into the same Merkle root.
To ensure a users profile is complete, a relay or user will hash their updated entire Merkle tree and verify that it matches the entire tree hash on-chain. If the entire tree hashes match, then the user data is complete. If the whole tree hash does not match, then a relay or user can tell other relays their latest leaf number and request the missing branch until the whole tree hash matches. In order to keep track of all the new Merkle roots being added every week or so, Nostr relays must become Bitcoin full nodes. Nostr 2.0 relays are indirectly paid to store the Bitcoin blockchain while enhancing the security of Bitcoin and Nostr.
Nostr Storage Limits: A Users Rule of Thumb
Since relayers have the right to choose what to store, unlike Bitcoin full nodes, Nostr relays may lose some user data. Therefore, users should only store data on the Nostr relay, if users can make backups locally. Web5s self-hosted service allows users to sync backups to all their local devices, which will reduce the risk for users who are concerned about using Nostr. At the end of the day, only the blockchain is where data is truly immutable. That said, Nostr is a fairly safe hybrid solution that will still work well for many applications. The trade-offs are listed below:
Three layers of trust minimization
Layer 1: Immutable and expensive data storage that is extremely difficult to censor. (The blocks on the chain synchronize all Bitcoin full nodes)
Layer 2: Mutable and cheap data storage, moderately difficult censorship. (off-chain Merkle tree and on-chain hash, synchronous Nostr relay on demand)
Local data storage is synced to all local devices and is easily auditable. (local centralization)
Fundamental trade-offs between blockchains based on Nakamoto consensus and Nostr
The more Nostr relays storing data for a particular address, the harder it is to censor that data. This means that popular data hosted by many Nostr relays may be harder to censor than unpopular data that is rarely downloaded.
On the other hand, the Nakamoto consensus blockchain prevents censorship based on the age of the data. The longer data exists in the blockchain, the more difficult it is to delete it using a 51% attack.
ZKCSP and Lightning Network Payment Nostr Relay Using Proof-of-Retrievability
Since we can verify that certain forks belong to specific users, Nostr relays can be paid every time they pass a small piece of data to a user. In order to achieve this, the user needs to download the head of the blockchain (as in SPV) to be able to perform the typical functions of a light wallet. The user will leverage the light wallets SPV functionality to fetch a specific transaction from the chain, which will include the user profiles Merkle root and the entire tree hash in its OP_RETURN. Users can now pay the relay to download that users profile branch by branch, and verify each branch by hashing them to match the Merkle root on-chain.
In order to send sats (the smallest unit of Bitcoin) to the Nostr relay, in exchange for providing data, we use an evolution of Gregory Maxwells (famous Bitcoin Core developer) ZKCP design (Zero-Knowledge Conditional Payment) [1], namely ZKCSP : Proof of Retrievability [2] combined with the Lightning Network.
According to the description of the ZKCSP white paper:
“…no need for a trusted third party…We also implemented the ZKCSP protocol for the proof-of-retrievability case, where the client pays the server for proof that the client’s data is stored correctly on the server.” [2]
A user initiates a Lightning smart contract with several financiers.
Users send requests to surrounding financiers. The financier signs the request.
Users send requests signed by financiers directly to the Nostr relays connected to those financiers.
Users now initiate ZKCSP constructs to ensure that the Nostr relay will only get paid from financiers after the correctly requested data has been delivered.
Once step 3 occurs, the user will make amendments on top of the original request signed by the financier before initiating ZKCSP construction in step 4. The user will add an extra on top of the original request, specifying the amount to deduct from both the users and the financiers balances (they must be the same amount, plus the financiers fee), which the user will then append to the original message content to sign.
If the user specifies to send more sats than they own, or more than the financier has frozen on that Nostr relay, then the Nostr relay will reject the request as the relay will not be able to get paid.
In this way, users can connect with many Nostr relays while freezing their funds with only a few funders. A similar approach could be taken with the Lightning Network, where trust-minimized financiers are permissionless intermediaries between users and merchants. Normal P2P lightning jumps are also available in Nostr 2.0, but this might be useful if routing and channel balancing fail frequently.
Whitelist Unlock Paid Nostr Relay
Nostr relays can whitelist certain keys if they wish to store data viewed by all of these users. If Nostr relays are unable to whitelist users wishing to store data, they will store any data sent to them. If users can always send data to relays for free, then users will never need to pay for Nostr relays. Nostr can offer paid options only if the relayer has the option to refuse to store data that cannot be paid for. Free relays still exist, but the option for paid relays does not currently exist.
Instead of trying to store all Nostr data for free, a paid Nostr relay can use a whitelist to selectively store all the data its paying users view on a daily basis. Some Nostr relays will continue to operate on a free model, but at the largest scale this is not sustainable as most free relays are just enthusiastic enthusiasts. Whitelisting (even if we were able to securely assign a public key to each Nostr profile) giving the Nostr relay the ability to decide which data to store would not be possible.
One public key per profile to unlock the whitelist feature: Bitcoin address becomes your Nostr public key
This system finally allows us to assign a public key to each configuration file.
There is no benefit to users creating new public keys for each post... as they are all associated with their profiles! This is not the same as a Bitcoin address. Unlike Bitcoin, having users have multiple public keys within the same application does not improve privacy.
The public key of the Nostr profile must match the public key of the Bitcoin transaction containing the weekly hash (the Merkle root of all the users posts and the entire tree hash). Unlike Nostr users who use Schnorr signatures, they will need to use a Bitcoin wallet (mobile wallet/light wallet or full node) to sign.
The beauty of this is that each Nostr account will be represented by its Bitcoin address, meaning users can send payments directly to Nostr accounts without requesting two different public keys. This reduces the cognitive cost of new users in the system. Instead of using usernames, users still need to send each other public keys or DNS to find each other on Nostr.
If other Nostr applications use different public keys, they can still be attached to the same decentralized identity (DID) - this way, the way your account is identified remains consistent across applications. However, this Nostr consensus rule will limit the use of only one public key per profile on each Nostr application.
DHT acts as a leaderboard for peer discovery.
Relays can use a Distributed Hash Table (DHT) to find other Relays. Relays can share their whitelist in a distributed hash table by listing the public key where the data is stored. This way, relays with missing forks of data for a certain public key can scan the DHT and connect directly to the IP addresses of other relays claiming to store those missing forks. Relays can then download missing branches directly from these Nostr relays.
Relays will also be able to find the most active relayers by checking how many previous ZKCSP transactions those relayers have resolved on-chain - recent and all time. In this system, all Nostr relays become full nodes, so auditing previous transactions of other relays will be painless. This would make forging trust expensive, since on-chain transactions always require transaction fees. If a Nostr relay opens many channels to build trust with itself in order to gain the trust of other relays, he will have to pay a lot of transaction fees to maintain the fake reputation every week. After the attacker fails to supply the missing branch, the timeout will cause the relay to disconnect - so this is only a temporary, expensive attack (just like a 51% attack is a temporary, expensive attack).
The DHT is not as anonymous as mining, as each Nostr relays public key will be listed next to the IP address in the DHT, but it will avoid the need for relays to blindly send requests across the network - at a large enough scale, This will overload the network. Nostr relayers looking for more privacy can use a VPN or other IP protection service.
Users will not have access to this same system of trust because they are not full nodes. However, users can rely on it.
Financiers are indirectly connected to hundreds of Nostr relays
Since users automatically store all block headers in their light wallets, users can see who the most active miners are. Miners becoming financiers will enable users to filter out the most popular miners so they dont have to blindly tie funds to random financiers that have no correlation to the viability of the network.
in conclusion
in conclusion
The Lightning Network would not exist without Bitcoin’s Nakamoto consensus blockchain, as users would have nowhere to store their bundled proofs of off-chain transactions.
Just like users bundle all these Lightning Network transactions together and put small proofs into the blockchain, we will bundle all Nostr data and put small proofs into the blockchain. In the same way that the Lightning Network provides instant payments at the second layer, Nostr may be able to provide data storage at the second layer without the risk of an insecure sidechain.
It uses the same approach as the Lightning Network, with Bitcoins Nakamoto consensus blockchain on the first layer and Nostr+ZKCSP Lightning on the second layer.
