The Ultimate Guide to Filecoin: Digging Deep into the Filecoin White Paper
Editor's Note: This article comes fromBlue Fox Notes (ID: lanhubiji)Editor's Note: This article comes from
Blue Fox Notes (ID: lanhubiji)
Blue Fox Notes (ID: lanhubiji)
, the author of this article "Vasa", translated by "SIEN" of the "Blue Fox Notes" community, and reprinted by Odaily with authorization.
Foreword: According to the plan, Filecoin will go live on the testnet on December 11, and the mainnet will go live in March 2020. As a decentralized storage project, Filecoin builds a storage and retrieval market, similar to the decentralized Airbnb in the storage market. Can it bring impact and subversion to the centralized storage market? Technically, what exactly is it? How to interpret it?
Since the decentralization revolution started in 2009, many promising projects have emerged and changed the way we view the world and the way we live. One of them is Protocol Labs, which has spawned amazing projects like IPFS.
This is where Filecoin comes in. Since its launch, Filecoin has garnered a lot of attention in the community. But because of its token economy (crowdfunding and investment strategy), it lost quite a few supporters. Apparently, some people seem to be unhappy with its plans.
Before we dive into its core technology, let's analyze the current state of the file storage market.
secondary title
State of the file storage market
text
Today, Amazon S3 is the giant of file storage on the internet. there are many reasons:
Very cheap: $0.023 per GB of storage, 0.04 cents per 10,000 read requests;
very fast
Reliable: Indeed, it had several major outages that took large parts of the internet offline. But it still has 99.9% uptime.
In a world where we already have such great cloud storage services, any competitor must have something better, or at least on par. On a small scale, decentralized networks don't do a great job.
Technical overview
However, if IPFS is adopted on a large scale (more so than BitTorrent), then it may prove to be a better version of the Internet, and it will also open up a whole new economy.
secondary title
Technical overview
There are four sections:
In-depth study of the Filecoin protocol
Other issues (not covered in the white paper)
Possible Improvements to the Filecoin Protocol
secondary title
How does the Filecoin network work?
There are 3 groups of users in Filecoin: clients, storage miners, and retrieval miners.
Customers pay for the service of storing and retrieving data. They can choose from available service providers. If they want to store private data, they need to encrypt it before submitting it to the service provider.
Storage miners store customers' data for rewards. They decide for themselves how much space to provide for storage. After the client and the storage miner reach an agreement, the miner is obliged to provide proof of its stored data on an ongoing basis. Everyone can look at this proof and be sure that the storage miner is reliable.
Retrieval miners provide data to clients upon their request. They can fetch data from clients or storage miners. Retrieval miners and clients exchange data and tokens using micropayments: data is sliced into shards, and clients pay a small amount of tokens for each shard. Retrieval miners can also act as storage miners at the same time.
Finally, the network represents all full nodes validating the actions of clients and miners. These nodes count available storage, check storage proofs, and fix data errors.
Some terms in this article:
Shards: A shard refers to a portion of data that a client stores in a decentralized storage network. For example, data (perhaps a picture of a cat) can be intentionally split into many shards, and each shard can be stored into a different storage miner.
Sector: A sector is some disk space provided by storage miners to the network (it can be considered as a unique ID, which is associated with a specific part of the disk space of a specific storage miner). Miners store pieces of customers' data in their sectors and earn tokens for their services. In order to store shards, storage miners must pledge their sectors to the network.
Allocation Table: An allocation table is a data structure that keeps track of fragments and their allocated sectors. The allocation table is updated every block on the ledger, and its Merkle root is stored in the latest block. In practice, an allocation table is used to keep the state of the DSN for quick lookups when verifying proofs.
Order: An order is a statement to request or provide a service. Clients submit buy orders to the market to request services (including requesting data storage in the storage market and data retrieval in the retrieval market), and miners submit sell orders to provide services.
Users express their interest by placing an order. A customer submits a buy order and specifies the price they want to pay. Miners submit sell orders and specify the price they want to charge. When the buy and sell orders are matched, both the client and the miner sign the transaction order agreement and submit it to the blockchain.
storage market
Together, buy and sell orders form the Storage Market (the market for file storage) and the Retrieval Market (the market for file retrieval). We can delve into these markets and see how they work.
secondary title
storage market
It is a decentralized exchange run by the network, where all sell and buy orders are stored on the blockchain and used to store data on the Filecoin network.
Clients submit buy orders to the storage order book using the PUT protocol. Customers must deposit tokens in a designated order and specify the number of copies they want to store. Customers can submit multiple orders and can also specify copy quantities in one order. Higher redundancy (Blue Fox Notes: This refers to a greater number of replications) can produce higher fault tolerance for storage failures.
Storage Miners guarantee their storage to the network by depositing pledged tokens, whose guarantees are guaranteed transactions in the blockchain through Manage.PledgeSector. Staking tokens are staked for as long as the storage miner is willing to provide the service, and tokens are returned if the miner generates proof of data storage they committed to.
If some storage proofs fail, the storage miner loses staked tokens proportional to it.
Once the guaranteed transaction appears on the blockchain (and thus in the allocation table), miners can offer their storage in the storage market: they set a price and submit a sell order to the market order book via Put.AddOrders.
When a sell order matches a buy order (via Put.MatchOrders), the client sends the data shard to the miner.
When a piece is received, the miner runs Put.ReceivePiece. After receiving the data, both the miner and the client sign the transaction order and submit it to the blockchain (where the market order book is stored).
Storage Miners' storage is divided into sectors, each sector contains shards assigned to the miner. The network keeps track of each Storage Miner's sector through an allocation table. At this time (when the transaction order agreement is signed), the network distributes the data to the miners and leaves a record on the distribution table.
When a storage miner sector is full, the sector is sealed. Sealing is a slow and orderly operation that turns the data in a sector into a copy, which is the only physical copy of the data associated with the storage miner's public key. Sealing is a necessary operation during Proof-of-Replication.
When storage miners are assigned data, they must repeatedly generate proofs of replication to ensure they are storing the data. Proofs are published on the blockchain and verified by the network.
If any proof is missing or invalid, the network will take part of the miner's mortgage tokens as a punishment;
If a large number of proofs are missing or invalid (defined by the system parameter Δfault), the network will consider the storage miners to be faulty, settle the order as failure, and then re-introduce the data fragments into the market as new orders;
If each Storage Miner fails to store the shard, the shard is lost and the client receives a refund.
secondary title
search market
This is an off-chain peer-to-peer marketplace where clients and retrieval miners discover each other. Once clients and miners agree on a price, they start trading data and tokens in micropayments on a case-by-case basis.
A client submits a buy order to the retrieved market order book. Retrieval miners check to see if their orders match the corresponding buy orders from clients.
Summarize
secondary title
The image below shows all the activity happening on the network
image description
Execution example of the Filecoin network, grouped by participants and time-ordered by row
secondary title
An in-depth study of the Filecoin protocol
Filecoin introduces the concept of a decentralized storage network (DSN). A DSN is a scheme for describing a network of independent customers and storage providers. DSN aggregates storage provided by multiple independent storage providers and coordinates itself to provide customers with data storage and data retrieval services.
Coordination is decentralized and does not require a trusted third party: the secure operation of these systems is achieved through agreements that coordinate and verify the actions of various participants. (Blue Fox Note: Unlike Airbnb’s scheduling through a centralized company, it does not need to trust a third party)
DSN can adopt different strategies for coordination according to system requirements, including Byzantine agreement, gossip agreement, or CRDT.
DSN involves the realization of 3 functions: storage, acquisition and management. "Deposit" allows customers to store data under a unique identifier. "Get" allows clients to retrieve data using a unique identifier. "Management" is the administration of the decentralized storage marketplace network, which measures space available for rent, audits storage providers, and fixes possible data failures. Governance agreements are typically run by storage providers along with customers or auditors of the network.
A DSN has several properties. The first two are mandatory.
Data integrity means that customers always receive the same data they deposit, and storage providers cannot reassure customers if they provide incorrect data.
Retrievability means that customers can retrieve their data over time.
Other properties of DSN:
Public verifiability, which allows anyone in the network to verify that data is stored without knowing it.
Auditability, which allows verification that data was stored for the correct period of time.
Incentive-compatible, it aims to reward good service providers and punish bad ones.
Enabling confidentiality: Its customers who wish to store their data privately must encrypt their data before submitting it to the network.
fault tolerance
A DSN should be fault tolerant to two types of possible errors:
management fault tolerance
This is Byzantine fault tolerance caused by participants (storage providers, clients & auditors) in the governance protocol. The DSN mechanism relies on the fault tolerance of the protocol it manages. Violations of the fault-tolerance assumptions of management errors can compromise the liveness and security of the system.
For example, consider the DSB mechanism, where the governance protocol requires Byzantine agreement (since nodes can lie to the auditor) to audit storage providers (if they store all data that should be stored according to the agreed conditions).
In such a protocol, the network collects proofs of storage from storage providers and runs a Byzantine agreement to verify the validity of these proofs. If the Byzantine agreement can tolerate at most f errors out of a total of n nodes, then our DSN can tolerate f < n/2 error nodes. In cases where these assumptions are violated, auditing can be compromised, rendering the system useless.
Storage fault tolerance is also Byzantine fault tolerance, and they prevent clients from retrieving data: that is, storage miners lose data fragments, and retrieval miners stop providing services. If its "stored" data is stored on m independent storage providers (n in total), and it is fault-tolerant to at most f Byzantine providers, then a successful "stored" execution is (f,m) − fault tolerance. The parameters f and m depend on the protocol implementation; the protocol designer can fix f and m or let the user choose, and extend Put(data) to Put(data,f,m).
consensus algorithm
If there are fewer than f faulty storage providers, the "get" of the storage data is performed successfully. For example, consider the simple scenario where the "storage" protocol is designed so that each storage provider stores all data. In this scheme, m=n, and f=m-1.
So, will f=m-1 always be? no. Some schemes can be designed to use erasure coding, where each storage provider stores a specific portion of the data such that x out of a total of m storage providers are required to retrieve the data; in this case, f = mx.
consensus algorithm
Filecoin's DSN protocol can be implemented on top of any consensus protocol, which allows verification of Filecoin's proofs. Proof-of-work mechanisms typically require solving puzzles whose answers cannot be reused or require a large amount of computation to find. (Note from Blue Fox Notes: That is, no memory, starting again every time)
non-reusable work
Most permissionless blockchains require miners to solve difficult computational puzzles, such as reversing hash functions. Often, the answers to these difficult questions are useless and have no intrinsic value other than providing security for the network. Some blockchains, such as Ethereum (executing smart contract logic) and Primecoin (finding new prime numbers) try to use some of their computing power to do useful work.
wasted work
Solving puzzles can be very expensive in terms of machine and energy consumption costs, especially if those puzzles rely solely on computing power. When mining algorithms are embarrassingly parallel, the main factor in solving puzzles is computing power.
try to reduce waste
Ideally, most of the network's resources should be spent on useful work. Several efforts are also trying to require miners to use more energy-efficient solutions. For example, Spacemint requires miners to dedicate disk space instead of computation. Although more energy efficient, these disks are still "wasted" because they are filled with random data.
Other efforts include the use of traditional PoS-based Byzantine agreement as an alternative to solving puzzles. Among them, stakeholders of token pledge vote in the system for the next block according to their proportional token proportion.
The work of Filecoin miners is not to perform wasteful PoW proof calculations, they generate PoST (Proof-of-Spacetime) to participate in the consensus.
useful work
We argue that the work done by miners in a consensus protocol is useful if, in addition to securing the blockchain, the results of computations are also valuable to the network.
Filecoin proposes a useful work-of-work consensus protocol in which the probability that the network elects miners to produce new blocks is proportional to the proportion of their current storage space in the network. The design of the Filecoin protocol is such that miners would rather invest in storage than invest in computing power for parallel mining. Miners provide storage and reuse computation to prove that data is being stored in order to participate in consensus.
Modeling Mining Capabilities
power fault tolerance
Power Fault Tolerance is a form of abstraction that refactors Byzantine Fault Tolerance in terms of the influence of participants on the outcome of the protocol.
Each participant controls some power, where n is the total power in the network and f is the fractional power, controlled by wrong parties or malicious actors.
Power in Filecoin
In Filecoin, the power p of a miner M at time t is the sum of M's storage allocations. M's influence I is the ratio of M's power to the total power of the entire network. In Filecoin, power has the following properties:
public
The total amount of storage currently in use in the network is public. By reading the blockchain, anyone can calculate each miner's storage allocation, and therefore, anyone can calculate each miner's power and the total power in the network at any point in time. (Blue Fox Note: The power here, similar to the influence in its network, can be directly used to elect miners who generate blocks.)
publicly verifiable
variable
witch attack
At any point in time, miners can add new storage to the network by committing to new sectors and filling the sectors. In this way, miners can change the proportion of power they hold over time.
We also need a mechanism to prevent three types of attacks where malicious miners can exploit storage they do not actually provide for rewards: Sybil Attack, Outsourcing Attack, Generation Attack .
witch attack
By creating multiple sybil identities, a malicious miner can pretend that they are storing more copies than they actually are (and get paid for doing so), but actually only store the data once.
outsourcing attack
By relying on quickly fetching data from other storage providers, malicious miners promise to store more data than they can actually store.
generate attack
Malicious miners can claim to store vast amounts of data that they efficiently generate on demand using a small program. If the program is smaller than what it claims to store, this increases the probability that a malicious miner will win a block reward in Filecoin proportional to the storage the miner is currently using.
Storage providers have to convince their customers that they already store the data they paid to store. In effect, storage providers generate Proof-of-Storage (PoS) to be verified by the blockchain network or by the clients themselves.
In order to make storage behavior publicly verifiable, Filecoin introduces two consensus algorithms: Proof-of-Replication (PoRep) and Proof-of-Spacetime (PoSt), that is, proof of replication and proof of space-time.
Proof of Replication (PoRep) is a novel proof of storage that allows a server (prover P) to convince a user (verifier V) that some data D has been replicated to its own unique dedicated physical storage.
Our mechanism is an interactive protocol where the prover P:
(a) A commitment to store n different copies (physically separate copies) of some data D
(b) Convince verifier V that P did indeed store each replica via the challenge/response protocol. PoRep improves the PoR and PDP mechanisms to prevent Sybil attacks, outsourcing attacks, and generation attacks.
Proof-of-Spacetime: The PoS (Proof of Storage) mechanism allows users to check whether a storage provider is storing outsourced data during a challenge. How can we use a PoS (Proof of Storage) mechanism to prove that certain data was stored over a period of time?
A natural answer to this question is to require the user to repeatedly (eg, every minute) send challenges to the storage provider. However, the communication complexity required for each interaction can be a bottleneck in systems like Filecoin, where storage providers need to commit their proofs to the blockchain.
The intuition is to ask for a prover
Generate ordered proof-of-storage (proof-of-replication in Filecoin) as a way of determining time.
Recursive combinatorial execution to generate short proofs
PoSt icon
smart contract
The prover receives a random challenge (c)) from the verifier, and uses the output of the proof as the input of another for a specified number of iterations t, which in turn generates replica proofs. So make sure that all work done is reusable (as above).
PoSt & PoRep use zk-SNARKS, which makes the proof very concise and easy to verify.
smart contract
Smart contracts enable Filecoin users to write stateful programs that can spend tokens, request storage/retrieval of data in the marketplace, and verify proof-of-storage. Users can interact with smart contracts by sending transactions to the ledger, which can trigger function calls in the contract. We extended the smart contract system to support Filecoin-specific operations, such as market operations, proof verification.
Filecoin supports specific data storage contracts, as well as more general smart contracts.
document contract
We allow users to program the conditions under which they provide storage services. A few examples are worth mentioning:
contract with miners
Clients can designate service miners in advance without participating in the market.
payment policy
Clients can design different reward strategies for miners. For example, contracts can be set to pay miners higher and higher fees over time, or contracts can set storage prices informed by trusted oracles.
Contracts can allow miners to deposit tokens to pay for storage/retrieval on behalf of their users.
smart contract
more complex operations
Clients can create contracts that allow data upgrades.
smart contract
Users can associate programs with their transactions, like other systems (such as Ethereum) that do not directly depend on the use of storage. We can foresee these applications: DNS (Blue Fox Note: Decentralized Domain Name System), asset tracking and crowdfunding platforms.
Cross-chain interaction
Filecoin on other platforms: Other blockchain systems, such as Bitcoin, Zcash, and especially Ethereum and Tezos, allow developers to write smart contracts; however, these platforms offer little storage capacity and are extremely costly.
We plan to provide a bridge to provide storage and retrieval support for these platforms. We noticed that IPFS is already used by several smart contracts as a way to reference and distribute content. Adding support for Filecoin will allow these systems to guarantee storage of IPFS content in exchange for Filecoin tokens.
Other platforms in Filecoin: We plan to provide bridges to connect other blockchain services with Filecoin. For example, integration with zcash would allow sending requests to store private data.
secondary title
some other questions
Here we list some potential issues that are not adequately discussed in the white paper:
Search Market Scalability
Micropayment systems (retrieval markets) incur a lot of overhead on the retrieval protocol. To achieve retrieval speeds that match today's centralized infrastructure, Filecoin and IPFS need massive adoption to create a dense network of state channels. (Blue Fox Notes: If the search market is large, then its micropayment needs higher throughput support)
Censorship (illegal content)
As we’ve seen in the past with Napster and the Priate Bay, a lack of censorship will eventually lead to illegal content on the web, bringing the dark web to light. A possible solution is an AI-driven protocol that can learn over time and automatically detect illegal content and take necessary action.
But in order for the network to be a shared governance network, the protocol needs to be governed by the users themselves (introducing Byzantine behavior) to decide whether the content needs to be acted upon.
Open source?
Extrapolating from the above issues, it is also possible that initially in order to protect the network from illegal content, it may have been managed by Protocol Labs. This probably means closed software, free to use but not open for modification.
Token volatility
Considering that Filecoin will be listed on the exchange, how feasible is a micropayment system (storage and retrieval will design micropayments) in this market? (Blue Fox Note: The author means that Filecoin tokens are volatile in the market, and it is difficult to be used as a currency for micropayments, and stable coins are required).
Judging from the current market maturity and decentralized field, tokens are more like investment vehicles than utility tools. This is one of the biggest reasons why we don't see much adoption of token-based projects today.
secondary title
Possibility of Filecoin Protocol Improvement
Here we list possible improvements in the Filecoin protocol.
Tahor-LAFS Encryption Scheme
When adding value, the client first encrypts it (with a symmetric key), then divides it into manageable sized shares, and then Erasure Codes for redundancy. (Blue Fox Notes: EC coding, also known as erasure code, can add n copies of original data to m copies of data, and can restore the original data to any n copies of data in n+m copies).
So, for example, "2 of 3" erasure coding means that there are a total of 3 copies, any 2 of which are sufficient to reconstruct the original data. These shares can be called shares, stored on specific storage nodes. Storage nodes are shared databases; users do not rely on them for data integrity or confidentiality.
Ultimately, the encryption key and some information that helps in discovering the correct storage node become part of the "capabilities string". Importantly, the capability string is both necessary and sufficient for retrieving values from the Grid. It will fail if too many nodes become unavailable or offline and you cannot get enough shares retrieved.
