Distributed search
Our ambition is to make ipfs-search into a truly distributed search engine. We are grateful To Nina for the work she put in to researching distributed search on behalf of ipfs-search. The following is adapted from the results, first published on Nina’s homepage.
What does it mean for search to be distributed?
Distributed search engines
Sadly most that existed have gone defunct, leaving seekstorm (former FAROO (Propietary, English), Seeks (Open Source, English) and YaCy (Fully distributed).
IPFS search engines
Noetic Noetic searches IPFS content as indexed by conventional search engines. Noetic has had no commits over the past three years.
Trinity Trinity is a more promising but currently unfunctional effort to make a search engine and has had no commits for a year.
ipfssearch.xyz ipfsearch.xyz searches decentrally but requires a pre-compiled database.
dweb.page dweb.page uses IOTA, a permissionless distributed ledger allowing every single user to run their own search engine.
Source
The path to distributed search
Learning paths of technology evolutions which we travelled for getting transforming ideas for dweb search.
Distributed technology
Blockchain
Hashgraph
DAG
Holochain
Assistive technologies
Zero knowledge proofs
Smart contracts
Voting
Gossip about gossip
Current implementations
IPFS
BTFS
FileCoin
The usual peer crawling
Internet-facing demo
Current IPFS Search architecture
Good enough indexing
Technologies for distributed search
IPFS
Overlay networks
Parsing
Distributing the index on an IPFS Cluster
Querying the index
Yggdrasil
Testing
Moonshot for realizing distributed search
Provider nodes that wish to participate, parse and index only the files they have added to a dweb (DHT hashes) and that have world file permissions.
This local index is put on an (IPFS) cluster.
A query can use the distributed index.
Initial search functionalities are basic boolean search, to begin with.
Settings functionality anticipates tuning.
In the future, one can add to the search engine functionality with extensions.