Alex Potsides 978eb3676f
feat: async peerstore backed by datastores (#1058)
We have a peerstore that keeps all data for all observed peers in memory with no eviction.

This is fine when you don't discover many peers but when using the DHT you encounter a significant number of peers so our peer storage grows and grows over time.

We have a persistent peer store, but it just periodically writes peers into the datastore to be read at startup, still keeping them in memory.

It also means a restart doesn't give you any temporary reprieve from the memory leak as the previously observed peer data is read into memory at startup.

This change refactors the peerstore to use a datastore by default, reading and writing peer info as it arrives.  It can be configured with a MemoryDatastore if desired.

It was necessary to change the peerstore and *book interfaces to be asynchronous since the datastore api is asynchronous.

BREAKING CHANGE: `libp2p.handle`, `libp2p.registrar.register` and the peerstore methods have become async
2022-01-20 12:03:35 +00:00
..
2021-08-20 09:13:21 +02:00
2021-02-10 21:00:40 +01:00

Publish Subscribe

Publish Subscribe is also included on the stack. Currently, we have two PubSub implementation available libp2p-floodsub and libp2p-gossipsub, with many more being researched at research-pubsub.

We've seen many interesting use cases appear with this, here are some highlights:

0. Set up the example

Before moving into the examples, you should run npm install on the top level js-libp2p folder, in order to install all the dependencies needed for this example. In addition, you will need to install the example related dependencies by doing cd examples && npm install. Once the install finishes, you should move into the example folder with cd pubsub.

1. Setting up a simple PubSub network on top of libp2p

For this example, we will use MulticastDNS for automatic Peer Discovery. This example is based the previous examples found in Discovery Mechanisms. You can find the complete version at 1.js.

Using PubSub is super simple, you only need to provide the implementation of your choice and you are ready to go. No need for extra configuration.

First, let's update our libp2p configuration with a pubsub implementation.

const Libp2p = require('libp2p')
const Gossipsub = require('libp2p-gossipsub')

const node = await Libp2p.create({
  addresses: {
    listen: ['/ip4/0.0.0.0/tcp/0']
  },
  modules: {
    transport: [ TCP ],
    streamMuxer: [ Mplex ],
    connEncryption: [ NOISE ],
    // we add the Pubsub module we want
    pubsub: Gossipsub
  }
})

Once that is done, we only need to create a few libp2p nodes, connect them and everything is ready to start using pubsub.

const { fromString } = require('uint8arrays/from-string')
const { toString } = require('uint8arrays/to-string')
const topic = 'news'

const node1 = nodes[0]
const node2 = nodes[1]

// Add node's 2 data to the PeerStore
await node1.peerStore.addressBook.set(node2.peerId, node2.multiaddrs)
await node1.dial(node2.peerId)

node1.pubsub.on(topic, (msg) => {
  console.log(`node1 received: ${toString(msg.data)}`)
})
await node1.pubsub.subscribe(topic)

// Will not receive own published messages by default
node2.pubsub.on(topic, (msg) => {
  console.log(`node2 received: ${toString(msg.data)}`)
})
await node2.pubsub.subscribe(topic)

// node2 publishes "news" every second
setInterval(() => {
  node2.pubsub.publish(topic, fromString('Bird bird bird, bird is the word!'))
}, 1000)

The output of the program should look like:

> node 1.js
connected to QmWpvkKm6qHLhoxpWrTswY6UMNWDyn8hN265Qp9ZYvgS82
node1 received: Bird bird bird, bird is the word!
node1 received: Bird bird bird, bird is the word!

You can change the pubsub emitSelf option if you want the publishing node to receive its own messages.

const defaults = {
  config: {
    pubsub: {
      enabled: true,
      emitSelf: true
    }
  }
}

The output of the program should look like:

> node 1.js
connected to QmWpvkKm6qHLhoxpWrTswY6UMNWDyn8hN265Qp9ZYvgS82
node1 received: Bird bird bird, bird is the word!
node2 received: Bird bird bird, bird is the word!
node1 received: Bird bird bird, bird is the word!
node2 received: Bird bird bird, bird is the word!

2. Future work

libp2p/IPFS PubSub is enabling a whole set of Distributed Real Time applications using CRDT (Conflict-Free Replicated Data Types). It is still going through heavy research (and hacking) and we invite you to join the conversation at research-CRDT. Here is a list of some of the exciting examples: