We have a peerstore that keeps all data for all observed peers in memory with no eviction.
This is fine when you don't discover many peers but when using the DHT you encounter a significant number of peers so our peer storage grows and grows over time.
We have a persistent peer store, but it just periodically writes peers into the datastore to be read at startup, still keeping them in memory.
It also means a restart doesn't give you any temporary reprieve from the memory leak as the previously observed peer data is read into memory at startup.
This change refactors the peerstore to use a datastore by default, reading and writing peer info as it arrives. It can be configured with a MemoryDatastore if desired.
It was necessary to change the peerstore and *book interfaces to be asynchronous since the datastore api is asynchronous.
BREAKING CHANGE: `libp2p.handle`, `libp2p.registrar.register` and the peerstore methods have become async
When we've previously seen loads of peers and stored them in the
datastore we'll try to dial them as part of starting the autodial
component.
If we or our peers have bad network connections this can make
starting a libp2p node take ages so don't wait for a round of auto
dialing before considering the component started.
Small refactor of the component stats - adds a `TrackedMap` which encapsulates updating the metrics and means we don't need to null guard on `this._metrics` everywhere.
If metrics are not enabled a regular `Map` is used.
We use timeout controllers to ensure we're not dialling peers forever but we can end up registering lots of listeners for the `abort` event when peers have a lot of addresses.
In node this means we see an unhelpful `MaxListenersExceededWarning` in the console warning of a potential memory leak.
Increase the max number of listeners on the signal to silence the warning.
Implements the idea from #1060 - allows us to get some insight into what's happening in a libp2p node out side of just bandwidth stats.
Configures a few default metrics if metrics are enabled - current connections, the state of the dial queue, etc.
Also makes the `Metrics` class not depend on the `ConnectionManager` class, otherwise we can't collect simple metrics from the connection manager class due to the circular dependency.
If the `Promise.race` throws, execution of the function is terminated so the pending dial target is never removed from the map and we leak memory.
This can happen when there are invalid multiaddrs or when a peer reports more dialable addresses than the threshold.
Instead wrap the `Promise.race` in a `try/finally` which will always remove the pending dial target in the event of success or failure.
Looks like this project stopped running the `test:node` npm script when it was migrated to gh actions.
Re-enable it and fix all the related test failures.
Do not abort all attempts to find peers when `findPeers` on one router throws synchronously
Co-authored-by: Robert Kiel <robert.kiel@hoprnet.io>
Co-authored-by: achingbrain <alex@achingbrain.net>
Sometimes you encounter peers with lots of addresses. When this happens
you can attach more than 10x event listeners to the abort signal we
use to abort all the dials - this causes node to print a warning
which is misleading.
This PR increases the default number of listeners on the signal.
Fixes#900
Stop the dht before the connection manager, otherwise in-flight eviction pings fail and we move on to the next one when we should just abort them all.
Also pulls in the fix from #1039 and splits the auto-dialler out from the connection manager as during shutdown it can get into a weird state where it's simultaneously killing and creating connections so stop auto-dialling things before we cause connections to dip below the low watermark by killing existing connections.
Fixes: https://github.com/ipfs/js-ipfs/issues/3923
`interface-datastore` now only contains the interface definition,
`datastore-core` has the various implementations.
BREAKING CHANGE: datastore implementations provided to libp2p must be compliant with interface-datastore@6.0.0
* chore: remove ipfs-utils dep
We only use it for the env detection, so use [wherearewe](https://www.npmjs.com/package/wherearewe)
instead which is that, but pulled out into a tiny module.
The `TextDecoder` class is global everywhere we support so we don't
need to pull it in from `ipfs-utils` and it's been removed from v8
anyway.
* chore: update ipfs-http-client
* fix: give stats initial values
Otherwise the compiler cannot derive the type and thinks `stats.snapshot` returns `{}`
* fix: add type shape to moving averages as well
BREAKING CHANGES:
top level types were updated, multiaddr@9.0.0 is used, dialer and keychain internal property names changed and connectionManager minPeers is not supported anymore
