311 Commits

Author SHA1 Message Date
Cayman
8880eefa8f
fix: add successful stream peer to protobook (#1341)
* fix: add successful stream peer to protobook

If a protocol stream has been successfully negotiated and is to be
psased to the application, the peerstore should ensure that the peer is
registerd with that protocol.

* fix: upgrader test fix
2022-08-11 10:48:16 -05:00
Alex Potsides
f439d9b589
deps!: update all deps to support no-copy operations (#1335)
Updates all deps needed to support passing lists of byte arrays where they have been created from multiple input buffers.

When reading multiplexed data, all messages arrive in length-prefixed buffers, which means the first few bytes tell the consumer how many bytes long next chunk will be.

One length prefixed chunk can be delivered in several payloads from the underlying network transport. The first payload can also include the length prefix and some or all of the data, so we stitch these together in a `Uint8ArrayList` to avoid having to concatenate `Uint8Array`s together.

Previously once we'd received enough bytes to satisfy the length prefix we'd concatenate the bytes together, but this is a potentially expensive operation where transports have small message sizes so instead just pass the `Uint8ArrayList` to the consumer and let them decide wether to concatenate or not as some consumers will be smart enough to operate on lists of `Uint8Array`s instead of always requiring a contiguous block of memory.

BREAKING CHANGE: Streams are now `Duplex<Uint8ArrayList, Uint8ArrayList | Uint8Array>`
2022-08-11 13:21:04 +01:00
Alex Potsides
564f4b8aa7
deps: update it-length-prefix, uint8arraylist etc (#1317)
In order to support no-copy operations in streams, update all deps
to support streaming Uint8ArrayLists.
2022-08-03 14:15:35 +01:00
Alex Potsides
05e8e7ead9
fix: remove mplex prefix from muxer errors (#1304)
Muxer errors are now standard across implementations so do not
depend on the `"MPLEX_"` prefix
2022-07-25 13:37:31 +01:00
Alex Potsides
3c0fb13bab
fix: close streams when protocol limits are reached (#1301)
- If a stream is opened that exceeds inbound/outbound limits, reset that stream (if it is incoming) or abort and throw (if it is outgoing)
- Make the error message more helpful (say which protocol has breached the limit)
- Increase the default stream limits so we don't trigger this by accident when a remote dials us with a protocol we don't support
2022-07-22 13:57:01 +01:00
Alex Potsides
ba56c64662
fix: add timeout for circuit relay (#1294)
Make sure we don't potentially wait forever during incoming circuit relay handshakes.

Adds a timeout option to the hop config to control how long we will wait.
2022-07-15 16:36:31 +00:00
Alex Potsides
0bb1b802c8
feat: programmatically set agentVersion for use in identify (#1296)
If no `agentVersion` is provided for the Identify protocol, the default `AGENT_VERSION` will now be set to
* `js-libp2p/<libp2p.version> UserAgent=<process.version>` when running in Node.js
* `js-libp2p/<libp2p.version> UserAgent=<navigator.userAgent>` when running in the browser (also when running in a webworker)

Fixes #686
Supersedes #1240

Co-authored-by: Kevin Westphal <westphal@consider-it.de>
Co-authored-by: Kevin <56823591+6d7a@users.noreply.github.com>
2022-07-15 16:35:52 +00:00
Alex Potsides
750ed9c35f
fix: add timeout for incoming connections and build-in protocols (#1292)
Ensure that we don't wait forever for upgrading an inbound connection
to occur.

Note that transports should return an AbortableSource when passed an
AbortSignal so outbound connections to not need the same fix.

Also adds default timeouts for the ping, fetch, and identify protocols.
2022-07-14 13:35:12 +00:00
Cayman
b1b91398e2
fix: update muxer behavior (#1289)
- use `direction` in `muxerFactory.createStreamMuxer`
- use `muxer.close` instead of `muxer.streams.forEach(s => s.close())`
2022-07-14 12:03:06 +00:00
Alex Potsides
e6f646ed36
chore: update deps (#1285) 2022-07-01 18:33:17 +02:00
Alex Potsides
5af93883ce
chore: update deps (#1284)
Update libp2p deps
2022-06-29 06:59:51 +01:00
Alex Potsides
2836acc90f
fix: use keep-alive tag to reconnect to peers on startup (#1278)
Instead of trying to connect to every peer in the peer store when
we start a node, only connect to the peers that have been marked
with a `keep-alive` tag.
2022-06-28 08:05:16 +01:00
Alex Potsides
b1b2b216da
feat!: use tag values to choose which connections to close (#1276)
Uses peer tag values to select low-value connections to prune when we have too many connections open.

BREAKING CHANGE: `connectionManager.peerValue` has been removed, use `peerStore.tagPeer` instead
2022-06-27 15:34:03 +01:00
Alex Potsides
de30c2cec7
feat!: limit protocol streams per-connection (#1255)
* feat: limit protocol streams per-connection

Uses the `maxInboundStreams` and `maxOutboundStreams` of the `registrar.handle`
opts to limit the number of concurrent streams open on each connection
on a per-protocol basis.

Both values default to 1 so some tuning will be necessary to set
appropriate values for some protocols.

* chore: make error codes consistent

* chore: fix up examples
2022-06-17 15:46:31 +02:00
Alex Potsides
d4dd664071
feat!: update libp2p interfaces (#1252)
BREAKING CHANGE: uses new single-issue libp2p interface modules
2022-06-15 18:30:39 +01:00
Alex Potsides
b0472686d2
fix: wait for peer stats to be updated during test (#1238)
The peer stats update interval doesn't always align with the timing
in the test so make sure it's elapsed before asserting on the results.

Fixes #1219
2022-06-08 10:47:41 -05:00
Alex Potsides
f9073ecd21
fix: connection pruning (#1235)
Actually prune connections when we reach the connection limit
2022-06-08 16:20:56 +01:00
Alex Potsides
eee256db8a
fix: ensure streams are closed when protocol negotiation fails (#1236)
If an error is thrown during the initial stages of setting up a multiplexed stream, ensure we close the stream to free up any resources associated with it.
2022-06-08 08:29:32 +01:00
Alex Potsides
3babbbd75a
chore: update ipfs-http-client (#1234)
Updates to ESM version of ipfs-http-client
2022-06-06 15:04:00 +01:00
Alex Potsides
824720fb8f
fix: reduce identify message size limit (#1230)
Adds a config option to specify a maximum message size we'll accept
for an Identify message.

The default is 8KB, the same as go-libp2p - previously we fell back
to the default `maxMessageLength` option of `it-length-prefixed`
which is 4MB.

Also adds a default timeout for reading responses to identify
requests which is used if an AbortSignal is not passed in.

The default timeout also aligns with go-libp2p.
2022-05-31 17:10:40 +01:00
Alex Potsides
a1220d22f5
fix: time out slow reads (#1227)
There are a few places in the codebase where we send/receive data from the network without timeouts/abort controllers which means the user has to wait for the underlying socket to timeout which can take a long time depending on the platform, if at all.

This change ensures we can time out while running identify (both flavours), ping and fetch and adds tests to ensure there are no regressions.
2022-05-25 18:15:21 +01:00
Robert Kiel
35f9c0c793
fix: fix unintended aborts in dialer (#1185)
Fix a bug where `DialRequest` can abort wrong dial attempts.

Co-authored-by: Robert Kiel <robert.kiel@hoprnet.io>
2022-05-18 14:53:58 +01:00
Alex Potsides
d5386df684
fix: do upnp hole punch after startup (#1217)
The transport manager configures it's addresses during the `start`
phase so access them during `afterStart` so they'll be ready for use.
2022-05-18 13:19:31 +01:00
Alex Potsides
31480603f3
fix: simplify pnet exports (#1213)
Export the key generator from `libp2p/pnet`
2022-05-16 10:07:32 +01:00
Alex Potsides
4837430d8b
fix: encode enums correctly (#1210)
Updates protons and regenerates protobuf code to encode enums correctly
2022-05-10 12:35:33 +01:00
Alex Potsides
da3d19b309
fix: update interfaces (#1207)
Update to the latest interfaces version
2022-05-04 16:03:43 +01:00
Alex Potsides
a15254fdd4
fix: update to new interfaces (#1206)
Notably removes the `Dialer` interface as the `ConnectionManager` is now in charge of managing connections.
2022-05-04 10:19:04 +01:00
tuyennhv
d16817ca44
fix: emit peer:connect after all (#1171)
**Motivation**

In lodestar, when we handle "peer:connect" event, we dial the peer which gives another "peer:connect" event and it causes other issues

**Motivation**

In `onConnect` function, "peer:connect" event should be emitted after we add connection to the `connections` map so that when app dial the peer in "peer:connect" event handler, it uses the same/existing connection
2022-04-22 20:56:47 +01:00
Alex Potsides
fab4f1385c
fix: update pubsub interfaces (#1194)
Update to latest version of pubsub interface
2022-04-22 20:49:35 +01:00
Alex Potsides
5397137c65
fix: use placeholder dht/pubsub (#1193)
Instead of making the `.dht` and `.pubsub` properties optional, use dummy implementations that throw exceptions if they are not configured.

This way we don't have to null guard everywhere they are accessed.
2022-04-21 15:46:06 +01:00
Alex Potsides
147304449e
fix: expose getPublicKey (#1188)
This is used externally by IPFS so expose the method
2022-04-14 18:00:21 +01:00
Alex Potsides
c64a586a20
chore: update aegir to the latest version (#1186)
Removes boilerplate config that is no longer necessary
2022-04-09 09:26:25 +01:00
Alex Potsides
cc60cfde1a
fix: add transport manager to exports map and fix docs (#1182)
Addresses PR comments from #1172 - fixes syntax of examples in docs, adds the transport manager to the exports map and renames fault tolerance enum for consistency.
2022-03-29 15:39:50 +01:00
Alex Potsides
199395de4d
feat: convert to typescript (#1172)
Converts this module to typescript.

- Ecosystem modules renamed from (e.g.) `libp2p-tcp` to `@libp2p/tcp`
- Ecosystem module now have named exports
- Configuration has been updated, now pass instances of modules instead of classes:
- Some configuration keys have been renamed to make them more descriptive.  `transport` -> `transports`, `connEncryption` -> `connectionEncryption`.  In general where we pass multiple things, the key is now plural, e.g. `streamMuxer` -> `streamMuxers`, `contentRouting` -> `contentRouters`, etc.  Where we are configuring a singleton the config key is singular, e.g. `connProtector` -> `connectionProtector` etc.
- Properties of the `modules` config key have been moved to the root
- Properties of the `config` config key have been moved to the root
```js
// before
import Libp2p from 'libp2p'
import TCP from 'libp2p-tcp'

await Libp2p.create({
  modules: {
    transport: [
      TCP
    ],
  }
  config: {
    transport: {
      [TCP.tag]: {
        foo: 'bar'
      }
    },
    relay: {
      enabled: true,
      hop: {
        enabled: true,
        active: true
      }
    }
  }
})
```
```js
// after
import { createLibp2p } from 'libp2p'
import { TCP } from '@libp2p/tcp'

await createLibp2p({
  transports: [
    new TCP({ foo: 'bar' })
  ],
  relay: {
    enabled: true,
    hop: {
      enabled: true,
      active: true
    }
  }
})
```
- Use of `enabled` flag has been reduced - previously you could pass a module but disable it with config.  Now if you don't want a feature, just don't pass an implementation.   Eg:
```js
// before
await Libp2p.create({
  modules: {
    transport: [
      TCP
    ],
    pubsub: Gossipsub
  },
  config: {
    pubsub: {
      enabled: false
    }
  }
})
```
```js
// after
await createLibp2p({
  transports: [
    new TCP()
  ]
})
```
- `.multiaddrs` renamed to `.getMultiaddrs()` because it's not a property accessor, work is done by that method to calculate announce addresses, observed addresses, etc
- `/p2p/${peerId}` is now appended to all addresses returned by `.getMultiaddrs()` so they can be used opaquely (every consumer has to append the peer ID to the address to actually use it otherwise).  If you need low-level unadulterated addresses, call methods on the address manager.

BREAKING CHANGE: types are no longer hand crafted, this module is now ESM only
2022-03-28 14:30:27 +01:00
Alex Potsides
902f10d58d
fix: reject connections when not running (#1146)
When the node is shutting down, new connections can still be received.

If this happens we can end up writing into the datastore when it's
been closed which throws an error.

Instead, if we're not running, have the connection manager close new
incoming connections.
2022-01-26 10:52:23 +00:00
Alex Potsides
d44bd9094f
fix: await unhandle of protocols (#1144)
To allow us to shut down cleanly, we must wait the unhandling of protocols - this is because they write the new list of protocols into the datastore which might also be in the process of shutting down.
2022-01-25 19:56:56 +00:00
Alex Potsides
ff32eba6a0
feat: connection gater (#1142)
Port of https://github.com/libp2p/go-libp2p-core/blob/master/connmgr/gater.go

Adds a new configuration key `connectionGater` which allows denying the dialing of certain peers, individual multiaddrs and the creation of connections at certain points in the connection flow.

Fixes: https://github.com/libp2p/js-libp2p/issues/175
Refs: https://github.com/libp2p/js-libp2p/issues/744
Refs: https://github.com/libp2p/js-libp2p/issues/769

Co-authored-by: mzdws <8580712+mzdws@user.noreply.gitee.com>
2022-01-25 16:27:01 +00:00
Robert Kiel
9b22c6e2f9
fix: prevent auto-dialer from dialing self (#1104)
Co-authored-by: Robert Kiel <robert.kiel@hoprnet.io>
Co-authored-by: achingbrain <alex@achingbrain.net>
2022-01-24 17:59:14 +00:00
Spencer T Brody
d8ceb0bc66
feat: add fetch protocol (#1036)
Adds three methods to implement the `/libp2p/fetch/0.0.1` protocol:

* `libp2p.fetch(peerId, key) => Promise<Uint8Array>`
* `libp2p.fetchService.registerLookupFunction(prefix, lookupFunction)`
* `libp2p.fetchService.unRegisterLookupFunction(prefix, [lookupFunction])`

Co-authored-by: achingbrain <alex@achingbrain.net>
2022-01-24 17:07:11 +00:00
Alex Potsides
63aa480800
fix: try all peer addresses when dialing a relay (#1140)
The order of the addresses can affect our success rate in dialing a
relay - if it's a loopback address or similar it won't work.

Instead try dialing every address.
2022-01-21 11:57:09 -06:00
Alex Potsides
b7e87066a6
fix: make tests more reliable (#1139)
Try to use only public functions and properties to verify test behaviour
2022-01-21 12:58:31 +00:00
Alex Potsides
4c3bf01f35
chore: fix flaky tests (#1137)
These tests are flaky in CI, probably due to differences in timing introduced by #1058

Fixes #1134
2022-01-21 08:44:13 +00:00
Alex Potsides
978eb3676f
feat: async peerstore backed by datastores (#1058)
We have a peerstore that keeps all data for all observed peers in memory with no eviction.

This is fine when you don't discover many peers but when using the DHT you encounter a significant number of peers so our peer storage grows and grows over time.

We have a persistent peer store, but it just periodically writes peers into the datastore to be read at startup, still keeping them in memory.

It also means a restart doesn't give you any temporary reprieve from the memory leak as the previously observed peer data is read into memory at startup.

This change refactors the peerstore to use a datastore by default, reading and writing peer info as it arrives.  It can be configured with a MemoryDatastore if desired.

It was necessary to change the peerstore and *book interfaces to be asynchronous since the datastore api is asynchronous.

BREAKING CHANGE: `libp2p.handle`, `libp2p.registrar.register` and the peerstore methods have become async
2022-01-20 12:03:35 +00:00
Alex Potsides
0a4dc54d08
fix: remove abort-controller dep (#1095)
The `AbortController` class is supported by browsers and node 14+ - we only support node 16+ (e.g. LTS+Current) so the `abort-controller` module isn't needed any more.
2022-01-20 12:02:13 +00:00
Alex Potsides
5043cd5643
fix: cache build artefacts (#1091)
To speed up the build and make it more reliable, cache the node_modules
folder, dist, etc and re-use on each step.
2021-12-29 15:06:58 +01:00
Alex Potsides
2f0b311df7
feat: allow per-component metrics to be collected (#1061)
Implements the idea from #1060 - allows us to get some insight into what's happening in a libp2p node out side of just bandwidth stats.

Configures a few default metrics if metrics are enabled - current connections, the state of the dial queue, etc.

Also makes the `Metrics` class not depend on the `ConnectionManager` class, otherwise we can't collect simple metrics from the connection manager class due to the circular dependency.
2021-12-15 08:03:09 +00:00
Alex Potsides
1b46f47fdb
chore: run node tests in ci (#1057)
Looks like this project stopped running the `test:node` npm script when it was migrated to gh actions.

Re-enable it and fix all the related test failures.
2021-12-08 08:38:17 +00:00
Robert Kiel
3b683e7156
fix: fix uncaught promise rejection when finding peers (#1044)
Do not abort all attempts to find peers when `findPeers` on one router throws synchronously

Co-authored-by: Robert Kiel <robert.kiel@hoprnet.io>
Co-authored-by: achingbrain <alex@achingbrain.net>
2021-12-07 17:51:48 +00:00
Alex Potsides
b25e0fe531
fix: make error codes consistent (#1054)
Sometimes they are `NOT_FOUND`, sometimes `ERR_NOT_FOUND`, etc.

Move all error codes into `errors.js` and reference them from there.
2021-12-07 14:42:10 +00:00
Alex Potsides
9cbf36fcb5
chore: update peer id and libp2p crypto (#1042)
BREAKING CHANGE: requires node 15+
2021-12-02 10:11:23 +00:00