fluencelabs/dweb-transports
1
0
Fork 0
mirror of https://github.com/fluencelabs/dweb-transports synced 2025-03-15 10:30:48 +00:00
Code Issues Projects Releases Wiki Activity

Merge branch 'master' into IPFS

Browse Source
This commit is contained in:
Mitra Ardron 2018-07-25 18:28:37 -07:00
commit 1c5aeb8b62
17 changed files with 805 additions and 24205 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -1 +1,3 @@
node_modules
/package-lock.json
data.json

37
API.md
View File

@ -30,6 +30,7 @@ There are a set of classes:
* *TransportIPFS*: Connects to IPFS, currently (April 2018) via WebSocketsStar (WSS)
* *TransportYJS*: Implements shared lists, and dictionaries. Uses IPFS for transport
* *TransportWEBTORRENT*: Integrates to Feross's WebTorrent library
* *TransportGUN*: Integrates to the Gun DB
* *Transports*: manages the list of conencted transports, and directs api calls to them.
Calls are generally made through the Transports class which knows how to route them to underlying connections.
@ -172,12 +173,13 @@ verbose boolean - True for debugging output
Resolves to Array objects as stored on the list (see p_rawlist for format)
```
##### listmonitor (url, cb, {verbose})
##### listmonitor (url, cb, {verbose, current})
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
```
url Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
cb(obj) function(obj) Callback for each new item added to the list
verbose boolean - True for debugging output
verbose true for debugging output
current true to send existing members as well as new
obj is same format as p_rawlist or p_rawreverse
```
@ -323,6 +325,11 @@ returns instance of TransportIPFS if connected
returns instance of TransportWEBTORRENT if connected
```
##### static gun(verbose)
```
returns instance of TransportGUN if connected
```
##### static async p_resolveNames(urls)
See Naming below
```
@ -345,7 +352,7 @@ t: Add a Transport instance to _transports
##### static setup0(transports, options, verbose, cb)
Calls setup0 for each transport based on its short name. Specially handles ‘LOCAL’ as a transport pointing at a local http server (for testing).
```
transports Array of short names of transports e.g. [‘IPFS’,’HTTP’,’ORBITDB’]
transports Array of short names of transports e.g. [‘IPFS’,’HTTP’,’GUN’]
options Passed to setup0 on each transport
cb Callback to be called each time status changes
Returns: Array of transport instances
@ -400,7 +407,7 @@ static async p_rawstore(data, {verbose})|[urls]|Tries all and combines results
static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay})|data|See note
static async p_rawlist(urls, {verbose})|[sigs]|Tries all and combines results
static async p_rawadd(urls, sig, {verbose})||Tries on all urls, error if none succeed
static listmonitor(urls, cb, {verbose})||Tries on all urls (so note cb may be called multiple times)
static listmonitor(urls, cb, {verbose, current})||Tries on all urls (so note cb may be called multiple times)
static p_newlisturls(cl, {verbose})|[urls]|Tries all and combines results
static async p_f_createReadStream(urls, options)|f(opts)=>stream|Returns first success
static async p_get(urls, keys, {verbose})|currently (April 2018) returns on first success, TODO - will combine results and relay across transports
@ -411,7 +418,7 @@ static async p_getall(urls, {verbose})|dict|currently (April 2018) returns on fi
static async p_newdatabase(pubkey, {verbose})|{privateurls: [urls], publicurls: [urls]}|Tries all and combines results
static async p_newtable(pubkey, table, {verbose})|{privateurls: [urls], publicurls: [urls]}|Tries all and combines results
static async p_connection(urls, verbose)||Tries all parallel
static monitor(urls, cb, verbose)||Tries all sequentially
static monitor(urls, cb, {verbose, current})||Tries all sequentially
##### static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay})
Tries to fetch on all valid transports until successful. See Transport.p_rawfetch
@ -425,7 +432,7 @@ relay If first transport fails, try and retrieve on 2nd, then store on 1s
A utility class to support HTTP with or without TransportHTTP
e.g. `httptools.http().p_httpfetch("http://foo.com/bar", {method: 'GET'} )`
##### p_httpfetch(url, init, verbose)
##### p_httpfetch(url, init, {verbose)}
Fetch a url.
If the result
@ -518,6 +525,14 @@ supportFunctions:
supportFeatures:
fetch.range Not supported (currently April 2018)
## TransportGUN
A subclass of Transport for handling GUN connections (decentralized database)
supportURLS = `gun:*` (TODO: may in the future support `dweb:/gun/*`)
supportFunctions
`add, list, listmonitor, newlisturls, connection, get, set, getall, keys, newdatabase, newtable, monitor`
supportFeatures:
## Naming
Independently from the transport, the Transport library can resolve names if provided an appropriate callback.
See p_resolveNames(urls) and resolveNamesWith(cb)
@ -533,3 +548,13 @@ The format of names currently (April 2018) is under development but its likely t
`dweb:/arc/archive.org/details/foo`
to allow smooth integration with existing HTTP urls that are moving to decentralization.
## Adding a Transport
The following steps are needed to add a transport.
* Add a line to package.json/dependencies for any packages needed
* Add lines to index.js;
* Add function to return the instance to Transports.js
* Add to list in Transports.p_connect()
* Add to API.md
* Look for any "SEE-OTHER-ADDTRANSPORT" in case not on this list
* Edit a copy of the closest Transport to what you are building

16
Errors.js
View File

@ -34,6 +34,14 @@ class TimeoutError extends Error {
}
errors.TimeoutError = TimeoutError;
class IntentionallyUnimplementedError extends Error {
constructor(message) {
super(message || "Intentionally Unimplemented Function");
this.name = "IntentionallyUnimplementedError"
}
}
errors.IntentionallyUnimplementedError = IntentionallyUnimplementedError;
/*---- Below here are errors copied from previous Dweb-Transport and not currently used */
/*
@ -80,14 +88,6 @@ class AuthenticationError extends Error {
}
errors.AuthenticationError = AuthenticationError;
class IntentionallyUnimplementedError extends Error {
constructor(message) {
super(message || "Intentionally Unimplemented Function");
this.name = "IntentionallyUnimplementedError"
}
}
errors.IntentionallyUnimplementedError = IntentionallyUnimplementedError;
class DecryptionFailError extends Error {
constructor(message) {
super(message || "Decryption Failed");

38
Transport.js
View File

@ -15,7 +15,7 @@ class Transport {
Fields:
statuselement: If set is an HTML Element that should be adjusted to indicate status (this is managed by Transports, just stored on Transport)
statuscb: Callback when status changes
name: Short name of element e.g. HTTP IPFS WEBTORRENT
name: Short name of element e.g. HTTP IPFS WEBTORRENT GUN
*/
}
@ -140,6 +140,7 @@ class Transport {
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/
console.assert(false, "Intentionally undefined function Transport.p_rawfetch should have been subclassed");
return "UNIMPLEMENTED";
}
p_fetch() {
@ -200,7 +201,7 @@ class Transport {
throw new errors.ToBeImplementedError("Undefined function Transport.p_rawreverse");
}
listmonitor(url, callback, verbose) {
listmonitor(url, callback, {verbose=false, current=false}={}) {
/*
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
@ -307,6 +308,31 @@ class Transport {
return c;
}
async p_test_list({urlexpectedsubstring=undefined, verbose=false}={}) {
//TODO - this test doesn't work since we dont have Signature nor want to create dependency on it - when works, add to GUN & YJS
if (verbose) {console.log(this.name,"p_test_kvt")}
try {
let table = await this.p_newlisturls("NACL VERIFY:1234567LIST", {verbose});
let mapurl = table.publicurl;
if (verbose) console.log("newlisturls=",mapurl);
console.assert((!urlexpectedsubstring) || mapurl.includes(urlexpectedsubstring));
await this.p_rawadd(mapurl, "testvalue", {verbose});
let res = await this.p_rawlist(mapurl, {verbose});
console.assert(res.length===1 && res[0] === "testvalue");
await this.p_rawadd(mapurl, {foo: "bar"}, {verbose}); // Try adding an object
res = await this.p_rawlist(mapurl, {verbose});
console.assert(res.length === 2 && res[1].foo === "bar");
await this.p_rawadd(mapurl, [1,2,3], {verbose}); // Try setting to an array
res = await this.p_rawlist(mapurl, {verbose});
console.assert(res.length === 2 && res[2].length === 3 && res[2][1] === 2);
await delay(200);
if (verbose) console.log(this.name, "p_test_list complete")
} catch(err) {
console.log("Exception thrown in ", this.name, "p_test_list:", err.message);
throw err;
}
}
async p_test_kvt(urlexpectedsubstring, verbose=false) {
/*
Test the KeyValue functionality of any transport that supports it.
@ -314,7 +340,7 @@ class Transport {
*/
if (verbose) {console.log(this.name,"p_test_kvt")}
try {
let table = await this.p_newtable("NACL VERIFY:1234567","mytable", {verbose});
let table = await this.p_newtable("NACL VERIFY:1234567KVT","mytable", {verbose});
let mapurl = table.publicurl;
if (verbose) console.log("newtable=",mapurl);
console.assert(mapurl.includes(urlexpectedsubstring));
@ -328,11 +354,11 @@ class Transport {
res = await this.p_get(mapurl, "testkey3", {verbose});
console.assert(res[1] === 2);
res = await this.p_keys(mapurl, {verbose});
console.assert(res.includes("testkey") && res.includes("testkey3"));
res = await this.p_delete(mapurl, ["testkey"], {verbose});
console.assert(res.includes("testkey") && res.includes("testkey3") && res.length === 3);
await this.p_delete(mapurl, ["testkey"], {verbose});
res = await this.p_getall(mapurl, {verbose});
if (verbose) console.log("getall=>",res);
console.assert(res.testkey2.foo === "bar" && res.testkey3["1"] === 2 && !res.testkey1);
console.assert(res.testkey2.foo === "bar" && res.testkey3["1"] === 2 && !res.testkey);
await delay(200);
if (verbose) console.log(this.name, "p_test_kvt complete")
} catch(err) {

364
TransportGUN.js Normal file
View File

@ -0,0 +1,364 @@
/*
This Transport layers uses GUN.
*/
const Url = require('url');
process.env.GUN_ENV = "false";
const Gun = require('gun/gun.js'); // TODO-GUN switchback to gun/gun at some point to get minimized version
require('gun/lib/path.js');
// Other Dweb modules
const errors = require('./Errors'); // Standard Dweb Errors
const Transport = require('./Transport.js'); // Base class for TransportXyz
const Transports = require('./Transports'); // Manage all Transports that are loaded
const utils = require('./utils'); // Utility functions
// Utility packages (ours) And one-liners
//unused currently: function delay(ms, val) { return new Promise(resolve => {setTimeout(() => { resolve(val); },ms)})}
let defaultoptions = {
peers: [ "https://dweb.me:4246/gun" ]
//localstore: true #True is default TODO-GUN check if false turns it off, or defaults to a different store.
};
//To run a superpeer - cd wherever; node install gun; cd node_modules/gun; npm start - starts server by default on port 8080, or set an "env" - see http.js
//setenv GUN_ENV false; node examples/http.js 4246
//Make sure to open of the port (typically in /etc/ferm)
// TODO-GUN - copy example from systemctl here
/*
WORKING AROUND GUN WEIRNESS/SUBOPTIMAL (of course, whats weird/sub-optimal to me, might be ideal to someone else) - search the code to see where worked around
WORKAROUND-GUN-UNDERSCORE .once() and possibly .on() send an extra GUN internal field "_" which needs filtering. Reported and hopefully will get fixed
.once behaves differently on node or the browser - this is a bug https://github.com/amark/gun/issues/586 and for now this code doesnt work on Node
WORKAROUND-GUN-CURRENT: .once() and .on() deliver existing values as well as changes, reported & hopefully will get way to find just new ones.
WORKAROUND-GUN-DELETE: There is no way to delete an item, setting it to null is recorded and is by convention a deletion. BUT the field will still show up in .once and .on,
WORKAROUND-GUN-PROMISES: GUN is not promisified, there is only one place we care, and that is .once (since .on is called multiple times).
WORKAROUND-GUN-ERRORS: GUN does an unhelpful job with errors, for example returning undefined when it cant find something (e.g. if connection to superpeer is down),
for now just throw an error on undefined
Errors and Promises: Note that GUN's use of promises is seriously uexpected (aka weird), see https://gun.eco/docs/SEA#errors
instead of using .reject or throwing an error at async it puts the error in SEA.err, so how that works in async parallel context is anyone's guess
*/
class TransportGUN extends Transport {
/*
GUN specific transport - over IPFS
Fields:
gun: object returned when starting GUN
*/
constructor(options, verbose) {
super(options, verbose);
this.options = options; // Dictionary of options
this.gun = undefined;
this.name = "GUN"; // For console log etc
this.supportURLs = ['gun'];
this.supportFunctions = [ 'fetch', //'store'
'connection', 'get', 'set', 'getall', 'keys', 'newdatabase', 'newtable', 'monitor',
'add', 'list', 'listmonitor', 'newlisturls'];
this.status = Transport.STATUS_LOADED;
}
connection(url, verbose) {
/*
TODO-GUN need to determine what a "rooted" Url is in gun, is it specific to a superpeer for example
Utility function to get Gun object for this URL (note this isn't async)
url: URL string or structure, to find list of of form [gun|dweb]:/gun/<database>/<table>[/<key] but could be arbitrary gun path
resolves: Gun a connection to use for get's etc, undefined if fails
*/
url = Url.parse(url); // Accept string or Url structure
let patharray = url.pathname.split('/'); //[ 'gun', database, table ] but could be arbitrary length path
patharray.shift(); // Loose leading ""
patharray.shift(); // Loose "gun"
if (verbose) console.log("Path=", patharray);
return this.gun.path(patharray); // Not sure how this could become undefined as it will return g before the path is walked, but if do a lookup on this "g" then should get undefined
}
static setup0(options, verbose) {
/*
First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections.
options: { gun: { }, } Set of options - "gun" is used for those to pass direct to Gun
*/
let combinedoptions = Transport.mergeoptions(defaultoptions, options.gun);
console.log("GUN options %o", combinedoptions); // Log even if !verbose
let t = new TransportGUN(combinedoptions, verbose); // Note doesnt start IPFS or OrbitDB
t.gun = new Gun(t.options); // This doesnt connect, just creates db structure
Transports.addtransport(t);
return t;
}
async p_setup1(verbose, cb) {
/*
This sets up for GUN.
Throws: TODO-GUN-DOC document possible error behavior
*/
try {
this.status = Transport.STATUS_STARTING; // Should display, but probably not refreshed in most case
if (cb) cb(this);
//TODO-GUN-TEST - try connect and retrieve info then look at ._.opt.peers
await this.p_status(verbose);
} catch(err) {
console.error(this.name,"failed to start",err);
this.status = Transport.STATUS_FAILED;
}
if (cb) cb(this);
return this;
}
async p_status(verbose) {
/*
Return an integer for the status of a transport see Transport
*/
//TODO-GUN-TEST - try connect and retrieve info then look at ._.opt.peers
this.status = Transport.STATUS_CONNECTED; //TODO-GUN how do I know if/when I'm connected (see comment on p_setup1 as well)
return this.status;
}
// ===== DATA ======
async p_rawfetch(url, {verbose=false}={}) {
url = Url.parse(url); // Accept url as string or object
let g = this.connection(url, verbose); // Goes all the way to the key
let val = await this._p_once(g);
if (!val) throw new errors.TransportError("GUN unable to retrieve: "+url.href); // WORKAROUND-GUN-ERRORS - gun doesnt throw errors when it cant find something
return typeof val === "string" ? JSON.parse(val) : val; // This looks like it is sync (see same code on p_get and p_rawfetch)
}
// ===== LISTS ========
// noinspection JSCheckFunctionSignatures
async p_rawlist(url, {verbose=false}={}) {
/*
Fetch all the objects in a list, these are identified by the url of the public key used for signing.
(Note this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter
Returns a promise that resolves to the list.
Each item of the list is a dict: {"url": url, "date": date, "signature": signature, "signedby": signedby}
List items may have other data (e.g. reference ids of underlying transport)
:param string url: String with the url that identifies the list.
:param boolean verbose: true for debugging output
:resolve array: An array of objects as stored on the list.
*/
try {
let g = this.connection(url, verbose);
let data = await this._p_once(g);
let res = data ? Object.keys(data).filter(k => k !== '_').sort().map(k => data[k]) : []; //See WORKAROUND-GUN-UNDERSCORE
// .filter((obj) => (obj.signedby.includes(url))); // upper layers verify, which filters
if (verbose) console.log("GUN.p_rawlist found", ...utils.consolearr(res));
return res;
} catch(err) {
console.log("TransportGUN.p_rawlist failed",err.message);
throw(err);
}
}
listmonitor(url, callback, {verbose=false, current=false}={}) {
/*
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
url: string Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
callback: function(obj) Callback for each new item added to the list
obj is same format as p_rawlist or p_rawreverse
current true if should send list of existing elements
verbose: true for debugging output
*/
let g = this.connection(url, verbose);
if (!current) { // See WORKAROUND-GUN-CURRENT have to keep an extra copy to compare for which calls are new.
g.once(data => {
this.monitored = data ? Object.keys(data) : []; // Keep a copy - could actually just keep high water mark unless getting partial knowledge of state of array.
g.map().on((v, k) => {
if (!(this.monitored.includes(k)) && (k !== '_')) { //See WORKAROUND-GUN-UNDERSCORE
this.monitored.push(k);
callback(JSON.parse(v));
}
});
});
} else {
g.map().on((v, k) => callback("set", k, JSON.parse(v)));
}
}
// noinspection JSCheckFunctionSignatures
async p_rawadd(url, sig, {verbose=false}={}) {
/*
Store a new list item, it should be stored so that it can be retrieved either by "signedby" (using p_rawlist) or
by "url" (with p_rawreverse). The underlying transport does not need to guarantee the signature,
an invalid item on a list should be rejected on higher layers.
:param string url: String identifying list to post to
:param Signature sig: Signature object containing at least:
date - date of signing in ISO format,
urls - array of urls for the object being signed
signature - verifiable signature of date+urls
signedby - urls of public key used for the signature
:param boolean verbose: true for debugging output
:resolve undefined:
*/
// noinspection JSUnresolvedVariable
console.assert(url && sig.urls.length && sig.signature && sig.signedby.length, "TransportGUN.p_rawadd args", url, sig);
if (verbose) console.log("TransportGUN.p_rawadd", typeof url === "string" ? url : url.href, sig);
this.connection(url, verbose)
.set( JSON.stringify( sig.preflight( Object.assign({}, sig))));
}
// noinspection JSCheckFunctionSignatures
async p_newlisturls(cl, {verbose=false}={}) {
let u = await this._p_newgun(cl, {verbose});
return [ u, u];
}
//=======KEY VALUE TABLES ========
// noinspection JSMethodCanBeStatic
async _p_newgun(pubkey, {verbose=false}={}) {
if (pubkey.hasOwnProperty("keypair"))
pubkey = pubkey.keypair.signingexport();
// By this point pubkey should be an export of a public key of form xyz:abc where xyz
// specifies the type of public key (NACL VERIFY being the only kind we expect currently)
return `gun:/gun/${encodeURIComponent(pubkey)}`;
}
async p_newdatabase(pubkey, {verbose=false}={}) {
/*
Request a new database
For GUN it doesnt actually create anything, just generates the URLs
TODO-GUN simple version first - userid based on my keypair first, then switch to Gun's userid and its keypair
Include gun/sea.js; user.create(<alias>,<passphrase>); user.auth(<alias>,<passphrase>); # See gun.eco/docs/Auth
returns: {publicurl: "gun:/gun/<publickey>", privateurl: "gun:/gun/<publickey>">
*/
let u = await this._p_newgun(pubkey, {verbose});
return {publicurl: u, privateurl: u};
}
async p_newtable(pubkey, table, {verbose=false}={}) {
/*
Request a new table
For GUN it doesnt actually create anything, just generates the URLs
returns: {publicurl: "gun:/gun/<publickey>/<table>", privateurl: "gun:/gun/<publickey>/<table>">
*/
if (!pubkey) throw new errors.CodingError("p_newtable currently requires a pubkey");
let database = await this.p_newdatabase(pubkey, {verbose});
// If have use cases without a database, then call p_newdatabase first
return { privateurl: `${database.privateurl}/${table}`, publicurl: `${database.publicurl}/${table}`} // No action required to create it
}
async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table
/*
Set key values
keyvalues: string (key) in which case value should be set there OR
object in which case value is ignored
*/
let table = this.connection(url, verbose);
if (typeof keyvalues === "string") {
table.path(keyvalues).put(JSON.stringify(value));
} else {
// Store all key-value pairs without destroying any other key/value pairs previously set
console.assert(!Array.isArray(keyvalues), "TransportGUN - shouldnt be passsing an array as the keyvalues");
table.put(
Object.keys(keyvalues).reduce(
function(previous, key) { previous[key] = JSON.stringify(keyvalues[key]); return previous; },
{}
))
}
}
async p_get(url, keys, {verbose=false}={}) {
let table = this.connection(url, verbose);
if (Array.isArray(keys)) {
throw new errors.ToBeImplementedError("p_get(url, [keys]) isn't supported - because of ambiguity better to explicitly loop on set of keys or use getall and filter");
/*
return keys.reduce(function(previous, key) {
let val = table.get(key);
previous[key] = typeof val === "string" ? JSON.parse(val) : val; // Handle undefined
return previous;
}, {});
*/
} else {
let val = await this._p_once(table.get(keys)); // Resolves to value
return typeof val === "string" ? JSON.parse(val) : val; // This looks like it is sync (see same code on p_get and p_rawfetch)
}
}
async p_delete(url, keys, {verbose=false}={}) {
let table = this.connection(url, verbose);
if (typeof keys === "string") {
table.path(keys).put(null);
} else {
keys.map((key) => table.path(key).put(null)); // This looks like it is sync
}
}
//WORKAROUND-GUN-PROMISE suggest p_once as a good single addition
//TODO-GUN expand this to workaround Gun weirdness with errors.
_p_once(gun) { // Note in some cases (e.g. p_getall) this will resolve to a object, others a string/number (p_get)
return new Promise((resolve, reject) => gun.once(resolve));
}
async p_keys(url, {verbose=false}={}) {
let res = await this._p_once(this.connection(url, verbose));
return Object.keys(res)
.filter(k=> (k !== '_') && (res[k] !== null)); //See WORKAROUND-GUN-UNDERSCORE and WORKAROUND-GUN-DELETE
}
async p_getall(url, {verbose=false}={}) {
let res = await this._p_once(this.connection(url, verbose));
return Object.keys(res)
.filter(k=> (k !== '_') && res[k] !== null) //See WORKAROUND-GUN-UNDERSCORE and WORKAROUND-GUN-DELETE
.reduce( function(previous, key) { previous[key] = JSON.parse(res[key]); return previous; }, {});
}
async monitor(url, callback, {verbose=false, current=false}={}) {
/*
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_getall to get any more items not returned by p_getall.
Stack: KVT()|KVT.p_new => KVT.monitor => (a: Transports.monitor => GUN.monitor)(b: dispatchEvent)
url: string Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
callback: function({type, key, value}) Callback for each new item added to the list (type = "set"|"delete")
verbose: boolean - true for debugging output
current Send existing items to the callback as well
*/
let g = this.connection(url, verbose);
if (!current) { // See WORKAROUND-GUN-CURRENT have to keep an extra copy to compare for which calls are new.
g.once(data => {
this.monitored = Object.assign({},data); // Make a copy of data (this.monitored = data won't work as just points at same structure)
g.map().on((v, k) => {
if ((v !== this.monitored[k]) && (k !== '_')) { //See WORKAROUND-GUN-UNDERSCORE
this.monitored[k] = v;
callback("set", k, JSON.parse(v));
}
});
});
} else {
g.map().on((v, k) => callback("set", k, JSON.parse(v)));
}
}
static async p_test(verbose) {
if (verbose) {console.log("TransportGUN.test")}
try {
let t = this.setup0({}, verbose); //TODO-GUN when works with peers commented out, try passing peers: []
await t.p_setup1(verbose); // Not passing cb yet
await t.p_setup2(verbose); // Not passing cb yet - this one does nothing on GUN
// noinspection JSIgnoredPromiseFromCall
t.p_test_kvt("gun:/gun/NACL", {verbose});
//t.p_test_list("gun:/gun/NACL", {verbose}); //TODO test_list needs fixing to not create a dependency on Signature
} catch(err) {
console.log("Exception thrown in TransportGUN.test:", err.message);
throw err;
}
}
static async demo_bugs() {
let gun = new Gun();
gun.get('foo').get('bar').put('baz');
console.log("Expect {bar: 'baz'} but get {_:..., bar: 'baz'}");
gun.get('foo').once(data => console.log(data));
gun.get('zip').get('bar').set('alice');
console.log("Expect {12345: 'alice'} but get {_:..., 12345: 'alice'}");
gun.get('foo').once(data => console.log(data));
// Returns extra "_" field
}
}
Transports._transportclasses["GUN"] = TransportGUN;
exports = module.exports = TransportGUN;

57
TransportHTTP.js
View File

@ -144,6 +144,63 @@ class TransportHTTP extends Transport {
return [u,u];
}
// ============================== Stream support
/*
Code disabled until have a chance to test it with <VIDEO> tag etc, problem is that it returns p_createReadStream whch is async
if need sync, look at WebTorrent and how it buffers through a stream which can be returned immediately
*/
async p_f_createReadStream(url, {verbose=false, wanturl=false}={}) {
/*
Fetch bytes progressively, using a node.js readable stream, based on a url of the form:
No assumption is made about the data in terms of size or structure.
This is the initialisation step, which returns a function suitable for <VIDEO>
Returns a new Promise that resolves to function for a node.js readable stream.
Node.js readable stream docs: https://nodejs.org/api/stream.html#stream_readable_streams
:param string url: URL of object being retrieved of form magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents)
:param boolean verbose: true for debugging output
:resolves to: f({start, end}) => stream (The readable stream.)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/
if (verbose) console.log(this.name, "p_f_createreadstream", Url.parse(url).href);
try {
let self = this;
if (wanturl) {
return url;
} else {
return function (opts) { return self.p_createReadStream(url, opts, verbose); };
}
} catch(err) {
console.warn(`p_f_createReadStream failed on ${Url.parse(url).href} ${err.message}`);
throw(err);
}
}
async p_createReadStream(url, opts, verbose) {
/*
The function, encapsulated and inside another function by p_f_createReadStream (see docs)
NOTE THIS PROBABLY WONT WORK FOR <VIDEO> tags, but shouldnt be using it there anyway
:param file: Webtorrent "file" as returned by webtorrentfindfile
:param opts: { start: byte to start from; end: optional end byte }
:param boolean verbose: true for debugging output
:resolves to stream: The readable stream.
*/
if (verbose) console.log(this.name, "createreadstream", Url.parse(url).href, opts);
try {
return await httptools.p_GET(this._url(url, servercommands.rawfetch), Object.assign({wantstream: true}, opts));
} catch(err) {
console.warn(this.name, "caught error", err);
throw err;
}
}
// ============================== Key Value support

46
TransportIPFS.js
View File

@ -29,21 +29,19 @@ const Transports = require('./Transports'); // Manage all Transports that are lo
const utils = require('./utils'); // Utility functions
const defaultoptions = {
ipfs: {
repo: '/tmp/dweb_ipfsv2700', //TODO-IPFS think through where, esp for browser
//init: false,
//start: false,
//TODO-IPFS-Q how is this decentralized - can it run offline? Does it depend on star-signal.cloud.ipfs.team
config: {
// Addresses: { Swarm: [ '/dns4/star-signal.cloud.ipfs.team/wss/p2p-webrtc-star']}, // For Y - same as defaults
// Addresses: { Swarm: [ ] }, // Disable WebRTC to test browser crash, note disables Y so doesnt work.
//Addresses: {Swarm: ['/dns4/ws-star.discovery.libp2p.io/tcp/443/wss/p2p-websocket-star']}, // from https://github.com/ipfs/js-ipfs#faq 2017-12-05 as alternative to webrtc works sort-of
Bootstrap: ['/dns4/dweb.me/tcp/4245/wss/ipfs/QmPNgKEjC7wkpu3aHUzKKhZmbEfiGzL5TP1L8zZoHJyXZW'], // Supposedly connects to Dweb IPFS instance, but doesnt work (nor does ".../wss/...")
},
//init: true, // Comment out for Y
EXPERIMENTAL: {
pubsub: true
}
repo: '/tmp/dweb_ipfsv2700', //TODO-IPFS think through where, esp for browser
//init: false,
//start: false,
//TODO-IPFS-Q how is this decentralized - can it run offline? Does it depend on star-signal.cloud.ipfs.team
config: {
// Addresses: { Swarm: [ '/dns4/star-signal.cloud.ipfs.team/wss/p2p-webrtc-star']}, // For Y - same as defaults
// Addresses: { Swarm: [ ] }, // Disable WebRTC to test browser crash, note disables Y so doesnt work.
//Addresses: {Swarm: ['/dns4/ws-star.discovery.libp2p.io/tcp/443/wss/p2p-websocket-star']}, // from https://github.com/ipfs/js-ipfs#faq 2017-12-05 as alternative to webrtc works sort-of
Bootstrap: ['/dns4/dweb.me/tcp/4245/wss/ipfs/QmPNgKEjC7wkpu3aHUzKKhZmbEfiGzL5TP1L8zZoHJyXZW'], // Supposedly connects to Dweb IPFS instance, but doesnt work (nor does ".../wss/...")
},
//init: true, // Comment out for Y
EXPERIMENTAL: {
pubsub: true
}
};
@ -53,13 +51,13 @@ class TransportIPFS extends Transport {
Fields:
ipfs: object returned when starting IPFS
yarray: object returned when starting yarray
TODO - this is not complete
*/
constructor(options, verbose) {
super(options, verbose);
this.ipfs = undefined; // Undefined till start IPFS
this.options = options; // Dictionary of options { ipfs: {...}, "yarrays", yarray: {...} }
this.options = options; // Dictionary of options
this.name = "IPFS"; // For console log etc
this.supportURLs = ['ipfs'];
this.supportFunctions = ['fetch', 'store', 'createReadStream']; // Does not support reverse
@ -85,7 +83,7 @@ class TransportIPFS extends Transport {
*/
const self = this;
return new Promise((resolve, reject) => {
this.ipfs = new IPFS(this.options.ipfs);
this.ipfs = new IPFS(this.options);
this.ipfs.on('ready', () => {
//this._makepromises();
resolve();
@ -104,7 +102,7 @@ class TransportIPFS extends Transport {
/*
First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections.
*/
const combinedoptions = Transport.mergeoptions(defaultoptions, options);
const combinedoptions = Transport.mergeoptions(defaultoptions, options.ipfs);
if (verbose) console.log("IPFS loading options %o", combinedoptions);
const t = new TransportIPFS(combinedoptions, verbose); // Note doesnt start IPFS
Transports.addtransport(t);
@ -119,7 +117,7 @@ class TransportIPFS extends Transport {
await this.p_ipfsstart(verbose); // Throws Error("websocket error") and possibly others.
this.status = await this.p_status(verbose);
} catch(err) {
console.error("IPFS failed to connect",err);
console.error(this.name, "failed to connect", err);
this.status = Transport.STATUS_FAILED;
}
if (cb) cb(this);
@ -230,7 +228,7 @@ class TransportIPFS extends Transport {
:resolve buffer: Return the object being fetched. (may in the future return a stream and buffer externally)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/
if (verbose) console.log("IPFS p_rawfetch", utils.stringfrom(url));
if (verbose) console.log("IPFS p_rawfetch", Url.parse(url).href);
if (!url) throw new errors.CodingError("TransportIPFS.p_rawfetch: requires url");
const cid = TransportIPFS.cidFrom(url); // Throws TransportError if url bad
const ipfspath = TransportIPFS.ipfsFrom(url) // Need because dag.get has different requirement than file.cat
@ -245,7 +243,7 @@ class TransportIPFS extends Transport {
let buff;
//if (res.value instanceof DAGNode) { // Its file or something added with the HTTP API for example, TODO not yet handling multiple files
if (res.value.constructor.name === "DAGNode") { // Kludge to replace above, as its not matching the type against the "require" above.
if (verbose) console.log("IPFS p_rawfetch looks like its a file", url);
if (verbose) console.log("IPFS p_rawfetch looks like its a file", Url.parse(url).href);
//console.log("Case a or b" - we can tell the difference by looking at (res.value._links.length > 0) but dont need to
// as since we dont know if we are on node or browser best way is to try the files.cat and if it fails try the block to get an approximate file);
// Works on Node, but fails on Chrome, cant figure out how to get data from the DAGNode otherwise (its the wrong size)
@ -268,7 +266,7 @@ class TransportIPFS extends Transport {
} catch (err) { // TimeoutError or could be some other error from IPFS etc
console.log("Caught misc error in TransportIPFS.p_rawfetch trying IPFS", err.message);
try {
let ipfsurl = TransportIPFS.ipfsGatewayFrom(url)
let ipfsurl = TransportIPFS.ipfsGatewayFrom(url);
return await utils.p_timeout(
httptools.p_GET(ipfsurl), // Returns a buffer
timeoutMS, "Timed out IPFS fetch of "+ipfsurl)
@ -339,7 +337,7 @@ class TransportIPFS extends Transport {
verbose = true;
if (verbose) console.log("p_f_createReadStream",url);
const mh = TransportIPFS.multihashFrom(url);
const links = await this.ipfs.object.links(mh)
const links = await this.ipfs.object.links(mh);
let throughstream; //Holds pointer to stream between calls.
const self = this;
function crs(opts) { // This is a synchronous function

24
TransportWEBTORRENT.js
View File

@ -8,6 +8,7 @@ Y Lists have listeners and generate events - see docs at ...
const WebTorrent = require('webtorrent');
const stream = require('readable-stream');
const Url = require('url');
// Other Dweb modules
const errors = require('./Errors'); // Standard Dweb Errors
@ -15,7 +16,6 @@ const Transport = require('./Transport.js'); // Base class for TransportXyz
const Transports = require('./Transports'); // Manage all Transports that are loaded
let defaultoptions = {
webtorrent: {}
};
class TransportWEBTORRENT extends Transport {
@ -42,7 +42,7 @@ class TransportWEBTORRENT extends Transport {
*/
let self = this;
return new Promise((resolve, reject) => {
this.webtorrent = new WebTorrent(this.options.webtorrent);
this.webtorrent = new WebTorrent(this.options);
this.webtorrent.once("ready", () => {
console.log("WEBTORRENT READY");
resolve();
@ -58,8 +58,8 @@ class TransportWEBTORRENT extends Transport {
/*
First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections.
*/
let combinedoptions = Transport.mergeoptions(defaultoptions, options);
console.log("WebTorrent options %o", combinedoptions);
let combinedoptions = Transport.mergeoptions(defaultoptions, options.webtorrent);
if (verbose) console.log("WebTorrent options %o", combinedoptions); // Dont normally log options as its long
let t = new TransportWEBTORRENT(combinedoptions, verbose);
Transports.addtransport(t);
@ -73,7 +73,7 @@ class TransportWEBTORRENT extends Transport {
await this.p_webtorrentstart(verbose);
await this.p_status(verbose);
} catch(err) {
console.error("WebTorrent failed to connect",err);
console.error(this.name, "failed to connect", err);
this.status = Transport.STATUS_FAILED;
}
if (cb) cb(this);
@ -103,7 +103,7 @@ class TransportWEBTORRENT extends Transport {
throw new errors.CodingError("TransportWEBTORRENT.p_rawfetch: requires url");
}
const urlstring = (typeof url === "string" ? url : url.href)
const urlstring = (typeof url === "string" ? url : url.href);
const index = urlstring.indexOf('/');
if (index === -1) {
@ -216,7 +216,7 @@ class TransportWEBTORRENT extends Transport {
} catch(err) {
console.log(`p_fileFrom failed on ${url} ${err.message}`);
throw(err);
};
}
}
@ -236,7 +236,7 @@ class TransportWEBTORRENT extends Transport {
:resolves to: f({start, end}) => stream (The readable stream.)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/
if (verbose) console.log("TransportWEBTORRENT p_f_createreadstream %o", url);
if (verbose) console.log(this.name, "p_f_createreadstream", Url.parse(url).href);
try {
let filet = await this._p_fileTorrentFromUrl(url);
let self = this;
@ -248,7 +248,7 @@ class TransportWEBTORRENT extends Transport {
} catch(err) {
console.log(`p_f_createReadStream failed on ${url} ${err.message}`);
throw(err);
};
}
}
createReadStream(file, opts, verbose) {
@ -260,7 +260,7 @@ class TransportWEBTORRENT extends Transport {
:param boolean verbose: true for debugging output
:returns stream: The readable stream.
*/
if (verbose) console.log("TransportWEBTORRENT createreadstream %o %o", file.name, opts);
if (verbose) console.log(this.name, "createreadstream", file.name, opts);
let through;
try {
through = new stream.PassThrough();
@ -268,9 +268,9 @@ class TransportWEBTORRENT extends Transport {
fileStream.pipe(through);
return through;
} catch(err) {
console.log("TransportWEBTORRENT caught error", err)
console.log("TransportWEBTORRENT caught error", err);
if (typeof through.destroy === 'function')
through.destroy(err)
through.destroy(err);
else through.emit('error', err)
}
}

67
TransportYJS.js
View File

@ -25,7 +25,6 @@ const Transports = require('./Transports'); // Manage all Transports that are lo
const utils = require('./utils'); // Utility functions
let defaultoptions = {
yarray: { // Based on how IIIF uses them in bootstrap.js in ipfs-iiif-db repo
db: {
name: 'indexeddb', // leveldb in node
},
@ -33,21 +32,18 @@ let defaultoptions = {
name: 'ipfs',
//ipfs: ipfs, // Need to link IPFS here once created
},
}
};
class TransportYJS extends Transport {
/*
YJS specific transport - over IPFS, but could probably use other YJS transports
Fields:
ipfs: object returned when starting IPFS
yarray: object returned when starting yarray
Fields: TODO document this
*/
constructor(options, verbose) {
super(options, verbose);
this.options = options; // Dictionary of options { ipfs: {...}, "yarrays", yarray: {...} }
this.options = options; // Dictionary of options
this.name = "YJS"; // For console log etc
this.supportURLs = ['yjs'];
this.supportFunctions = ['fetch', 'add', 'list', 'listmonitor', 'newlisturls',
@ -70,7 +66,7 @@ class TransportYJS extends Transport {
if (verbose) console.log("Found Y for", url);
return this.yarrays[url];
} else {
let options = Transport.mergeoptions(this.options.yarray, {connector: {room: url}}, opts); // Copies options, ipfs will be set already
let options = Transport.mergeoptions(this.options, {connector: {room: url}}, opts); // Copies options, ipfs will be set already
if (verbose) console.log("Creating Y for", url); //"options=",options);
return this.yarrays[url] = await Y(options);
}
@ -103,7 +99,7 @@ class TransportYJS extends Transport {
/*
First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections.
*/
let combinedoptions = Transport.mergeoptions(defaultoptions, options);
let combinedoptions = Transport.mergeoptions(defaultoptions, options.yjs);
if (verbose) console.log("YJS options %o", combinedoptions); // Log even if !verbose
let t = new TransportYJS(combinedoptions, verbose); // Note doesnt start IPFS or Y
Transports.addtransport(t);
@ -119,11 +115,11 @@ class TransportYJS extends Transport {
try {
this.status = Transport.STATUS_STARTING; // Should display, but probably not refreshed in most case
if (cb) cb(this);
this.options.yarray.connector.ipfs = Transports.ipfs(verbose).ipfs; // Find an IPFS to use (IPFS's should be starting in p_setup1)
this.options.connector.ipfs = Transports.ipfs(verbose).ipfs; // Find an IPFS to use (IPFS's should be starting in p_setup1)
this.yarrays = {};
await this.p_status(verbose);
} catch(err) {
console.error("YJS failed to start",err);
console.error(this.name,"failed to start",err);
this.status = Transport.STATUS_FAILED;
}
if (cb) cb(this);
@ -135,10 +131,12 @@ class TransportYJS extends Transport {
Return a string for the status of a transport. No particular format, but keep it short as it will probably be in a small area of the screen.
For YJS, its online if IPFS is.
*/
this.status = (await this.options.yarray.connector.ipfs.isOnline()) ? Transport.STATUS_CONNECTED : Transport.STATUS_FAILED;
this.status = (await this.options.connector.ipfs.isOnline()) ? Transport.STATUS_CONNECTED : Transport.STATUS_FAILED;
return super.p_status(verbose);
}
// ======= LISTS ========
async p_rawlist(url, {verbose=false}={}) {
/*
Fetch all the objects in a list, these are identified by the url of the public key used for signing.
@ -163,7 +161,7 @@ class TransportYJS extends Transport {
}
}
listmonitor(url, callback, {verbose}) {
listmonitor(url, callback, {verbose=false, current=false}={}) {
/*
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
@ -174,6 +172,9 @@ class TransportYJS extends Transport {
*/
let y = this.yarrays[typeof url === "string" ? url : url.href];
console.assert(y,"Should always exist before calling listmonitor - async call p__yarray(url) to create");
if (current) {
y.share.array.toArray.map(callback);
}
y.share.array.observe((event) => {
if (event.type === 'insert') { // Currently ignoring deletions.
if (verbose) console.log('resources inserted', event.values);
@ -230,13 +231,12 @@ class TransportYJS extends Transport {
return [u,u];
}
// ======= KEY VALUE TABLES ========
// Support for Key-Value pairs as per
// https://docs.google.com/document/d/1yfmLRqKPxKwB939wIy9sSaa7GKOzM5PrCZ4W1jRGW6M/edit#
async p_newdatabase(pubkey, {verbose=false}={}) {
//if (pubkey instanceof Dweb.PublicPrivate)
if (pubkey.hasOwnProperty("keypair"))
pubkey = pubkey.keypair.signingexport()
pubkey = pubkey.keypair.signingexport();
// By this point pubkey should be an export of a public key of form xyz:abc where xyz
// specifies the type of public key (NACL VERIFY being the only kind we expect currently)
let u = `yjs:/yjs/${encodeURIComponent(pubkey)}`;
@ -252,7 +252,12 @@ class TransportYJS extends Transport {
return { privateurl: `${database.privateurl}/${table}`, publicurl: `${database.publicurl}/${table}`} // No action required to create it
}
async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table/key
async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table
/*
Set key values
keyvalues: string (key) in which case value should be set there OR
object in which case value is ignored
*/
let y = await this.p_connection(url, verbose);
if (typeof keyvalues === "string") {
y.share.map.set(keyvalues, JSON.stringify(value));
@ -300,9 +305,9 @@ class TransportYJS extends Transport {
_map: await this.p_getall(url, {verbose})
}; // Data struc is ok as SmartDict.p_fetch will pass to KVT constructor
}
async monitor(url, callback, verbose) {
async monitor(url, callback, {verbose=false, current=false}={}) {
/*
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_getall to get any more items not returned by p_getall.
Stack: KVT()|KVT.p_new => KVT.monitor => (a: Transports.monitor => YJS.monitor)(b: dispatchEvent)
:param url: string Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
@ -315,6 +320,14 @@ class TransportYJS extends Transport {
if (!y) {
throw new errors.CodingError("Should always exist before calling monitor - async call p__yarray(url) to create");
}
if (current) {
// Iterate over existing items with callback
y.share.map.keys()
.forEach(k => {
let val = y.share.map.get[k];
callback({type: "set", key: k, value: (typeof val === "string" ? JSON.parse(val) : val)});
})
}
y.share.map.observe((event) => {
if (['add','update'].includes(event.type)) { // Currently ignoring deletions.
if (verbose) console.log("YJS monitor:", url, event.type, event.name, event.value);
@ -332,6 +345,24 @@ class TransportYJS extends Transport {
})
}
static async p_test(opts={}, verbose=false) {
if (verbose) {console.log("TransportHTTP.test")}
try {
let transport = await this.p_setup(opts, verbose);
if (verbose) console.log("HTTP connected");
let res = await transport.p_info(verbose);
if (verbose) console.log("TransportHTTP info=",res);
res = await transport.p_status(verbose);
console.assert(res === Transport.STATUS_CONNECTED);
await transport.p_test_kvt("NACL%20VERIFY", verbose);
} catch(err) {
console.log("Exception thrown in TransportHTTP.test:", err.message);
throw err;
}
}
}
TransportYJS.Y = Y; // Allow node tests to find it
Transports._transportclasses["YJS"] = TransportYJS;

34
Transports.js
View File

@ -52,7 +52,10 @@ class Transports {
returns: Array of pairs of url & transport instance [ [ u1, t1], [u1, t2], [u2, t1]]
throws: CodingError if urls empty or [undefined...]
*/
console.assert((urls && urls[0]) || ["store", "newlisturls", "newdatabase", "newtable"].includes(func), "Coding Error: Transports.validFor called with invalid arguments: urls=", urls, "func=", func); // FOr debugging old calling patterns with [ undefined ]
if (!((urls && urls[0]) || ["store", "newlisturls", "newdatabase", "newtable"].includes(func))) {
console.warn("Transports.validFor called with invalid arguments: urls=", urls, "func=", func); // FOr debugging old calling patterns with [ undefined ]
return [];
}
if (!(urls && urls.length > 0)) {
return this._connected().filter((t) => (t.supports(undefined, func)))
.map((t) => [undefined, t]);
@ -68,6 +71,9 @@ class Transports {
// Need a async version of this for serviceworker and TransportsProxy
return this.validFor(urls, func, options).map((ut) => ut[0]);
}
// SEE-OTHER-ADDTRANSPORT
static http(verbose) {
// Find an http transport if it exists, so for example YJS can use it.
return Transports._connected().find((t) => t.name === "HTTP")
@ -82,6 +88,12 @@ class Transports {
return Transports._connected().find((t) => t.name === "WEBTORRENT")
}
static gun(verbose) {
// Find a GUN transport if it exists
return Transports._connected().find((t) => t.name === "GUN")
}
static async p_resolveNames(urls) {
/* If and only if TransportNAME was loaded (it might not be as it depends on higher level classes like Domain and SmartDict)
then resolve urls that might be names, returning a modified array.
@ -133,7 +145,7 @@ class Transports {
urls = await this.p_resolveNames(urls); // If naming is loaded then convert to a name
let tt = this.validFor(urls, "list"); // Valid connected transports that support "store"
if (!tt.length) {
throw new errors.TransportError('Transports.p_rawlist: Cant find transport for urls:'+urls.join(','));
throw new errors.TransportError('Transports.p_rawlist: Cant find transport to "list" urls:'+urls.join(','));
}
let errs = [];
let ttlines = await Promise.all(tt.map(async function([url, t]) {
@ -169,10 +181,12 @@ class Transports {
throws: CodingError if urls empty or [undefined ... ]
*/
let verbose = opts.verbose;
urls = await this.p_resolveNames(urls); // If naming is loaded then convert to a name
let tt = this.validFor(urls, "fetch"); //[ [Url,t],[Url,t]] throws CodingError on empty /undefined urls
if (!urls.length) throw new errors.TransportError("Transports.p_rawfetch given an empty list of urls");
let resolvedurls = await this.p_resolveNames(urls); // If naming is loaded then convert to a name
if (!resolvedurls.length) throw new errors.TransportError("Transports.p_rawfetch none of the urls resolved: " + urls);
let tt = this.validFor(resolvedurls, "fetch"); //[ [Url,t],[Url,t]] throws CodingError on empty /undefined urls
if (!tt.length) {
throw new errors.TransportError("Transports.p_fetch cant find any transport for urls: " + urls);
throw new errors.TransportError("Transports.p_rawfetch cant find any transport for urls: " + resolvedurls);
}
//With multiple transports, it should return when the first one returns something.
let errs = [];
@ -206,6 +220,8 @@ class Transports {
returns: undefined
throws: TransportError with message being concatenated messages of transports if NONE of them succeed.
*/
//TODO-REFACTOR remove dependecy on the object having a .preflight, this should be handled one layer up.
//TODO-REFACTOR requires changes in: dweb-transports: TransportXyz, Transport, API.md; dweb-objects: CommonList.js, test.js; dweb-serviceworker/TransportsProxy.js;
//TODO-MULTI-GATEWAY might be smarter about not waiting but Promise.race is inappropriate as returns after a failure as well.
urls = await this.p_resolveNames(urls); // If naming is loaded then convert to a name
let tt = this.validFor(urls, "add"); // Valid connected transports that support "store"
@ -442,14 +458,16 @@ class Transports {
.map(([u, t]) => t.p_connection(u, verbose)));
}
static monitor(urls, cb, verbose) {
static monitor(urls, cb, {verbose=false, current=false}={}) {
/*
Add a listmonitor for each transport - note this means if multiple transports support it, then will get duplicate events back if everyone else is notifying all of them.
Stack: KVT()|KVT.p_new => KVT.monitor => (a: Transports.monitor => YJS.monitor)(b: dispatchEvent)
cb: function({type, key, value})
current: If true then then send all current entries as well
*/
//Cant' its async. urls = await this.p_resolveNames(urls); // If naming is loaded then convert to a name
this.validFor(urls, "monitor")
.map(([u, t]) => t.monitor(u, cb, verbose));
.map(([u, t]) => t.monitor(u, cb, {verbose, current}));
}
// Setup and connection
@ -530,7 +548,7 @@ class Transports {
let tabbrevs = options.transports; // Array of transport abbreviations
this._optionspaused = (options.paused || []).map(n => n.toUpperCase()); // Array of transports paused - defaults to none, upper cased
if (!(tabbrevs && tabbrevs.length)) { tabbrevs = options.defaulttransports || [] }
if (! tabbrevs.length) { tabbrevs = ["HTTP", "YJS", "IPFS", "WEBTORRENT"]; }
if (! tabbrevs.length) { tabbrevs = ["HTTP", "YJS", "IPFS", "WEBTORRENT", "GUN"]; } // SEE-OTHER-ADDTRANSPORT
tabbrevs = tabbrevs.map(n => n.toUpperCase());
let transports = this.setup0(tabbrevs, options, verbose);
if (options.statuscb) {

15008
dist/dweb-transports-bundle.js vendored
View File

File diff suppressed because one or more lines are too long

25
httptools.js
View File

@ -22,6 +22,15 @@ if (typeof(fetch) === "undefined") {
httptools = {};
async function loopfetch(req, ms, count, what) {
/*
A workaround for a nasty Chrome issue which fails if there is a (cross-origin?) fetch of more than 6 files. See other WORKAROUND-CHROME-CROSSORIGINFETCH
Loops at longer and longer intervals trying
req: Request
ms: Initial wait between polls
count: Max number of times to try (0 means just once)
what: Name of what retrieving for log (usually file name or URL)
returns Response:
*/
let lasterr;
let loopguard = (typeof window != "undefined") && window.loopguard; // Optional global parameter, will cancel any loops if changes
while (count-- && (loopguard === ((typeof window != "undefined") && window.loopguard)) ) {
@ -43,16 +52,17 @@ async function loopfetch(req, ms, count, what) {
}
}
httptools.p_httpfetch = async function(httpurl, init, verbose) { // Embrace and extend "fetch" to check result etc.
httptools.p_httpfetch = async function(httpurl, init, {verbose=false, wantstream=false}={}) { // Embrace and extend "fetch" to check result etc.
/*
Fetch a url
url: optional (depends on command)
httpurl: optional (depends on command)
init: {headers}
resolves to: data as text or json depending on Content-Type header
throws: TransportError if fails to fetch
*/
try {
if (verbose) console.log("httpurl=%s init=%o", httpurl, init);
if (verbose) console.log("p_httpfetch:", httpurl, JSON.stringify(init));
//console.log('CTX=',init["headers"].get('Content-Type'))
// Using window.fetch, because it doesn't appear to be in scope otherwise in the browser.
let req = new Request(httpurl, init);
@ -62,7 +72,9 @@ httptools.p_httpfetch = async function(httpurl, init, verbose) { // Embrace and
// Note response.body gets a stream and response.blob gets a blob and response.arrayBuffer gets a buffer.
if (response.ok) {
let contenttype = response.headers.get('Content-Type');
if (contenttype === "application/json") {
if (wantstream) {
return response.body; // Note property while json() or text() are functions
} else if (contenttype === "application/json") {
return response.json(); // promise resolving to JSON
} else if (contenttype.startsWith("text")) { // Note in particular this is used for responses to store
return response.text();
@ -89,6 +101,7 @@ httptools.p_GET = async function(httpurl, opts={}) {
Throws TransportError if fails
opts {
start, end, // Range of bytes wanted - inclusive i.e. 0,1023 is 1024 bytes
wantstream, // Return a stream rather than data
verbose }
resolves to: URL that can be used to fetch the resource, of form contenthash:/contenthash/Q123
*/
@ -102,7 +115,7 @@ httptools.p_GET = async function(httpurl, opts={}) {
redirect: 'follow', // Chrome defaults to manual
keepalive: true // Keep alive - mostly we'll be going back to same places a lot
};
return await httptools.p_httpfetch(httpurl, init, opts.verbose); // This s a real http url
return await httptools.p_httpfetch(httpurl, init, {verbose: opts.verbose, wantstream: opts.wantstream}); // This s a real http url
}
httptools.p_POST = async function(httpurl, type, data, verbose) {
// Locate and return a block, based on its url
@ -121,7 +134,7 @@ httptools.p_POST = async function(httpurl, type, data, verbose) {
redirect: 'follow', // Chrome defaults to manual
keepalive: true // Keep alive - mostly we'll be going back to same places a lot
};
return await httptools.p_httpfetch(httpurl, init, verbose);
return await httptools.p_httpfetch(httpurl, init, {verbose});
}
exports = module.exports = httptools;

2
index.js
View File

@ -1,9 +1,11 @@
// Order is significant as should search earlier ones first
// put IPFS before Webtorrent for showcasing, as Webtorrent works in some cases IPFS doesnt so that way we exercise both
const DwebTransports = require("./Transports.js");
// SEE-OTHER-ADDTRANSPORT
require("./TransportHTTP.js"); // Can access via window.DwebTransports._transportclasses["HTTP"]
require("./TransportIPFS.js");
require("./TransportYJS.js");
require("./TransportWEBTORRENT.js");
require("./TransportGUN.js");
if (typeof window !== "undefined") { window.DwebTransports = DwebTransports; }
exports = module.exports = DwebTransports;

9249
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

14
package.json
View File

@ -12,14 +12,16 @@
},
"dependencies": {
"cids": "latest",
"gun": "latest",
"ipfs": "latest",
"ipfs-unixfs": "^0.1.15",
"node-fetch": "latest",
"node-fetch": "^2.2.0",
"readable-stream": "latest",
"webpack": "^4.16.2",
"webtorrent": "^0.99.3",
"y-array": "latest",
"y-indexeddb": "latest",
"y-ipfs-connector": "latest",
"y-ipfs-connector": "^2.3.0",
"y-map": "latest",
"y-memory": "latest",
"y-text": "latest",
@ -28,8 +30,10 @@
"description": "Internet Archive Decentralized Web Transports Library",
"devDependencies": {
"browserify": "^14.5.0",
"chai": "latest",
"uglifyjs-webpack-plugin": "latest",
"watchify": "^3.11.0",
"chai": "latest"
"webpack-cli": "^3.1.0"
},
"homepage": "https://github.com/internetarchive/dweb-transports#readme",
"keywords": [],
@ -41,9 +45,9 @@
"url": "git://github.com/internetarchive/dweb-transports.git"
},
"scripts": {
"build": "webpack --mode development",
"build": "webpack --mode production",
"test": "cd src; node ./test.js",
"help": "echo 'test (test it)'; echo 'build (creates dweb-transports-bundle)'"
},
"version": "0.0.1"
"version": "0.1.1"
}

13
test.js Normal file
View File

@ -0,0 +1,13 @@
const DwebTransports = require('./index.js');
async function p_test({verbose=true, transport=["GUN"]}={}) {
if (Array.isArray(transport)) {
for (tname of transport) {
await p_test({verbose, transport: tname}); // Note this is going to run in parallel
}
} else {
let tclass = DwebTransports._transportclasses[transport];
await tclass.p_test({verbose});
}
}
p_test();

14
webpack.config.js
View File

@ -1,3 +1,4 @@
const UglifyJsPlugin = require('uglifyjs-webpack-plugin');
module.exports = {
entry: {
'dweb-transports': './index.js',
@ -23,5 +24,14 @@ module.exports = {
alias: {
zlib: 'browserify-zlib-next'
}
}
}
},
plugins: [
new UglifyJsPlugin({
uglifyOptions: {
compress: {
unused: false
}
}
})
]
}