Update API and tidy ups flowing from that

This commit is contained in:
Mitra Ardron 2018-04-23 10:49:53 +10:00
parent 15f920dbdf
commit 7cf34c16f2
9 changed files with 243 additions and 57 deletions

1
.gitignore vendored
View File

@ -1,2 +1,3 @@
node_modules
dweb_transports_bundle.js
dweb-transports-bundle.js

191
API.md
View File

@ -3,7 +3,7 @@ Mitra Ardron, Internet Archive, mitra@mitra.biz
This doc provides a concise API specification for the Dweb Javascript Libraries.
It was last revised (to match the code) on 9 March 2018.
It was last revised (to match the code) on 23 April 2018.
If you find any discrepancies please add an issue here.
@ -27,7 +27,7 @@ The Transport layer provides a layer that is intended to be independent of the u
There are a set of classes:
* *Transport*: superclass of each supported transport,
* *TransportHTTP*: Connects to generic http servers, and handles contenthash: through a known gateway
* *TransportIPFS*: Connects to IPFS, currently via WebSocketsStar (WSS)
* *TransportIPFS*: Connects to IPFS, currently (April 2018) via WebSocketsStar (WSS)
* *TransportYJS*: Implements shared lists, and dictionaries. Uses IPFS for transport
* *TransportWEBTORRENT*: Integrates to Feross's WebTorrent library
* *Transports*: manages the list of conencted transports, and directs api calls to them.
@ -47,7 +47,7 @@ Fields| 
options | Holds options passed to constructor
name|Short name of transport e.g. “HTTP”, “IPFS”
supportURLs|Array of url prefixes supported e.g. [ipfs,http]
supportFunctions|Array of functions supported on those urls, current full list would be: `['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase", "listmonitor"]`
supportFunctions|Array of functions supported on those urls, current (April 2018) full list would be: `['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase", "listmonitor"]`
status|Numeric indication of transport status: Started(0); Failed(1); Starting(2); Loaded(3)
###Setup of a transport
@ -83,12 +83,22 @@ cb (t)=>void If set, will be called back as status changes (so could be multip
Resolves to the Transport instance
```
#####p_setup(options, verbose, cb)
#####async p_setup(options, verbose, cb)
A deprecated utility to simply setup0 then p_setup1 then p_setup2 to allow a transport to be started
in one step, normally `Transports.p_setup` should be called instead.
#####p_status (verbose)
Return a numeric code for the status of a transport.
####togglePaused(cb)
Switch the state of the transport between STATUS_CONNECTED and STATUS_PAUSED,
in the paused state it will not be used for transport but, in some cases, will still do background tasks like serving files.
```
cb(transport)=>void a callback called after this is run, may be used for example to change the UI
```
#####async p_status (verbose)
Check the status of the underlying transport. This may update the "status" field from the underlying transport.
```
returns: a numeric code for the status of a transport.
```
Code|Name|Means
---|---|---
@ -119,7 +129,7 @@ timeoutMS Max time to wait on transports that support it (IPFS for fetch)
start,end Inclusive byte range wanted (must be supported, uses a "slice" on output if transport ignores it.
relay If first transport fails, try and retrieve on 2nd, then store on 1st, and so on.
verbose boolean - True for debugging output
Resolves to string The object being fetched, (note currently returned as a string, may refactor to return Buffer)
Resolves to string The object being fetched, (note currently (April 2018) returned as a string, may refactor to return Buffer)
throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
```
@ -162,11 +172,12 @@ verbose boolean - True for debugging output
Resolves to Array objects as stored on the list (see p_rawlist for format)
```
#####listmonitor (url, cb)
#####listmonitor (url, cb, {verbose})
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
```
url Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
cb(obj) function(obj) Callback for each new item added to the list
verbose boolean - True for debugging output
obj is same format as p_rawlist or p_rawreverse
```
@ -236,7 +247,7 @@ returns: Dictionary of Key:Value pairs, note take care if this could
#####static async p_f_createReadStream(url, {wanturl, verbose})
Provide a function of the form needed by <VIDEO> tag and renderMedia library etc
```
url Urlsof stream
url Urls of stream
wanturl True if want the URL of the stream (for service workers)
returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file
```
@ -244,8 +255,9 @@ returns f(opts) => stream returning bytes from opts.start || start of file to op
#####supports(url, funcl)
Determines if the Transport supports urls of this form. For example TransportIPFS supports URLs starting ipfs:
```
url identifier of resource to fetch or list
Returns True if this Transport supports that type of URL
url identifier of resource to fetch or list
Returns True if this Transport supports that type of URL
throw TransportError if invalid URL
```
#####p_info()
@ -270,21 +282,30 @@ returns Array of transports that are connected (i.e. status=STATUS_CONNECTED)
```
resolves to: Array of names transports that are connected (i.e. status=STATUS_CONNECTED)
```
#####static async p_connectedNamesParm
#####static async p_connectedNamesParm()
```
resolves to: part of URL string for transports e.g. 'transport=HTTP&transport=IPFS"
```
#####static async p_statuses()
```
resolves to: a dictionary of statuses of transports e.g. { TransportHTTP: STATUS_CONNECTED }
```
#####static validFor(urls, func, options) {
Finds an array or Transports that are STARTED and can support this URL.
```
urls: Array of urls
func: Function to check support for: fetch, store, add, list, listmonitor, reverse
- see supportFunctions on each Transport class
- see supportFunctions on each Transport class
options For future use
Returns: Array of pairs of url & transport instance [ [ u1, t1], [u1, t2], [u2, t1]]
```
#####static async p_urlsValidFor(urls, func, options) {
Async version of validFor for serviceworker and TransportsProxy
#####static http(verbose)
```
returns instance of TransportHTTP if connected
@ -295,14 +316,21 @@ returns instance of TransportHTTP if connected
returns instance of TransportIPFS if connected
```
#####static webtorrent(verbose)
```
returns instance of TransportWEBTORRENT if connected
```
#####static async p_resolveNames(urls)
See Naming below
```
urls mix of urls and names
names if namingcb is set, will convert any names to URLS (this requires higher level libraries)
```
#####static async resolveNamesWith(cb)
Called by higher level libraries that provide name resolution function
Called by higher level libraries that provide name resolution function.
See Naming below
```
cb(urls) => urls Provide callback function
```
@ -327,11 +355,12 @@ Call p_setup1 on all transports that were created in setup0(). Completes when a
#####static async p_setup2(verbose, cb)
Call p_setup2 on all transports that were created in setup0(). Completes when all setups are complete.
#####static asunc refreshstatus(t)
#####static async refreshstatus(t)
Set the class of t.statuselement (if set) to transportstatus0..transportstatus4 depending on its status.
```
t Instance of transport
```
####static async p_connect({options}, verbose)
Main connection process for a browser based application,
```
@ -343,6 +372,7 @@ options {
}
```
#####static async p_urlsFrom(url)
Utility to convert to urls form wanted for Transports functions, e.g. from user input
```
@ -350,6 +380,13 @@ url: Array of urls, or string representing url or representing array of urls
return: Array of strings representing url
```
#####static async p_httpfetchurl(url)
Return URLS suitable for caller to pass to fetch.
```
url: Array of urls, or string representing url or representing array of urls
return: Array of strings representing url
```
###Multi transport calls
Each of the following is attempted across multiple transports
For parameters refer to underlying Transport call
@ -360,14 +397,14 @@ static async p_rawstore(data, {verbose})|[urls]|Tries all and combines results
static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay})|data|See note
static async p_rawlist(urls, {verbose})|[sigs]|Tries all and combines results
static async p_rawadd(urls, sig, {verbose})||Tries on all urls, error if none succeed
static listmonitor(urls, cb)||Tries on all urls (so note cb may be called multiple times)
static listmonitor(urls, cb, {verbose})||Tries on all urls (so note cb may be called multiple times)
static p_newlisturls(cl, {verbose})|[urls]|Tries all and combines results
static async p_f_createReadStream(urls, options)|f(opts)=>stream|Returns first success
static async p_get(urls, keys, {verbose})|currently returns on first success, TODO - will combine results and relay across transports
static async p_get(urls, keys, {verbose})|currently (April 2018) returns on first success, TODO - will combine results and relay across transports
static async p_set(urls, keyvalues, value, {verbose})|Tries all, error if none succeed
static async p_delete(urls, keys, {verbose})|Tries all, error if none succeed
static async p_keys(urls, {verbose}|[keys]|currently returns on first success, TODO - will combine results and relay across transports
static async p_getall(urls, {verbose})|dict|currently returns on first success, TODO - will combine results and relay across transports
static async p_keys(urls, {verbose}|[keys]|currently (April 2018) returns on first success, TODO - will combine results and relay across transports
static async p_getall(urls, {verbose})|dict|currently (April 2018) returns on first success, TODO - will combine results and relay across transports
static async p_newdatabase(pubkey, {verbose})|{privateurls: [urls], publicurls: [urls]}|Tries all and combines results
static async p_newtable(pubkey, table, {verbose})|{privateurls: [urls], publicurls: [urls]}|Tries all and combines results
static async p_connection(urls, verbose)||Tries all parallel
@ -381,6 +418,116 @@ start,end Inclusive byte range wanted - passed to
relay If first transport fails, try and retrieve on 2nd, then store on 1st, and so on.
```
###TODO documentation
* options to each Transport
* Other useful functions on each transport esp http p_GET etc
##TransportHTTP
A subclass of Transport for handling HTTP connections - both directly and for contenthash: urls.
It looks at `options { http }` for its options.
Option|Default|Meaning
------|-------|-------
urlbase|https://gateway.dweb.me:443|Connect to gateway.dweb.me for contenthash urls
supportURLS = `http:*, https:*, contenthash:*` (TODO: may in the future support `dweb:/contenthash/*`)
supportFunctions:
`fetch, store, add, list, reverse, newlisturls, get, set, keys, getall, delete, newtable, newdatabase`
supportFeatures:
`fetch.range` i.e. it will fetch a range of bytes if specified in {start, end} to p_rawfetch()
###Other useful functions
The HTTP Transport can be used for utility functions as well as via the Transports interface.
e.g. Transports.http().p_httpfetch("http://foo.com/bar", {method: 'GET'} )
#####p_httpfetch(url, init, verbose)
Fetch a url.
If the result
url: HTTP or HTTPS url
init: Init parameter to fetch (see for docs)
verbose: boolean for debugging
returns: Depends on mime type;
If application/json returns a Object,
If text/* returns text
Oherwise Buffer
#####p_GET(url, {start, end, verbose})
Shortcut to do a HTTP/POST get, sets `mode: cors, redirect: follow, keepalive: true, cache: default`
start: First byte to retrieve
end: Last byte to retrieve (undefined means end of file)
Note that it passes start and end as the Range header, most servers support it,
but it does not (yet) explicitly check the result.
#####p_POST(url, type, data, verbose)
Shortcut to do a HTTP/HTTPS POST. sets same options as p_GET
data: Data to send to fetch, typically the body,
type: Currently not passed as header{Content-type} because fetch appears to ignore it.
##TransportIPFS
A subclass of Transport for handling IPFS connections
The code works, however there are a number of key IPFS issues <TODO document here> that mean that not
every IPFS `CID` will be resolvable, since:
* IPFS javascript doesnt yet support CIDv1 so can't recognize the CIDs starting with "z"
* WSS only can find blocks known about by peers connected to the same star.
* urlstore on the archive.org gateway is not advertising the files in a way that WSS finds them.
* Protocol Labs (IPFS)) have been unable to get WSS to connect to our gateway machine
Other parts of this code work but just note that usage should be aware that problems may be to do
with your specific configuration and use cases.
It looks at `options { ipfs }` for its options.
Option|Default|Meaning
------|-------|-------
repo|/tmp/dweb_ipfsv2700|Local file system (only relevant if running under Node)
config Addresses |Swarm: ['/dns4/ws-star.discovery.libp2p.io/tcp/443/wss/p2p-websocket-star']}|Connects currently (April 2018) to ipfs.io, this may change.
EXPERIMENTAL pubsub|true|Requires pubsub for list monitoring etc
supportURLS = `ipfs:*` (TODO: may in the future support `dweb:/ipfs/*`)
SupportFunctions (note YJS uses IPFS and supports some other functions):
`fetch, store`
SupportFeatures:
fetch.range Not supported (currently April 2018))
Currently there is code for p_f_createReadStream. It works but because of some other IPFS issues is disabled.
##TransportYJS
A subclass of Transport for handling YJS connections.
Uses IPFS as its transport (other transports are possible with YJS and may be worth experimenting with)
supportURLS = `yjs:*` (TODO: may in the future support `dweb:/yjs/*`)
supportFunctions (note YJS uses IPFS and supports some other functions):
`fetch, add, list, listmonitor, newlisturls, connection, get, set, getall, keys, newdatabase, newtable, monitor`
supportFeatures:
fetch.range Not supported (currently April 2018)
##TransportWEBTORRENT
A subclass of Transport for handling WEBTORRENT connections (similar to, with interworking with BitTorrent)
Note that currently (April 2018) this won't work properly in the ServiceWorker (SW) because SW dont support WebRTC
When used with a SW, it will attempt to retrieve from the http backup URL that is in many magnet links.
In the SW it will also generate errors about trackers because the only reason to use trackers is to get the WebRTC links.
supportURLS = `magnet:*` (TODO: may in the future support `dweb:/magnet/*`)
supportFunctions:
`fetch, createReadStream`
supportFeatures:
fetch.range Not supported (currently April 2018)
##Naming
Independently from the transport, the Transport library can resolve names if provided an appropriate callback.
See p_resolveNames(urls) and resolveNamesWith(cb)
In practice this means that an application should do.
```
require('dweb-transports)
require('dweb-objects/Domain) # Sets the callbacks.
```
When setup this way, then calls to most functions that take an array of urls will first try and expand names.
The format of names currently (April 2018) is under development but its likely to be something like
`dweb:/arc/archive.org/details/foo`
to allow smooth integration with existing HTTP urls that are moving to decentralization.

View File

@ -19,31 +19,47 @@ class Transport {
*/
}
setup0(options, verbose) {
static setup0(options, verbose) {
/*
First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections.
*/
throw new errors.IntentionallyUnimplementedError("Intentionally undefined function Transport.setup0 should have been subclassed");
}
p_setup1(options, verbose) { return this; }
p_setup2(options, verbose) { return this; }
p_setup1(options, verbose) {
/*
Setup the resource and open any P2P connections etc required to be done just once. Asynchronous and should leave status=STATUS_STARTING until it resolves, or STATUS_FAILED if fails.
cb (t)=>void If set, will be called back as status changes (so could be multiple times)
Resolves to the Transport instance
*/
return this;
}
p_setup2(options, verbose) {
/*
Works like p_setup1 but runs after p_setup1 has completed for all transports. This allows for example YJS to wait for IPFS to be connected in TransportIPFS.setup1() and then connect itself using the IPFS object.
cb (t)=>void If set, will be called back as status changes (so could be multiple times)
Resolves to the Transport instance
*/
return this;
}
static async p_setup(options, verbose, cb) {
/*
Setup the resource and open any P2P connections etc required to be done just once.
In almost all cases this will call the constructor of the subclass
/*
A deprecated utility to simply setup0 then p_setup1 then p_setup2 to allow a transport to be started in one step, normally Transports.p_setup should be called instead.
*/
let t = await this.setup0(options, verbose) // Sync version that doesnt connect
.p_setup1(verbose, cb); // And connect
:param obj options: Data structure required by underlying transport layer (format determined by that layer)
:param boolean verbose: true for debugging output
:resolve Transport: Instance of subclass of Transport
*/
let t = await this.setup0(options, verbose) // Sync version that doesnt connect
.p_setup1(verbose, cb); // And connect
return t.p_setup2(verbose, cb); // And connect
return t.p_setup2(verbose, cb); // And connect
}
togglePaused(cb) { //TODO-API //TODO-SW move to Transports > TransportsProxy > UI
/*
Switch the state of the transport between STATUS_CONNECTED and STATUS_PAUSED,
in the paused state it will not be used for transport but, in some cases, will still do background tasks like serving files.
cb(transport)=>void a callback called after this is run, may be used for example to change the UI
*/
switch (this.status) {
case Transport.STATUS_CONNECTED:
this.status = Transport.STATUS_PAUSED;
@ -55,6 +71,14 @@ class Transport {
if (cb) cb(this);
}
async p_status(verbose) {
/*
Check the status of the underlying transport. This may update the "status" field from the underlying transport.
returns: a numeric code for the status of a transport.
*/
return this.status;
}
supports(url, func) {
/*
Determine if this transport supports a certain set of URLs and a func
@ -99,15 +123,19 @@ class Transport {
//noinspection JSUnusedLocalSymbols
p_rawfetch(url, {verbose=false}={}) {
p_rawfetch(url, {timeoutMS=undefined, start=undefined, end=undefined, relay=false, verbose=false}={}) {
/*
Fetch some bytes based on a url, no assumption is made about the data in terms of size or structure.
Where required by the underlying transport it should retrieve a number if its "blocks" and concatenate them.
Returns a new Promise that resolves currently to a string.
There may also be need for a streaming version of this call, at this point undefined.
:param string url: URL of object being retrieved
:param boolean verbose: true for debugging output
:param string url: URL of object being retrieved
:param verbose: true for debugging output
:param timeoutMS Max time to wait on transports that support it (IPFS for fetch)
:param start,end Inclusive byte range wanted (must be supported, uses a "slice" on output if transport ignores it.
:param relay If first transport fails, try and retrieve on 2nd, then store on 1st, and so on.
:resolve string: Return the object being fetched, (note currently returned as a string, may refactor to return Buffer)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/
@ -249,6 +277,15 @@ class Transport {
*/
throw new errors.ToBeImplementedError("Undefined function Transport.p_keys");
}
static async p_f_createReadStream(url, {wanturl=false, verbose=false}) {
/*
Provide a function of the form needed by tag and renderMedia library etc
url Urls of stream
wanturl True if want the URL of the stream (for service workers)
returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file
*/
}
// ------ UTILITY FUNCTIONS, NOT REQD TO BE SUBCLASSED ----
static mergeoptions(a) {

View File

@ -82,12 +82,12 @@ class TransportHTTP extends Transport {
console.log(this.name, ": Error in p_status.info",err.message);
this.status = Transport.STATUS_FAILED;
}
return this.status;
return super.p_status(verbose);
}
async p_httpfetch(httpurl, init, verbose) { // Embrace and extend "fetch" to check result etc.
/*
Fetch a url based from default server at command/multihash
Fetch a url
url: optional (depends on command)
resolves to: data as text or json depending on Content-Type header
@ -105,7 +105,7 @@ class TransportHTTP extends Transport {
if (contenttype === "application/json") {
return response.json(); // promise resolving to JSON
} else if (contenttype.startsWith("text")) { // Note in particular this is used for responses to store
return response.text(); // promise resolving to arrayBuffer (was returning text, but distorts binaries (e.g. jpegs)
return response.text();
} else { // Typically application/octetStream when don't know what fetching
return new Buffer(await response.arrayBuffer()); // Convert arrayBuffer to Buffer which is much more usable currently
}

View File

@ -130,7 +130,7 @@ class TransportIPFS extends Transport {
Return a numeric code for the status of a transport.
*/
this.status = (await this.ipfs.isOnline()) ? Transport.STATUS_CONNECTED : Transport.STATUS_FAILED;
return this.status;
return super.p_status(verbose);
}
// Everything else - unless documented here - should be opaque to the actual structure of a CID

View File

@ -91,7 +91,7 @@ class TransportWEBTORRENT extends Transport {
} else {
this.status = Transport.STATUS_FAILED;
}
return this.status;
return super.p_status(verbose);
}
webtorrentparseurl(url) {

View File

@ -136,7 +136,7 @@ class TransportYJS extends Transport {
For YJS, its online if IPFS is.
*/
this.status = (await this.options.yarray.connector.ipfs.isOnline()) ? Transport.STATUS_CONNECTED : Transport.STATUS_FAILED;
return this.status;
return super.p_status(verbose);
}
async p_rawlist(url, {verbose=false}={}) {
@ -163,7 +163,7 @@ class TransportYJS extends Transport {
}
}
listmonitor(url, callback, verbose) {
listmonitor(url, callback, {verbose}) {
/*
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.

View File

@ -20,23 +20,19 @@ class Transports {
}
static async p_connectedNames() {
/*
Return an array of the names of connected transports
resolves to: an array of the names of connected transports
*/
return this._connected().map(t => t.name);
}
static async p_connectedNamesParm() { // Doesnt strictly need to be async, but for consistency with Proxy it has to be.
return (await this.p_connectedNames()).map(n => "transport="+n).join('&')
}
static async p_statuses(t) { //TODO-API
static async p_statuses() { //TODO-API
/*
Return a dictionary of statuses of transports
resolves to: a dictionary of statuses of transports, e.g. { TransportHTTP: STATUS_CONNECTED }
*/
return this._transports.map((t) => { return {"name": t.name, "status": t.status}})
}
static async p_urlsValidFor(urls, func, options) {
// Need a async version of this for serviceworker and TransportsProxy
return this.validFor(urls, func, options).map((ut) => ut[0]);
}
static validFor(urls, func, options) {
/*
Finds an array or Transports that can support this URL.
@ -45,6 +41,7 @@ class Transports {
urls: Array of urls
func: Function to check support for: fetch, store, add, list, listmonitor, reverse - see supportFunctions on each Transport class
options: For future use
returns: Array of pairs of url & transport instance [ [ u1, t1], [u1, t2], [u2, t1]]
*/
console.assert((urls && urls[0]) || ["store", "newlisturls", "newdatabase", "newtable"].includes(func), "Transports.validFor failed - coding error - urls=", urls, "func=", func); // FOr debugging old calling patterns with [ undefined ]
@ -59,6 +56,10 @@ class Transports {
.map((t) => [url, t]))); // [[ u, t1], [u, t2]]
}
}
static async p_urlsValidFor(urls, func, options) {
// Need a async version of this for serviceworker and TransportsProxy
return this.validFor(urls, func, options).map((ut) => ut[0]);
}
static http(verbose) {
// Find an http transport if it exists, so for example YJS can use it.
return Transports._connected().find((t) => t.name === "HTTP")
@ -221,13 +222,13 @@ class Transports {
}
static listmonitor(urls, cb) {
static listmonitor(urls, cb, opts={}) {
/*
Add a listmonitor for each transport - note this means if multiple transports support it, then will get duplicate events back if everyone else is notifying all of them.
*/
// Note cant do p_resolveNames since sync but should know real urls of resource by here.
this.validFor(urls, "listmonitor")
.map(([u, t]) => t.listmonitor(u, cb));
.map(([u, t]) => t.listmonitor(u, cb, opts));
}
static async p_newlisturls(cl, {verbose=false}={}) {
@ -559,7 +560,7 @@ class Transports {
return url;
}
static async p_httpfetchurls(urls) {
static async p_httpfetchurl(urls) {
/*
Utility to take a array of Transport urls, convert back to a single url that can be used for a fetch, typically
this is done when cant handle a stream, so want to give the url to the <VIDEO> tag.

View File

@ -5,7 +5,7 @@
<!--Note there are copies of this in dweb-transports repo and dweb-transport repo-->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>DWEB Transports example page</title>
<script type="text/javascript" src="./dist/dweb_transports_bundle.js"></script>
<script type="text/javascript" src="./dist/dweb-transports-bundle.js"></script>
<script type="text/javascript" src='https://cloud.tinymce.com/stable/tinymce.min.js'></script><!-- TinyMCE -->
<script type="text/javascript">