fluencelabs/dweb-transports
1
0
Fork 0
mirror of https://github.com/fluencelabs/dweb-transports synced 2025-03-15 18:30:49 +00:00
Code Issues Projects Releases Wiki Activity

Documentation, comments, and removing extraneous logging

Browse Source
This commit is contained in:
Mitra Ardron 2018-05-30 20:11:33 -07:00
parent d56508d889
commit 1d35e12ee7
10 changed files with 93 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

159
API.md
View File

@ -1,4 +1,4 @@
#dweb-transports API # dweb-transports API
Mitra Ardron, Internet Archive, mitra@mitra.biz Mitra Ardron, Internet Archive, mitra@mitra.biz
This doc provides a concise API specification for the Dweb Javascript Transports Libraries. This doc provides a concise API specification for the Dweb Javascript Transports Libraries.
@ -7,7 +7,7 @@ It was last revised (to match the code) on 23 April 2018.
If you find any discrepancies please add an issue here. If you find any discrepancies please add an issue here.
##General API notes and conventions ## General API notes and conventions
We use a naming convention that anything starting “p_” returns a promise so you know to "await" it if you want a result. We use a naming convention that anything starting “p_” returns a promise so you know to "await" it if you want a result.
Ideally functions should take a String, Buffer or where applicable Object as parameters with automatic conversion. Ideally functions should take a String, Buffer or where applicable Object as parameters with automatic conversion.
@ -20,7 +20,7 @@ which is intended to be a good way to see what is happening.
Note: I am gradually (March2018) changing the API to take an opts {} dict which includes verbose as one field. This process is incomplete, but I’m happy to see it accelerated if there is any code built on this, just let mitra@archive.org know. Note: I am gradually (March2018) changing the API to take an opts {} dict which includes verbose as one field. This process is incomplete, but I’m happy to see it accelerated if there is any code built on this, just let mitra@archive.org know.
##Overview ## Overview
The Transport layer provides a layer that is intended to be independent of the underlying storage/transport mechanism. The Transport layer provides a layer that is intended to be independent of the underlying storage/transport mechanism.
@ -40,7 +40,7 @@ The actual urls used might change as the Decentralized Web universe reaches cons
This spec will in the future probably add a variation that sends events as the block is retrieved to allow for streaming. This spec will in the future probably add a variation that sends events as the block is retrieved to allow for streaming.
##Transport Class ## Transport Class
Fields|  Fields| 
:---|--- :---|---
@ -50,11 +50,11 @@ supportURLs|Array of url prefixes supported e.g. [‘ipfs’,’http’]
supportFunctions|Array of functions supported on those urls, current (April 2018) full list would be: `['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase", "listmonitor"]` supportFunctions|Array of functions supported on those urls, current (April 2018) full list would be: `['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase", "listmonitor"]`
status|Numeric indication of transport status: Started(0); Failed(1); Starting(2); Loaded(3) status|Numeric indication of transport status: Started(0); Failed(1); Starting(2); Loaded(3)
###Setup of a transport ### Setup of a transport
Transport setup is split into 3 parts, this allows the Transports class to do the first phase on all the transports synchronously, Transport setup is split into 3 parts, this allows the Transports class to do the first phase on all the transports synchronously,
then asynchronously (or at a later point) try and connect to all of them in parallel. then asynchronously (or at a later point) try and connect to all of them in parallel.
#####static setup0 (options, verbose) ##### static setup0 (options, verbose)
First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections. In almost all cases this will call the constructor of the subclass First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections. In almost all cases this will call the constructor of the subclass
Should be synchronous and leave `status=STATUS_LOADED` Should be synchronous and leave `status=STATUS_LOADED`
``` ```
@ -66,7 +66,7 @@ Default options should be set in each transport, but can be overwritten,
for example to overwrite the options for HTTP call it with for example to overwrite the options for HTTP call it with
`options={ http: { urlbase: “https://gateway.dweb.me:443/” } }` `options={ http: { urlbase: “https://gateway.dweb.me:443/” } }`
#####async p_setup1 (verbose, cb) ##### async p_setup1 (verbose, cb)
Setup the resource and open any P2P connections etc required to be done just once. Setup the resource and open any P2P connections etc required to be done just once.
Asynchronous and should leave `status=STATUS_STARTING` until it resolves, or `STATUS_FAILED` if fails. Asynchronous and should leave `status=STATUS_STARTING` until it resolves, or `STATUS_FAILED` if fails.
``` ```
@ -74,7 +74,7 @@ cb (t)=>void If set, will be called back as status changes (so could be multip
Resolves to the Transport instance Resolves to the Transport instance
``` ```
#####async p_setup2 (verbose, cb) ##### async p_setup2 (verbose, cb)
Works like p_setup1 but runs after p_setup1 has completed for all transports. Works like p_setup1 but runs after p_setup1 has completed for all transports.
This allows for example YJS to wait for IPFS to be connected in TransportIPFS.setup1() This allows for example YJS to wait for IPFS to be connected in TransportIPFS.setup1()
and then connect itself using the IPFS object. and then connect itself using the IPFS object.
@ -83,18 +83,18 @@ cb (t)=>void If set, will be called back as status changes (so could be multip
Resolves to the Transport instance Resolves to the Transport instance
``` ```
#####async p_setup(options, verbose, cb) ##### async p_setup(options, verbose, cb)
A deprecated utility to simply setup0 then p_setup1 then p_setup2 to allow a transport to be started A deprecated utility to simply setup0 then p_setup1 then p_setup2 to allow a transport to be started
in one step, normally `Transports.p_setup` should be called instead. in one step, normally `Transports.p_setup` should be called instead.
####togglePaused(cb) #### togglePaused(cb)
Switch the state of the transport between STATUS_CONNECTED and STATUS_PAUSED, Switch the state of the transport between STATUS_CONNECTED and STATUS_PAUSED,
in the paused state it will not be used for transport but, in some cases, will still do background tasks like serving files. in the paused state it will not be used for transport but, in some cases, will still do background tasks like serving files.
``` ```
cb(transport)=>void a callback called after this is run, may be used for example to change the UI cb(transport)=>void a callback called after this is run, may be used for example to change the UI
``` ```
#####async p_status (verbose) ##### async p_status (verbose)
Check the status of the underlying transport. This may update the "status" field from the underlying transport. Check the status of the underlying transport. This may update the "status" field from the underlying transport.
``` ```
returns: a numeric code for the status of a transport. returns: a numeric code for the status of a transport.
@ -108,8 +108,8 @@ Code|Name|Means
3|STATUS_LOADED|Code loaded but havent tried to connect 3|STATUS_LOADED|Code loaded but havent tried to connect
4|STATUS_PAUSED|It was launched, probably connected, but now paused so will be ignored by validfor() 4|STATUS_PAUSED|It was launched, probably connected, but now paused so will be ignored by validfor()
###Transport: General storage and retrieval of objects ### Transport: General storage and retrieval of objects
#####p_rawstore(data, {verbose}) ##### p_rawstore(data, {verbose})
Store a opaque blob of data onto the decentralised transport. Store a opaque blob of data onto the decentralised transport.
``` ```
data string|Buffer data to store - no assumptions made to size or content data string|Buffer data to store - no assumptions made to size or content
@ -117,7 +117,7 @@ verbose boolean - True for debugging output
Resolves to url of data stored Resolves to url of data stored
``` ```
#####p_rawfetch(url, {timeoutMS, start, end, relay, verbose}) ##### p_rawfetch(url, {timeoutMS, start, end, relay, verbose})
Fetch some bytes based on a url, no assumption is made about the data in terms of size or structure. Fetch some bytes based on a url, no assumption is made about the data in terms of size or structure.
Where required by the underlying transport it should retrieve a number if its "blocks" and concatenate them. Where required by the underlying transport it should retrieve a number if its "blocks" and concatenate them.
@ -133,8 +133,8 @@ Resolves to string The object being fetched, (note currently (April 2018) retur
throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
``` ```
###Transport: Handling lists ### Transport: Handling lists
#####p_rawadd(url, sig, {verbose}) ##### p_rawadd(url, sig, {verbose})
Store a new list item, it should be stored so that it can be retrieved either by "signedby" (using p_rawlist) Store a new list item, it should be stored so that it can be retrieved either by "signedby" (using p_rawlist)
or by "url" (with p_rawreverse). or by "url" (with p_rawreverse).
@ -150,7 +150,7 @@ sig Signature data structure (see below - contains url, date, signedby, s
verbose boolean - True for debugging output verbose boolean - True for debugging output
``` ```
#####p_rawlist(url, {verbose}) ##### p_rawlist(url, {verbose})
Fetch all the objects in a list, these are identified by the url of the public key used for signing. Fetch all the objects in a list, these are identified by the url of the public key used for signing.
Note this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter. Note this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter.
List items may have other data (e.g. reference ids of underlying transport) List items may have other data (e.g. reference ids of underlying transport)
@ -162,7 +162,7 @@ Resolves to Array: An array of objects as stored on the list. Each of which is
``` ```
Each item of the list is a dict: {"url": url, "date": date, "signature": signature, "signedby": signedby} Each item of the list is a dict: {"url": url, "date": date, "signature": signature, "signedby": signedby}
#####p_rawreverse (url, {verbose}) ##### p_rawreverse (url, {verbose})
Similar to p_rawlist, but return the list item of all the places where the object url has been listed. Similar to p_rawlist, but return the list item of all the places where the object url has been listed.
(not supported by most transports) (not supported by most transports)
``` ```
@ -172,7 +172,7 @@ verbose boolean - True for debugging output
Resolves to Array objects as stored on the list (see p_rawlist for format) Resolves to Array objects as stored on the list (see p_rawlist for format)
``` ```
#####listmonitor (url, cb, {verbose}) ##### listmonitor (url, cb, {verbose})
Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist. Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
``` ```
url Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd url Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
@ -181,7 +181,7 @@ verbose boolean - True for debugging output
obj is same format as p_rawlist or p_rawreverse obj is same format as p_rawlist or p_rawreverse
``` ```
#####async p_newlisturls(cl, {verbose}) ##### async p_newlisturls(cl, {verbose})
Obtain a pair of URLs for a new list. The transport can use information in the cl to generate this or create something random (the former is encouraged since it means repeat tests might not generate new lists). Possession of the publicurl should be sufficient to read the list, the privateurl should be required to read (for some transports they will be identical, and higher layers should check for example that a signature is signed. Obtain a pair of URLs for a new list. The transport can use information in the cl to generate this or create something random (the former is encouraged since it means repeat tests might not generate new lists). Possession of the publicurl should be sufficient to read the list, the privateurl should be required to read (for some transports they will be identical, and higher layers should check for example that a signature is signed.
``` ```
cl CommonList instance that can be used as a seed for the URL cl CommonList instance that can be used as a seed for the URL
@ -189,8 +189,8 @@ verbose boolean - True for debugging output
Returns [privateurl, publicurl] Returns [privateurl, publicurl]
``` ```
###Transport: Support for KeyValueTable ### Transport: Support for KeyValueTable
#####async p_newdatabase(pubkey, {verbose}) { ##### async p_newdatabase(pubkey, {verbose}) {
Create a new database based on some existing object Create a new database based on some existing object
``` ```
pubkey: Something that is, or has a pubkey, by default support Dweb.PublicPrivate, KeyPair pubkey: Something that is, or has a pubkey, by default support Dweb.PublicPrivate, KeyPair
@ -198,7 +198,7 @@ pubkey: Something that is, or has a pubkey, by default support Dweb.PublicPr
returns: {publicurl, privateurl} which may be the same if there is no write authentication returns: {publicurl, privateurl} which may be the same if there is no write authentication
``` ```
#####async p_newtable(pubkey, table, {verbose}) { ##### async p_newtable(pubkey, table, {verbose}) {
Create a new table, Create a new table,
``` ```
pubkey: Is or has a pubkey (see p_newdatabase) pubkey: Is or has a pubkey (see p_newdatabase)
@ -206,7 +206,7 @@ table: String representing the table - unique to the database
returns: {privateurl, publicurl} which may be the same if there is no write authentication returns: {privateurl, publicurl} which may be the same if there is no write authentication
``` ```
#####async p_set(url, keyvalues, value, {verbose}) ##### async p_set(url, keyvalues, value, {verbose})
Set one or more keys in a table. Set one or more keys in a table.
``` ```
url: URL of the table url: URL of the table
@ -214,7 +214,7 @@ keyvalues: String representing a single key OR dictionary of keys
value: String or other object to be stored (its not defined yet what objects should be supported, e.g. any object ? value: String or other object to be stored (its not defined yet what objects should be supported, e.g. any object ?
``` ```
#####async p_get(url, keys, {verbose}) ##### async p_get(url, keys, {verbose})
Get one or more keys from a table Get one or more keys from a table
``` ```
url: URL of the table url: URL of the table
@ -222,29 +222,29 @@ keys: Array of keys
returns: Dictionary of values found (undefined if not found) returns: Dictionary of values found (undefined if not found)
``` ```
#####async p_delete(url, keys, {verbose}) ##### async p_delete(url, keys, {verbose})
Delete one or more keys from a table Delete one or more keys from a table
``` ```
url: URL of the table url: URL of the table
keys: Array of keys keys: Array of keys
``` ```
#####async p_keys(url, {verbose}) ##### async p_keys(url, {verbose})
Return a list of keys in a table (suitable for iterating through) Return a list of keys in a table (suitable for iterating through)
``` ```
url: URL of the table url: URL of the table
returns: Array of strings returns: Array of strings
``` ```
#####async p_getall(url, {verbose}) ##### async p_getall(url, {verbose})
Return a dictionary representing the table Return a dictionary representing the table
``` ```
url: URL of the table url: URL of the table
returns: Dictionary of Key:Value pairs, note take care if this could be large. returns: Dictionary of Key:Value pairs, note take care if this could be large.
``` ```
###Transports - other functions ### Transports - other functions
#####static async p_f_createReadStream(url, {wanturl, verbose}) ##### static async p_f_createReadStream(url, {wanturl, verbose})
Provide a function of the form needed by <VIDEO> tag and renderMedia library etc Provide a function of the form needed by <VIDEO> tag and renderMedia library etc
``` ```
url Urls of stream url Urls of stream
@ -252,7 +252,7 @@ wanturl True if want the URL of the stream (for service workers)
returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file
``` ```
#####supports(url, funcl) ##### supports(url, funcl)
Determines if the Transport supports url’s of this form. For example TransportIPFS supports URLs starting ipfs: Determines if the Transport supports url’s of this form. For example TransportIPFS supports URLs starting ipfs:
``` ```
url identifier of resource to fetch or list url identifier of resource to fetch or list
@ -260,13 +260,13 @@ Returns True if this Transport supports that type of URL
throw TransportError if invalid URL throw TransportError if invalid URL
``` ```
#####p_info() ##### p_info()
Return a JSON with info about the server. Return a JSON with info about the server.
##Transports class ## Transports class
The Transports Class manages multiple transports The Transports Class manages multiple transports
#####Properties ##### Properties
``` ```
_transports List of transports loaded (internal) _transports List of transports loaded (internal)
namingcb If set will be called cb(urls) => urls to convert to urls from names. namingcb If set will be called cb(urls) => urls to convert to urls from names.
@ -275,26 +275,26 @@ _optionspaused Saves paused option for setup
``` ```
#####static _connected() ##### static _connected()
``` ```
returns Array of transports that are connected (i.e. status=STATUS_CONNECTED) returns Array of transports that are connected (i.e. status=STATUS_CONNECTED)
``` ```
#####static async p_connectedNames() ##### static async p_connectedNames()
``` ```
resolves to: Array of names transports that are connected (i.e. status=STATUS_CONNECTED) resolves to: Array of names transports that are connected (i.e. status=STATUS_CONNECTED)
``` ```
#####static async p_connectedNamesParm() ##### static async p_connectedNamesParm()
``` ```
resolves to: part of URL string for transports e.g. 'transport=HTTP&transport=IPFS" resolves to: part of URL string for transports e.g. 'transport=HTTP&transport=IPFS"
``` ```
#####static async p_statuses() ##### static async p_statuses()
``` ```
resolves to: a dictionary of statuses of transports e.g. { TransportHTTP: STATUS_CONNECTED } resolves to: a dictionary of statuses of transports e.g. { TransportHTTP: STATUS_CONNECTED }
``` ```
#####static validFor(urls, func, options) { ##### static validFor(urls, func, options) {
Finds an array or Transports that are STARTED and can support this URL. Finds an array or Transports that are STARTED and can support this URL.
``` ```
urls: Array of urls urls: Array of urls
@ -304,45 +304,45 @@ options For future use
Returns: Array of pairs of url & transport instance [ [ u1, t1], [u1, t2], [u2, t1]] Returns: Array of pairs of url & transport instance [ [ u1, t1], [u1, t2], [u2, t1]]
``` ```
#####static async p_urlsValidFor(urls, func, options) { ##### static async p_urlsValidFor(urls, func, options) {
Async version of validFor for serviceworker and TransportsProxy Async version of validFor for serviceworker and TransportsProxy
#####static http(verbose) ##### static http(verbose)
``` ```
returns instance of TransportHTTP if connected returns instance of TransportHTTP if connected
``` ```
#####static ipfs(verbose) ##### static ipfs(verbose)
``` ```
returns instance of TransportIPFS if connected returns instance of TransportIPFS if connected
``` ```
#####static webtorrent(verbose) ##### static webtorrent(verbose)
``` ```
returns instance of TransportWEBTORRENT if connected returns instance of TransportWEBTORRENT if connected
``` ```
#####static async p_resolveNames(urls) ##### static async p_resolveNames(urls)
See Naming below See Naming below
``` ```
urls mix of urls and names urls mix of urls and names
names if namingcb is set, will convert any names to URLS (this requires higher level libraries) names if namingcb is set, will convert any names to URLS (this requires higher level libraries)
``` ```
#####static async resolveNamesWith(cb) ##### static async resolveNamesWith(cb)
Called by higher level libraries that provide name resolution function. Called by higher level libraries that provide name resolution function.
See Naming below See Naming below
``` ```
cb(urls) => urls Provide callback function cb(urls) => urls Provide callback function
``` ```
#####static addtransport(t) ##### static addtransport(t)
``` ```
t: Add a Transport instance to _transports t: Add a Transport instance to _transports
``` ```
#####static setup0(transports, options, verbose, cb) ##### static setup0(transports, options, verbose, cb)
Calls setup0 for each transport based on its short name. Specially handles ‘LOCAL’ as a transport pointing at a local http server (for testing). Calls setup0 for each transport based on its short name. Specially handles ‘LOCAL’ as a transport pointing at a local http server (for testing).
``` ```
transports Array of short names of transports e.g. [‘IPFS’,’HTTP’,’ORBITDB’] transports Array of short names of transports e.g. [‘IPFS’,’HTTP’,’ORBITDB’]
@ -351,19 +351,19 @@ cb Callback to be called each time status changes
Returns: Array of transport instances Returns: Array of transport instances
``` ```
#####static async p_setup1(verbose, cb) ##### static async p_setup1(verbose, cb)
Call p_setup1 on all transports that were created in setup0(). Completes when all setups are complete. Call p_setup1 on all transports that were created in setup0(). Completes when all setups are complete.
#####static async p_setup2(verbose, cb) ##### static async p_setup2(verbose, cb)
Call p_setup2 on all transports that were created in setup0(). Completes when all setups are complete. Call p_setup2 on all transports that were created in setup0(). Completes when all setups are complete.
#####static async refreshstatus(t) ##### static async refreshstatus(t)
Set the class of t.statuselement (if set) to transportstatus0..transportstatus4 depending on its status. Set the class of t.statuselement (if set) to transportstatus0..transportstatus4 depending on its status.
``` ```
t Instance of transport t Instance of transport
``` ```
####static async p_connect({options}, verbose) #### static async p_connect({options}, verbose)
Main connection process for a browser based application, Main connection process for a browser based application,
``` ```
options { options {
@ -376,21 +376,21 @@ options {
``` ```
#####static async p_urlsFrom(url) ##### static async p_urlsFrom(url)
Utility to convert to urls form wanted for Transports functions, e.g. from user input Utility to convert to urls form wanted for Transports functions, e.g. from user input
``` ```
url: Array of urls, or string representing url or representing array of urls url: Array of urls, or string representing url or representing array of urls
return: Array of strings representing url return: Array of strings representing url
``` ```
#####static async p_httpfetchurl(url) ##### static async p_httpfetchurl(url)
Return URLS suitable for caller to pass to fetch. Return URLS suitable for caller to pass to fetch.
``` ```
url: Array of urls, or string representing url or representing array of urls url: Array of urls, or string representing url or representing array of urls
return: Array of strings representing url return: Array of strings representing url
``` ```
###Multi transport calls ### Multi transport calls
Each of the following is attempted across multiple transports Each of the following is attempted across multiple transports
For parameters refer to underlying Transport call For parameters refer to underlying Transport call
@ -413,7 +413,7 @@ static async p_newtable(pubkey, table, {verbose})|{privateurls: [urls], publicur
static async p_connection(urls, verbose)||Tries all parallel static async p_connection(urls, verbose)||Tries all parallel
static monitor(urls, cb, verbose)||Tries all sequentially static monitor(urls, cb, verbose)||Tries all sequentially
#####static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay}) ##### static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay})
Tries to fetch on all valid transports until successful. See Transport.p_rawfetch Tries to fetch on all valid transports until successful. See Transport.p_rawfetch
``` ```
timeoutMS: Max time to wait on transports that support it (IPFS for fetch) timeoutMS: Max time to wait on transports that support it (IPFS for fetch)
@ -421,26 +421,11 @@ start,end Inclusive byte range wanted - passed to
relay If first transport fails, try and retrieve on 2nd, then store on 1st, and so on. relay If first transport fails, try and retrieve on 2nd, then store on 1st, and so on.
``` ```
##TransportHTTP ## httptools
A subclass of Transport for handling HTTP connections - both directly and for contenthash: urls. A utility class to support HTTP with or without TransportHTTP
e.g. `httptools.http().p_httpfetch("http://foo.com/bar", {method: 'GET'} )`
It looks at `options { http }` for its options. ##### p_httpfetch(url, init, verbose)
Option|Default|Meaning
------|-------|-------
urlbase|https://gateway.dweb.me:443|Connect to gateway.dweb.me for contenthash urls
supportURLS = `http:*, https:*, contenthash:*` (TODO: may in the future support `dweb:/contenthash/*`)
supportFunctions:
`fetch, store, add, list, reverse, newlisturls, get, set, keys, getall, delete, newtable, newdatabase`
supportFeatures:
`fetch.range` i.e. it will fetch a range of bytes if specified in {start, end} to p_rawfetch()
###Other useful functions
The HTTP Transport can be used for utility functions as well as via the Transports interface.
e.g. Transports.http().p_httpfetch("http://foo.com/bar", {method: 'GET'} )
#####p_httpfetch(url, init, verbose)
Fetch a url. Fetch a url.
If the result If the result
@ -452,7 +437,7 @@ returns: Depends on mime type;
If text/* returns text If text/* returns text
Oherwise Buffer Oherwise Buffer
#####p_GET(url, {start, end, verbose}) ##### p_GET(url, {start, end, verbose})
Shortcut to do a HTTP/POST get, sets `mode: cors, redirect: follow, keepalive: true, cache: default` Shortcut to do a HTTP/POST get, sets `mode: cors, redirect: follow, keepalive: true, cache: default`
start: First byte to retrieve start: First byte to retrieve
@ -461,13 +446,27 @@ end: Last byte to retrieve (undefined means end of file)
Note that it passes start and end as the Range header, most servers support it, Note that it passes start and end as the Range header, most servers support it,
but it does not (yet) explicitly check the result. but it does not (yet) explicitly check the result.
#####p_POST(url, type, data, verbose) ##### p_POST(url, type, data, verbose)
Shortcut to do a HTTP/HTTPS POST. sets same options as p_GET Shortcut to do a HTTP/HTTPS POST. sets same options as p_GET
data: Data to send to fetch, typically the body, data: Data to send to fetch, typically the body,
type: Currently not passed as header{Content-type} because fetch appears to ignore it. type: Currently not passed as header{Content-type} because fetch appears to ignore it.
##TransportIPFS
## TransportHTTP
A subclass of Transport for handling HTTP connections - both directly and for contenthash: urls.
It looks at `options { http }` for its options.
Option|Default|Meaning
------|-------|-------
urlbase|https://gateway.dweb.me:443|Connect to gateway.dweb.me for contenthash urls
supportURLS|http:*, https:*, contenthash:*}| (TODO: may in the future support `dweb:/contenthash/*`)
supportFunctions|fetch, store, add, list, reverse, newlisturls, get, set, keys, getall, delete, newtable, newdatabase|
supportFeatures|fetch.range|it will fetch a range of bytes if specified in {start, end} to p_rawfetch()
## TransportIPFS
A subclass of Transport for handling IPFS connections A subclass of Transport for handling IPFS connections
The code works, however there are a number of key IPFS issues <TODO document here> that mean that not The code works, however there are a number of key IPFS issues <TODO document here> that mean that not
@ -496,7 +495,7 @@ SupportFeatures:
Currently there is code for p_f_createReadStream. It works but because of some other IPFS issues is disabled. Currently there is code for p_f_createReadStream. It works but because of some other IPFS issues is disabled.
##TransportYJS ## TransportYJS
A subclass of Transport for handling YJS connections. A subclass of Transport for handling YJS connections.
Uses IPFS as its transport (other transports are possible with YJS and may be worth experimenting with) Uses IPFS as its transport (other transports are possible with YJS and may be worth experimenting with)
@ -506,7 +505,7 @@ supportFunctions (note YJS uses IPFS and supports some other functions):
supportFeatures: supportFeatures:
fetch.range Not supported (currently April 2018) fetch.range Not supported (currently April 2018)
##TransportWEBTORRENT ## TransportWEBTORRENT
A subclass of Transport for handling WEBTORRENT connections (similar to, with interworking with BitTorrent) A subclass of Transport for handling WEBTORRENT connections (similar to, with interworking with BitTorrent)
Note that currently (April 2018) this won't work properly in the ServiceWorker (SW) because SW dont support WebRTC Note that currently (April 2018) this won't work properly in the ServiceWorker (SW) because SW dont support WebRTC
@ -519,7 +518,7 @@ supportFunctions:
supportFeatures: supportFeatures:
fetch.range Not supported (currently April 2018) fetch.range Not supported (currently April 2018)
##Naming ## Naming
Independently from the transport, the Transport library can resolve names if provided an appropriate callback. Independently from the transport, the Transport library can resolve names if provided an appropriate callback.
See p_resolveNames(urls) and resolveNamesWith(cb) See p_resolveNames(urls) and resolveNamesWith(cb)

4
README.md
View File

@ -18,8 +18,8 @@ decentralized transports.
to your package.json file in the dependencies section. to your package.json file in the dependencies section.
* `npm install dweb-transports` will install the dependencies including IPFS & WebTorrent * `npm install dweb-transports` will install the dependencies including IPFS & WebTorrent
`const DwebTransports = require(dweb-transport)` will add all Transports to a Javascript file. `const DwebTransports = require("dweb-transports")` will add all Transports to a Javascript file.
* TODO writeup how to require only some of the transports. * TODO-API writeup how to require only some of the transports.
* Then see usage API below * Then see usage API below
### Installation and usage in the Browser ### Installation and usage in the Browser

2
Transport.js
View File

@ -53,7 +53,7 @@ class Transport {
return t.p_setup2(verbose, cb); // And connect return t.p_setup2(verbose, cb); // And connect
} }
togglePaused(cb) { //TODO-SW move to Transports > TransportsProxy > UI togglePaused(cb) {
/* /*
Switch the state of the transport between STATUS_CONNECTED and STATUS_PAUSED, Switch the state of the transport between STATUS_CONNECTED and STATUS_PAUSED,
in the paused state it will not be used for transport but, in some cases, will still do background tasks like serving files. in the paused state it will not be used for transport but, in some cases, will still do background tasks like serving files.

10
TransportHTTP.js
View File

@ -89,8 +89,8 @@ class TransportHTTP extends Transport {
//if (!(url && url.includes(':') )) //if (!(url && url.includes(':') ))
// throw new errors.CodingError("TransportHTTP.p_rawfetch bad url: "+url); // throw new errors.CodingError("TransportHTTP.p_rawfetch bad url: "+url);
if (((typeof url === "string") ? url : url.href).includes('/getall/table')) { if (((typeof url === "string") ? url : url.href).includes('/getall/table')) {
console.log("XXX@176 - probably dont want to be calling p_rawfetch on a KeyValueTable, especially since dont know if its keyvaluetable or subclass"); //TODO-NAMING throw new Error("Probably dont want to be calling p_rawfetch on a KeyValueTable, especially since dont know if its keyvaluetable or subclass"); //TODO-NAMING
return { return { // I'm not sure what this return would have done - looks half finished to me?
table: "keyvaluetable", table: "keyvaluetable",
} }
} else { } else {
@ -122,7 +122,7 @@ class TransportHTTP extends Transport {
} }
p_rawadd(url, sig, {verbose=false}={}) { //TODO-BACKPORT turn date into ISO before adding p_rawadd(url, sig, {verbose=false}={}) {
//verbose=true; //verbose=true;
if (!url || !sig) throw new errors.CodingError("TransportHTTP.p_rawadd: invalid parms",url, sig); if (!url || !sig) throw new errors.CodingError("TransportHTTP.p_rawadd: invalid parms",url, sig);
if (verbose) console.log("rawadd", url, sig); if (verbose) console.log("rawadd", url, sig);
@ -166,7 +166,7 @@ class TransportHTTP extends Transport {
} }
//TODO-KEYVALUE needs signing with private key of list //TODO-KEYVALUE needs signing with private key of list
async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table/key //TODO-KEYVALUE-API async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table/key
if (!url || !keyvalues) throw new errors.CodingError("TransportHTTP.p_set: invalid parms",url, keyvalyes); if (!url || !keyvalues) throw new errors.CodingError("TransportHTTP.p_set: invalid parms",url, keyvalyes);
if (verbose) console.log("p_set", url, keyvalues, value); if (verbose) console.log("p_set", url, keyvalues, value);
if (typeof keyvalues === "string") { if (typeof keyvalues === "string") {
@ -188,7 +188,7 @@ class TransportHTTP extends Transport {
return Array.isArray(keys) ? res : res[keys] return Array.isArray(keys) ? res : res[keys]
} }
async p_delete(url, keys, {verbose=false}={}) { //TODO-KEYVALUE-API need to think this one through async p_delete(url, keys, {verbose=false}={}) {
if (!url && keys) throw new errors.CodingError("TransportHTTP.p_get: requires url and at least one key"); if (!url && keys) throw new errors.CodingError("TransportHTTP.p_get: requires url and at least one key");
let parmstr = keys.map(k => this._keyparm(k)).join('&'); let parmstr = keys.map(k => this._keyparm(k)).join('&');
await httptools.p_GET(this._url(url, servercommands.delete, parmstr), {verbose}); await httptools.p_GET(this._url(url, servercommands.delete, parmstr), {verbose});

2
TransportIPFS.js
View File

@ -190,7 +190,7 @@ class TransportIPFS extends Transport {
throw new errors.CodingError(`TransportIPFS.ipfsFrom: Cant convert url ${url} into a path starting /ipfs/`); throw new errors.CodingError(`TransportIPFS.ipfsFrom: Cant convert url ${url} into a path starting /ipfs/`);
} }
static ipfsGatewayFrom(url) { //TODO-API static ipfsGatewayFrom(url) {
/* /*
url: CID, Url, or a string url: CID, Url, or a string
returns: https://ipfs.io/ipfs/<cid> returns: https://ipfs.io/ipfs/<cid>

1
TransportWEBTORRENT.js
View File

@ -239,7 +239,6 @@ class TransportWEBTORRENT extends Transport {
let filet = await this._p_fileTorrentFromUrl(url); let filet = await this._p_fileTorrentFromUrl(url);
let self = this; let self = this;
if (wanturl) { if (wanturl) {
console.log("XXX@WT:242 returning",url)
return url; return url;
} else { } else {
return function (opts) { return self.createReadStream(filet, opts, verbose); }; return function (opts) { return self.createReadStream(filet, opts, verbose); };

6
TransportYJS.js
View File

@ -252,7 +252,7 @@ class TransportYJS extends Transport {
return { privateurl: `${database.privateurl}/${table}`, publicurl: `${database.publicurl}/${table}`} // No action required to create it return { privateurl: `${database.privateurl}/${table}`, publicurl: `${database.publicurl}/${table}`} // No action required to create it
} }
async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table/key //TODO-KEYVALUE-API async p_set(url, keyvalues, value, {verbose=false}={}) { // url = yjs:/yjs/database/table/key
let y = await this.p_connection(url, verbose); let y = await this.p_connection(url, verbose);
if (typeof keyvalues === "string") { if (typeof keyvalues === "string") {
y.share.map.set(keyvalues, JSON.stringify(value)); y.share.map.set(keyvalues, JSON.stringify(value));
@ -272,11 +272,11 @@ class TransportYJS extends Transport {
return typeof val === "string" ? JSON.parse(val) : val; // Surprisingly this is sync, the p_connection should have synchronised return typeof val === "string" ? JSON.parse(val) : val; // Surprisingly this is sync, the p_connection should have synchronised
} }
} }
async p_get(url, keys, {verbose=false}={}) { //TODO-KEYVALUE-API - return dict or single async p_get(url, keys, {verbose=false}={}) {
return this._p_get(await this.p_connection(url, verbose), keys, {verbose}); return this._p_get(await this.p_connection(url, verbose), keys, {verbose});
} }
async p_delete(url, keys, {verbose=false}={}) { //TODO-KEYVALUE-API async p_delete(url, keys, {verbose=false}={}) {
let y = await this.p_connection(url, verbose); let y = await this.p_connection(url, verbose);
if (typeof keys === "string") { if (typeof keys === "string") {
y.share.map.delete(keys); y.share.map.delete(keys);

4
Transports.js
View File

@ -328,7 +328,7 @@ class Transports {
} }
} }
static async p_delete(urls, keys, {verbose=false}={}) { //TODO-KEYVALUE-API static async p_delete(urls, keys, {verbose=false}={}) {
/* Delete a key or a list of keys /* Delete a key or a list of keys
kv: Either dict or a string kv: Either dict or a string
value: if kv is a string, this is the value to set value: if kv is a string, this is the value to set
@ -555,7 +555,7 @@ class Transports {
if (verbose) console.groupEnd("p_connect ---"); if (verbose) console.groupEnd("p_connect ---");
} }
static async p_urlsFrom(url) { //TODO backport to main repo - copy from htmlutils to utils static async p_urlsFrom(url) {
/* Utility to convert to urls form wanted for Transports functions, e.g. from user input /* Utility to convert to urls form wanted for Transports functions, e.g. from user input
url: Array of urls, or string representing url or representing array of urls url: Array of urls, or string representing url or representing array of urls
return: Array of strings representing url return: Array of strings representing url

2
httptools.js
View File

@ -1,8 +1,6 @@
const nodefetch = require('node-fetch'); // Note, were using node-fetch-npm which had a warning in webpack see https://github.com/bitinn/node-fetch/issues/421 and is intended for clients const nodefetch = require('node-fetch'); // Note, were using node-fetch-npm which had a warning in webpack see https://github.com/bitinn/node-fetch/issues/421 and is intended for clients
const errors = require('./Errors'); // Standard Dweb Errors const errors = require('./Errors'); // Standard Dweb Errors
//TODO-API move separate out httptools to own part of API.md
//var fetch,Headers,Request; //var fetch,Headers,Request;
//if (typeof(Window) === "undefined") { //if (typeof(Window) === "undefined") {
if (typeof(fetch) === "undefined") { if (typeof(fetch) === "undefined") {

2
test_ipfs.js
View File

@ -93,9 +93,7 @@ async function test_files_cat(cid, expected, expectfailure) {
if (expectfailure && !tryexpectedfailures) return; if (expectfailure && !tryexpectedfailures) return;
console.log("testing via files.cat") console.log("testing via files.cat")
cid = ipfsFrom(cid); // Turn it into the /ipfs/Q... form that files.cat now expects cid = ipfsFrom(cid); // Turn it into the /ipfs/Q... form that files.cat now expects
console.log("XXX@95 passing cid=",cid);
buff = await ipfs.files.cat(cid); //Error: Groups are not supported in the blocks case - never returns from this call. buff = await ipfs.files.cat(cid); //Error: Groups are not supported in the blocks case - never returns from this call.
console.log("XXX@97")
check_result("files.cat", buff, expected, expectfailure); check_result("files.cat", buff, expected, expectfailure);
} catch(err) { } catch(err) {
console.log("Error thrown in files.cat", err); console.log("Error thrown in files.cat", err);