Enable streams in http, and add support for preferredTransports as parameter to createReadStream (as it was in p_f_createReadStream)

This commit is contained in:
Mitra Ardron 2019-04-14 09:06:03 +10:00
parent 729bab2728
commit e7971a73df
5 changed files with 38 additions and 9 deletions

32
API.md
View File

@ -239,14 +239,26 @@ returns: Dictionary of Key:Value pairs, note take care if this could
``` ```
### Transports - other functions ### Transports - other functions
##### static async p_f_createReadStream(url, {wanturl}) ##### static async p_f_createReadStream(url, {wanturl, preferredTransports=[] })
Provide a function of the form needed by <VIDEO> tag and renderMedia library etc Provide a function of the form needed by <VIDEO> tag and renderMedia library etc
``` ```
url Urls of stream url Urls of stream
wanturl True if want the URL of the stream (for service workers) wanturl True if want the URL of the stream (for service workers)
preferredTransports: preferred order to select stream transports (usually determined by application)
returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file
``` ```
##### static createReadStream(urls, opts, cb)
Different interface, more suitable when just want a stream, now.
```
urls: Url or [urls] of the stream
opts{
start, end: First and last byte wanted (default to 0...last)
preferredTransports: preferred order to select stream transports (usually determined by application)
}
returns open readable stream from the net via cb or promise
```
##### supports(url, funcl) ##### supports(url, funcl)
Determines if the Transport supports urls of this form. For example TransportIPFS supports URLs starting ipfs: Determines if the Transport supports urls of this form. For example TransportIPFS supports URLs starting ipfs:
``` ```
@ -500,7 +512,8 @@ SupportFunctions (note YJS uses IPFS and supports some other functions):
SupportFeatures: SupportFeatures:
fetch.range Not supported (currently April 2018)) fetch.range Not supported (currently April 2018))
Currently there is code for p_f_createReadStream. It works but because of some other IPFS issues is disabled. Currently there is code for p_f_createReadStream. It works but because IPFS cannot return an error even if it
cannot open the stream, IPFS is usually set as the last choice transport for streams.
## TransportYJS ## TransportYJS
A subclass of Transport for handling YJS connections. A subclass of Transport for handling YJS connections.
@ -520,8 +533,10 @@ When used with a SW, it will attempt to retrieve from the http backup URL that i
In the SW it will also generate errors about trackers because the only reason to use trackers is to get the WebRTC links. In the SW it will also generate errors about trackers because the only reason to use trackers is to get the WebRTC links.
supportURLS = `magnet:*` (TODO: may in the future support `dweb:/magnet/*`) supportURLS = `magnet:*` (TODO: may in the future support `dweb:/magnet/*`)
supportFunctions: supportFunctions:
`fetch, createReadStream` `fetch`, `createReadStream`
supportFeatures: supportFeatures:
fetch.range Not supported (currently April 2018) fetch.range Not supported (currently April 2018)
@ -529,10 +544,17 @@ supportFeatures:
A subclass of Transport for handling GUN connections (decentralized database) A subclass of Transport for handling GUN connections (decentralized database)
supportURLS = `gun:*` (TODO: may in the future support `dweb:/gun/*`) supportURLS = `gun:*` (TODO: may in the future support `dweb:/gun/*`)
supportFunctions
`add, list, listmonitor, newlisturls, connection, get, set, getall, keys, newdatabase, newtable, monitor` supportFunctions = `add`, `list`, `listmonitor`, `newlisturls`, `connection`, `get`, `set`, `getall`, `keys`, `newdatabase`, `newtable`, `monitor`
supportFeatures: supportFeatures:
## TransportWOLK
A subclass of Transport for handling the WOLK transport layer (decentralized, block chain based, incentivised storage)
supportURLs = ['wolk'];
supportFunctions = [ 'fetch', 'connection', 'get', 'set', ]; // 'store' - requires chunkdata; 'createReadStream' not implemented
## Naming ## Naming
Independently from the transport, the Transport library can resolve names if provided an appropriate callback. Independently from the transport, the Transport library can resolve names if provided an appropriate callback.
See p_resolveNames(urls) and resolveNamesWith(cb) See p_resolveNames(urls) and resolveNamesWith(cb)

View File

@ -31,7 +31,7 @@ class TransportHTTP extends Transport {
this.options = options; this.options = options;
this.urlbase = options.urlbase; this.urlbase = options.urlbase;
this.supportURLs = ['contenthash', 'http','https']; this.supportURLs = ['contenthash', 'http','https'];
this.supportFunctions = ['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase"]; //Does not support: listmonitor - reverse is disabled somewhere not sure if here or caller this.supportFunctions = ['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase", "createReadStream"]; //Does not support: listmonitor - reverse is disabled somewhere not sure if here or caller
// noinspection JSUnusedGlobalSymbols // noinspection JSUnusedGlobalSymbols
this.supportFeatures = ['fetch.range']; this.supportFeatures = ['fetch.range'];
this.name = "HTTP"; // For console log etc this.name = "HTTP"; // For console log etc
@ -164,6 +164,7 @@ class TransportHTTP extends Transport {
Node.js readable stream docs: https://nodejs.org/api/stream.html#stream_readable_streams Node.js readable stream docs: https://nodejs.org/api/stream.html#stream_readable_streams
:param string url: URL of object being retrieved of form magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents) :param string url: URL of object being retrieved of form magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents)
:param boolean wanturl True if want the URL of the stream (for service workers)
:resolves to: f({start, end}) => stream (The readable stream.) :resolves to: f({start, end}) => stream (The readable stream.)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise :throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/ */

View File

@ -362,6 +362,7 @@ class TransportIPFS extends Transport {
:param string url: URL of object being retrieved of form: :param string url: URL of object being retrieved of form:
magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents) magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents)
ipfs:/ipfs/Q123 ipfs:/ipfs/Q123
:param boolean wanturl True if want the URL of the stream (for service workers)
:resolves to: f({start, end}) => stream (The readable stream.) :resolves to: f({start, end}) => stream (The readable stream.)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise :throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/ */

View File

@ -235,6 +235,7 @@ class TransportWEBTORRENT extends Transport {
Node.js readable stream docs: https://nodejs.org/api/stream.html#stream_readable_streams Node.js readable stream docs: https://nodejs.org/api/stream.html#stream_readable_streams
:param string url: URL of object being retrieved of form magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents) :param string url: URL of object being retrieved of form magnet:xyzabc/path/to/file (Where xyzabc is the typical magnet uri contents)
:param boolean wanturl True if want the URL of the stream (for service workers)
:resolves to: f({start, end}) => stream (The readable stream.) :resolves to: f({start, end}) => stream (The readable stream.)
:throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise :throws: TransportError if url invalid - note this happens immediately, not as a catch in the promise
*/ */

View File

@ -299,6 +299,7 @@ class Transports {
static async p_f_createReadStream(urls, {wanturl=false, preferredTransports=[]}={}) { // Note options is options for selecting a stream, not the start/end in a createReadStream call static async p_f_createReadStream(urls, {wanturl=false, preferredTransports=[]}={}) { // Note options is options for selecting a stream, not the start/end in a createReadStream call
/* /*
urls: Url or [urls] of the stream urls: Url or [urls] of the stream
wanturl True if want the URL of the stream (for service workers)
returns: f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file returns: f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file
*/ */
let tt = this.validFor(urls, "createReadStream", {}); //[ [Url,t],[Url,t]] // Can pass options TODO-STREAM support options in validFor let tt = this.validFor(urls, "createReadStream", {}); //[ [Url,t],[Url,t]] // Can pass options TODO-STREAM support options in validFor
@ -331,16 +332,19 @@ class Transports {
debugtransports("Opening stream to %o failed on all transports", urls); debugtransports("Opening stream to %o failed on all transports", urls);
throw new errors.TransportError(errs.map((err)=>err.message).join(', ')); //Throw err with combined messages if none succeed throw new errors.TransportError(errs.map((err)=>err.message).join(', ')); //Throw err with combined messages if none succeed
} }
static createReadStream(urls, opts, cb) { //TODO-API static createReadStream(urls, opts, cb) {
/* /*
Different interface, more suitable when just want a stream, now. Different interface, more suitable when just want a stream, now.
urls: Url or [urls] of the stream urls: Url or [urls] of the stream
opts{start, end}: First and last byte wanted (default to 0...last) opts{
start, end: First and last byte wanted (default to 0...last)
preferredTransports: preferred order to select stream transports (usually determined by application)
}
cb(err, stream): Called with open readable stream from the net. cb(err, stream): Called with open readable stream from the net.
Returns promise if no cb Returns promise if no cb
*/ */
if (typeof opts === "function") { cb = opts; opts = {start: 0}; } // Allow skipping opts if (typeof opts === "function") { cb = opts; opts = {start: 0}; } // Allow skipping opts
DwebTransports.p_f_createReadStream(urls) DwebTransports.p_f_createReadStream(urls, {preferredTransports: (opts.preferredTransports || [])})
.then(f => { .then(f => {
let s = f(opts); let s = f(opts);
if (cb) { cb(null, s); } else { return(s); }; // Callback or resolve stream if (cb) { cb(null, s); } else { return(s); }; // Callback or resolve stream