Documentation and associated tweaks

API.md

@@ -0,0 +1,377 @@
+#dweb-transports  API
+Mitra Ardron, Internet Archive,  mitra@mitra.biz
+
+This doc provides a concise API specification for the Dweb Javascript Libraries. 
+
+It was last revised (to match the code) on 9 March 2018. 
+
+If you find any discrepancies please add an issue here.
+
+##General API notes and conventions
+We use a naming convention that anything starting “p_” returns a promise so you know to "await" it if you want a result.
+
+Ideally functions should take a String, Buffer or where applicable Object as parameters with automatic conversion. 
+And anything that takes a URL should take either a string or parsed URL object. 
+
+The verbose parameter is a boolean that is an indicator of a need to output to the console. Normally it will be passed down to called functions and default to false.
+
+Note that example_block.html collects this from the URL and passes it to the library, 
+which is intended to be a good way to see what is happening.
+
+Note: I am gradually (March2018) changing the API to take an opts {} dict which includes verbose as one field. This process is incomplete, but I’m happy to see it accelerated if there is any code built on this, just let mitra@archive.org know.
+
+##Overview
+
+The Transport layer provides a layer that is intended to be independent of the underlying storage/transport mechanism.
+
+There is a class “Transport”, which should be subclassed for each supported transport and implement each of these functions, and there is a “Transports” class which manages the list of conencted transports, and directs api calls to them. 
+
+Documents are retrieved by a list of URLs, where in each URL, the left side helps identify the transport, and the right side can be the internal format of the underlying transport BUT must be identifiable e.g. ipfs:/ipfs/12345abc or https://gateway.dweb.me/content/contenthash/Qm..., this format will evolve if a standard URL for the decentralized space is defined. 
+
+The actual urls used might change as the Decentralized Web universe reaches consensus (for example dweb:/ipfs/Q123 is an alternative seen in some places)
+
+This spec will in the future probably add a variation that sends events as the block is retrieved to allow for streaming.
+
+##Transport Class
+
+Fields| 
+:---|---
+options | Holds options passed to constructor
+name|Short name of transport e.g. “HTTP”, “IPFS”
+supportURLs|Array of url prefixes supported e.g. [‘ipfs’,’http’]
+supportFunctions|Array of functions supported on those urls, current full list would be: `['fetch', 'store', 'add', 'list', 'reverse', 'newlisturls', "get", "set", "keys", "getall", "delete", "newtable", "newdatabase", "listmonitor"]`
+status|Numeric indication of transport status: Started(0); Failed(1); Starting(2); Loaded(3)
+
+###Setup of a transport
+Transport setup is split into 3 parts, this allows the Transports class to do the first phase on all the transports synchronously, 
+then asynchronously (or at a later point) try and connect to all of them in parallel. 
+
+#####static setup0 (options, verbose)
+First part of setup, create obj, add to Transports but dont attempt to connect, typically called instead of p_setup if want to parallelize connections.  In almost all cases this will call the constructor of the subclass
+Should be synchronous and leave `status=STATUS_LOADED`
+```
+options        Object fields including those needed by transport layer
+verbose        boolean - true for debugging output
+Resolves to    Instance of subclass of Transport
+```
+Default options should be set in each transport, but can be overwritten, 
+for example to overwrite the options for HTTP call it with  
+`options={ http: { urlbase: “https://gateway.dweb.me:443/” } }`
+
+#####async p_setup1 (verbose, cb) 
+Setup the resource and open any P2P connections etc required to be done just once. 
+Asynchronous and should leave `status=STATUS_STARTING` until it resolves, or `STATUS_FAILED` if fails.
+```
+cb (t)=>void   If set, will be called back as status changes (so could be multiple times) 
+Resolves to    the Transport instance
+```
+
+#####async p_setup2 (verbose, cb) 
+Works like p_setup1 but runs after p_setup1 has completed for all transports. 
+This allows for example YJS to wait for IPFS to be connected in TransportIPFS.setup1() 
+and then connect itself using the IPFS object. 
+```
+cb (t)=>void   If set, will be called back as status changes (so could be multiple times) 
+Resolves to    the Transport instance
+```
+
+#####p_setup(options, verbose, cb) 
+A deprecated utility to simply setup0 then p_setup1 then p_setup2 to allow a transport to be started
+in one step, normally `Transports.p_setup` should be called instead.
+
+#####p_status (verbose)
+Return a numeric code for the status of a transport.
+
+Code|Name|Means
+---|---|---
+0|STATUS_CONNECTED|Connected and can be used
+1|STATUS_FAILED|Setup process failed, can rerun if required
+2|STATUS_STARTING|Part way through the setup process
+3|STATUS_LOADED|Code loaded but havent tried to connect
+4|STATUS_PAUSED|It was launched, probably connected, but now paused so will be ignored by validfor()
+
+###Transport: General storage and retrieval of objects
+#####p_rawstore(data, {verbose})
+Store a opaque blob of data onto the decentralised transport.
+```
+data         string|Buffer data to store - no assumptions made to size or content
+verbose      boolean - True for debugging output
+Resolves to  url of data stored
+```
+
+#####p_rawfetch(url, {timeoutMS, start, end, relay, verbose})
+Fetch some bytes based on a url, no assumption is made about the data in terms of size or structure.
+
+Where required by the underlying transport it should retrieve a number if its "blocks" and concatenate them.
+
+There may also be need for a streaming version of this call, at this point undefined.
+```
+url          string url of object being retrieved in form returned by link or p_rawstore
+timeoutMS    Max time to wait on transports that support it (IPFS for fetch)
+start,end    Inclusive byte range wanted (must be supported, uses a "slice" on output if transport ignores it.
+relay        If first transport fails, try and retrieve on 2nd, then store on 1st, and so on.
+verbose      boolean - True for debugging output
+Resolves to  string The object being fetched, (note currently returned as a string, may refactor to return Buffer)
+throws:      TransportError if url invalid - note this happens immediately, not as a catch in the promise
+```
+
+###Transport: Handling lists
+#####p_rawadd(url, sig, {verbose})
+Store a new list item, it should be stored so that it can be retrieved either by "signedby" (using p_rawlist) 
+or by "url"  (with p_rawreverse). 
+
+The underlying transport can, but does not need to verify the signature, 
+an invalid item on a list should be rejected by higher layers.
+```
+url        String identifying list to add sig to.
+sig        Signature data structure (see below - contains url, date, signedby, signature)
+    date        - date of signing in ISO format,
+    urls        - array of urls for the object being signed
+    signature   - verifiable signature of date+urls
+    signedby    - url of data structure (typically CommonList) holding public key used for the signature
+verbose        boolean - True for debugging output
+```
+
+#####p_rawlist(url, {verbose})
+Fetch all the objects in a list, these are identified by the url of the public key used for signing.
+Note this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter. 
+List items may have other data (e.g. reference ids of underlying transport)
+```
+url         String with the url that identifies the list
+            (this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter
+verbose     boolean - True for debugging output
+Resolves to Array: An array of objects as stored on the list. Each of which is …..
+```
+Each item of the list is a dict: {"url": url, "date": date, "signature": signature, "signedby": signedby}
+
+#####p_rawreverse (url, {verbose})
+Similar to p_rawlist, but return the list item of all the places where the object url has been listed.
+(not supported by most transports)
+```
+url         String with the url that identifies the object put on a list
+            This is the “url” parameter of p_rawadd
+verbose     boolean - True for debugging output
+Resolves to Array objects as stored on the list (see p_rawlist for format)
+```
+
+#####listmonitor (url, cb) 
+Setup a callback called whenever an item is added to a list, typically it would be called immediately after a p_rawlist to get any more items not returned by p_rawlist.
+```
+url        Identifier of list (as used by p_rawlist and "signedby" parameter of p_rawadd
+cb(obj)    function(obj)  Callback for each new item added to the list
+           obj is same format as p_rawlist or p_rawreverse
+```
+
+#####async p_newlisturls(cl, {verbose}) 
+Obtain a pair of URLs for a new list. The transport can use information in the cl to generate this or create something random (the former is encouraged since it means repeat tests might not generate new lists). Possession of the publicurl should be sufficient to read the list, the privateurl should be required to read (for some transports they will be identical, and higher layers should check for example that a signature is signed.  
+```
+cl          CommonList instance that can be used as a seed for the URL
+verbose     boolean - True for debugging output
+Returns     [privateurl, publicurl]
+```
+
+###Transport: Support for KeyValueTable
+#####async p_newdatabase(pubkey, {verbose}) {
+Create a new database based on some existing object
+```
+pubkey:     Something that is, or has a pubkey, by default support Dweb.PublicPrivate, KeyPair 
+            or an array of strings as in the output of keypair.publicexport()
+returns:    {publicurl, privateurl} which may be the same if there is no write authentication
+```
+
+#####async p_newtable(pubkey, table, {verbose}) {
+Create a new table,
+```
+pubkey:     Is or has a pubkey (see p_newdatabase)
+table:      String representing the table - unique to the database
+returns:    {privateurl, publicurl} which may be the same if there is no write authentication
+```
+
+#####async p_set(url, keyvalues, value, {verbose}) 
+Set one or more keys in a table.
+```
+url:            URL of the table
+keyvalues:      String representing a single key OR dictionary of keys
+value:          String or other object to be stored (its not defined yet what objects should be supported, e.g. any object ?
+```
+
+#####async p_get(url, keys, {verbose})
+Get one or more keys from a table
+```
+url:            URL of the table
+keys:           Array of keys
+returns:        Dictionary of values found (undefined if not found)
+```
+
+#####async p_delete(url, keys, {verbose})
+Delete one or more keys from a table
+```
+url:            URL of the table
+keys:           Array of keys
+```
+
+#####async p_keys(url, {verbose})
+Return a list of keys in a table (suitable for iterating through)
+```
+url:            URL of the table
+returns:            Array of strings
+```
+
+#####async p_getall(url, {verbose})
+Return a dictionary representing the table
+```
+url:            URL of the table
+returns:            Dictionary of Key:Value pairs, note take care if this could be large.
+```
+
+###Transports - other functions
+#####static async p_f_createReadStream(url, verbose, options)
+Provide a function of the form needed by <VIDEO> tag and renderMedia library etc
+```
+url    Urlsof stream
+returns f(opts) => stream returning bytes from opts.start || start of file to opts.end-1 || end of file
+```
+
+#####supports(url, funcl)
+Determines if the Transport supports url’s of this form. For example TransportIPFS supports URLs starting ipfs:
+```
+url        identifier of resource to fetch or list
+Returns        True if this Transport supports that type of URL
+```
+
+#####p_info()
+Return a JSON with info about the server. 
+
+##Transports class
+The Transports Class manages multiple transports 
+
+#####Properties
+```
+_transports         List of transports loaded (internal)
+namingcb            If set will be called cb(urls) => urls  to convert to urls from names. 
+_transportclasses   All classes whose code is loaded e.g. {HTTP: TransportHTTP, IPFS: TransportIPFS}
+```
+
+#####static _connected()
+```
+returns Array of transports that are connected (i.e. status=STATUS_CONNECTED)
+```
+
+#####static _connectedNames()
+```
+returns Array of names transports that are connected (i.e. status=STATUS_CONNECTED)
+```
+#####static connectedNamesParm
+```
+returns part of URL string for transports e.g. 'transport=HTTP&transport=IPFS"
+```
+
+#####static validFor(urls, func, options) {
+Finds an array or Transports that are STARTED and can support this URL. 
+```
+urls:       Array of urls
+func:       Function to check support for: fetch, store, add, list, listmonitor, reverse
+        - see supportFunctions on each Transport class
+options     For future use
+Returns:    Array of pairs of url & transport instance [ [ u1, t1], [u1, t2], [u2, t1]]
+```
+
+#####static http(verbose)
+```
+returns instance of TransportHTTP if connected
+```
+
+#####static ipfs(verbose)
+```
+returns instance of TransportIPFS if connected
+```
+
+#####static async p_resolveNames(urls)
+```
+urls    mix of urls and names
+names   if namingcb is set, will convert any names to URLS (this requires higher level libraries)
+```
+
+#####static async resolveNamesWith(cb)
+Called by higher level libraries that provide name resolution function
+```
+cb(urls) => urls    Provide callback function 
+```
+
+#####static addtransport(t)
+```
+t:        Add a Transport instance to _transports
+```
+
+#####static setup0(transports, options, verbose, cb)
+Calls setup0 for each transport based on its short name. Specially handles ‘LOCAL’ as a transport pointing at a local http server (for testing).
+```
+transports      Array of short names of transports e.g. [‘IPFS’,’HTTP’,’ORBITDB’]
+options         Passed to setup0 on each transport
+cb              Callback to be called each time status changes
+Returns:        Array of transport instances
+```
+
+#####static async p_setup1(verbose, cb)
+Call p_setup1 on all transports that were created in setup0().  Completes when all setups are complete. 
+
+#####static async p_setup2(verbose, cb)
+Call p_setup2 on all transports that were created in setup0().  Completes when all setups are complete. 
+
+#####static asunc refreshstatus(t)
+Set the class of t.statuselement (if set) to transportstatus0..transportstatus4 depending on its status.
+```
+t   Instance of transport
+```
+####static async p_connect({options}, verbose)
+Main connection process for a browser based application, 
+```
+options {
+    transports  Array of abbreviations of transports e.g. ["HTTP","IPFS"] as provided in URL
+    defaulttransports   Array of abbreviations of transports to use if transports is unset
+    statuselement   HTML element to build status display under
+    ...             Entire options is passed to each setup0 and will include options for each Transport
+}
+
+```
+#####static urlsFrom(url)
+Utility to convert to urls form wanted for Transports functions, e.g. from user input
+```
+url:    Array of urls, or string representing url or representing array of urls
+return: Array of strings representing url
+```
+
+###Multi transport calls 
+Each of the following is attempted across multiple transports
+For parameters refer to underlying Transport call
+
+Call|Returns|Behavior
+---|---|---
+static async p_rawstore(data, {verbose})|[urls]|Tries all and combines results
+static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay})|data|See note
+static async p_rawlist(urls, {verbose})|[sigs]|Tries all and combines results
+static async p_rawadd(urls, sig, {verbose})||Tries on all urls, error if none succeed
+static listmonitor(urls, cb)||Tries on all urls (so note cb may be called multiple times)
+static p_newlisturls(cl, {verbose})|[urls]|Tries all and combines results
+static async p_f_createReadStream(urls, verbose, options)|f(opts)=>stream|Returns first success
+static async p_get(urls, keys, {verbose})|currently returns on first success, TODO - will combine results and relay across transports
+static async p_set(urls, keyvalues, value, {verbose})|Tries all, error if none succeed
+static async p_delete(urls, keys, {verbose})|Tries all, error if none succeed
+static async p_keys(urls, {verbose}|[keys]|currently returns on first success, TODO - will combine results and relay across transports
+static async p_getall(urls, {verbose})|dict|currently returns on first success, TODO - will combine results and relay across transports
+static async p_newdatabase(pubkey, {verbose})|{privateurls: [urls], publicurls: [urls]}|Tries all and combines results
+static async p_newtable(pubkey, table, {verbose})|{privateurls: [urls], publicurls: [urls]}|Tries all and combines results
+static async p_connection(urls, verbose)||Tries all parallel
+static monitor(urls, cb, verbose)||Tries all sequentially
+
+#####static async p_rawfetch(urls, {timeoutMS, start, end, verbose, relay})
+Tries to fetch on all valid transports until successful. See Transport.p_rawfetch
+```
+timeoutMS:   Max time to wait on transports that support it (IPFS for fetch)
+start,end    Inclusive byte range wanted - passed to 
+relay        If first transport fails, try and retrieve on 2nd, then store on 1st, and so on.
+```
+
+###TODO documentation
+* options to each Transport
+* Other useful functions on each transport esp http p_GET etc

README.md

@@ -29,25 +29,29 @@ to your package.json file in the dependencies section.
 * Clone this repo and cd to it.
 * `npm bundle` will create dist/dweb_transports_bundle.js
 * Add `<SCRIPT type="text/javascript" src="dweb_transports_bundle.js"></SCRIPT>` to your `<HEAD>`
-* TODO need to split out necessary from htmlutils so p_connect exposed
 
 Then code like this should work. 
 
 ```
-async function main() {
+async function main(url) {
     try {
-        // Connect to a set of transports, that the user can override
+        // and if not found will use the defaulttransports specified here.
-        // p_connect will look for "?transport=HTTP" etc to override defaults
+        await DwebTransports.p_connect({
-        await p_connect({defaulttransport: ["HTTP", "IPFS"]});
+            statuselement: document.getElementById("statuselement"),    // Where to build status indicator
-        // Code you want to happen after connection goes here
+            defaulttransports: ["HTTP","IPFS"],                         // Default transports if not specified
-        
+            transports: searchparams.getAll("transport")    // Allow override default from URL parameters
-        // Fetch some data from a service, specifiying two places to get it
+        }, verbose);                                        // Pass verbose global parameter from command line
-        foo = await DwebTransports.p_fetch(["http://bar.com/foo.json", "ipfs:/ipfs/Q1abc"], verbose);                
+        // Any code you want to run after connected to transports goes here.
     } catch(err) {
+        console.log("App Error:", err);
         alert(err.message);
     }
 }
 var searchparams = new URL(window.location.href).searchParams;
-// Allow specifying ?verbose=true in URL to get debugging 
+// Allow specifying ?verbose=true in URL to get debugging, and transport=HTTP etc
 var verbose = searchparams.get("verbose") || false;
-```
+```
+
+See [example_block.html](./example_block.html) for an example of connecting, storing and retrieving.
+
+

Transport.js

@@ -28,7 +28,7 @@ class Transport {
     p_setup1(options, verbose) { return this; }
     p_setup2(options, verbose) { return this; }
 
-    static async p_setup(options, verbose) {
+    static async p_setup(options, verbose, cb) {
             /*
             Setup the resource and open any P2P connections etc required to be done just once.
             In almost all cases this will call the constructor of the subclass
@@ -38,9 +38,9 @@ class Transport {
             :resolve Transport: Instance of subclass of Transport
              */
             let t = await this.setup0(options, verbose) // Sync version that doesnt connect
-                .p_setup1(verbose); // And connect
+                .p_setup1(verbose, cb); // And connect
 
-            return t.p_setup2(verbose);     // And connect
+            return t.p_setup2(verbose, cb);     // And connect
     }
     togglePaused(cb) { //TODO-API
         switch (this.status) {
@@ -73,7 +73,7 @@ class Transport {
             && (!func || this.supportFunctions.includes(func)))
     }
 
-    p_rawstore(data, verbose) {
+    p_rawstore(data, opts) {
         /*
         Store a blob of data onto the decentralised transport.
         Returns a promise that resolves to the url of the data
@@ -85,9 +85,9 @@ class Transport {
         throw new errors.ToBeImplementedError("Intentionally undefined function Transport.p_rawstore should have been subclassed");
     }
 
-    async p_rawstoreCaught(data, verbose) {
+    async p_rawstoreCaught(data, {verbose}) {
         try {
-            return await this.p_rawstore(data, verbose);
+            return await this.p_rawstore(data, {verbose});
         } catch (err) {
 
         }
@@ -117,7 +117,7 @@ class Transport {
         throw new errors.ToBeImplementedError("Undefined function Transport.p_fetch - may define higher level semantics here (see Python)");
     }
 
-    p_rawadd(url, sig, verbose) {
+    p_rawadd(url, sig, {verbose=false}={}) {
         /*
         Store a new list item, ideally it should be stored so that it can be retrieved either by "signedby" (using p_rawlist) or
         by "url" (with p_rawreverse). The underlying transport does not need to guarantee the signature,
@@ -131,7 +131,7 @@ class Transport {
         throw new errors.ToBeImplementedError("Undefined function Transport.p_rawadd");
     }
 
-    p_rawlist(url, verbose) {
+    p_rawlist(url, {verbose=false}={}) {
         /*
         Fetch all the objects in a list, these are identified by the url of the public key used for signing.
         (Note this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter
@@ -149,7 +149,7 @@ class Transport {
     p_list() {
         throw new Error("Undefined function Transport.p_list");
     }
-    p_newlisturls(cl, verbose) {
+    p_newlisturls(cl, {verbose=false}={}) {
         /*
         Must be implemented by any list, return a pair of URLS that may be the same, private and public links to the list.
         returns: ( privateurl, publicurl) e.g. yjs:xyz/abc or orbitdb:a123
@@ -158,7 +158,7 @@ class Transport {
     }
 
     //noinspection JSUnusedGlobalSymbols
-    p_rawreverse(url, verbose) {
+    p_rawreverse(url, {verbose=false}={}) {
         /*
         Similar to p_rawlist, but return the list item of all the places where the object url has been listed.
         The url here corresponds to the "url" parameter of p_rawadd
@@ -188,7 +188,7 @@ class Transport {
     // Support for Key-Value pairs as per
     // https://docs.google.com/document/d/1yfmLRqKPxKwB939wIy9sSaa7GKOzM5PrCZ4W1jRGW6M/edit#
 
-    async p_newdatabase(pubkey, verbose) {
+    async p_newdatabase(pubkey, {verbose=false}={}) {
         /*
          Create a new database based on some existing object
          pubkey:    Something that is, or has a pubkey, by default support Dweb.PublicPrivate, KeyPair or an array of strings as in the output of keypair.publicexport()
@@ -198,7 +198,7 @@ class Transport {
     }
     //TODO maybe change the listmonitor / monitor code for to use "on" and the structure of PP.events
     //TODO but note https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy about Proxy which might be suitable, prob not as doesnt map well to lists
-    async p_newtable(pubkey, table, verbose) {
+    async p_newtable(pubkey, table, {verbose=false}={}) {
         /*
         Create a new table,
         pubkey: Is or has a pubkey (see p_newdatabase)
@@ -208,7 +208,7 @@ class Transport {
         throw new errors.ToBeImplementedError("Undefined function Transport.p_newtable");
     }
 
-    async p_set(url, keyvalues, value, verbose) {  // url = yjs:/yjs/database/table/key
+    async p_set(url, keyvalues, value, {verbose=false}={}) {  // url = yjs:/yjs/database/table/key
         /*
         Set one or more keys in a table.
         url:    URL of the table
@@ -217,7 +217,7 @@ class Transport {
          */
         throw new errors.ToBeImplementedError("Undefined function Transport.p_set");
     }
-    async p_get(url, keys, verbose) {
+    async p_get(url, keys, {verbose=false}={}) {
         /* Get one or more keys from a table
         url:    URL of the table
         keys:   Array of keys
@@ -226,7 +226,7 @@ class Transport {
         throw new errors.ToBeImplementedError("Undefined function Transport.p_get");
     }
 
-    async p_delete(url, keys, verbose) {
+    async p_delete(url, keys, {verbose=false}={}) {
         /* Delete one or more keys from a table
         url:    URL of the table
         keys:   Array of keys
@@ -234,14 +234,14 @@ class Transport {
         throw new errors.ToBeImplementedError("Undefined function Transport.p_delete");
     }
 
-    async p_keys(url, verbose) {
+    async p_keys(url, {verbose=false}={}) {
         /* Return a list of keys in a table (suitable for iterating through)
         url:    URL of the table
         returns:    Array of strings
          */
         throw new errors.ToBeImplementedError("Undefined function Transport.p_keys");
     }
-    async p_getall(url, verbose) {
+    async p_getall(url, {verbose=false}={}) {
         /* Return a dictionary representing the table
         url:    URL of the table
         returns:    Dictionary of Key:Value pairs, note take care if this could be large.
@@ -276,23 +276,23 @@ class Transport {
          */
         if (verbose) {console.log(this.name,"p_test_kvt")}
         try {
-            let table = await this.p_newtable("NACL VERIFY:1234567","mytable", verbose);
+            let table = await this.p_newtable("NACL VERIFY:1234567","mytable", {verbose});
             let mapurl = table.publicurl;
             if (verbose) console.log("newtable=",mapurl);
             console.assert(mapurl.includes(urlexpectedsubstring));
-            await this.p_set(mapurl, "testkey", "testvalue", verbose);
+            await this.p_set(mapurl, "testkey", "testvalue", {verbose});
-            let res = await this.p_get(mapurl, "testkey", verbose);
+            let res = await this.p_get(mapurl, "testkey", {verbose});
             console.assert(res === "testvalue");
-            await this.p_set(mapurl, "testkey2", {foo: "bar"}, verbose);   // Try setting to an object
+            await this.p_set(mapurl, "testkey2", {foo: "bar"}, {verbose});   // Try setting to an object
-            res = await this.p_get(mapurl, "testkey2", verbose);
+            res = await this.p_get(mapurl, "testkey2", {verbose});
             console.assert(res.foo === "bar");
-            await this.p_set(mapurl, "testkey3", [1,2,3], verbose);    // Try setting to an array
+            await this.p_set(mapurl, "testkey3", [1,2,3], {verbose});    // Try setting to an array
-            res = await this.p_get(mapurl, "testkey3", verbose);
+            res = await this.p_get(mapurl, "testkey3", {verbose});
             console.assert(res[1] === 2);
-            res = await this.p_keys(mapurl);
+            res = await this.p_keys(mapurl, {verbose});
             console.assert(res.includes("testkey") && res.includes("testkey3"));
-            res = await this.p_delete(mapurl, ["testkey"]);
+            res = await this.p_delete(mapurl, ["testkey"], {verbose});
-            res = await this.p_getall(mapurl, verbose);
+            res = await this.p_getall(mapurl, {verbose});
             if (verbose) console.log("getall=>",res);
             console.assert(res.testkey2.foo === "bar" && res.testkey3["1"] === 2 && !res.testkey1);
             await delay(200);

TransportHTTP.js

@@ -70,10 +70,9 @@ class TransportHTTP extends Transport {
         return this;
     }
 
-    async p_status(verbose) {    //TODO-BACKPORT
+    async p_status(verbose) {
         /*
-        Return a string for the status of a transport. No particular format, but keep it short as it will probably be in a small area of the screen.
+        Return a numeric code for the status of a transport.
-        resolves to: String representing type connected (always HTTP) and online if online.
          */
         try {
             this.info = await this.p_info(verbose);
@@ -194,7 +193,7 @@ class TransportHTTP extends Transport {
         }
     }
 
-    p_rawlist(url, verbose) {
+    p_rawlist(url, {verbose=false}={}) {
         // obj being loaded
         // Locate and return a block, based on its url
         if (!url) throw new errors.CodingError("TransportHTTP.p_rawlist: requires url");
@@ -202,7 +201,7 @@ class TransportHTTP extends Transport {
     }
     rawreverse() { throw new errors.ToBeImplementedError("Undefined function TransportHTTP.rawreverse"); }
 
-    async p_rawstore(data, verbose) {
+    async p_rawstore(data, {verbose=false}={}) {
         /*
         Store data on http server,
         data:   string
@@ -218,7 +217,7 @@ class TransportHTTP extends Transport {
 
     }
 
-    p_rawadd(url, sig, verbose) { //TODO-BACKPORT turn date into ISO before adding
+    p_rawadd(url, sig, {verbose=false}={}) { //TODO-BACKPORT turn date into ISO before adding
         //verbose=true;
         if (!url || !sig) throw new errors.CodingError("TransportHTTP.p_rawadd: invalid parms",url, sig);
         if (verbose) console.log("rawadd", url, sig);
@@ -226,7 +225,7 @@ class TransportHTTP extends Transport {
         return this.p_POST(this._url(url, servercommands.rawadd), "application/json", value, verbose); // Returns immediately
     }
 
-    p_newlisturls(cl, verbose) {
+    p_newlisturls(cl, {verbose=false}={}) {
        let  u = cl._publicurls.map(urlstr => Url.parse(urlstr))
             .find(parsedurl =>
                 (parsedurl.protocol === "https" && parsedurl.host === "gateway.dweb.me" && parsedurl.pathname.includes('/content/rawfetch'))
@@ -242,7 +241,7 @@ class TransportHTTP extends Transport {
 
     // Support for Key-Value pairs as per
     // https://docs.google.com/document/d/1yfmLRqKPxKwB939wIy9sSaa7GKOzM5PrCZ4W1jRGW6M/edit#
-    async p_newdatabase(pubkey, verbose) {
+    async p_newdatabase(pubkey, {verbose=false}={}) {
         //if (pubkey instanceof Dweb.PublicPrivate)
         if (pubkey.hasOwnProperty("keypair"))
             pubkey = pubkey.keypair.signingexport()
@@ -253,15 +252,15 @@ class TransportHTTP extends Transport {
     }
 
 
-    async p_newtable(pubkey, table, verbose) {
+    async p_newtable(pubkey, table, {verbose=false}={}) {
         if (!pubkey) throw new errors.CodingError("p_newtable currently requires a pubkey");
-        let database = await this.p_newdatabase(pubkey, verbose);
+        let database = await this.p_newdatabase(pubkey, {verbose});
         // If have use cases without a database, then call p_newdatabase first
         return { privateurl: `${database.privateurl}/${table}`,  publicurl: `${database.publicurl}/${table}`}  // No action required to create it
     }
 
     //TODO-KEYVALUE needs signing with private key of list
-    async p_set(url, keyvalues, value, verbose) {  // url = yjs:/yjs/database/table/key   //TODO-KEYVALUE-API
+    async p_set(url, keyvalues, value, {verbose=false}={}) {  // url = yjs:/yjs/database/table/key   //TODO-KEYVALUE-API
         if (!url || !keyvalues) throw new errors.CodingError("TransportHTTP.p_set: invalid parms",url, keyvalyes);
         if (verbose) console.log("p_set", url, keyvalues, value);
         if (typeof keyvalues === "string") {
@@ -276,25 +275,24 @@ class TransportHTTP extends Transport {
     _keyparm(key) {
         return `key=${encodeURIComponent(key)}`
     }
-    //TODO-KEYALUE got to here on KEYVALUE in HTTP
+    async p_get(url, keys, {verbose=false}={}) {
-    async p_get(url, keys, verbose) {
         if (!url && keys) throw new errors.CodingError("TransportHTTP.p_get: requires url and at least one key");
         let parmstr =Array.isArray(keys)  ?  keys.map(k => this._keyparm(k)).join('&') : this._keyparm(keys)
         let res = await this.p_GET(this._url(url, servercommands.get, parmstr), {verbose});
         return Array.isArray(keys) ? res : res[keys]
     }
 
-    async p_delete(url, keys, verbose) {  //TODO-KEYVALUE-API need to think this one through
+    async p_delete(url, keys, {verbose=false}={}) {  //TODO-KEYVALUE-API need to think this one through
         if (!url && keys) throw new errors.CodingError("TransportHTTP.p_get: requires url and at least one key");
         let parmstr =  keys.map(k => this._keyparm(k)).join('&');
         await this.p_GET(this._url(url, servercommands.delete, parmstr), {verbose});
     }
 
-    async p_keys(url, verbose) {
+    async p_keys(url, {verbose=false}={}) {
         if (!url && keys) throw new errors.CodingError("TransportHTTP.p_get: requires url and at least one key");
         return await this.p_GET(this._url(url, servercommands.keys), {verbose});
     }
-    async p_getall(url, verbose) {
+    async p_getall(url, {verbose=false}={}) {
         if (!url && keys) throw new errors.CodingError("TransportHTTP.p_get: requires url and at least one key");
         return await this.p_GET(this._url(url, servercommands.getall), {verbose});
     }
@@ -302,12 +300,12 @@ class TransportHTTP extends Transport {
     async p_rawfetch(url, verbose) {
         return {
             table: "keyvaluetable",
-            _map: await this.p_getall(url, verbose)
+            _map: await this.p_getall(url, {verbose})
         };   // Data struc is ok as SmartDict.p_fetch will pass to KVT constructor
     }
     */
 
-    p_info(verbose) { return this.p_GET(`${this.urlbase}/info`, {verbose}); } //TODO-BACKPORT
+    p_info(verbose) { return this.p_GET(`${this.urlbase}/info`, {verbose}); }
 
     static async p_test(opts={}, verbose=false) {
         if (verbose) {console.log("TransportHTTP.test")}

TransportIPFS.js

@@ -127,7 +127,7 @@ class TransportIPFS extends Transport {
 
     async p_status(verbose) {
         /*
-        Return a string for the status of a transport. No particular format, but keep it short as it will probably be in a small area of the screen.
+            Return a numeric code for the status of a transport.
          */
         this.status =  (await this.ipfs.isOnline()) ? Transport.STATUS_CONNECTED : Transport.STATUS_FAILED;
         return this.status;
@@ -253,7 +253,7 @@ class TransportIPFS extends Transport {
         }
     }
 
-    async p_rawstore(data, verbose) {
+    async p_rawstore(data, {verbose}) {
         /*
         Store a blob of data onto the decentralised transport.
         Returns a promise that resolves to the url of the data
@@ -268,9 +268,7 @@ class TransportIPFS extends Transport {
         //https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/DAG.md#dagput
         //let res = await this.ipfs.dag.put(buf,{ format: 'dag-cbor', hashAlg: 'sha2-256' });
         const res = (await this.ipfs.files.add(buf,{ "cid-version": 1, hashAlg: 'sha2-256'}))[0];
-        //TODO-IPFS has been suggested to move this to files.add with no filename.
         return TransportIPFS.urlFrom(res);
-        //return this.ipfs.files.put(buf).then((block) => TransportIPFS.urlFrom(block.cid));
     }
 
     // Based on https://github.com/ipfs/js-ipfs/pull/1231/files
@@ -350,7 +348,7 @@ class TransportIPFS extends Transport {
             const qbf = "The quick brown fox";
             const qbf_url = "ipfs:/ipfs/zdpuAscRnisRkYnEyJAp1LydQ3po25rCEDPPEDMymYRfN1yPK"; // Expected url
             const testurl = "1114";  // Just a predictable number can work with
-            const url = await transport.p_rawstore(qbf, verbose);
+            const url = await transport.p_rawstore(qbf, {verbose});
             if (verbose) console.log("rawstore returned", url);
             const newcid = TransportIPFS.cidFrom(url);  // Its a CID which has a buffer in it
             console.assert(url === qbf_url, "url should match url from rawstore");

TransportYJS.js

@@ -139,7 +139,7 @@ class TransportYJS extends Transport {
         return this.status;
     }
 
-    async p_rawlist(url, verbose) {
+    async p_rawlist(url, {verbose=false}={}) {
     /*
     Fetch all the objects in a list, these are identified by the url of the public key used for signing.
     (Note this is the 'signedby' parameter of the p_rawadd call, not the 'url' parameter
@@ -197,7 +197,7 @@ class TransportYJS extends Transport {
         //TODO-REVERSE this needs implementing once list structure on IPFS more certain
         throw new errors.ToBeImplementedError("Undefined function TransportYJS.rawreverse"); }
 
-    async p_rawadd(url, sig, verbose) {
+    async p_rawadd(url, sig, {verbose=false}={}) {
         /*
         Store a new list item, it should be stored so that it can be retrieved either by "signedby" (using p_rawlist) or
         by "url" (with p_rawreverse). The underlying transport does not need to guarantee the signature,
@@ -219,7 +219,7 @@ class TransportYJS extends Transport {
         y.share.array.push([value]);
     }
 
-    p_newlisturls(cl, verbose) {
+    p_newlisturls(cl, {verbose=false}={}) {
         let  u = cl._publicurls.map(urlstr => Url.parse(urlstr))
             .find(parsedurl =>
                 (parsedurl.protocol === "ipfs" && parsedurl.pathname.includes('/ipfs/'))
@@ -233,7 +233,7 @@ class TransportYJS extends Transport {
 
     // Support for Key-Value pairs as per
     // https://docs.google.com/document/d/1yfmLRqKPxKwB939wIy9sSaa7GKOzM5PrCZ4W1jRGW6M/edit#
-    async p_newdatabase(pubkey, verbose) {
+    async p_newdatabase(pubkey, {verbose=false}={}) {
         //if (pubkey instanceof Dweb.PublicPrivate)
         if (pubkey.hasOwnProperty("keypair"))
             pubkey = pubkey.keypair.signingexport()
@@ -245,14 +245,14 @@ class TransportYJS extends Transport {
 
     //TODO maybe change the listmonitor / monitor code for to use "on" and the structure of PP.events
     //TODO but note https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy about Proxy which might be suitable, prob not as doesnt map well to lists
-    async p_newtable(pubkey, table, verbose) {
+    async p_newtable(pubkey, table, {verbose=false}={}) {
         if (!pubkey) throw new errors.CodingError("p_newtable currently requires a pubkey");
-        let database = await this.p_newdatabase(pubkey, verbose);
+        let database = await this.p_newdatabase(pubkey, {verbose});
         // If have use cases without a database, then call p_newdatabase first
         return { privateurl: `${database.privateurl}/${table}`,  publicurl: `${database.publicurl}/${table}`}  // No action required to create it
     }
 
-    async p_set(url, keyvalues, value, verbose) {  // url = yjs:/yjs/database/table/key   //TODO-KEYVALUE-API
+    async p_set(url, keyvalues, value, {verbose=false}={}) {  // url = yjs:/yjs/database/table/key   //TODO-KEYVALUE-API
         let y = await this.p_connection(url, verbose);
         if (typeof keyvalues === "string") {
             y.share.map.set(keyvalues, JSON.stringify(value));
@@ -260,7 +260,7 @@ class TransportYJS extends Transport {
             Object.keys(keyvalues).map((key) => y.share.map.set(key, keyvalues[key]));
         }
     }
-    _p_get(y, keys, verbose) {
+    _p_get(y, keys, {verbose=false}={}) {
         if (Array.isArray(keys)) {
             return keys.reduce(function(previous, key) {
                 let val = y.share.map.get(key);
@@ -272,11 +272,11 @@ class TransportYJS extends Transport {
             return typeof val === "string" ? JSON.parse(val) : val;  // Surprisingly this is sync, the p_connection should have synchronised
         }
     }
-    async p_get(url, keys, verbose) {  //TODO-KEYVALUE-API - return dict or single
+    async p_get(url, keys, {verbose=false}={}) {  //TODO-KEYVALUE-API - return dict or single
-        return this._p_get(await this.p_connection(url, verbose), keys);
+        return this._p_get(await this.p_connection(url, verbose), keys, {verbose});
     }
 
-    async p_delete(url, keys, verbose) {  //TODO-KEYVALUE-API
+    async p_delete(url, keys, {verbose=false}={}) {  //TODO-KEYVALUE-API
         let y = await this.p_connection(url, verbose);
         if (typeof keys === "string") {
             y.share.map.delete(keys);
@@ -285,19 +285,19 @@ class TransportYJS extends Transport {
         }
     }
 
-    async p_keys(url, verbose) {
+    async p_keys(url, {verbose=false}={}) {
         let y = await this.p_connection(url, verbose);
         return y.share.map.keys();   // Surprisingly this is sync, the p_connection should have synchronised
     }
-    async p_getall(url, verbose) {
+    async p_getall(url, {verbose=false}={}) {
         let y = await this.p_connection(url, verbose);
         let keys = y.share.map.keys();   // Surprisingly this is sync, the p_connection should have synchronised
-        return this._p_get(y, keys);
+        return this._p_get(y, keys. {verbose});
     }
     async p_rawfetch(url, {verbose=false}={}) {
         return { // See identical structure in TransportHTTP
             table: "keyvaluetable",         //TODO-KEYVALUE its unclear if this is the best way, as maybe want to know the real type of table e.g. domain
-            _map: await this.p_getall(url, verbose)
+            _map: await this.p_getall(url, {verbose})
         };   // Data struc is ok as SmartDict.p_fetch will pass to KVT constructor
     }
     async monitor(url, callback, verbose) {

Transports.js

@@ -1,5 +1,6 @@
 const Url = require('url');
 const errors = require('./Errors');
+const utils = require('./utils');
 
 /*
 Handles multiple transports, API should be (almost) the same as for an individual transport)
@@ -18,6 +19,9 @@ class Transports {
         return this._transports.filter((t) => (!t.status));
     }
     static connectedNames() {
+        /*
+        Return an array of the names of connected transports
+         */
         return this._connected().map(t => t.name);
     }
     static connectedNamesParm() {
@@ -58,23 +62,23 @@ class Transports {
         /* If and only if TransportNAME was loaded (it might not be as it depends on higher level classes like Domain and SmartDict)
             then resolve urls that might be names, returning a modified array.
          */
-        if (this.namingcb) {        //
+        if (this.namingcb) {
             return await this.namingcb(urls);  // Array of resolved urls
         } else {
             return urls;
         }
     }
-    static resolveNamesWith(cb) {
+    static resolveNamesWith(cb) { //TODO-API
         // Set a callback for p_resolveNames
         this.namingcb = cb;
     }
 
-    static async _p_rawstore(tt, data, verbose) {
+    static async _p_rawstore(tt, data, {verbose}) {
         // Internal method to store at known transports
         let errs = [];
         let rr = await Promise.all(tt.map(async function(t) {
             try {
-                return await t.p_rawstore(data, verbose); //url
+                return await t.p_rawstore(data, {verbose}); //url
             } catch(err) {
                 console.log("Could not rawstore to", t.name, err.message);
                 errs.push(err);
@@ -88,7 +92,7 @@ class Transports {
         return rr;
 
     }
-    static async p_rawstore(data, verbose) {
+    static async p_rawstore(data, {verbose}) {
         /*
         data: Raw data to store - typically a string, but its passed on unmodified here
         returns:    Array of urls of where stored
@@ -99,9 +103,9 @@ class Transports {
         if (!tt.length) {
             throw new errors.TransportError('Transports.p_rawstore: Cant find transport for store');
         }
-        return this._p_rawstore(tt, data, verbose);
+        return this._p_rawstore(tt, data, {verbose});
     }
-    static async p_rawlist(urls, verbose) {
+    static async p_rawlist(urls, {verbose=false}={}) {
         urls = await this.p_resolveNames(urls); // If naming is loaded then convert to a name
         let tt = this.validFor(urls, "list"); // Valid connected transports that support "store"
         if (!tt.length) {
@@ -110,7 +114,7 @@ class Transports {
         let errs = [];
         let ttlines = await Promise.all(tt.map(async function([url, t]) {
             try {
-                return await t.p_rawlist(url, verbose); // [sig]
+                return await t.p_rawlist(url, {verbose}); // [sig]
             } catch(err) {
                 console.log("Could not rawlist ", url, "from", t.name, err.message);
                 errs.push(err);
@@ -151,10 +155,10 @@ class Transports {
         for (const [url, t] of tt) {
             try {
                 let data = await t.p_rawfetch(url, opts);   // throws errors if fails or timesout
-                //TODO-MULTI-GATEWAY working here
+                //TODO-MULTI-GATEWAY working here - it doesnt quite work yet as the "Add" on browser gets different url than on server
                 if (opts.relay && failedtransports.length) {
                     console.log(`Relaying ${data.length} bytes from ${typeof url === "string" ? url : url.href} to ${failedtransports.map(t=>t.name)}`);
-                    this._p_rawstore(failedtransports, data, verbose)
+                    this._p_rawstore(failedtransports, data, {verbose})
                         .then(uu => console.log(`Relayed to ${uu}`)); // Happening async, not waiting and dont care if fails
                 }
                 //END TODO-MULTI-GATEWAY
@@ -170,7 +174,7 @@ class Transports {
         throw new errors.TransportError(errs.map((err)=>err.message).join(', '));  //Throw err with combined messages if none succeed
     }
 
-    static async p_rawadd(urls, sig, verbose) {
+    static async p_rawadd(urls, sig, {verbose=false}={}) {
         /*
         urls: of lists to add to
         sig: Sig to add
@@ -186,7 +190,7 @@ class Transports {
         let errs = [];
         await Promise.all(tt.map(async function([u, t]) {
             try {
-                await t.p_rawadd(u, sig, verbose); //undefined
+                await t.p_rawadd(u, sig, {verbose}); //undefined
                 return undefined;
             } catch(err) {
                 console.log("Could not rawlist ", u, "from", t.name, err.message);
@@ -211,12 +215,12 @@ class Transports {
             .map(([u, t]) => t.listmonitor(u, cb));
     }
 
-    static async p_newlisturls(cl, verbose) {
+    static async p_newlisturls(cl, {verbose=false}={}) {
         // Create a new list in any transport layer that supports lists.
         // cl is a CommonList or subclass and can be used by the Transport to get info for choosing the list URL (normally it won't use it)
         // Note that normally the CL will not have been stored yet, so you can't use its urls.
         let uuu = await Promise.all(this.validFor(undefined, "newlisturls")
-            .map(([u, t]) => t.p_newlisturls(cl, verbose)) );   // [ [ priv, pub] [ priv, pub] [priv pub] ]
+            .map(([u, t]) => t.p_newlisturls(cl, {verbose})) );   // [ [ priv, pub] [ priv, pub] [priv pub] ]
         return [uuu.map(uu=>uu[0]), uuu.map(uu=>uu[1])];    // [[ priv priv priv ] [ pub pub pub ] ]
     }
 
@@ -249,7 +253,7 @@ class Transports {
 
 // KeyValue support ===========================================
 
-    static async p_get(urls, keys, verbose) {
+    static async p_get(urls, keys, {verbose=false}={}) {
         /*
         Fetch the values for a url and one or more keys, transports act on the data, typically storing it.
         urls:	array of urls to retrieve (any are valid)
@@ -265,7 +269,7 @@ class Transports {
         let errs = [];
         for (const [url, t] of tt) {
             try {
-                return await t.p_get(url, keys, verbose); //TODO-MULTI-GATEWAY potentially copy from success to failed URLs.
+                return await t.p_get(url, keys, {verbose}); //TODO-MULTI-GATEWAY potentially copy from success to failed URLs.
             } catch (err) {
                 errs.push(err);
                 console.log("Could not retrieve ", url.href, "from", t.name, err.message);
@@ -274,7 +278,7 @@ class Transports {
         }
         throw new errors.TransportError(errs.map((err)=>err.message).join(', '));  //Throw err with combined messages if none succeed
     }
-    static async p_set(urls, keyvalues, value, verbose) {
+    static async p_set(urls, keyvalues, value, {verbose=false}={}) {
         /* Set a series of key/values or a single value
          keyvalues:    Either dict or a string
          value: if kv is a string, this is the value to set
@@ -289,7 +293,7 @@ class Transports {
         let success = false;
         await Promise.all(tt.map(async function([url, t]) {
             try {
-                await t.p_set(url, keyvalues, value, verbose);
+                await t.p_set(url, keyvalues, value, {verbose});
                 success = true; // Any one success will return true
             } catch(err) {
                 console.log("Could not rawstore to", t.name, err.message);
@@ -301,7 +305,7 @@ class Transports {
         }
     }
 
-    static async p_delete(urls, keys, verbose) {  //TODO-KEYVALUE-API
+    static async p_delete(urls, keys, {verbose=false}={}) {  //TODO-KEYVALUE-API
         /* Delete a key or a list of keys
          kv:    Either dict or a string
          value: if kv is a string, this is the value to set
@@ -316,7 +320,7 @@ class Transports {
         let success = false;
         await Promise.all(tt.map(async function([url, t]) {
             try {
-                await t.p_delete(url, keys, verbose);
+                await t.p_delete(url, keys, {verbose});
                 success = true; // Any one success will return true
             } catch(err) {
                 console.log("Could not rawstore to", t.name, err.message);
@@ -327,7 +331,7 @@ class Transports {
             throw new errors.TransportError(errs.map((err)=>err.message).join(', ')); // New error with concatenated messages
         }
     }
-    static async p_keys(urls, verbose) {
+    static async p_keys(urls, {verbose=false}={}) {
         /*
         Fetch the values for a url and one or more keys, transports act on the data, typically storing it.
         urls:	array of urls to retrieve (any are valid)
@@ -344,7 +348,7 @@ class Transports {
         let errs = [];
         for (const [url, t] of tt) {
             try {
-                return await t.p_keys(url, verbose); //TODO-MULTI-GATEWAY potentially copy from success to failed URLs.
+                return await t.p_keys(url, {verbose}); //TODO-MULTI-GATEWAY potentially copy from success to failed URLs.
             } catch (err) {
                 errs.push(err);
                 console.log("Could not retrieve keys for", url.href, "from", t.name, err.message);
@@ -354,7 +358,7 @@ class Transports {
         throw new errors.TransportError(errs.map((err)=>err.message).join(', '));  //Throw err with combined messages if none succeed
     }
 
-    static async p_getall(urls, verbose) {
+    static async p_getall(urls, {verbose=false}={}) {
         /*
         Fetch the values for a url and one or more keys, transports act on the data, typically storing it.
         urls:	array of urls to retrieve (any are valid)
@@ -371,7 +375,7 @@ class Transports {
         let errs = [];
         for (const [url, t] of tt) {
             try {
-                return await t.p_getall(url, verbose); //TODO-MULTI-GATEWAY potentially copy from success to failed URLs.
+                return await t.p_getall(url, {verbose}); //TODO-MULTI-GATEWAY potentially copy from success to failed URLs.
             } catch (err) {
                 errs.push(err);
                 console.log("Could not retrieve all keys for", url.href, "from", t.name, err.message);
@@ -381,25 +385,25 @@ class Transports {
         throw new errors.TransportError(errs.map((err)=>err.message).join(', '));  //Throw err with combined messages if none succeed
     }
 
-    static async p_newdatabase(pubkey, verbose) {
+    static async p_newdatabase(pubkey, {verbose=false}={}) {
         /*
             Create a new database in any transport layer that supports databases (key value pairs).
             pubkey: CommonList, KeyPair, or exported public key
             resolves to: [ privateurl, publicurl]
          */
         let uuu = await Promise.all(this.validFor(undefined, "newdatabase")
-            .map(([u, t]) => t.p_newdatabase(pubkey, verbose)) );   // [ { privateurl, publicurl} { privateurl, publicurl} { privateurl, publicurl} ]
+            .map(([u, t]) => t.p_newdatabase(pubkey, {verbose})) );   // [ { privateurl, publicurl} { privateurl, publicurl} { privateurl, publicurl} ]
         return { privateurls: uuu.map(uu=>uu.privateurl), publicurls: uuu.map(uu=>uu.publicurl) };    // { privateurls: [], publicurls: [] }
     }
 
-    static async p_newtable(pubkey, table, verbose) {
+    static async p_newtable(pubkey, table, {verbose=false}={}) {
         /*
             Create a new table in any transport layer that supports the function (key value pairs).
             pubkey: CommonList, KeyPair, or exported public key
             resolves to: [ privateurl, publicurl]
          */
         let uuu = await Promise.all(this.validFor(undefined, "newtable")
-            .map(([u, t]) => t.p_newtable(pubkey, table, verbose)) );   // [ [ priv, pub] [ priv, pub] [priv pub] ]
+            .map(([u, t]) => t.p_newtable(pubkey, table, {verbose})) );   // [ [ priv, pub] [ priv, pub] [priv pub] ]
         return { privateurls: uuu.map(uu=>uu.privateurl), publicurls: uuu.map(uu=>uu.publicurl)};    // {privateurls: [ priv priv priv ], publicurls: [ pub pub pub ] }
     }
 
@@ -423,6 +427,8 @@ class Transports {
             .map(([u, t]) => t.monitor(u, cb, verbose));
     }
 
+    // Setup and connection
+
     static addtransport(t) {
         /*
         Add a transport to _transports,
@@ -468,6 +474,69 @@ class Transports {
         // Does all setup1a before setup1b since 1b can rely on ones with 1a, e.g. YJS relies on IPFS
         await Promise.all(this._transports.map((t) => t.p_setup2(verbose, cb)))
     }
+
+    static async refreshstatus(t) {
+        let statusclasses = ["transportstatus0","transportstatus1","transportstatus2","transportstatus3","transportstatus4"];
+        let el = t.statuselement;
+        if (el) {
+            el.classList.remove(...statusclasses);
+            el.classList.add(statusclasses[t.status]);
+        }
+    }
+
+    static async p_connect(options, verbose) {
+        /*
+            This is a standardish starting process, feel free to subclass or replace !
+            It will connect to a set of standard transports and is intended to work inside a browser.
+            options = { defaulttransports: ["IPFS"]; statuselement: el }
+         */
+        if (verbose) console.group("p_connect ---");
+        try {
+            options = options || {};
+            let setupoptions = {};
+            let tabbrevs = options.transports;    // Array of transport abbreviations
+            if (!tabbrevs.length) { tabbrevs = options.defaulttransports || [] }
+            if (!tabbrevs.length) { tabbrevs = ["HTTP", "YJS", "IPFS", "WEBTORRENT"]; }
+            tabbrevs = tabbrevs.map(n => n.toUpperCase());
+            let transports = this.setup0(tabbrevs, options, verbose);
+            if (!!options.statuselement) {
+                while (statuselement.lastChild) {el.removeChild(el.lastChild); }   // Remove any exist status
+                statuselement.appendChild(
+                    utils.createElement("UL", {}, transports.map(t => {
+                            let el = utils.createElement("LI",
+                                {onclick: "this.source.togglePaused(DwebTransports.refreshstatus);", source: t},
+                                t.name);
+                            t.statuselement = el;   // Save status element on transport
+                            return el;
+                        }
+                    )));
+            }
+            await this.p_setup1(verbose, this.refreshstatus);
+            await this.p_setup2(verbose, this.refreshstatus);
+        } catch(err) {
+            console.error("ERROR in p_connect:",err.message);
+            throw(err);
+        }
+        if (verbose) console.groupEnd("p_connect ---");
+    }
+
+    static urlsFrom(url) {    //TODO backport to main repo - copy from htmlutils to utils
+        /* Utility to convert to urls form wanted for Transports functions, e.g. from user input
+        url:    Array of urls, or string representing url or representing array of urls
+        return: Array of strings representing url
+         */
+        if (typeof(url) === "string") {
+            if (url[0] === '[')
+                url = JSON.parse(url);
+            else if (url.includes(','))
+                url = url.split(',');
+            else
+                url = [ url ];
+        }
+        if (!Array.isArray(url)) throw new Error(`Unparsable url: ${url}`);
+        return url;
+    }
+
     static async test(verbose) {
         if (verbose) {console.log("Transports.test")}
         try {
@@ -478,7 +547,7 @@ class Transports {
             if (verbose) console.log("rawlist returned ", ...utils.consolearr(res));
             transport.listmonitor(testurl, (obj) => console.log("Monitored", obj), verbose);
             let sig = new Dweb.Signature({urls: ["123"], date: new Date(Date.now()), signature: "Joe Smith", signedby: [testurl]}, verbose);
-            await transport.p_rawadd(testurl, sig, verbose);
+            await transport.p_rawadd(testurl, sig, {verbose});
             if (verbose) console.log("TransportIPFS.p_rawadd returned ");
             res = await transport.p_rawlist(testurl, verbose);
             if (verbose) console.log("rawlist returned ", ...utils.consolearr(res)); // Note not showing return
@@ -488,23 +557,23 @@ class Transports {
             */
             //console.log("TransportYJS test complete");
             /* TODO-KEYVALUE reenable these tests,s but catch http examples
-            let db = await this.p_newdatabase("TESTNOTREALLYAKEY", verbose);    // { privateurls, publicurls }
+            let db = await this.p_newdatabase("TESTNOTREALLYAKEY", {verbose});    // { privateurls, publicurls }
             console.assert(db.privateurls[0] === "yjs:/yjs/TESTNOTREALLYAKEY");
-            let table = await this.p_newtable("TESTNOTREALLYAKEY","TESTTABLE", verbose);         // { privateurls, publicurls }
+            let table = await this.p_newtable("TESTNOTREALLYAKEY","TESTTABLE", {verbose});         // { privateurls, publicurls }
             let mapurls = table.publicurls;
             console.assert(mapurls[0] === "yjs:/yjs/TESTNOTREALLYAKEY/TESTTABLE");
-            await this.p_set(mapurls, "testkey", "testvalue", verbose);
+            await this.p_set(mapurls, "testkey", "testvalue", {verbose});
-            let res = await this.p_get(mapurls, "testkey", verbose);
+            let res = await this.p_get(mapurls, "testkey", {verbose});
             console.assert(res === "testvalue");
-            await this.p_set(mapurls, "testkey2", {foo: "bar"}, verbose);
+            await this.p_set(mapurls, "testkey2", {foo: "bar"}, {verbose});
-            res = await this.p_get(mapurls, "testkey2", verbose);
+            res = await this.p_get(mapurls, "testkey2", {verbose});
             console.assert(res.foo === "bar");
-            await this.p_set(mapurls, "testkey3", [1,2,3], verbose);
+            await this.p_set(mapurls, "testkey3", [1,2,3], {verbose});
-            res = await this.p_get(mapurls, "testkey3", verbose);
+            res = await this.p_get(mapurls, "testkey3", {verbose});
             console.assert(res[1] === 2);
             res = await this.p_keys(mapurls);
             console.assert(res.length === 3 && res.includes("testkey3"));
-            res = await this.p_getall(mapurls, verbose);
+            res = await this.p_getall(mapurls, {verbose});
             console.assert(res.testkey2.foo === "bar");
             */
 
@@ -516,7 +585,7 @@ class Transports {
 
 }
 Transports._transports = [];    // Array of transport instances connected
-Transports.namingcb = undefined;
+Transports.namingcb = undefined;    // Will be defined by the naming component (turns URLs for names into URLs for transport)
 Transports._transportclasses = {};  // Pointers to classes whose code is loaded.
 
 exports = module.exports = Transports;

example_block.html

@@ -2,15 +2,11 @@
 <!--suppress HtmlUnknownTarget -->
 <html lang="en">
 <head>
+    <!--Note there are copies of this in dweb-transports repo and dweb-transport repo-->
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     <title>DWEB Transports example page</title>
-
     <script type="text/javascript" src="./dist/dweb_transports_bundle.js"></script>
     <script type="text/javascript" src='https://cloud.tinymce.com/stable/tinymce.min.js'></script><!-- TinyMCE -->
-    <!--TODO What parts of example_styles.css, htmlutils.js and loginutils.js are needed ?-->
-    <link rel='stylesheet' href='example_styles.css'>
-    <script type="text/javascript" src="htmlutils.js"></script><!--objectbrowser functionality for debugging-->
-    <script type="text/javascript" src="loginutils.js"></script><!--Login tools (used for p_connect) -->
     <script type="text/javascript">
 
     tinymce.init({
@@ -26,7 +22,7 @@
             let content = tinyMCE.get('mytextarea').getContent();
             // alert(content);
             let el = document.getElementById("retrievalarea");
-            let urls = DwebTransports.p_rawstore(content, verbose)
+            let urls = DwebTransports.p_rawstore(content, {verbose})
                 .then((urls) => el.value = urls)
                 .catch((err) => {console.error("saveoncallback", err); alert(err)});
         }
@@ -34,8 +30,8 @@
 
     function fetchit() {
         let el = document.getElementById("retrievalarea");
-        let urls = urlsFrom(el.value);
+        let urls = DwebTransports.urlsFrom(el.value);
-        fetchanddisplay(urls);
+        fetchanddisplay(urls); //asynchronous
     }
 
     async function fetchanddisplay(urls) {
@@ -54,14 +50,13 @@
         }
     }
     async function main(url) {
-        //This is the "main()", starts a transport, checks its status, and if appropriate to app opens a URL passed
         try {
-            // p_connect will look in searchparams for e.g.: ?transport=HTTP&transport=WEBTORRENT
             // and if not found will use the defaulttransports specified here.
-            await p_connect({
+            await DwebTransports.p_connect({
-                defaulttransports: ["HTTP","IPFS"],
+                statuselement: document.getElementById("statuselement"),    // Where to build status indicator
-                statuselement: document.getElementById("statuselement")
+                defaulttransports: ["HTTP","IPFS"],                         // Default transports if not specified
-            }); // Default startup beavior from loginutils.js // {http: {urlbase: "http://localhost:4244"}} for local
+                transports: searchparams.getAll("transport")    // Allow override default from URL parameters
+            }, verbose);
             // Any code you want to run after connected to transports goes here.
             if (url) fetchanddisplay(url);
         } catch(err) {
@@ -72,18 +67,24 @@
 
 
     </script>
-    <script type="text/css">
+    <style type="text/css">
+        /* Application specific styles */
         .button { border: 1px black solid; background-color:#dddddd;  padding-bottom:0.1em; padding-top:0.1em;}
-    </script>
+        .propval {display:table-cell; background-color:#dddddd;border: 1px dotted grey; padding-left: 0.3em; padding-right: 0.3em;}
+        .displayedblock {border: 1px #e0e0e0 solid; background-color:#f0f0f0; box-shadow: 1px 1px 2px #cccccc; padding-top:0.1em; padding-bottom: 0.1em; margin-top: 10px; margin-bottom: 5px; display:inline-block;}
+        /* These are required for Dweb statuses */
+        #statuselement { margin: 0 0 3px 10px; float: right; clear: right; text-align: right; position: relative; }  /* "floatright" Grouped in top right corner */
+        #statuselement li {display: table-row; padding-top: 3px; padding-right: 0; margin: 0 0 3px 3px;}
+        .transportstatus0 {color: lawngreen} /*STATUS_CONNECTED*/
+        .transportstatus1 {color: red} /*STATUS_FAILED*/
+        .transportstatus2 {color: yellow} /*STATUS_STARTING*/
+        .transportstatus3 {color: black} /*STATUS_LOADED*/
+        .transportstatus4 {color: purple} /*STATUS_PAUSED*/
+    </style>
 </head>
 <body>
 <!--Network status box - this is standard for any app -->
-<!--TODO replace next div with following-->
+<div id="statuselement"></div>
-<div class="floatright" style="position:relative;">
-    <ul id="transportstatuses"><li class='template vertical_li' onclick="transportclick(this);"><span name='name'></span></li></ul>
-</div><!--End of standard network status and login panel-->
-
-<div class="floatright statuselement" style="position:relative;" id="statuselement"></div>
 <h1>Dweb - Transports Example - unstructured data store and retrieval</h1>
 <h4>Create something here, and when its saved its urls will appear below</h4>
 <form method="post">

example_styles.css

@@ -1,92 +0,0 @@
-/* Styles used for all the examples */
-
-/* Styles used for tables of properties, especially by (now obsolete) objbrowser.js*/
-.props { padding-left: 0; } /*Move back directly under type name*/
-.prop {display:table-row; border: 1px dotted grey; } /*ignoring border*/
-.propname {display:table-cell; font-weight:bold;background-color:#cccccc;border: 1px dotted grey; padding-left:0.3em; padding-right:0.3em} /*width:15em */
-.examplesacademic .propname {background-color:#f0f0f0; border: 0; }
-.propval {display:table-cell; background-color:#dddddd;border: 1px dotted grey; padding-left: 0.3em; padding-right: 0.3em;} /*width:15em;*/
-.propval span {white-space: pre;} /*Dont want white-space in e.g. img*/
-.proplinks {padding-left: 0.5em;} /*List of sub-objects inside propval*/
-.propobj {border-color: black; border-top: 2px solid; padding-bottom:0.3em; padding-top:0.3em; }
-.classname {font-weight: bold;} /* Used for class name in objbrowser */
-.propurls {display: table; padding: 0px;} /* Ignore ul padding default which is large */
-.propurl {display: table-row} /* propurl inside propurls */
-/*
-    Styles for appearing/disappearing parts based on hover or active
-    hoveronly appear only when they are active, hoverclue only when not active
-    the :active :hover :focus are to ensure it works on mobile devices as well
-*/
-.container form.hoveronly, .container div.hoveronly, .container span.hoveronly,.container i.hoveronly{display: none}
-.container:active span.hoveronly,.container:active span.hoveronly,.container:hover span.hoveronly{display: inline}
-.container:active i.hoveronly,.container:focus i.hoveronly,.container:hover i.hoveronly{display: inline-block}
-.container:active form.hoveronly,.container:focus form.hoveronly,.container:hover form.hoveronly{display: block}
-.container:active div.hoveronly,.container:focus div.hoveronly,.container:hover div.hoveronly{display: block}
-.container span.hoverclue{display: inline}/* Text in headline to disappear when main part open*/
-.container:active .hoverclue,.container:focus .hoverclue,.container:hover .hoverclue{display: none}
-
-/*
-   Some specific styles used by difference examples
- */
-.floatright {margin: 0 0 3px 10px; float: right; clear: right; text-align: right} /* Grouped in top right corner */
-.examplesarchive .floatright { position: fixed; z-index: 999999; top: 100px; right: 10px; font-size: 50% }
-.examplesarchive .floatright ul {display: grid;  grid-template-columns: repeat(auto-fit, minmax(7em, 1fr));  grid-auto-flow: dense;}
-.floatleft {margin: 0 0 3px 10px;  clear: left; float: left; text-align: left} /* Grouped in top left corner */
-.floatright li {margin: 0 0 3px 10px; }
-.floatright ul {margin: 0 0 3px 10px; }
-.examplesarchive .floatright ul { width: 150px;} /* Wide enogh for two indicators */
-.examplesarchive .floatright li { padding-top: 1px; margin: 0 0 3px 3px;} /* Wide enogh for two indicators */
-.vertical_li {display: table-row; padding-top: 3px; padding-right: 0; margin: 0 0 3px 3px;}
-.said {display: block; border-bottom: 1px black solid;} /* Each item added to list in example_list */
-.button { border: 1px black solid; background-color:#dddddd;  padding-bottom:0.1em;  display:inline;} /* Buttons */
-.displayedblock {border: 1px #e0e0e0 solid; background-color:#f0f0f0; box-shadow: 1px 1px 2px #cccccc; padding-top:0.1em; padding-bottom: 0.1em; margin-top: 10px; margin-bottom: 5px; display:inline-block;}
-.displayedblockheader {background-color:#e0e0e0; padding-top:0.1em; padding-bottom: 0.1em; margin-top: 3px; margin-bottom: 3px;}
-.dialogform .propval {white-space: pre; display:block; background-color:#dddddd;border: 1px dotted grey; padding-left: 0.3em; padding-top: 0.3em; padding-bottom: 0.1em; margin-top: 0.5em; margin-left: 0.5em; margin-right: 0.5em; margin-bottom: 0.5em;  }
-.dialogform .button { border: 1px black solid; background-color:#dddddd;  padding-bottom: 0.1em; padding-right: 0.3em; padding-left: 0.3em; margin-top: 0.5em; margin-left: 0.5em; margin-right: 0.5em; margin-bottom: 0.5em;}
-.iconopener {width:20px;}
-.inline_li {display:inline-grid; padding-left: 1em; padding-right: 1em;} /*A single login in a list of KeyChains or Keys */
-.inline_ul {display:inline; -webkit-padding-start: 0;} /*A List of KeyChains or Keys*/
-.keylist_icon {height: 12px; margin-left: 12px; margin-right: 3px;}
-.inline {display:inline;}
-.hidden {display:none}
-.overdiv {position: absolute; top: 1em; left: 1em; z-index:10; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; } /* Dialogue box over other*/
-.template {display: none}
-.tooltip { position: relative; display: inline-block; } /*border-bottom: 1px dotted black; */
-.tooltip .tooltiptext { visibility: hidden; z-index: 1;  opacity: 0; transition: opacity 1s;
-    width: 250px; background-color: #555; color: #fff; text-align: center; border-radius: 6px; padding: 5px 0;
-    position: absolute; bottom: 125%; left: 50%; margin-left: -125px;  }
-.tooltip .tooltiptext::after { content: ""; position: absolute; top: 100%; left: 50%;
-    margin-left: -5px; border-width: 5px; border-style: solid; border-color: #555 transparent transparent transparent;   }
-.tooltip:hover .tooltiptext { visibility: visible; opacity: 1; }
-.propval .tooltiptext {white-space: normal}
-
-/*
-    Academic playing around without effecting others
- */
-.examplesacademic .fileslist { text-align: center;}
-.academic_demo_img {display:table-cell; margin-right:10px;}
-.academic_demo_link {display: table-cell;}
-.academic_demo_a {display:table-row; text-align: left;}
-.academic_demo_meta {background-color:#f0f0f0; border: 0; vertical-align: middle; display:block; text-align:left; margin-bottom:10px; }
-.academic_demo_file {margin-bottom: 20px;}
-.examplesacademic #searchresults [name=title] {font-weight: bold;}
-.examplesacademic #metadataresults [name=doi_org_metadata_title] {font-weight: bold;}
-.examplesacademic #searchresults li {text-align: left; margin-bottom: 10px;}
-.examplesacademic #searchresults span {display: inline-block;}/*Try and keep each part together*/
-
-.exampleskeys .inline_ul {display: grid;  grid-template-columns: repeat(auto-fit, minmax(7em, 1fr));  grid-auto-flow: dense;}
-
-.examples #vl_target {display: block;}
-.examples #loginform {position: absolute; top: 0.5em; right: 1em; z-index:5; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; }
-.examples #registrationform {position: absolute; top: 1em; right: 1.5em; z-index:10; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; }/* z > loginform*/
-.examples #keychain {position: absolute; top: 0.5em; right: 1em; z-index:5; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; width: 450px;}
-.examples #keynew_form {position: absolute; top: 1em; right: 1.5em; z-index:7; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; width: 450px;}/*z > keychain*/
-.examples #locknew_form {position: absolute; top: 1em; right: 1.5em; z-index:7; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; width: 450px;}/*z > keychain*/
-.examples #lock_div {position: absolute; top: 0.5em; right: 1em; z-index:7; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; width: 450px;}/*z > keychain*/
-.examples #tokennew_form {position: absolute; top: 0.5em; right: 1em; z-index:10; background-color: #f0f0ff; border: 1px #e0e0e0 solid; padding: 5px; width: 450px; text-align:left;}/*z > lock_div*/
-
-.transportstatus0 {color: lawngreen}
-.transportstatus1 {color: red}
-.transportstatus2 {color: yellow}
-.transportstatus3 {color: black}
-.transportstatus4 {color: purple}

htmlutils.js

@@ -1,292 +0,0 @@
-/*
-    This file is a set of utility functions used in the manipulation of HTML pages
-    There is nothing specific to Dweb at all here, feel free to copy and modify.
- */
-
-
-function urlsFrom(url) {
-    /* Convert to urls,
-    url:    Array of urls, or string representing url or representing array of urls
-    return: Array of strings representing url
-     */
-    if (typeof(url) === "string") {
-        if (url[0] === '[')
-            url = JSON.parse(url);
-        else if (url.includes(','))
-            url = url.split(',');
-        else
-            url = [ url ];
-    }
-    if (!Array.isArray(url)) throw new Error(`Unparsable url: ${url}`);
-    return url;
-}
-
-function elementFrom(el) {
-    /* Make sure we have an element, if passed a string then find the element with that id.
-      el:       Element or string being the id of an element.
-      returns:  element.
-     */
-    return (typeof(el) === "string") ? document.getElementById(el) : el;
-}
-async function p_resolveobj(url) {
-    /*
-    Asynchronously find an object
-    url:    An object, or a url representing an object, or an array of urls.
-    returns: Object
-    throws: Error if can't resolve object
-    */
-    try {
-        if (typeof url === 'string')
-            url = [ url ];
-        if (Array.isArray(url))
-            url = await Dweb.SmartDict.p_fetch(url, verbose);
-        return url;
-    } catch(err) {
-        console.log("p_resolveobj: Cannot resolve",url);
-        throw err;
-    }
-}
-
-function form2dict(frm) {
-    /* Convert a form into a dictionary
-       its mindblowing that Javascript even at EC6 doesnt have this !
-   */
-    let res = {};
-    /* Find the element if we are given a string */
-    /* Note this is not the usual getElementById since forms have their own array */
-    let el_form = (typeof frm === "string") ? document.forms.namedItem(frm) : frm;
-    for (let el of el_form.elements) {
-        if (el.type !== "submit") {
-            res[el.name] = el.value;
-        }
-    }
-    return res;
-}
-
-function togglevis(el, displayvis) {
-    /*
-        Toggle the visibility of an item
-        el element, its id, or an array of elements
-        displayvis is one of "inline" "block" "inlineblock"
-     */
-    if (Array.isArray(el)) {
-        el.map((e) => togglevis(e, displayvis))
-    } else {
-        el = elementFrom(el);
-        el.style.display = (el.style && el.style.display === "none" ? displayvis : "none");
-    }
-}
-
-function setstatus(msg) {
-    // Display the message in a Status DIV (usually top right corner, but could be anywhere example wants it)
-    document.getElementById("status").innerHTML=msg;
-}
-
-function deletechildren(el, keeptemplate) {
-    /*
-    Remove all children from a node
-    :param el:  An HTML element, or a string with id of an HTML element
-    */
-    if (typeof keeptemplate === "undefined") keeptemplate=true;
-    el = (typeof(el) === "string") ? document.getElementById(el) : el;
-    // Carefull - this deletes from the end, because template if it exists will be firstChild
-    while (el.lastChild && !(keeptemplate && el.lastChild.classList && el.lastChild.classList.contains("template"))) {
-        // Note that deletechildren is also used on Span's to remove the children before replacing with text.
-        el.removeChild(el.lastChild);
-    }
-    return el; // For chaining
-}
-
-function replacetext(el, text) {
-    /* Replace the text of el with text, removing all other children
-    :param el:  An HTML element, or a string with id of an HTML element
-    */
-    el = elementFrom(el);
-    //console.log("replacetext",text,el.constructor.name) // Uncomment to get class name of different things want to edit
-    if (el instanceof HTMLImageElement) {
-        el.src = text;
-    } else {
-        deletechildren(el);
-        return el.appendChild(document.createTextNode(text))
-    }
-}
-
-function replacetexts(el, ...dict) {
-    /*
-    Replace the text of all inner nodes of el from the dict
-    Note this intentionally doesnt allow html as the values of the dict since probably from a network call and could be faked as "bad" html
-
-    Rules apply to el or any children.
-    <x name="yyy"></x>  Adds an inner text node with oo.yyy
-    <x name="aaa_bbb"></x> Replaces with ooo.aaa.bbb
-    <x name="aaa"><y name="bbb"></y></x> Replaces with ooo.aaa.bbb
-    <x name="aaa"><y class="template"/></x> Has y replecated withe each of ooo.aaa[]
-    <x value="ccc"></x>  Has value replaced with ooo.ccc (typically used for <option> etc inside <form>, arrays will be JSON stringified
-    <a href="ddd"></x> Has href replaced with ooo.ddd
-    <img name="eee"/> Has src replaced with ooo.eee (in replacetext)
-
-    This is intended to be easy to use, not necessarily bomb-proof, odd structures of HTML or oo may cause unpredictable results.
-
-    :param el:  An HTML element, or a string with id of an HTML element
-    :param dict: Multiple arguments, each a dictionary or object,  or a single array argument
-     */
-    // First combine with a raw dict so that "prop" doesnt get functions and handles dict like things
-    el = elementFrom(el);
-    if (Array.isArray(dict[0])) {
-        _replacetexts("", el, dict[0])
-    } else if (typeof dict[0] === "string") {   // Its just text string, no field name, so look for "_"
-        _replacetexts("", el, {"_": dict[0]});
-    } else {
-        el.source = dict[0];    // Usually used with one object, if append fields its usually just calculated for display
-        _replacetexts("", el, Object.assign({}, ...dict))
-    }
-    return el;
-}
-function _replacetexts(prefix, el, oo) {
-    /*
-    Inner function for replacetexts to allow crawling depth of oo
-     */
-    if (Array.isArray(oo)) {    // Add a templated element for each member of array
-        deletechildren(el);
-        oo.map((f) => addtemplatedchild(el, {}, f))
-    } else {
-        for (let prop in oo) {
-            try {
-                let p = prefix + prop;
-                let val = oo[prop];
-                if (val instanceof Date) {  // Convert here because otherwise treated as an object
-                    val = val.toString();
-                }
-                if (typeof val === "object" && !Array.isArray(val)) {
-                    // Look for current level, longer names e.g. prefixprop_xyz
-                    Array.prototype.slice.call(el.querySelectorAll(`[name=${p}]`)).map((i) => replacetexts(i, val) );
-                    //Commented out prefix version since runs into problems with complex nested objects such as "ipfs" in transports
-                    //_replacetexts(`${p}_`, el, val);
-                    // And nowif found any prefixprop look at xyz under it
-                    Array.prototype.slice.call(el.querySelectorAll(`[name=${p}]`)).map((i) => _replacetexts("", i, val));
-                }
-                else if (typeof val === "object" && Array.isArray(val)) {   // Exand an array into sub tags
-                    if (el.getAttribute("value") === p) el.value = JSON.stringify(val); //Do the parent as well if its e.g. an option, convert to something that urlsFrom will understand
-                    let dests = el.querySelectorAll(`[name=${p}]`);
-                    Array.prototype.slice.call(dests).map((i) => replacetexts(i, val));
-                } else {
-                    if (el.getAttribute("name") === p) replacetext(el, val); //Do the parent as well
-                    if (el.getAttribute("value") === p) el.value = val; //Do the parent as well
-                    Array.prototype.slice.call(el.querySelectorAll(`[name=${p}]`)).map((i) => replacetext(i, val));  // <span name="text">...val...</span>
-                    if (el.getAttribute("href") === p) el.href = val;
-                    Array.prototype.slice.call(el.querySelectorAll(`[href=${p}]`)).map((i) => i.href = val);
-                    Array.prototype.slice.call(el.querySelectorAll(`[value=${p}]`)).map((i) => i.value = val);
-                }
-            } catch(err) {
-                if (verbose) console.log("Unable to _replacetexts on",prefix,prop,err.message); // Continue loop its probably just some property of some nested object
-            }
-        }
-    }
-}
-
-function addtemplatedchild(el, {templatename="template"}={}, ...dict) {
-    /*
-    Standardised tool to add fields to html,  add that as the last child (or children) of el
-    The slightly convulated way of doing this is because of the limited set of functions available
-    Note this has to be done with care, as "dict" may be user supplied and contain HTML or other malicious content
-
-    el: An HTML element, or a string with the id of one.
-    html: html to add under outerelement
-    dict: Dictionary with parameters to replace in html, it looks for nodes with name="xyz" and replaces text inside it with dict[xyz]
-    */
-    el = elementFrom(el);
-    let el_li = el.getElementsByClassName(templatename)[0].cloneNode(true);   // Copy first child with class=Template
-    el_li.classList.remove("template");                                 // Remove the "template" class so it displays
-    replacetexts(el_li, ...dict);                          // Safe since only replace text - sets el_li.source to dict
-    el.appendChild(el_li);
-    return el_li;
-}
-
-function show(el, displayvalue) {
-    displayvalue = displayvalue || "";
-    if (Array.isArray(el)) el.map((e) => show(e, displayvalue));
-    elementFrom(el).style.display = displayvalue;
-}
-
-function hide(el) {
-    if (Array.isArray(el)) el.map((e) => hide(e));
-    elementFrom(el).style.display = "none";
-}
-
-async function p_httpget(url, headers) {
-    //https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch
-    /* Simple Get of a URL, resolves to either json or text depending on mimetype */
-    h = new Headers( headers ? headers : {} );
-    try {
-        let response = await fetch(new Request(url, {
-            method: 'GET',
-            headers: h,
-            mode: 'cors',
-            cache: 'default',
-            redirect: 'follow',  // Chrome defaults to manual
-        })); // A promise, throws (on Chrome, untested on Ffox or Node) TypeError: Failed to fetch)
-        if (response.ok) {
-            if (response.headers.get('Content-Type') === "application/json") {  // It should always be JSON
-                return response.json(); // promise resolving to JSON
-            } else {
-                return response.text(); // promise resolving to text
-            }
-        }
-        throw new Error(`Transport Error ${response.status}: ${response.statusText}`); // Should be TransportError but out of scope
-    } catch(err) {
-            // Error here is particularly unhelpful - if rejected during the COrs process it throws a TypeError
-            console.log("Probably misleading error from fetch:", url, err);
-            throw new Error(`Transport error thrown by ${url}`)
-    }
-}
-
-function display_blob(bb, options) {
-    /*
-        Display a blob of data
-        Typical usage display_blob(await Dweb.Transports.p_rawfetch(urls), {type: "application/pdf"})
-        bb: Data to display, either as blob or something that can be passed to Blob([bb]) e.g. a buffer
-        options:    {
-            type: mimetype  (required if bb not already a blob)
-            target:     Where to display e.g. "_blank" or "_self"
-        }
-        TODO probably extend this to do a download which has some code in archive/*js to handle
-     */
-    // See https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL
-    // and https://stackoverflow.com/questions/3665115/create-a-file-in-memory-for-user-to-download-not-through-server
-    if (!(bb instanceof Blob)) {
-        console.log("display_blob: creating with type",options.type);
-        bb = new Blob([bb], {type: options.type})
-    }
-    console.log("display_blob:",typeof bb);
-    // This next code is bizarre combination needed to open a blob from within an HTML window.
-    let a = window.document.createElement('a');
-    //bb = new Blob([datapdf], {type: 'application/pdf'});
-    let objectURL = URL.createObjectURL(bb);
-    a.href = objectURL;
-    a.target= (options && options.target) || "_blank";                      // Open in new window by default
-    document.body.appendChild(a);
-    a.click();
-    document.body.removeChild(a);
-    //URL.revokeObjectURL(objectURL)    //TODO figure out when can do this - maybe last one, or maybe dont care?
-}
-
-//------- For dealing with MCE editor ----------
-
-function seteditor(content) {
-    tinyMCE.activeEditor.setContent(content);   // Set actual MCE editing text
-    tinyMCE.activeEditor.setDirty(true);        // Allow saving this restored text
-}
-
-function starteditor() {
-    //TODO maybe add some options that can override fields if needed (e.g. with a Object.assign
-    tinymce.init({
-        selector: '#mytextarea',
-        menubar: "true",
-        plugins: [ "save",
-            'advlist autolink lists link image charmap print preview anchor',
-            'searchreplace visualblocks code fullscreen',
-            'insertdatetime media table contextmenu paste code' ],
-        toolbar: 'save | undo redo | insert | styleselect | bold italic | alignleft aligncenter alignright alignjustify | bullist numlist outdent indent | link image',
-        save_onsavecallback: () => p_updatecontent(tinyMCE.get('mytextarea').getContent())  // This function must be provided
-    });
-}

loginutils.js

@@ -1,272 +0,0 @@
-/*
-    A set of common scripts for use in Dweb code to add key management and login functionality
-    See example_versions.html for some usage and example_keys.html for instructions.
-
-    Requires Dweb to point to top level (e.g. by loading src="dweb_bundled.js"
-
-    keychains_ul should be something like:
-    <ul id="keychains_ul"><li class='template vertical_li' onclick='keychain_click(this);'><span name='name'>PLACEHOLDER</span></li></ul>
-
-    Needs a login and registration HTML
-
-    Naming conventions:
-
-
- */
-
-
-// Array of images can use
-const icon_images = {   //!SEE-OTHER-KC-CLASSES
-    acl: "noun_1093404_cc.png",
-    kp: "noun_1146472_cc.png",
-    "tok": "noun_708669_cc.png",
-    "vl": "log.gif",    // TODO replace by better icon
-    "locked": "noun_1093404_cc.png",
-    "unlocked": "noun_1093404_cc_unlocked.png",
-}; //If change here - see also Keys on KeyChain code below
-
-function logout_click() {
-    /* Logout button clicked  - logged out*/
-    Dweb.KeyChain.logout();         // Empty keychains
-    deletechildren("keychains_ul");    // Delete any visible children
-    hide('logout');                 // Nothing remains to logout so hide button
-}
-
-async function p_login(dict) {
-    /* common routine to login someone by name and passphrase, and display in KeyChains list - note
-    :param dict: {name: passprase: } or name of form containing it.
-    :resolves to: undefined
-     */
-    // concatenate them so that a common passphrase isn't sufficient to guess an id.
-    if (typeof(dict) === "string") dict = form2dict(dict);
-    try {
-        let passphrase = dict.name + "/" + dict.passphrase;
-        let kc = await Dweb.KeyChain.p_new({name: dict.name}, {passphrase: passphrase}, verbose);
-        addtemplatedchild("keychains_ul", {}, kc);      // returns el, but unused
-        show('logout');                             // And show the logout button
-    } catch(err) {
-        console.log("Unable to _login",err);
-        alert(err);
-    }
-}
-
-async function registrationsubmit() {
-    /* User has filled in and submitted the registration button */
-    if (verbose) console.group("p_registrationsubmit ---");
-    hide('registrationform');                                // Hide after submission
-    await p_login("registrationform");         // { name, passphrase }
-    if (verbose) console.groupEnd("p_registrationsubmit ---");
-}
-
-async function loginformsubmit() {
-    /* Login button clicked - User has logged in */
-    // At the moment this is identical behavior to p_registrationsubmit, but that could change
-    //TODO - check if user already exists, require "registration if not
-    if (verbose) console.group("loginformsubmit ---");
-    hide('loginform');                           // Hide after submission
-    await p_login("loginform");    // { name, passphrase }
-    if (verbose) console.groupEnd("loginformsubmit -GROUP--");
-}
-
-function _showkeyorlock(el, obj) {
-    // Utility function to add new or existing element to Key List
-    addtemplatedchild(el, {}, obj, {objsym: icon_images[obj.table === "sd" && obj.token ? "tok" : obj.table ] });
-}
-
-function keychain_click(el) {
-    /* Click on a KeyChain i.e. on a login - display list of Keys and Locks for that login */
-    let kc = el.source;                                         // Find KeyChain clicked on
-    show('keychain');                                               // Show the div 'keys' for showing keylist
-    let el_keychain_header = replacetexts("keychain_header", kc);  // Set name fields etc in keylistdiv, sets source - dont set in keychain as will get list wrong
-    deletechildren("keychain_ul");                               // Delete any locks or eys currently displayed
-    kc.addEventListener("insert", (event) => {                  // Setup a listener that will trigger when anything added to the list and update HTML
-        if (verbose) console.log("keychain.eventlistener",event);
-        let sig = event.detail;
-        if (Dweb.utils.intersects(kc._publicurls, el_keychain_header.source._publicurls))   // Check its still this KeyChain being displayed in keylist
-            sig.p_fetchdata({verbose})                            // Get the data from a sig, its not done automatically as in other cases could be large
-                .then((obj) => _showkeyorlock("keychain_ul", obj))             // Show on the list
-    });
-    kc.p_list_then_elements({verbose, ignoreerrors: true})                            // Retrieve the keys for the keylist - ignore any cant decrypt
-        .then(() => kc._keys.map((key)=> _showkeyorlock("keychain_ul", key)));  // And add to the HTML
-}
-async function kcitem_click(el) { //!SEE-OTHER-KC-CLASSES
-    // Clicked on a key or a lock, determine which and forward
-    el = elementFrom(el);
-    let obj = el.source;
-    if (obj instanceof Dweb.AccessControlList)
-        await p_lock_click(el);
-    else if (obj instanceof Dweb.KeyPair)
-        key_click(el);
-    else if (obj instanceof Dweb.VersionList)
-        await p_versionlist_click(el);
-    else if ((obj instanceof Dweb.SmartDict) && obj.token)  // Its a token - like a key
-        token_click(el);
-     else
-        throw new errors.ToBeImplementedError(`kcitem_click doesnt support ${obj.constructor.name}`)
-}
-
-//!SEE-OTHER-KC-CLASSES
-
-// Clicked on a key, display a prompt to copy it for sharing
-function locklink_click(el) {
-    window.prompt("Copy to clipboard for locking (Ctrl-C + OK)", elementFrom(el).source._urls);
-}
-function key_click(el) {
-    window.prompt("Copy to clipboard for sharing (Ctrl-C + OK)", elementFrom(el).source._publicurls);
-}
-function token_click(el) {
-    window.prompt("Copy to clipboard for sharing (Ctrl-C + OK)", elementFrom(el).source.viewer);
-}
-async function p_versionlist_click(el) {
-    // If there is a vl_target element then use it - e.g. for editing a VersionList, otherwise offer dialog to copy the URL
-    let target = elementFrom("vl_target");
-    if (target) {
-        await p_vl_target_display(target, elementFrom(el).source);    // Application dependent
-    } else {
-        window.prompt("Copy to clipboard for sharing (Ctrl-C + OK)", elementFrom(el).source._urls);  // In some cases should load form
-    }
-}
-
-async function keynew_click() {
-    if (verbose) console.group("keynew_click ---");
-    hide('keynew_form');
-    let dict = form2dict("keynew_form"); //name
-    let keychain = document.getElementById('keychain_header').source;   // Keychain of parent of this dialog
-    let key = new Dweb.KeyPair({name: dict.name, key: {keygen: true}, _acl: keychain}, verbose ); // Doesnt store
-    _showkeyorlock("keychain_ul", key);   // Put in UI, as listmonitor response will be deduplicated.
-    await keychain.p_push(key, verbose);    // Will store on the way
-    if (verbose) console.groupEnd("keynew_click ---");
-}
-
-async function locknew_click() {
-    if (verbose) console.group("locknew_click ---");
-    hide('locknew_form');
-    let dict = form2dict("locknew_form"); //name
-    let keychain = document.getElementById('keychain_header').source;  // The KeyChain being added to.
-    let res = await Dweb.AccessControlList.p_new({name: dict.name, _acl: keychain}, true, {keygen: true}, verbose, null, keychain )    //(data, master, key, verbose, options, kc)
-        .then((acl) => _showkeyorlock("keychain_ul", acl)); // Put in UI, as listmonitor return rejected as duplicate
-    if (verbose) console.groupEnd("locknew_click ---");
-    return res;
-}
-
-async function p_lock_click(el) {
-    if (verbose) console.group("p_lock_click ---");
-    let acl = el.source;                                    // The ACL clicked on
-    show('lock_div');                                     // Show the HTML with a list of tokens in ACL
-    let el_lockheader = replacetexts("lock_header", acl);     // Set name fields etc in keylistdiv, sets source
-    deletechildren("lock_ul");                               // Remove any existing HTML children
-    try {
-        let toks = await acl.p_tokens();                                       // Retrieve the keys for the keylist
-        toks.map((tok) => _showkeyorlock("lock_ul", tok));   // And add to the HTML
-        acl.addEventListener("insert", (event) => {                  // Setup a listener that will trigger when anything added to the list and update HTML
-            if (verbose) console.log("lock.eventlistener",event);
-            let sig = event.detail;
-            if (Dweb.utils.intersects(acl._publicurls, el_lockheader.source._publicurls))  // Check its still this ACL being displayed in keylist
-                sig.p_fetchdata({verbose})                    // Get the data from a sig, its not done automatically as in other cases could be large
-                    .then((tok) => _showkeyorlock("lock_ul", tok))           // Show on the list
-        });
-    } catch(err) {
-        console.log("p_lock_click: failed",err.message);
-        throw err;
-    }
-    if (verbose) console.groupEnd("p_lock_click ---");
-}
-
-async function tokennew_click() { //Called by "Add" button on new token dialog
-    //TODO this allows duplicates, shouldnt add if viewer matches
-    if (verbose) console.group("tokennew_click ---");
-    hide('tokennew_form');
-    let dict = form2dict("tokennew_form"); //url
-    let acl = document.getElementById('lock_header').source;
-    let tok = await acl.p_add_acle(urlsFrom(dict.urls), {name: dict["name"]}, verbose);
-    _showkeyorlock("lock_ul", tok) // Push to visual list, as listmonitor will be a duplicate
-    if (verbose) console.groupEnd("tokennew_click ---");
-}
-
-
-/* Need stuff below this line */
-
-function createElement(tag, attrs, children) {        // Note arguments is set to tag, attrs, child1, child2 etc
-    var element = document.createElement(tag);
-    for (let name in attrs) {
-        let attrname = (name.toLowerCase() === "classname" ? "class" : name);
-        if (name === "dangerouslySetInnerHTML") {
-            element.innerHTML = attrs[name]["__html"];
-            delete attrs.dangerouslySetInnerHTML;
-        }
-        if (attrs.hasOwnProperty(name)) {
-            let value = attrs[name];
-            if (value === true) {
-                element.setAttribute(attrname, name);
-            } else if (typeof value === "object" && !Array.isArray(value)) { // e.g. style: {{fontSize: "124px"}}
-                if (["style"].includes(attrname)) {
-                    for (let k in value) {
-                        element[attrname][k] = value[k];
-                    }
-                } else {
-                    // Assume we are really trying to set the value to an object, allow it
-                    element[attrname] = value;  // Wont let us use setAttribute(attrname, value) unclear if because unknow attribute or object
-                }
-            } else if (value !== false && value != null) {
-                element.setAttribute(attrname, value.toString());
-            }
-        }
-    }
-    for (let i = 2; i < arguments.length; i++) { // Everything after attrs
-        let child = arguments[i];
-        if (!child) {
-        } else if (Array.isArray(child)) {
-            child.map((c) => element.appendChild(c.nodeType == null ?
-                document.createTextNode(c.toString()) : c))
-        }
-        else {
-            element.appendChild(
-                child.nodeType == null ?
-                    document.createTextNode(child.toString()) : child);
-        }
-    }
-    return element;
-}
-
-async function p_connect(options) {
-    /*
-        This is a standardish starting process, feel free to copy and reuse !
-        options = { defaulttransports: ["IPFS"]; statuselement: el }
-     */
-    if (verbose) console.group("p_connect ---");
-    try {
-        options = options || {};
-        let setupoptions = {};
-        let tabbrevs = searchparams.getAll("transport");    // Array of transports
-        if (!tabbrevs.length) { tabbrevs = options.defaulttransports || [] }
-        if (!tabbrevs.length) { tabbrevs = ["HTTP", "YJS", "IPFS", "WEBTORRENT"]; }
-        tabbrevs = tabbrevs.map(n => n.toUpperCase());
-        let transports = DwebTransports.setup0(tabbrevs, options, verbose);
-        while (statuselement.lastChild) { el.removeChild(el.lastChild); }   // Remove any exist status
-        statuselement.appendChild(
-            createElement("UL",{}, transports.map(t => {
-                el = createElement("LI", {onclick: "transportclick(this);", source: t}, t.name)
-                t.statuselement = el;   // Save status element on transport
-                return el;
-            }
-        )));
-        await DwebTransports.p_setup1(verbose, refreshstatus);
-        await DwebTransports.p_setup2(verbose, refreshstatus);
-    } catch(err) {
-        console.error("ERROR in p_connect:",err.message);
-        throw(err);
-    }
-    if (verbose) console.groupEnd("p_connect ---");
-}
-
-function refreshstatus(t) {
-    statusclasses = ["transportstatus0","transportstatus1","transportstatus2","transportstatus3","transportstatus4"];
-    let el = t.statuselement;
-    el.classList.remove(...statusclasses);
-    el.classList.add(statusclasses[t.status]);
-}
-function transportclick(el) { // Similar routines to this could display status, toggle activity etc.
-    t = el.source;
-    console.log("Clicked on Transport",t.name);
-    t.togglePaused(refreshstatus);
-}

utils.js

@@ -7,72 +7,6 @@ utils = {}; //utility functions
 // Utility function to print a array of items but just show number and last.
 utils.consolearr  = (arr) => ((arr && arr.length >0) ? [arr.length+" items inc:", arr[arr.length-1]] : arr );
 
-/*
-//Return true if two shortish arrays a and b intersect or if b is not an array, then if b is in a
-//Note there are better solutions exist for longer arrays
-//This is intended for comparing two sets of probably equal, but possibly just intersecting URLs
-utils.intersects = (a,b) =>  (Array.isArray(b) ? a.some(x => b.includes(x)) : a.includes(b));
-
-
-utils.mergeTypedArraysUnsafe = function(a, b) { // Take care of inability to concatenate typed arrays such as Uint8
-    //http://stackoverflow.com/questions/14071463/how-can-i-merge-typedarrays-in-javascript also has a safe version
-    const c = new a.constructor(a.length + b.length);
-    c.set(a);
-    c.set(b, a.length);
-    return c;
-};
-*/
-/*
-//TODO-STREAM, use this code and return stream from p_rawfetch that this can be applied to
-utils.p_streamToBuffer = function(stream, verbose) {
-    // resolve to a promise that returns a stream.
-    // Note this comes form one example ...
-    // There is another example https://github.com/ipfs/js-ipfs/blob/master/examples/exchange-files-in-browser/public/js/app.js#L102 very different
-    return new Promise((resolve, reject) => {
-        try {
-            let chunks = [];
-            stream
-                .on('data', (chunk) => { if (verbose) console.log('on', chunk.length); chunks.push(chunk); })
-                .once('end', () => { if (verbose) console.log('end chunks', chunks.length); resolve(Buffer.concat(chunks)); })
-                .on('error', (err) => { // Note error behavior untested currently
-                    console.log("Error event in p_streamToBuffer",err);
-                    reject(new errors.TransportError('Error in stream'))
-                });
-            stream.resume();
-        } catch (err) {
-            console.log("Error thrown in p_streamToBuffer", err);
-            reject(err);
-        }
-    })
-};
-*/
-/*
-//TODO-STREAM, use this code and return stream from p_rawfetch that this can be applied to
-//TODO-STREAM debugging in streamToBuffer above, copy to here when fixed above
-utils.p_streamToBlob = function(stream, mimeType, verbose) {
-    // resolve to a promise that returns a stream - currently untested as using Buffer
-    return new Promise((resolve, reject) => {
-        try {
-            let chunks = [];
-            stream
-                .on('data', (chunk)=>chunks.push(chunk))
-                .once('end', () =>
-                    resolve(mimeType
-                        ? new Blob(chunks, { type: mimeType })
-                        : new Blob(chunks)))
-                .on('error', (err) => { // Note error behavior untested currently
-                    console.log("Error event in p_streamToBuffer",err);
-                    reject(new errors.TransportError('Error in stream'))
-                });
-            stream.resume();
-        } catch(err) {
-            console.log("Error thrown in p_streamToBlob",err);
-            reject(err);
-        }
-    })
-};
-*/
-
 utils.stringfrom = function(foo, hints={}) {
     try {
         // Generic way to turn anything into a string
@@ -85,17 +19,7 @@ utils.stringfrom = function(foo, hints={}) {
         throw new errors.CodingError(`Unable to turn ${foo} into a string ${err.message}`)
     }
 };
-/*
-utils.objectfrom = function(data, hints={}) {
-    // Generic way to turn something into a object (typically expecting a string, or a buffer)
-    return (typeof data === "string" || data instanceof Buffer) ? JSON.parse(data) : data;
-}
 
-utils.keyFilter = function(dic, keys) {
-    // Utility to return a new dic containing each of keys (equivalent to python { dic[k] for k in keys }
-    return keys.reduce(function(prev, key) { prev[key] = dic[key]; return prev; }, {});
-}
-*/
 utils.p_timeout = function(promise, ms, errorstr) {
     /* In a certain period, timeout and reject
     promise:    A promise we want to watch to completion
@@ -115,6 +39,48 @@ utils.p_timeout = function(promise, ms, errorstr) {
     ]);
 }
 
-/*TODO backport these to utils.js in main repo */
+utils.createElement = function(tag, attrs, children) {        // Note arguments is set to tag, attrs, child1, child2 etc
+    // Note identical version in dweb-transport/js/utils.js and dweb-transports/utils.js
+    var element = document.createElement(tag);
+    for (let name in attrs) {
+        let attrname = (name.toLowerCase() === "classname" ? "class" : name);
+        if (name === "dangerouslySetInnerHTML") {
+            element.innerHTML = attrs[name]["__html"];
+            delete attrs.dangerouslySetInnerHTML;
+        }
+        if (attrs.hasOwnProperty(name)) {
+            let value = attrs[name];
+            if (value === true) {
+                element.setAttribute(attrname, name);
+            } else if (typeof value === "object" && !Array.isArray(value)) {
+                if (["style"].includes(attrname)) {  // e.g. style: {fontSize: "124px"}
+                    for (let k in value) {
+                        element[attrname][k] = value[k];
+                    }
+                } else {
+                    // Assume we are really trying to set the value to an object, allow it
+                    element[attrname] = value;  // Wont let us use setAttribute(attrname, value) unclear if because unknow attribute or object
+                }
+            } else if (value !== false && value != null) {
+                element.setAttribute(attrname, value.toString());
+            }
+        }
+    }
+    for (let i = 2; i < arguments.length; i++) { // Everything after attrs
+        let child = arguments[i];
+        if (!child) {
+        } else if (Array.isArray(child)) {
+            child.map((c) => element.appendChild(c.nodeType == null ?
+                document.createTextNode(c.toString()) : c))
+        }
+        else {
+            element.appendChild(
+                child.nodeType == null ?
+                    document.createTextNode(child.toString()) : child);
+        }
+    }
+    return element;
+}
+
 
 exports = module.exports = utils;