js-IPFS 0.50.0 runs in shared webworkers and has faster pinning

by Alex Potsides on 2020-09-14

πŸ”¦ Highlights

share an IPFS node between multiple tabs and pin files more quickly

js-IPFS@0.50.0 has landed with the ability to share a node between multiple browser tabs and greatly improved pinning performance.

We’re also phasing out use of Node.js Buffers as data types in favour of standard JavaScript Uint8Arrays.

Read on for the full details!

🀝 Share a node between browser tabs

An IPFS node makes lots of connections to other nodes on the network, and more so since delegate nodes were turned on by default. This is to ensure you have the greatest chance of finding content on the network, and so other people have the greatest chance of finding your content on the network.

This does not come without a price though, maintaining multiple connections can be resource-intensive and in some cases the browser will limit the number of concurrent connections you can have have.

This can be a problem in web browsers if the user opens your app in two tabs, suddenly you have two nodes running with twice the number of open connections. Worse, they are sharing a datastore and the same peer ID.

Help is at hand in the shape of the ipfs-message-port-client and ipfs-message-port-server which allow you to run one IPFS node in a SharedWorker and share it between multiple tabs within your application.

There will be a more in depth post here on this subject soon but in the meantime check out the browser-sharing-node-across-tabs example to see how to use it!

πŸ“Œ Pinning performance

When you add a piece of content to your local IPFS node, it’s pinned in place to prevent the blocks that make up your files being deleted during garbage collection. The pin is placed in a collection of pins we call a pinset.

The datastructure behind this pinset is a DAG, much like the structures that represent the files and folders you’d added to IPFS. The root CID of the DAG is stored in the datastore and all the blocks that make up the DAG are stored in the blockstore.

The pinset consists of a number of buckets in a tree structure, with each bucket containing a max of 8,192 items and each layer containing max 256 buckets. After the first bucket is full, pins are distributed between buckets.

When garbage collection runs, all nodes in the DAG are traversed and the blocks that correspond to their CIDs are exempted from deletion.

As you add and remove pins, this DAG grows and shrinks. CIDs of intermediate nodes within the DAG are recalculated as the structure changes. As the DAG gets larger this can become expensive and it hurts application performance for very large pinsets.

js-ipfs@0.50.0 has changed the default storage of pins to use the datastore instead of a DAG and has seen a corresponding speedup as the number of pinned blocks in your repo increases:

In the diagram above you can see that as the number of pinned items increases, so does the time it takes to add the next pin. There’s a steep increase at 8,192 pins, which is when the first bucket is considered full and multiple buckets are created which then involves more operations to add the next pin.

The performance of the approach taken by js-ipfs@0.50.0 compares very favourably to that of previous versions and is essentially only limited by the peformance of the underlying datastore since it has switched to simple puts and gets without the overhead of creating a data structure.

🍫 Uint8Arrays

In the beginning there were Arrays. Simple arrays that could hold all sorts of mixed types, could not be optimised very well and were an abstraction over blocks of memory.

Then Node.js came along and introduced the Buffer - suddenly JavaScript developers could access memory (sort of) directly! These things held numbers with an integer value range of 0-255 and were blazingly fast. JavaScript was starting to look like a proper language that you could do resource intensive work in.

The authors of the ECMAScript standard took note and introduced TypedArrays, of which there are many variations but the one we are most interested in is the Uint8Array.

These types of arrays hold numbers with an integer value range of 0-255 and support an API very similar to that of Node.js Buffers, which should come as no surprise as since v3.0.0 of Node.js, Buffers have been a subclass of Uint8Arrays.

These are exciting times so going forwards as JavaScript becomes more capable and the browser support arrives we are reducing our dependence on core Node.js libraries and utilities. Part of that is removing all use of Node.js Buffers in our codebase.

With js-ipfs@0.50.0 you should stop relying on Node.js Buffers to be returned from any part of the Core-API, instead you should code against the Uint8Array interface.

Some modules that we depend on will still return Buffers, which we pass on to avoid any conversion cost but we hope to remove or refactor these over time. In order to remain forwards-compatible you should not use Node.js Buffer methods on any of these returned values.

For example in the code below we create a Buffer from the string ‘Hello’, add it to IPFS then immediately cat it and call toString() on the chunks. This takes advantage of the fact that the Buffer we’ve added is utf8 encoded. Buffer.toString() takes an encoding argument which is utf8 by default, so the below code works but it’s only by coincidence:

const { cid } = await ipfs.add(Buffer.from('Hello'))

for await (const chunk of ipfs.cat(cid)) {
  console.info(chunk.toString()) // prints 'Hello'
}

Instead we would use the TextEncoder and TextDecoder classes. These are explicit in the encoding/decoding they use (also utf8 by default) so are safer to use:

const encoder = new TextEncoder()
const decoder = new TextDecoder()

const { cid } = await ipfs.add(encoder.encode('Hello'))

for await (const chunk of ipfs.cat(cid)) {
  console.info(decoder.decode(chunk)) // prints 'Hello'
}

✨New features

πŸ”¨ Breaking changes

πŸ— API Changes

Core API & HTTP API Client

For further reading, see the Core API Docs.

πŸ—ΊοΈ What’s next?

Check out the js-IPFS Project Roadmap which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

😍 Huge thank you to everyone that made this release possible

πŸ™ŒπŸ½ Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

Comments