js-IPFS 0.48.0 released with connectivity improvements and smaller blockstore

by Alex Potsides on 2020-07-20

πŸ”¦ Highlights

delegates turned on by default, smaller, faster blockstore and a more intuitive API

js-IPFS@0.48.0 is hot off the press with much better default connectivity, a smaller blockstore and a more intuitive API

🧭 Delegate nodes on by default

JS IPFS has traditionally primarily targeted the browser, and the browser is a bad place to be if you want to be on the DHT. You typically aren’t on a page long enough to make or respond to DHT queries, nor are you diallable, so even if you were able to advertise yourself as a provider for a given block, the chances are no-one can connect to you to retrieve that block which results in a degraded service for everyone. Worse, the way you find more peers and content is via the DHT so you’re kind of stuck.

There are several ways to give in-browser IPFS nodes a better experience on the network, one of those is Delegate Nodes. A Delegate Node is a network peer that performs certain actions on behalf of other nodes on the network. In this case it will make DHT queries on our behalf so we can find more peers and more content than ever before.

js-IPFS@0.48.0 enables delegate nodes in the configuration by default, which means you should see far more peers than you have previously and be able to find content faster and more reliably.

By default it uses public delegate nodes to give you the best out-of-the-box experience. These nodes are a shared commons but have no availability guarantees and are potentially a source of resource contention. If you are deploying JS IPFS in a production environment you should host your own delegate nodes and configure JS IPFS accordingly.

πŸ“ DHT configuration

The full DHT implementation for JS IPFS with all the changes made in Go IPFS 0.5 will not arrive until later this year, but for the time being you can run the experimental DHT implementation. This implementation is incomplete so some features may not work as intended but you should be able to use it to resolve content and find peers though there may be some performance degredation on your node over time.

You can enable the DHT for JS IPFS daemons via the command line. To put your node into client mode run:

$ jsipfs config Routing.Type dhtclient

Then restart your daemon process and you’ll be running the DHT.

To do the same thing in your application, use the libp2p config option:

const IPFS = require('ipfs')

const node = await IPFS.create({
  libp2p: {
    config: {
      dht: {
        enabled: true,
        clientMode: true
      }
    }
  }
})

DHT peers operate in either client mode or server mode. DHT clients can make queries to find content and other peers but will not advertise themselves as providers of content or answer any queries. You might be in client mode for any number of reasons but the primary one is that most DHT peers are behind a NAT firewall which means peers on other networks cannot dial them via ipfs.swarm.connect. Unless you know your node will have a public address, you should run it in client mode.

Go IPFS nodes use the libp2p-autonat package to determine if they are diallable by peers on external networks or not - if they are, they upgrade themselves from DHT clients to DHT servers. Autonat support is on the way for JS IPFS but until it lands it will only operate in client mode, which is a stepping stone on the way to full DHT support.

🧱 Smaller, faster blockstore

In the early days of IPFS, all CIDs were v0. That meant they were a bare multihash - a byte array prefixed with some prefixed bytes that told you what sort of hash the rest of the bytes represented (sha2-256, blake2s-128 etc) and how many of those bytes were present. The multihash was created by hashing the data in a block which was then stored in the block store contained within the IPFS repo.

Later v1 CIDs arrived and they added a version number and a codec to the byte array, but the CID still contained the multihash - a block can correspond to multiple CIDs, as long as they contain the same multihash.

The blockstore was turning CIDs into byte arrays and using these to generate the key for a block, which meant the same block might get stored against a v0 CID and a v1 CID. Since the block data is the same, the repo was also doing a double-lookup on each passed CID - once as a v0 CID and if the block was not found, again as a v1 CID.

With the release of js-IPFS@0.48.0, all blocks are now stored against the base32 encoded multihash extracted from the CID. This means no more duplication and no more double-lookups, but it’s come at the cost of needing to do a repo migration from v7 to v8 to ensure that all your blocks are stored under the correct key.

You may notice this when starting up your node:

$ jsipfs daemon
Initializing IPFS daemon...
js-ipfs version: 0.48.0
System version: x64/darwin
Node.js version: 12.16.1
Incompatible repo version. Migration needed. Pass --migrate for automatic migration

Just pass --migrate and your blockstore will be converted:

$ jsipfs daemon --migrate
Initializing IPFS daemon...
js-ipfs version: 0.48.0
System version: x64/darwin
Node.js version: 12.16.1
Swarm listening on /ip4/127.0.0.1/tcp/4002/p2p/QmSRS11FZeMHvqe5wZamurpgj2Jmm9SYX76t7ZtJ7Tt74d
Swarm listening on /ip4/192.168.1.109/tcp/4002/p2p/QmSRS11FZeMHvqe5wZamurpgj2Jmm9SYX76t7ZtJ7Tt74d
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/p2p/QmSRS11FZeMHvqe5wZamurpgj2Jmm9SYX76t7ZtJ7Tt74d
API listening on /ip4/127.0.0.1/tcp/5002/http
Gateway (read only) listening on /ip4/127.0.0.1/tcp/9090/http
Web UI available at http://127.0.0.1:5002/webui
Daemon is ready

When used as a module in the browser, or an application, options.repoAutoMigrate is enabled by default so the upgrade will happen invisibly in the background; all you will notice is a one-off slightly longer startup time.

πŸ—ΊοΈ A more intuitive API

As the IPFS ecosystem grows more and more developers become interested in the project and start using our APIs. A lot of them have grown organically over time and not all of them have had equal amounts of time invested in them.

The following changes only affect the core API and ipfs-http-client. The actual HTTP API and CLI remain unchanged.

ipfs.add()

Over time we’ve tried to remove the requirement to understand other frameworks and weird facets of the JavaScript language to start using IPFS. We removed pull-streams to let developers focus on the natural primitives of the environment they were developing for - e.g. streams in node and Files/Blobs in the browser. We removed the requirement to convert Strings to Buffers to let people simply add stringified JSON as a file to IPFS.

We refactored the whole API from callbacks to Promises, then from returning Arrays to AsyncIterators to allow streaming of enormous amounts of data without using external libraries.

In that last piece we lost a little bit of usability, as the humble ipfs.add API takes all manner of arguments and always returns an AsyncIterator:

for await (const file of ipfs.add('My file content')) {
  // Wait, what?  I only added one file, why am I in a loop?
}

With js-IPFS@0.48.0, ipfs.add now returns a single item. This seemingly innocuous change brings a raft of usability improvements as a very common question is ‘I added a file, and got back [object AsyncGenerator], what is that?’ and then you have to get the whiteboard and the pens out and before you know it you’ve gone down a long explanation cul de sac when all they wanted to do was get a CID back.

const file = await ipfs.add('My file content')
// Ahh, much better

But you can still add whole directories of files if you want to, just call ipfs.addAll:

const files = [{
  path: '/foo/file1.txt',
  content: 'file 1'
}, {
  path: '/foo/file2.txt',
  content: 'file 2'
}]

for await (const file of ipfs.addAll(files)) {
  // All the files!
}

APIs with optional arguments

Recently we released a change that allowed passing AbortSignals to all API methods. This necessitated adding an options object to every API call that didn’t have one already. This left us in the weird situation where some arguments were optional, but were not in the options argument. Worse, the actions of some API calls changed dramatically depending on whether you passed an option or not. For example ipfs.bootstrap.rm([multiaddr]) would completely empty the bootstrap list if you didn’t pass a Multiaddr.

All this leads to weird behaviour and subtle bugs when you pass things like undefined in for an optional arg position and don’t pass an options argument, as well as knotty, error-prone internal code that tries to guess what you passed based on type or properties of the objects where their types are the same.

With js-IPFS@0.48.0, all optional parameters to API methods now go in the options object. All APIs that have dramatic changes in behaviour have been split into more intuitive commands.

For example:

// before
ipfs.bootstrap.add('/ip4/...') // adds a multiaddr to the bootstrap list
ipfs.bootstrap.add({ default: true }) // restores default bootstrappers

// after
ipfs.bootstrap.add('/ip4/...') // adds a multiaddr to the bootstrap list
ipfs.bootstrap.reset() // restores default bootstrappers
// before
ipfs.bootstrap.rm('/ip4/...') // removes a multiaddr from the bootstrap list
ipfs.bootstrap.rm({ all: true }) // empties bootstrapper list

// after
ipfs.bootstrap.rm('/ip4/...') // removes a multiaddr from the bootstrap list
ipfs.bootstrap.clear() // empties bootstrapper list

See the API Changes section below for the full rundown.

✨New features

🦟 Bugs fixed

πŸ— API Changes

Core API & HTTP API Client

For further reading, see the Core API Docs.

πŸ—ΊοΈ What’s next?

Check out the js-IPFS Project Roadmap which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

😍 Huge thank you to everyone that made this release possible

πŸ™ŒπŸ½ Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

Comments