js-IPFS 0.48.0 released with connectivity improvements and smaller blockstore

Alex Potsides
js-IPFS 0.48.0 released with connectivity improvements and smaller blockstore
#js-ipfs #blockstore
Share this item:

# 🔦 Highlights

delegates turned on by default, smaller, faster blockstore and a more intuitive API

js-IPFS@0.48.0 is hot off the press with much better default connectivity, a smaller blockstore and a more intuitive API

# 🧭 Delegate nodes on by default

JS IPFS has traditionally primarily targeted the browser, and the browser is a bad place to be if you want to be on the DHT (opens new window). You typically aren't on a page long enough to make or respond to DHT (opens new window) queries, nor are you diallable, so even if you were able to advertise yourself as a provider for a given block, the chances are no-one can connect to you to retrieve that block which results in a degraded service for everyone. Worse, the way you find more peers and content is via the DHT (opens new window) so you're kind of stuck.

There are several ways to give in-browser IPFS nodes a better experience on the network, one of those is Delegate Nodes (opens new window). A Delegate Node is a network peer that performs certain actions on behalf of other nodes on the network. In this case it will make DHT (opens new window) queries on our behalf so we can find more peers and more content than ever before.

js-IPFS@0.48.0 enables delegate nodes in the configuration by default, which means you should see far more peers than you have previously and be able to find content faster and more reliably.

By default it uses public delegate nodes (opens new window) to give you the best out-of-the-box experience. These nodes are a shared commons but have no availability guarantees and are potentially a source of resource contention. If you are deploying JS IPFS in a production environment you should host your own delegate nodes and configure JS IPFS (opens new window) accordingly.

# 🏓 DHT configuration

The full DHT (opens new window) implementation for JS IPFS with all the changes made in Go IPFS 0.5 (opens new window) will not arrive until later this year, but for the time being you can run the experimental DHT (opens new window) implementation. This implementation is incomplete so some features may not work as intended but you should be able to use it to resolve content and find peers though there may be some performance degredation on your node over time.

You can enable the DHT (opens new window) for JS IPFS daemons via the command line. To put your node into client mode run:

$ jsipfs config Routing.Type dhtclient

Then restart your daemon process and you'll be running the DHT (opens new window).

To do the same thing in your application, use the libp2p config option:

const IPFS = require('ipfs')

const node = await IPFS.create({
  libp2p: {
    config: {
      dht: {
        enabled: true,
        clientMode: true,
      },
    },
  },
})

DHT peers operate in either client mode or server mode (opens new window). DHT clients can make queries to find content and other peers but will not advertise themselves as providers of content or answer any queries. You might be in client mode for any number of reasons but the primary one is that most DHT peers are behind a NAT (opens new window) firewall which means peers on other networks cannot dial them via ipfs.swarm.connect (opens new window). Unless you know your node will have a public address, you should run it in client mode.

Go IPFS nodes use the libp2p-autonat (opens new window) package to determine if they are diallable by peers on external networks or not - if they are, they upgrade themselves from DHT clients to DHT servers. Autonat support is on the way for JS IPFS (opens new window) but until it lands it will only operate in client mode, which is a stepping stone on the way to full DHT support.

# 🧱 Smaller, faster blockstore

In the early days of IPFS, all CID (opens new window)s were v0. That meant they were a bare multihash (opens new window) - a byte array prefixed with some prefixed bytes that told you what sort of hash the rest of the bytes represented (sha2-256, blake2s-128 etc) and how many of those bytes were present. The multihash (opens new window) was created by hashing the data in a block (opens new window) which was then stored in the block store contained within the IPFS repo (opens new window).

Later v1 CID (opens new window)s arrived and they added a version number and a codec to the byte array, but the CID (opens new window) still contained the multihash (opens new window) - a block can correspond to multiple CID (opens new window)s, as long as they contain the same multihash (opens new window).

The blockstore was turning CID (opens new window)s into byte arrays and using these to generate the key for a block, which meant the same block might get stored against a v0 CID (opens new window) and a v1 CID (opens new window). Since the block data is the same, the repo was also doing a double-lookup on each passed CID (opens new window) - once as a v0 CID (opens new window) and if the block was not found, again as a v1 CID (opens new window).

With the release of js-IPFS@0.48.0, all blocks are now stored against the base32 encoded multihash (opens new window) extracted from the CID (opens new window). This means no more duplication and no more double-lookups, but it's come at the cost of needing to do a repo migration from v7 to v8 to ensure that all your blocks are stored under the correct key.

You may notice this when starting up your node:

$ jsipfs daemon
Initializing IPFS daemon...
js-ipfs version: 0.48.0
System version: x64/darwin
Node.js version: 12.16.1
Incompatible repo version. Migration needed. Pass --migrate for automatic migration

Just pass --migrate and your blockstore will be converted:

$ jsipfs daemon --migrate
Initializing IPFS daemon...
js-ipfs version: 0.48.0
System version: x64/darwin
Node.js version: 12.16.1
Swarm listening on /ip4/127.0.0.1/tcp/4002/p2p/QmSRS11FZeMHvqe5wZamurpgj2Jmm9SYX76t7ZtJ7Tt74d
Swarm listening on /ip4/192.168.1.109/tcp/4002/p2p/QmSRS11FZeMHvqe5wZamurpgj2Jmm9SYX76t7ZtJ7Tt74d
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/p2p/QmSRS11FZeMHvqe5wZamurpgj2Jmm9SYX76t7ZtJ7Tt74d
API listening on /ip4/127.0.0.1/tcp/5002/http
Gateway (read only) listening on /ip4/127.0.0.1/tcp/9090/http
Web UI available at http://127.0.0.1:5002/webui
Daemon is ready

When used as a module (opens new window) in the browser, or an application, options.repoAutoMigrate (opens new window) is enabled by default so the upgrade will happen invisibly in the background; all you will notice is a one-off slightly longer startup time.

# 🗺️ A more intuitive API

As the IPFS ecosystem grows more and more developers become interested in the project and start using our APIs. A lot of them have grown organically over time and not all of them have had equal amounts of time invested in them.

The following changes only affect the core (opens new window) API and ipfs-http-client (opens new window). The actual HTTP API (opens new window) and CLI (opens new window) remain unchanged.

# ipfs.add()

Over time we've tried to remove the requirement to understand other frameworks and weird facets of the JavaScript language to start using IPFS. We removed pull-streams (opens new window) to let developers focus on the natural primitives of the environment they were developing for - e.g. streams (opens new window) in node and Files (opens new window)/Blobs (opens new window) in the browser. We removed the requirement to convert Strings (opens new window) to Buffers (opens new window) to let people simply add stringified JSON as a file to IPFS.

We refactored the whole API from callbacks (opens new window) to Promises (opens new window), then from returning Arrays (opens new window) to AsyncIterators (opens new window) to allow streaming of enormous amounts of data without using external libraries.

In that last piece we lost a little bit of usability, as the humble ipfs.add API takes all manner of arguments and always returns an AsyncIterator:

for await (const file of ipfs.add('My file content')) {
  // Wait, what?  I only added one file, why am I in a loop?
}

With js-IPFS@0.48.0, ipfs.add now returns a single item. This seemingly innocuous change brings a raft of usability improvements as a very common question is 'I added a file, and got back [object AsyncGenerator], what is that?' and then you have to get the whiteboard and the pens out and before you know it you've gone down a long explanation cul de sac when all they wanted to do was get a CID (opens new window) back.

const file = await ipfs.add('My file content')
// Ahh, much better

But you can still add whole directories of files if you want to, just call ipfs.addAll:

const files = [
  {
    path: '/foo/file1.txt',
    content: 'file 1',
  },
  {
    path: '/foo/file2.txt',
    content: 'file 2',
  },
]

for await (const file of ipfs.addAll(files)) {
  // All the files!
}

# APIs with optional arguments

Recently we released a change (opens new window) that allowed passing AbortSignal (opens new window)s to all API methods. This necessitated adding an options object to every API call that didn't have one already. This left us in the weird situation where some arguments were optional, but were not in the options argument. Worse, the actions of some API calls changed dramatically depending on whether you passed an option or not. For example ipfs.bootstrap.rm([multiaddr]) would completely empty the bootstrap list if you didn't pass a Multiaddr (opens new window).

All this leads to weird behaviour and subtle bugs when you pass things like undefined in for an optional arg position and don't pass an options argument, as well as knotty, error-prone internal code that tries to guess what you passed based on type or properties of the objects where their types are the same.

With js-IPFS@0.48.0, all optional parameters to API methods now go in the options object. All APIs that have dramatic changes in behaviour have been split into more intuitive commands.

For example:

// before
ipfs.bootstrap.add('/ip4/...') // adds a multiaddr to the bootstrap list
ipfs.bootstrap.add({ default: true }) // restores default bootstrappers

// after
ipfs.bootstrap.add('/ip4/...') // adds a multiaddr to the bootstrap list
ipfs.bootstrap.reset() // restores default bootstrappers

// before
ipfs.bootstrap.rm('/ip4/...') // removes a multiaddr from the bootstrap list
ipfs.bootstrap.rm({ all: true }) // empties bootstrapper list

// after
ipfs.bootstrap.rm('/ip4/...') // removes a multiaddr from the bootstrap list
ipfs.bootstrap.clear() // empties bootstrapper list

See the API Changes section below for the full rundown.

# ✨New features

# 🦟 Bugs fixed

# 🏗 API Changes

# Core API & HTTP API Client

  • ipfs.add only works on single items - a Uint8Array, a String, an AsyncIterable<Uint8Array> etc
  • ipfs.addAll works on multiple items
  • ipfs.bitswap.wantlist([peer], [options]) is split into:
    • ipfs.bitswap.wantlist([options])
    • ipfs.bitswap.wantlistForPeer(peer, [options])
  • ipfs.bootstrap.add([addr], [options]) is split into:
    • ipfs.bootstrap.add(addr, [options]) - add a bootstrap node
    • ipfs.bootstrap.reset() - restore the default list of bootstrap nodes
  • ipfs.bootstrap.rm([addr], [options]) is split into:
    • ipfs.bootstrap.rm(addr, [options]) - remove a bootstrap node
    • ipfs.bootstrap.clear([options]) - empty the bootstrap list
  • ipfs.config.get([key]) is split into:
    • ipfs.config.get(key) - return a value for a config key
    • ipfs.config.getAll() - return the whole config
  • ipfs.dag.resolve returns Promise<{ cid, remainderPath } instead of AsyncIterator<{ value, remainderPath }>
    • Previously the core api returned an async iterator and the http client returned a simple promise
  • ipfs.dag.get(cid, [path], [options]) becomes ipfs.dag.get(cid, [options])
    • path is moved into the options object
  • ipfs.dag.tree(cid, [path], [options]) becomes ipfs.dag.tree(cid, [options])
    • path is moved into the options object
  • ipfs.dag.resolve(cid, [path], [options]) becomes ipfs.dag.resolve(cid, [options])
    • path is moved into the options object
  • ipfs.files.flush([path], [options]) becomes ipfs.files.flush(path, [options])
  • ipfs.files.ls([path], [options]) becomes ipfs.files.ls(path, [options])
  • ipfs.object.new([template], [options]) becomes ipfs.object.new([options])
    • template is moved into the options object
  • ipfs.pin.ls([paths], [options]) becomes ipfs.pin.ls([options])
    • paths is moved into the options object
  • ipfs.refs.local now returns a v1 CID with the raw codec for every block and not the original CID by which it was added to the blockstore - nb for the ipfs-http-client this is only true when running against js-ipfs

For further reading, see the Core API Docs (opens new window).

# 🗺️ What’s next?

Check out the js-IPFS Project Roadmap (opens new window) which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

# 😍 Huge thank you to everyone that made this release possible

# 🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

  • Check the issues with the help wanted label in the js-IPFS repo (opens new window)
  • Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute: https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands
  • Hack with IPFS and show us what you made! The All Hands call is also the perfect venue for demos, join in and show us what you built
  • Join the discussion at https://discuss.ipfs.tech/ and help users finding their answers.
  • Join the 🚀 IPFS Core Implementations Weekly Sync 🛰 (opens new window) and be part of the action!

# ⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.tech (opens new window). We are also available at the #ipfs channel on Freenode.