js-IPFS 0.53.0 arrives with experimental gRPC server and libp2p-0.30.x

by Alex Potsides on 2021-01-19

🔦 Highlights

full-duplex streaming browser APIs and the latest libp2p

js-IPFS@0.53.0 has launched into the stratosphere with an experimental gRPC-over-websockets server that means the http client has true full-duplex streaming for the ipfs.add API

↔️ gRPC over websockets

In the beginning, go-IPFS shipped with an HTTP API, which js-IPFS also implemented with the aim of being cross-compatible.

This HTTP API allows you to orchestrate a locally running node from a language or environment that may not be able to run a fully-fledged IPFS node, or it may be preferable to have a single system-wide node shared between multiple applications.

The go-IPFS HTTP API supports streaming responses, and returns the message from any error that occurs during a streaming response as an HTTP trailer.

When the request is made from the browser or from node via the node-fetch module, HTTP trailers are not available. Bugs are open against FireFox and Chromium to address this but they are unlikely to be fixed in any reasonable time frame, partially due to a security concern around allowing trailers to change the status of a response after that response has been sent.

The effect of all this is that if you are reading a streaming response from the server, and an error occurs on the server, from the client’s point of view the stream just stops with no error message, which is less than ideal.

That’s the response end, from the request end, we’d like to be able to send requests of arbitrary length and start processing the response before the request has finished sending (think pubsub, or giving the user server-driven progress notifications when adding large or multiple files).

The Fetch API allows for sending request bodies as WhatWG ReadableStreams but no browser currently supports this (Chrome recently announced an intent to ship this feature which is great news, but FireFox so far has made little progress so we’re not there yet).

If ever implemented widely, this will allow us to stream uploads and downloads, but there’s no guarantee it’ll be full-duplex (e.g., both ends streaming at the same time).

HTTP/2 supports full-duplex streaming, but currently it requires ALPN, which requires TLS, which requires certificates, which is overkill if you’re only communicating over the loopback interface.

Where does this leave us?

We want full-duplex streaming over the API; we want the safety of typed inputs and outputs.

gRPC gives us the typing, (yay!) and there is gRPC-web which is designed to work over HTTP, but it does not support bi-directional streaming, and while there are plans afoot, there is nothing we can use today.

There is a way we can do bi-directional, full-duplex streaming today: WebSockets.

The gRPC-web team have no plans to publish even an experimental spec for gRPC-web-over-websockets due to concerns about compatibility with existing HTTP infrastructure - proxies and the like, all of which are not relevant to our use-case.

The engineering team at improbable.io have published a golang implementation of gRPC-web that wraps a gRPC server but also provides a websocket transport with HTTP fallback.

js-IPFS@0.53.0 ships with a JavaScript port of this gRPC-web-over-websockets server. Phew!

To use it we’ve shipped a new client called ipfs-client (hat tip to @brosenan for very kindly donating the module name) - a fully Core API compatible client. It combines the ipfs-http-client and the new ipfs-grpc-client, using gRPC for implemented methods and falling back to HTTP for methods that have not been ported to gRPC yet.

const createClient = require('ipfs-client')

const client = createClient({
  grpc: '/ipv4/',
  http: '/ipv4/'

const id = await client.id()

A few caveats:

  1. This only works against js-IPFS for now
  2. It only works over the loopback address (not localhost) as loopback is considered a secure context, otherwise it would require TLS
  3. Only the ipfs.add API has full-duplex streaming enabled for this release - expect a lot more in future!

There’s a new example called ipfs-client-add-files that you can use as a sandbox to experiment with this new client.

If you fire it up, you’ll see progress and file import events appearing before the upload request has completed, something that’s not been possible with the ipfs-http-client until now.

What’s next? We’re going to extend the gRPC implementation to all streaming methods and get a server into go-IPFS so you can use the same client with that implementation too.

The full-duplex streaming capability opens up all sorts of interesting possibilities, for example, opening up the full libp2p API over HTTP instead of the limited subset that we currently support.

Speaking of libp2p:

☎️ libp2p@0.30.x

The 0.53.0 release of js-IPFS ships with libp2p@0.30.x which gives us TypeScript type definitions, Auto Relay support, and improved Peer advertising and dialer mechanics.

Expect a blog post here soon with a deep dive, but in the mean time see the libp2p@0.30.x release notes for more details.

✨New features

🔨 Breaking changes

🕷️ Bug fixes

🗺️ What’s next?

Check out the js-IPFS Project Roadmap which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

😍 Huge thank you to everyone that made this release possible

🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.