ipfs:// URL support in
CURL 8.4.0 (opens new window) shipped with built-in support for
curl to seamlessly integrate with the user's preferred IPFS gateway (opens new window) through the
IPFS_GATEWAY environment variable or a
gateway file. Best of all, these capabilities are available for immediate use today:
$ export IPFS_GATEWAY="http://127.0.0.1:8080" # local (trusted) gateway provided by ipfs daemon like Kubo $ curl ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey hello from IPFS
In this blog post, we will:
- explore the journey of implementing IPFS URI support in CURL,
- delve into the mechanics of how CURL locates an IPFS gateway,
- learn how to be immune to malicious gateways,
- and finally, provide practical CURL examples for leveraging IPFS URLs for either deserialized or verifiable responses.
# A brief history
Supporting IPFS in CURL has been attempted before (opens new window) as a CURL library feature. Some discussions lead to a belief that this should be implemented in the CURL tool itself, not its library. A renewed implementation attempt (opens new window) took the tool-side approach which ultimately was accepted and is available right now in CURL 8.4.0!
The support of IPFS in CURL is effectively consisting of two implementation details.
- CURL tries to find a locally installed or configured gateway.
- It then rewrites an
ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzito a gateway URL. This is how curl handles it internally, you see nothing of this URL rewriting.
If you have IPFS installed locally then running
curl ipfs:// will Just Work™. If not, CURL will return an error with details about how to set up the gateway preference. This ensures the user agency is respected, no third-party gateway is used as implicit default.
ipfs:// URL support is so important?
https://ipfs.io/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi equally acceptable?
Or why isn't a local URL
Both addresses are tied to a specific location.
IPFS is a modular suite of protocols purpose built for the organization and transfer of content-addressed (opens new window) data. It shouldn't matter where the content is. Content Identifier (CID (opens new window)) is all that is required. The "where" part is implementation detail an IPFS system takes care of. Hardcoding a location in addition to a CID (like a specific HTTP gateway) limits end users to IPFS resources available through that one specific, centralized point of entry.
If we pull the URL apart we see:
Users of the IPFS system should not care about the where part, nor be coerced to use a specific, hard-coded entry point into the system.
Public gateways like
ipfs.io are always owned by some entity and could get censored or shut down at any time. Many gateways will not allow playback of deserialized videos or only respond to CIDs from allowlists to reduce costs. Other gateways will block specific CIDs from resolving in specific jurisdictions for legal reasons. Community-run public gateways will have limits and throttle usage.
These are not limitations of IPFS but purely a limitation a specific gateway has set through custom configuration. IPFS user should always have ability to avoid such limitations if they choose to self-host and run their own IPFS node with a local gateway (opens new window).
# How does CURL find an IPFS Gateway?
Any IPFS implementation that has support for IPIP-280 (opens new window) exposes an IPFS gateway that CURL (and ffmpeg (opens new window)) can use. At the moment of writing that's just Kubo (opens new window).
CURL 8.4.0 and greater looks for a gateway in the following order:
IPFS_GATEWAY, if set it's used.
~/.ipfs/gatewayfile, where it reads the first line.
If a gateway hint is found at any of those places, and if that is a valid HTTP URL, then CURL will use it. If not, then you'll be getting an error message pointing to the CURL documentation related to IPFS (opens new window) to help you further.
One can specify any IPFS gateway that is in compliance with Gateway Specifications (opens new window). It is highly recommended to use a local gateway, as it provides the best security guarantees.
# Malicious gateways and data integrity?
Requesting deserialized responses and delegating hash verification to a third-party gateway comes with risks. It is possible that a public gateway is malicious. Or, that a well-known and respected gateway gets hacked and changed to return payload that does not match requested CID. How can one protect themselves against that?
If deserialized responses are necessary, one should run their own gateway in a local, controlled environment. Every block of data retrieved though self-hosted IPFS gateway is verified to match the hash from CID. For the maximum flexibility and security, find an implementation that provides the gateway endpoint (i.e. Kubo (opens new window)) and run it yourself!
When using a third-party gateway that one can't fully trust, the only secure option is to request verifiable response types (opens new window) such as application/vnd.ipld.raw (opens new window) (a single block) or application/vnd.ipld.car (opens new window) (multiple blocks in CAR archive). Both allow to locally verify if the data returned by gateway match the requested CID, removing the surface for Man-in-the-middle attacks (opens new window).
# CURL Examples
# Deserialized responses
By default, a trusted local gateway acts as a bridge between traditional HTTP clients and IPFS.
It performs necessary hash verification, UnixFS deserialization and return reassembled files to the client, as if they were stored in a traditional HTTP server. This means all validation happens on the gateway, and clients trust that the gateway is correctly validating content-addressed data before returning it to them.
# Downloading a file from IPFS with CURL
$ curl ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey -o out.txt
If curl responds with
curl: IPFS automatic gateway detection failure, make sure
IPFS_GATEWAY is set (see examples below).
# Explicitly specifying a gateway
To use local gateway on custom port 48080:
$ export IPFS_GATEWAY=http://127.0.0.1:48080 $ curl ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey hello from IPFS
When setting environment variable is not feasible, one can use
$ curl --ipfs-gateway http://127.0.0.1:48080 ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey hello from IPFS
# Following subdomain redirects
By default, the URL resolution in
curl does not follow HTTP redirects and assumes the endpoint implements deserializing path gateway (opens new window), or at the very least, the trustless gateway (opens new window).
curl at a subdomain gateway (opens new window) (like
https://dweb.link or the
http://localhost:8080 provided by a local Kubo node (opens new window)) one has to pass
-L in the curl command to follow the redirect.
$ IPFS_GATEWAY=https://localhost:8080 curl -s -L ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey hello from IPFS
# Piping and streaming responses
Deserialized response returned by CURL can be piped directly to a video player:
$ curl ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi | ffplay -
# Verifiable responses
By explicitly requesting application/vnd.ipld.raw (opens new window) (a block) or application/vnd.ipld.car (opens new window) (a stream of blocks) responses, by means defined in Trustless Gateway Specification (opens new window), the user is able to fetch raw content-addressed data and perform hash verification themselves (opens new window).
# Fetching and verifying a directory from an untrusted gateway
Requesting trustless and verifiable (opens new window) CAR response via
Accept HTTP header:
$ export IPFS_GATEWAY="https://ipfs.io" # using untrusted public gateway $ curl -H "Accept: application/vnd.ipld.car" "ipfs://bafybeiakou6e7hnx4ms2yangplzl6viapsoyo6phlee6bwrg4j2xt37m3q" > dag.car
Then, CAR can be moved around and imported into some other IPFS node:
$ ipfs dag import dag.car
$ npm i -g ipfs-car $ ipfs-car unpack dag.car --output dag.out $ ls dag.out 1007 - Sustainable - alt.txt 1007 - Sustainable - transcript.txt 1007 - Sustainable.png
# What's next?
More places supporting IPFS addresses. Everyone can integrate
ipns:// URL support into their application. See specifications proposed in IPIP-280 (opens new window) for technical details. We are tracking potential project (opens new window) where an integration makes sense! If you feel up to the challenge, don't hesitate to drop a comment in one of the potential projects (opens new window) for IPFS URL integration or find us on:
- Matrix (opens new window), Discord (opens new window) or Slack (opens new window)
- Discussion Forum (opens new window)
Or one of the other many places where the IPFS community (opens new window) is active.