How to store IPFS hash using bytes32?

Question

Following Q/A (What datatype should I use for an IPFS address hash?) recommend us to use bytes to store IPFS hash.

I was using following example(https://github.com/AdrianClv/ethereum-ipfs/blob/master/NotSoSimpleStorage.sol), which uses string to store IPFS hash that costs around 110,000 gas price, which seems pretty expensive.

[Q] Does using bytes instead of string in order to store IPFS hash cost cheaper? I observe that storing bytes instead of string costs very close to string (110,000 gas). Since both datatypes storage seems expensive should I use events to store them?

Is there any example/tutorial related to store IPFS hash using bytes?

Would this work:

myContract.insertHash("QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vz");

contract Example_bytes {
    bytes[] list;
    function insertHash(bytes ipfsHash) {
       list.push(ipfsHash); //costs around 110,000 gas. 
    }
}

contract Example_string {
    struct hashes{
         string hash;
    }

    hashes[] list;
    function insertHash(string ipfsHash) {
       list.push(hashes{hash: ipfsHash); //costs around 110,000 gas. 
    }
}

Answer 1

Your example shows storing an IPFS identity using it's alphanumeric encoding (Qm...), which is the same Base58 encoding that Bitcoin uses. However, what it's representing at its core is a number (the hash). Storing the identifier in the Base58 format needs to be a String because it includes letters (and what actually gets saved is the ASCII code for each alphanumeric character in the identifier). That means you need 46 bytes to store QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vz from your example.

However, that identifier can also be expressed in hexadecimal as 12207D5A99F603F231D53A4F39D1521F98D2E8BB279CF29BEBFD0687DC98458E7F89, which is only 34 bytes long (takes 68 characters to write out in hexadecimal, since every two characters in hex is a byte of data).

But, both of those are greater than 32 bytes, which is the max fixed-size byte array, so they're going to need to use a dynamically-sized byte array to store (bytes or string, both of which are expensive, as you noted).

BUT, that IPFS hash is actually two concatenated pieces. It's a multihash identifier, so the first two bytes indicate the hash function being used and the size. 0x12 is sha2, 0x20 is 256-bits long. Currently, that's the only format IPFS uses, so you could just chop off the first two bytes, which leaves you with a 32-byte value, which is small enough to fit in a bytes32 fixed-size byte array, and you save some space there (and when retrieving either your contract can re-attach 0x1220 to the front of it, or your clients need to be smart enough to do that after retrieving the value).

If you want to make sure your code is future-proof, though, you probably want to save that hash function code and size, which you could combine with the hash as a struct:

struct Multihash {
  bytes32 hash
  uint8 hash_function
  uint8 size
}

That will work with any multihash format, as long as size is less than or equal to 32 (any bigger and the actual payload won't fit in the hash property). This struct will take two storage slots (two 32-byte chunks) to store, since the two uint8 pieces can be put in one slot. You could also add up to 30 bytes of additional data to this struct without taking any more storage cost.

Answer 2

Here are some js functions for stripping and re-adding the first two bytes containing the hash function and size, suitable for web3.

import bs58 from 'bs58'

// Return bytes32 hex string from base58 encoded ipfs hash,
// stripping leading 2 bytes from 34 byte IPFS hash
// Assume IPFS defaults: function:0x12=sha2, size:0x20=256 bits
// E.g. "QmNSUYVKDSvPUnRLKmuxk9diJ6yS96r1TrAXzjTiBcCLAL" -->
// "0x017dfd85d4f6cb4dcd715a88101f7b1f06cd1e009b2327a0809d01eb9c91f231"

getBytes32FromIpfsHash(ipfsListing) {
  return "0x"+bs58.decode(ipfsListing).slice(2).toString('hex')
}

// Return base58 encoded ipfs hash from bytes32 hex string,
// E.g. "0x017dfd85d4f6cb4dcd715a88101f7b1f06cd1e009b2327a0809d01eb9c91f231"
// --> "QmNSUYVKDSvPUnRLKmuxk9diJ6yS96r1TrAXzjTiBcCLAL"

getIpfsHashFromBytes32(bytes32Hex) {
  // Add our default ipfs values for first 2 bytes:
  // function:0x12=sha2, size:0x20=256 bits
  // and cut off leading "0x"
  const hashHex = "1220" + bytes32Hex.slice(2)
  const hashBytes = Buffer.from(hashHex, 'hex');
  const hashStr = bs58.encode(hashBytes)
  return hashStr
}

Here is the function used in context, calling a Listing contract deployed with Truffle.

submitListing(ipfsListing, ethPrice, units) {
  return new Promise((resolve, reject) => {
    this.listingContract.setProvider(window.web3.currentProvider)
    window.web3.eth.getAccounts((error, accounts) => {
      this.listingContract.deployed().then((instance) => {
        let weiToGive = window.web3.toWei(ethPrice, 'ether')
        return instance.create(
          this.getBytes32FromIpfsHash(ipfsListing), /*** IPFS here ***/
          weiToGive,
          units,
          {from: accounts[0]})
      }).then((result) => {
        resolve(result)
      }).catch((error) => {
        console.error("Error submitting to the Ethereum blockchain: " + error)
        reject(error)
      })
    })
  })

Taken from my work on the Origin Demo Dapp here: https://github.com/OriginProtocol/origin-js/blob/1cfc84d4693974bbf18e345e6c0def843321130c/src/services/contract-service.js#L102-L128

Answer 3

This answer is just Python implementation of @MidnightLightning's accepted answer above. I have used Web3.py.

from web3.auto import w3

def _ipfs_to_bytes32(hash_str: str):
    """Ipfs hash is converted into bytes32 format."""
    bytes_array = base58.b58decode(hash_str)
    b = bytes_array[2:]
    return binascii.hexlify(b).decode("utf-8")

def ipfs_to_bytes32(ipfs_hash: str) -> str:
    """bytes32 is converted back into Ipfs hash format."""
    ipfs_hash_bytes32 = _ipfs_to_bytes32(ipfs_hash)
    return w3.toBytes(hexstr=ipfs_hash_bytes32)

def bytes32_to_ipfs(bytes_array):
    """Convert bytes_array into IPFS hash format."""
    merge = Qm + bytes_array
    return base58.b58encode(merge).decode("utf-8")

if __name__ == "__main__":
    ipfs_hash = "QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vd"
    ipfs_bytes32 = ipfs_to_bytes32(ipfs_hash)
    _ipfs_hash = bytes32_to_ipfs(ipfs_bytes32)
    assert ipfs_hash == _ipfs_hash  # They should be equal to each other

Answer 4

I have handled similiar situation with this util function in web3.py:

import base58

def convertIpfsBytes32(hash_string):           
  bytes_array = base58.b58decode(hash_string) 
  return bytes_array[2:]

You need base58 module. Concept is same as accepted answer.

Answer 5

If you have no time to research on the algorithm, I suggest using content-hash node module. You can easily convert many format of string such as: CID v0, CID v1, base58, base32.

example:

 const ipfs = 'QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG'
const cidV1 = contentHash.helpers.cidV0ToV1Base32(ipfs)
 // 'bafybeibj6lixxzqtsb45ysdjnupvqkufgdvzqbnvmhw2kf7cfkesy7r7d4'

refer to: https://github.com/ensdomains/content-hash

Answer 6

Here's a more complete example using the js-multihash library:

MyContract.sol

pragma solidity ^0.4.24;

contract MyContract {

  event AddFile(address indexed owner, bytes32 digest, bytes2 hashFunction, uint8 size, bytes4 storageEngine);

  function addFile(bytes32 _digest, bytes2 _hashFunction, uint8 _size, bytes4 _storageEnginge) public {
    emit AddFile(msg.sender, _digest, _hashFunction, _size, _storageEngine);
  }
}

Javascript

import Web3 from 'web3'
import multihashes from 'multihashes'
import ipfsAPI from 'ipfs-api'

var web3 = window.web3
web3 = new Web3(web3.currentProvider)

// Utility functions:
const utils = {
  ipfs2multihash (hash) {
    let mh = multihashes.fromB58String(Buffer.from(hash))
    return {
      hashFunction: '0x' + mh.slice(0, 2).toString('hex'),
      digest: '0x' + mh.slice(2).toString('hex'),
      size: mh.length - 2
    }
  },

  multihash2hash (hashFunction, digest, size, storageEngine) {
    storageEngine = web3.toAscii(storageEngine)

    if (storageEngine === 'ipfs') {
      hashFunction = hashFunction.substr(2)
      digest = digest.substr(2)
      return {
        hash: multihashes.toB58String(multihashes.fromHexString(hashFunction + digest)),
        engine: storageEngine
      }
    }

    throw new Error('Unknown storage engine:', storageEngine)
  }
}

// ... code to instantiate contract
// ... code to get the file buffer

ipfs.add(buffer)
  .then((response) => {
    console.log('ipfs hash:', response[0].hash)

    // Prepare data
    let mh = utils.ipfs2multihash(response[0].hash)
    let storageEnginge = web3.fromAscii('ipfs')

    // Call contract
    myContractInstance.addFile.sendTransaction(mh.digest, mh.hashFunction, mh.size, storageEnginge, {from: myAccount, gas: 1000000}, (error, result) => {
      if (error) throw error
      console.log(result)
    })
  })

Reading the data

let fileFilter = myContract.AddFile({
    owner: myAccount
  }, {
       fromBlock: 0,
       toBlock: 'latest'
     }).watch((error, log) => {
      if (error) reject(error)

      console.log('file log:', log)

      let hash = utils.multihash2hash(log.args.hashFunction, log.args.digest, log.args.size, log.args.storageEngine)
      console.log('Hash:', hash)

      fileFilter.stopWatching()
    })

Answer 7

All of the answers here are needlessly complicated. If, in the end, you're going to store 64 bytes anyway, why not simply store the first 32 bytes of the IPFS hash in one string32 and the remainder of the hash in another string32? No need to convert when storing, therefore less gas to process. Simply concatenate to reconstitute the hash, therefore cheaper to process. Same gas cost to store (64 bytes). And, more future proof. Unlike the selected solution, the length of the IPFS hash change grow (or shrink) with no change needed to the code.