HTTP

To use HTTP retrievals, you only need to point your config file to an HTTP endpoint that hosts the epoch-NNN.car file, which supports HTTP range requests.

version: 1
epoch: 100
data:
  car:
    uri: https://files.old-faithful.net/100/epoch-100.car
indexes:
  cid_to_offset_and_size:
    uri: https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-cid-to-offset-and-size.index
  slot_to_cid:
    uri: https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-slot-to-cid.index
  sig_to_cid:
    uri:  https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-sig-to-cid.index
  sig_exists:
    uri: https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-sig-exists.index

Split retrievals (advanced)

If your remote storage solution cannot handle large monolithic files like the 600 GiB files we produce, you can also use split retrievals in HTTP mode, similar to how they work in Filecoin v1 retrievals.

You can use go-fil-dataprep to split the files in any size you want. This tool will also generate metadata.yaml. After you have generated metadata.yaml and the split, you need to create a map from this tool's “piece CIDs” and the URI of the specific pieces.

A sample config file for this use case would look like this (NOTE: this is just a sample; we currently do not host split retrievals on https://files.old-faithful.net):

version: 1
epoch: 100
data:
  car:
    from_pieces:
      metadata:
        uri: https://files.old-faithful.net/100/metadata.yaml
      piece_to_uri:
        baga6ea4seaqiogvxkc4cjgbimal52zgaqwo5ugcvy2bc72nrerro3wtzy47jepi:
          uri: https://files.old-faithful.net/100/baga6ea4seaqiogvxkc4cjgbimal52zgaqwo5ugcvy2bc72nrerro3wtzy47jepi.car
        baga6ea4seaqowdba7qkhqxw3umullabn7mx377we7vu3klguccuderfqclzpsiy:
          uri: https://files.old-faithful.net/100/baga6ea4seaqowdba7qkhqxw3umullabn7mx377we7vu3klguccuderfqclzpsiy.car
        baga6ea4seaqhky3ydiwp5toduhxpzt7oxm5cctuvrjbmkq5gzf3s6arayindoii:
          uri: https://files.old-faithful.net/100/baga6ea4seaqhky3ydiwp5toduhxpzt7oxm5cctuvrjbmkq5gzf3s6arayindoii.car
indexes:
  cid_to_offset_and_size:
    uri: https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-cid-to-offset-and-size.index
  slot_to_cid:
    uri: https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-slot-to-cid.index
  sig_to_cid:
    uri:  https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-sig-to-cid.index
  sig_exists:
    uri: https://files.old-faithful.net/100/epoch-100-bafyreibqt2nvroysxlxctgb52xxn27ectsllv2xyka4qar7ga6vupmbs3i-mainnet-sig-exists.index