Troubleshooting Miner Score

This page describes how to troubleshoot the situation when Spark reports low Retrieval Success Rate score for your miner ID.

What Spark requires from SPs

To provide payload retrievals, storage providers need to:

• Announce their deals to IPNI.
• Serve payload retrievals, preferably over HTTP (booster-http). Spark supports Graphsync, too, but we plan to remove it in the short to medium term.

Documentation:

https://boost.filecoin.io/configuration/http-indexer-announcement

https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#payload-retrievals-car-and-raw

DDO deals

SPs that want to use DDO deals and get Spark to test the retrievability of their new DDO deals must meet the following requirements:

• They must advertise Graphsync retrievals to IPNI.
• Preferably, they should advertise both Graphsync and HTTP retrievals.

Spark will retrieve the content over HTTP if available, but Graphsync advertisements are needed to extract payload CID(s).

• We recommend creating a new MinerID for their DDO operations to reduce the likelihood that Spark's new DDO retrieval checking pipeline will negatively influence Spark RSR of their existing f05 deals.

Check Spark’s list of deals eligible for retrieval testing

You can get a summary of eligible deals in Spark’s database at this URL - replace :f0id with your f0 address:

https://stats.filspark.com/miner/:f0id/deals/eligible/summary

If there are no eligible deals in Spark’s database:

• At the moment, we are ingesting new deals once a week on Monday. If your miner started the operation or is accepting FIL+ deals this week, it won’t be tested until next week.
• Spark is testing only FIL+ deals (verified deals) where the deal proposal has a Label set to a string starting with bafy, bafk or Qm (we interpret it as the payload CID). See https://blog.filstation.app/posts/how-spark-discovers-content-stored-in-fil-deals.
• Spark does not support DDO (direct data onboarding) deals yet. We are actively working on adding support for DDO deals; you can track the progress here: https://github.com/space-meridian/roadmap/issues/115

Check that you are advertising retrievals to IPNI (cid.contact)

1. Find the Peer ID of your miner
2. Request the following URL to check if cid.contact is ingesting your announcements (replace :PEERID with your Peer ID):
Copy

https://cid.contact/providers/:PEERID

Give Spark enough time to pick one of your deals for testing

Every 20-minute round, Spark picks around 1000 deals to test from the database of eligible deals at random. The more deals your miner has, the more likely it will be included in the round. In practice, Spark usually tests all storage providers within 24 hours.

Inspect recent measurements

We provide a Node.js script to fetch measurements submitted in the last ~16 hours.

1. Install the latest Node.js LTS version from https://nodejs.org/en/download/.
2. Download spark-evaluate source code from GitHub: https://github.com/filecoin-station/spark-evaluate/archive/refs/heads/main.zip
3. Extract the ZIP archive and open your Terminal in the directory with the source code.
4. Run the following command to install dependencies:
Copy

npm ci

5. Head to https://auth.node.glif.io/ to get yourself a Glif API Key. When you have it, in your terminal, run
Copy

export GLIF_TOKEN=<your glif token>

6. Run the script to list recent smart-contract events and fetch the measurements from IPFS. Replace f0xyz with your miner ID.
Copy

node bin/fetch-recent-miner-measurements.js f0xyz

💡
The script takes many minutes to complete. Feel free to interrupt it at any time by pressing Ctrl+C. When interrupted this way, the script will finish processing the measurements that have been already downloaded and write the partial data.
7. Run the script to evaluate measurements:
Copy

node bin/evaluate-measurements.js measurements-f0xyz.ndjson

8. After that script finishes, you will find these new files in your local working directory:
• measurements-f0xyz.evaluation.txt with human-readable evaluation summary.
• measurements-f0xyz.evaluation.ndjson contains all fields of all measurements for this miner. The file is in newline-delimited JSON format, which makes it easy to use tools like jq to further process the data.
• measurements-f0xyz.ndjson contains all measurements submitted for this miner. The file is in newline-delimited JSON format, which makes it easy to use tools like jq to further process the data.

Example summary

Timestamp                CID                                                                    Protocol   RetrievalResult
2024-05-24T02:43:30.087Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki            http       BAD_GATEWAY
2024-05-24T02:43:33.123Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki            http       BAD_GATEWAY
2024-05-24T02:43:39.298Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki            http       BAD_GATEWAY
2024-05-24T02:44:29.340Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki                       IPNI_ERROR_404

Run a Spark retrieval check locally

You can run Spark checker’s code to perform a single retrieval check from your machine. This is useful to quickly verify changes in your miner setup.

1. Download the latest Spark Checker source code from GitHub: https://github.com/filecoin-station/spark/archive/refs/heads/main.zip.
2. Extract the ZIP archive.
3. Download the zinnia binary for your operating system & processor architecture from GitHub: https://github.com/filecoin-station/zinnia/releases/latest.
4. Place the binary into the root directory of spark source code you created in the previous step.
5. In Spark’s root directory, find the file manual-check.js and edit the values in cid and minerId variables. These two variables define the retrieval task that will be manually checked.
6. Open your Terminal in the directory with the source code and run the following command:

Unix (Linux, macOS)

Copy

./zinnia run manual-check.js 

Windows

Copy

./zinnia.exe run manual-check.js 

Example retrieval check

Output from testing retrieval of bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki from miner f010479:

❯ zinnia run manual-check.js
Calling Filecoin JSON-RPC to get PeerId of miner f010479
Found peer id: 12D3KooWHKeaNCnYByQUMS2n5PAZ1KZ9xKXqsb4bhpxVJ6bBJg5V
Querying IPNI to find retrieval providers for bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki
IPNI returned 5 provider results
Fetching: ipfs://bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki?dag-scope=block&protocols=http&providers=%2Fdns%2Ff010479.twinquasar.io%2Ftcp%2F443%2Fhttps
{
  providerId: "12D3KooWHKeaNCnYByQUMS2n5PAZ1KZ9xKXqsb4bhpxVJ6bBJg5V",
  indexerResult: "OK",
  protocol: "http",
  providerAddress: "/dns/f010479.twinquasar.io/tcp/443/https",
  statusCode: 200,
  byteLength: NaN,
  carChecksum: "12202d73dcf0ed477140f9a14003cfca02c856671d8435a8fcc697ed3fe49114ba3f",
  endAt: 2024-05-27T08:29:35.413Z
}

Reach out to us

Spark’s mission is to improve retrievability of data stored on Filecoin. Our team is happy to help you understand why Spark cannot retrieve data from your miner and what options you have to fix the problem. You can find us on Filecoin Slack in the channel #station or write us an email to [email protected].

On this page

• Troubleshooting Miner Score
• What Spark requires from SPs
• DDO deals
• Check Spark’s list of deals eligible for retrieval testing
• Check that you are advertising retrievals to IPNI (cid.contact)
• Give Spark enough time to pick one of your deals for testing
• Inspect recent measurements
• Example summary
• Run a Spark retrieval check locally
• Example retrieval check
• Reach out to us

← Previous

Reward

Next →

Time To First Byte