Troubleshooting Miner Score
This page is a work in progress.
This page describes how to troubleshoot the situation when Spark reports low Retrieval Success Rate score for your miner ID.
Check Sparkās list of deals eligible for retrieval testing
You can get a summary of eligible deals in Sparkās database at this URL - replace :f0id with your f0 address:
https://stats.filspark.com/miner/:f0id/deals/eligible/summaryIf there are no eligible deals in Sparkās database:
- At the moment, we are ingesting new deals once a week on Monday. If your miner started the operation or is accepting FIL+ deals this week, it wonāt be tested until next week.
- Spark is testing only FIL+ deals (verified deals) where the deal proposal has aĀ
LabelĀ set to a string starting withĀbafy,ĀbafkĀ orĀQmĀ (we interpret it as the payload CID). SeeĀ https://blog.filstation.app/posts/how-spark-discovers-content-stored-in-fil-deals. - Spark does not support DDO (direct data onboarding) deals yet. We are actively working on adding support for DDO deals; you can track the progress here: https://github.com/space-meridian/roadmap/issues/115
Check that you are advertising retrievals to IPNI (cid.contact)
- Find the Peer ID of your miner
- Request the following URL to check if cid.contact is ingesting your announcements (replace
:PEERIDwith your Peer ID):
https://cid.contact/providers/:PEERIDInspect recent measurements
We provide a Node.js script to fetch measurements submitted in the last ~16 hours.
- Install the latest Node.js LTS version from https://nodejs.org/en/download/.
- Download spark-evaluate source code from GitHub: https://github.com/filecoin-station/spark-evaluate/archive/refs/heads/main.zip
- Extract the ZIP archive and open your Terminal in the directory with the source code.
- Run the following command to install dependencies:
- Head to https://auth.node.glif.io/ to get yourself a Glif API Key. When you have it, in your terminal, run
- Run the script to list recent smart-contract events and fetch the measurements from IPFS. Replace
f0xyzwith your miner ID. - After the script finishes, you will find two new files in your local working directory:
measurements-f0xyz.txtwith a human-readable summary.measurements-f0xyz.ndjsoncontains all fields of all measurements for this miner. The file is in newline-delimited JSON format, which makes it easy to use tools like jq to further process the data.
npm ciexport GLIF_TOKEN=<your glif token>node bin/fetch-recent-miner-measurements.js f0xyzThe script takes many minutes to complete. Feel free to interrupt it at any time by pressing Ctrl+C. When interrupted this way, the script will finish processing the measurements that have been already downloaded and write the partial data.
Example summary
Timestamp CID Protocol RetrievalResult
2024-05-24T02:43:30.087Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki http BAD_GATEWAY
2024-05-24T02:43:33.123Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki http BAD_GATEWAY
2024-05-24T02:43:39.298Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki http BAD_GATEWAY
2024-05-24T02:44:29.340Z bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki IPNI_ERROR_404Run a Spark retrieval check locally
You can run Spark checkerās code to perform a single retrieval check from your machine. This is useful to quickly verify changes in your miner setup.
- Download the latest Spark Checker source code from GitHub: https://github.com/filecoin-station/spark/archive/refs/heads/main.zip.
- Extract the ZIP archive.
- Download the
zinniabinary for your operating system & processor architecture from GitHub: https://github.com/filecoin-station/zinnia/releases/latest. - Place the binary into the root directory of spark source code you created in the previous step.
- In Sparkās root directory, find the file
manual-check.jsand edit the values incidandminerIdvariables. These two variables define the retrieval task that will be manually checked. - Open your Terminal in the directory with the source code and run the following command:
Unix (Linux, macOS)
./zinnia run manual-check.js Windows
./zinnia.exe run manual-check.js Example retrieval check
Output from testing retrieval of bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki from miner f010479:
āÆ zinnia run manual-check.js
Calling Filecoin JSON-RPC to get PeerId of miner f010479
Found peer id: 12D3KooWHKeaNCnYByQUMS2n5PAZ1KZ9xKXqsb4bhpxVJ6bBJg5V
Querying IPNI to find retrieval providers for bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki
IPNI returned 5 provider results
Fetching: ipfs://bafybeigpefr2xxsiigy5witxbmgrg3ok7tnrauz44z3utch5uxakljxlki?dag-scope=block&protocols=http&providers=%2Fdns%2Ff010479.twinquasar.io%2Ftcp%2F443%2Fhttps
{
providerId: "12D3KooWHKeaNCnYByQUMS2n5PAZ1KZ9xKXqsb4bhpxVJ6bBJg5V",
indexerResult: "OK",
protocol: "http",
providerAddress: "/dns/f010479.twinquasar.io/tcp/443/https",
statusCode: 200,
byteLength: NaN,
carChecksum: "12202d73dcf0ed477140f9a14003cfca02c856671d8435a8fcc697ed3fe49114ba3f",
endAt: 2024-05-27T08:29:35.413Z
}Additional resources
To provide payload retrievals, storage providers need to:
- Announce their deals to IPNI.
- Serve payload retrievals, preferably over HTTP (
booster-http). Spark supports Graphsync, too, but we plan to remove it in the short to medium term.
Documentation:
- https://boost.filecoin.io/configuration/http-indexer-announcement
- https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#payload-retrievals-car-and-raw
Reach out to us
Sparkās mission is to improve retrievability of data stored on Filecoin. Our team is happy to help you understand why Spark cannot retrieve data from your miner and what options you have to fix the problem. You can find us on Filecoin Slack in the channel #station or write us an email to [email protected].
ā Previous