We have a process called the CAA-indexer which is what sends these index.json files to the Internet Archive whenever a release is updated (such as when cover art is added, edited, or removed, or if you change other metadata on the release). That explains why @IvanDobsky adding a test image and removing it triggered us sending a new index.json file.
As for why it didn’t exist before, that’s unclear. There were times a year or two ago when we saw issues with the CAA-indexer, but it should be much more reliable now. Unfortunately some older updates may have gotten lost in the queue when it was less reliable, meaning the releases targeted by those updates don’t have an index.json file (or do, but out of date).
As I mentioned, changing the release metadata in some way is usually enough to trigger a reindex, though I don’t encourage making invalid edits to do it. You can also bring any broken releases to my attention on here, IRC, or by email and I can reindex them manually by prodding the CAA-indexer.
Sorry for the trouble! At some point we’d like to detect and fix all of these releases ourselves, but it’s lost in the shuffle of all our other priorities at the moment.