The answer I gave (which he said is how it is actually done) is to mirror how a cache works in hardware and periodically delete all references that are not held in the "cache" (which I stored as a priority queue).