Remove peers from the `node_id_to_descriptor` even without init