Avoid unnecessary immediate retake `per_peer_state` lock