Clean up at least some lifetimes on `SimpleRefPeerManager`