Set `ptr_for_ref` correctly when writing conversions for enum fields