[utils] Improve determine_ext (Closes #7593)