[downloader/rtmp] Fix arguments encoding and simplify retry logic (Closes #5528)