If there are any samples remaining in the output fifo from previous conversion calls, we have to output those samples first instead of doing direct output of the current samples.