swr_convert is not properly buffering packed input audio when the
output is not large enough, and when resampling is not actually needed
(same samplerate and no SWR_FLAG_RESAMPLE).
buf_set() is only handling the first channel and leaving the others as-is.
Sample program to reproduce the problem is here https://gist.github.com/2431768
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>