generic-library/ffmpeg
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Fixes avpicture_layout to not write past buffer end.

Browse Source 
avpicture_get_size() returns the size of buffer required for avpicture_layout.
For pseudo-paletted formats (gray8...) this size does not include the palette.
However, avpicture_layout doesn't know this and still writes the palette. Consequently,
avpicture_layout writes passed the length of the buffer. This fixes it
by fixing avpicture_layout so that it doesn't write the palette for these formats.

Signed-off-by: Matthew Einhorn <moiein2000@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
This commit is contained in:
Matthew Einhorn 2011-08-24 20:14:03 -04:00 committed by Michael Niedermayer
parent e31c5ebe11
commit e662b263d9
1 changed files with 10 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
libavcodec/imgconvert.c
View File

@ -337,6 +337,16 @@ int avpicture_layout(const AVPicture* src, enum PixelFormat pix_fmt, int width,
}
}
switch (pix_fmt) {
case PIX_FMT_RGB8:
case PIX_FMT_BGR8:
case PIX_FMT_RGB4_BYTE:
case PIX_FMT_BGR4_BYTE:
case PIX_FMT_GRAY8:
// do not include palette for these pseudo-paletted formats
return size;
}
if (desc->flags & PIX_FMT_PAL)
memcpy((unsigned char *)(((size_t)dest + 3) & ~3), src->data[1], 256 * 4);