generic-library/ffmpeg
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

rv34: fix invalid memory access for small video dimensions

Browse Source 
For small video dimensions calculations of the upper bound for pixel
access may result in negative value. Using an unsigned comparison
works only if the greater operand is non-negative. This is fixed by
doing edge emulation explicitly for such conditions.

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit aacf6b3a2f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
This commit is contained in:
John Brooks
2012-01-03 23:27:35 -07:00
committed by Michael Niedermayer
parent 68f1212696
commit fc9c5ad9ea
1 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
libavcodec/rv34.c
View File

@@ -690,8 +690,9 @@ static inline void rv34_mc(RV34DecContext *r, const int block_type,
srcY += src_y * s->linesize + src_x;
srcU += uvsrc_y * s->uvlinesize + uvsrc_x;
srcV += uvsrc_y * s->uvlinesize + uvsrc_x;
if( (unsigned)(src_x - !!lx*2) > s->h_edge_pos - !!lx*2 - (width <<3) - 4
|| (unsigned)(src_y - !!ly*2) > s->v_edge_pos - !!ly*2 - (height<<3) - 4){
if(s->h_edge_pos - (width << 3) < 6 || s->v_edge_pos - (height << 3) < 6 ||
(unsigned)(src_x - !!lx*2) > s->h_edge_pos - !!lx*2 - (width <<3) - 4 ||
(unsigned)(src_y - !!ly*2) > s->v_edge_pos - !!ly*2 - (height<<3) - 4) {
uint8_t *uvbuf = s->edge_emu_buffer + 22 * s->linesize;
srcY -= 2 + 2*s->linesize;