lzo: fix overflow checking in copy_backptr()

The check `src > dst' in the form `&c->out[-back] > c->out' invokes
pointer overflow, which is undefined behavior in C.

Remove the check.  Also replace `&c->out[-back] < c->out_start' with
a safe form `c->out - c->out_start < back' to avoid overflow.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit ca6c3f2c53)

Conflicts:

	libavutil/lzo.c
(cherry picked from commit ff712a262d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
This commit is contained in:
Xi Wang
2013-03-15 06:59:22 -04:00
committed by Michael Niedermayer
parent 90c8fa5221
commit 974c2ad87c

View File

@@ -118,10 +118,10 @@ static inline void memcpy_backptr(uint8_t *dst, int back, int cnt);
* cnt > back is valid, this will copy the bytes we just copied, * cnt > back is valid, this will copy the bytes we just copied,
* thus creating a repeating pattern with a period length of back. * thus creating a repeating pattern with a period length of back.
*/ */
static inline void copy_backptr(LZOContext *c, int back, int cnt) { static inline void copy_backptr(LZOContext *c, int back, int cnt)
register const uint8_t *src = &c->out[-back]; {
register uint8_t *dst = c->out; register uint8_t *dst = c->out;
if (src < c->out_start || src > dst) { if (dst - c->out_start < back) {
c->error |= AV_LZO_INVALID_BACKPTR; c->error |= AV_LZO_INVALID_BACKPTR;
return; return;
} }