[media] vivid: fix try_fmt behavior
authorHans Verkuil <hverkuil@xs4all.nl>
Mon, 6 Mar 2017 14:23:15 +0000 (11:23 -0300)
committerMauro Carvalho Chehab <mchehab@s-opensource.com>
Wed, 22 Mar 2017 13:16:17 +0000 (10:16 -0300)
commit5086924ad4f219ad6498b2cb217a98637b0b55b9
treeb19234532ffdef9582dc9fbd33a4a53f1cd2e210
parent733e009c37b6caa219d7057838582d1259c4554f
[media] vivid: fix try_fmt behavior

vivid_try_fmt_vid_cap() called tpg_calc_line_width to calculate the sizeimage
value, but that tpg function uses the current format, not the proposed (tried)
format.

Rewrote this code to calculate this correctly.

The vivid_try_fmt_vid_out() code was completely wrong w.r.t. sizeimage, and
neither did it take the vdownsampling[] factors into account.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Tested-by: Vincent Abriou <vincent.abriou@st.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
drivers/media/platform/vivid/vivid-vid-cap.c
drivers/media/platform/vivid/vivid-vid-out.c