common: netflix: merge patch for fix 29.97 and 23.97 output evenly [1/2]

PD#165744: netflix: merge patch for fix 29.97 and 23.97 output evenly

NEEDLEPLAT-2057:
[Netflix][NTS] Video judders are observed during VP9/HEVC playback [1/1]

[Problem]
[Netflix][NTS] Video judders are observed during VP9/HEVC playback.
The problem happens at 'Boardwalk Twirl Ride' scene with seek point at 760
 seconds. Seek before the point or the exact point will show stuttery video
 display but a seek after the location or a short seek (swim operation in
 NF's term) will make the playback smooth again. [Solution] The problem
 is identified to be an uneven PTS timestamp for sample frames. In order
 to get 29.97fps the timestamp of video samples are switching between a
 30fps rate and some samples with longer duration so by average it's
 29.97fps. The output rendering control has very strict timestamp comparison
 between system time and the timestamp of video frame. When system time passed
 the next video frame's timestamp, a new frame is toggled to display. For the
 uneven PTS case, some frames have over 2 vsync duration and the others have
 less than 2 vsync duration. Depending on the initial vsync offset, there is a
 chance to have frames toggled in 13221322 vsync sequence, instead of a normal
 always 2 vsync toggle one frame pattern. The change is to do a small adjustment
 to the system time, based on initial system time and video timestamp so the "phase"
 of the vsync can be set up to avoid such situation.
 [Platform]
 needle,stark
 [Test]
 Verify with Netflix Chime S1E8 29.97fps titles, 'Boardwalk Twirl Ride' scene.
 Change-Id: I073481ce9e39f49555480a139e3b32d8cc047e1c
 Signed-off-by: Tim Yao <tim.yao@amlog
 NEEDLEPLAT-2604: [Netflix] Video judders during playback [1/1]
 [JIRA] NEEDLEPLAT-2604
 [Problem]
 FRC caused video judders with 23.97fps source.
 [Solution]
 When source is 23.97 fps and output is 59.94hz, in order to make it
 friendly to TV 3:2 pulldown detection, the video frame output should follow a
 323232 patten, which means for each vsync, frames are repeated 3 times, then
 switch to a new frame and repeat twice. Such repeating pattern will make the
 average vsync toggle rate become 2.5 frame per vsync, to match 23.97 to 59.94
 frame conversion rate. And TV side can also detect such patterns and do its best
 recovery for MEMC processing. The problem with Netflix is the PTS of each vidoe
 frame is not incremented in a constant duration, but with some frames bigger then
 1/23.97 seconds and some are smaller. In Jira NEEDLEPLAT-2057 we have a similar
 processing for the case when source is 29.97fps and output is 59.94hz and for this
 Jira the reason is same. The fix for NEEDLEPLAT-2057 actually caused a side effect
 because it tried to set the initial vsync phase to 0.5, to maximumly avoid the
 problem, but for the 2:3 situation, a 0.5 phase make it worse because the average
 FRC ratio is 2.5, so an initial 0.5 phase will always make the output pattern not
 aligned at 232323 pattern when the duration of each frame is not even. This fix
 is to move the initial phase to 75% so both cases should work fine.
 [Test] Play typical HD source with 23.97fps and observe the playback smoothness.
 Also double check Chimera Boardwalk Twirl Ride from seeking point 02:34:14 and
 check the playback is smooth also.
 Change-Id: I95c35d4ffa563f74b9afa7ae08f7ef22d1227706
 Signed-off-by: Tim Yao <tim.yao@amlogic.com>
 NEEDLEPLAT-3173: [NTS]Frame stuttering on 4K 60fps [1/1]
 [JIRA] NEEDLEPLAT-3173
 [Problem] The test clip is 4K 60fps and the output is 4K
 59.94hz so a frame dropping always happen every 1000 frame. The clip is special
 in that the duration (PTS growing distance) between each frame is not same. The
 frame's PTS is incremened by (1530, 1530, 1440) to get an average of 1500 90K
 unit increase. (1500 = 90k / 60), which equals two 17ms dutation then followed
 by a 16ms duration frame to get an average of 16.6ms duration. The previous
 commit for NEEDLEPLAT-2604 and 2057 are the efforts to solve this problem by
 setting an initial phase, which can solve the problem for the uneven PTS for
 23.97 and 29.97fps cases. However, for 60fps source, the initial phase can not
 solve the problem because we can not maintain 1 vsync 1 frame switching always
 because eventually we need drop a frame for every 1000 frames. It means that
 during the playback, this initial phase will be shiftted. And when the vsync
 switching happens at those uneven PTS boundary, the same problem happens.
 [Solution] The change is a general frame switching improvement to avoid two
 situations: a) In one vsync, there are two frames toggled (frame dropping),
 but the next vsync there is no frame flipped because of its PTS has not arrived.
 This will cause unnecessary frame dropping and what we do is to move the second
 video frame to next vsync window. b) In one vsync, there are no frames be flipped
 (frame repeating), but the next vsync will have more than 1 frame flipped. It's
 also another frame dropping and what we do is to flip the next video frame in
 this vsync instead. So the idea is to make the frame switching at video display
 driver more smooth by avoid uneven frame flipping for each vsync. The timestamp
 of the test clip is not correct, but we can use this method to make output
 sequence smooth and avoid frame dropping.
 [Platform]
 needle,stark
 [Test]
 Verify with the special test clip according to the instruction. Verify
 NEEDLEPLAT-2604 and 2057 also.
Change-Id: Ic4dfc8aa243cf01acae296ac53fc2587583e601f
Signed-off-by: shuanglong.wang <shuanglong.wang@amlogic.com>

diff --git a/drivers/amlogic/media/frame_sync/tsync.c b/drivers/amlogic/media/frame_sync/tsync.c
index 34ed546ccdee738b8249e456755ce1d293f0fdae..cedceca9ef7cde12e852ba21054343cd0d62e0e7 100644 (file)
--- a/drivers/amlogic/media/frame_sync/tsync.c
+++ b/drivers/amlogic/media/frame_sync/tsync.c
@@ -763,6 +763,7 @@ void tsync_avevent_locked(enum avevent_e event, u32 param)
 
                if (slowsync_enable == 1) {     /* slow sync enable */
                        timestamp_pcrscr_set(param);
+                       set_pts_realign();
                        tsync_stat = TSYNC_STAT_PCRSCR_SETUP_VIDEO;
                } else {
                        if (tsync_stat == TSYNC_STAT_PCRSCR_SETUP_NONE) {
@@ -772,6 +773,7 @@ void tsync_avevent_locked(enum avevent_e event, u32 param)
                                                        VIDEO_HOLD_THRESHOLD);
                                } else
                                timestamp_pcrscr_set(param);
+                               set_pts_realign();
                                tsync_stat = TSYNC_STAT_PCRSCR_SETUP_VIDEO;
                        }
                }
@@ -785,6 +787,7 @@ void tsync_avevent_locked(enum avevent_e event, u32 param)
                                /* if this happens, then play */
                                tsync_stat = TSYNC_STAT_PCRSCR_SETUP_VIDEO;
                                timestamp_pcrscr_set(param);
+                               set_pts_realign();
                        }
                }
                if (/*tsync_mode == TSYNC_MODE_VMASTER && */ !vpause_flag)
@@ -836,11 +839,14 @@ void tsync_avevent_locked(enum avevent_e event, u32 param)
                                                          param - oldpts);
                }
                timestamp_vpts_set(param);
-               if (tsync_mode == TSYNC_MODE_VMASTER)
+               if (tsync_mode == TSYNC_MODE_VMASTER) {
                        timestamp_pcrscr_set(param);
-               else if (tsync_mode != oldmod
-                                && tsync_mode == TSYNC_MODE_AMASTER)
+                       set_pts_realign();
+               } else if (tsync_mode != oldmod
+                                && tsync_mode == TSYNC_MODE_AMASTER) {
                        timestamp_pcrscr_set(timestamp_apts_get());
+                       set_pts_realign();
+               }
        }
        break;
 
@@ -872,11 +878,14 @@ void tsync_avevent_locked(enum avevent_e event, u32 param)
                                                          param - oldpts);
                }
                timestamp_apts_set(param);
-               if (tsync_mode == TSYNC_MODE_AMASTER)
+               if (tsync_mode == TSYNC_MODE_AMASTER) {
                        timestamp_pcrscr_set(param);
-               else if (tsync_mode != oldmod
-                                && tsync_mode == TSYNC_MODE_VMASTER)
+                       set_pts_realign();
+               } else if (tsync_mode != oldmod
+                                && tsync_mode == TSYNC_MODE_VMASTER) {
                        timestamp_pcrscr_set(timestamp_vpts_get());
+                       set_pts_realign();
+               }
        }
        break;
 
@@ -925,10 +934,13 @@ void tsync_avevent_locked(enum avevent_e event, u32 param)
                                timestamp_pcrscr_set(param);
                        else
                                timestamp_pcrscr_set(vpts);
+                       set_pts_realign();
                        tsync_dec_reset_flag = 0;
                        tsync_dec_reset_video_start = 0;
-               } else if (tsync_mode == TSYNC_MODE_AMASTER)
+               } else if (tsync_mode == TSYNC_MODE_AMASTER) {
                        timestamp_pcrscr_set(param);
+                       set_pts_realign();
+               }
 
                tsync_stat = TSYNC_STAT_PCRSCR_SETUP_AUDIO;
 
@@ -1190,6 +1202,7 @@ int tsync_set_apts(unsigned int pts)
                                pr_info("pcr 0x%x, diff %d\n",
                                                t, pts - t);
                                timestamp_pcrscr_set(pts);
+                               set_pts_realign();
                        } else if ((!get_vsync_pts_inc_mode())
                                           && (abs(timestamp_apts_get() - t) >
                                                   100 * TIME_UNIT90K / 1000)) {
@@ -1199,6 +1212,7 @@ int tsync_set_apts(unsigned int pts)
                                pr_info("pcr 0x%x, diff %d\n",
                                                t, pts - t);
                                timestamp_pcrscr_set(pts);
+                               set_pts_realign();
                        }
                }
        } else if ((oldmod != tsync_mode) && (tsync_mode == TSYNC_MODE_VMASTER))
@@ -1509,6 +1523,7 @@ static ssize_t store_pcrscr(struct class *class,
                return -EINVAL;
 
        timestamp_pcrscr_set(pts);
+       set_pts_realign();
 
        return size;
 }

diff --git a/drivers/amlogic/media/video_processor/video_dev/amlvideo.c b/drivers/amlogic/media/video_processor/video_dev/amlvideo.c
index 0dee419503878c2a9c8966798ad6f3a84a5f0758..a6cefe9eb2820d2ffb7eb2d06091f706096661bd 100644 (file)
--- a/drivers/amlogic/media/video_processor/video_dev/amlvideo.c
+++ b/drivers/amlogic/media/video_processor/video_dev/amlvideo.c
@@ -509,6 +509,7 @@ static int vidioc_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
        struct vivi_dev *dev = video_drvdata(file);
        int ret = 0;
        u64 pts_us64 = 0;
+       struct vframe_s *next_vf;
 
        if (vfq_level(&dev->q_ready) > AMLVIDEO_POOL_SIZE - 1)
                return -EAGAIN;
@@ -537,6 +538,10 @@ static int vidioc_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
                AMLVIDEO_WARN("pts= %d, dev->vf->duration= %d\n",
                        dev->vf->pts, (DUR2PTS(dev->vf->duration)));
        }
+       next_vf = vf_peek(dev->vf_receiver_name);
+       dev->vf->next_vf_pts_valid = next_vf != NULL;
+       if (dev->vf->next_vf_pts_valid)
+               dev->vf->next_vf_pts = next_vf->pts;
 
        vfq_push(&dev->q_ready, dev->vf);
        p->index = 0;

diff --git a/drivers/amlogic/media/video_sink/video.c b/drivers/amlogic/media/video_sink/video.c
index 09d058e22d7e6c5fbabbb87af88e31bad01c8e5c..4544b7a68d7781aba72a48560ce7797ed1db309c 100644 (file)
--- a/drivers/amlogic/media/video_sink/video.c
+++ b/drivers/amlogic/media/video_sink/video.c
@@ -527,6 +527,18 @@ struct video_pm_state_s {
 
 #endif
 
+#define PTS_THROTTLE
+/* #define PTS_TRACE_DEBUG */
+/* #define PTS_TRACE_START */
+
+#ifdef PTS_TRACE_DEBUG
+static int pts_trace;
+static int pts_trace_his[16];
+static u32 pts_his[16];
+static u32 scr_his[16];
+static int pts_trace_his_rd;
+#endif
+
 static DEFINE_MUTEX(video_module_mutex);
 static DEFINE_SPINLOCK(lock);
 static u32 frame_par_ready_to_set, frame_par_force_to_set;
@@ -777,6 +789,10 @@ static u32 vsync_pts_100;
 static u32 vsync_freerun;
 static u32 vsync_slow_factor = 1;
 
+/* pts alignment */
+static bool vsync_pts_aligned;
+static s32 vsync_pts_align;
+
 /* frame rate calculate */
 static u32 last_frame_count;
 static u32 frame_count;
@@ -834,6 +850,12 @@ void set_freerun_mode(int mode)
 }
 EXPORT_SYMBOL(set_freerun_mode);
 
+void set_pts_realign(void)
+{
+       vsync_pts_aligned = false;
+}
+EXPORT_SYMBOL(set_pts_realign);
+
 static const enum f2v_vphase_type_e vpp_phase_table[4][3] = {
        {F2V_P2IT, F2V_P2IB, F2V_P2P},  /* VIDTYPE_PROGRESSIVE */
        {F2V_IT2IT, F2V_IT2IB, F2V_IT2P},       /* VIDTYPE_INTERLACE_TOP */
@@ -2084,6 +2106,23 @@ static void vsync_toggle_frame(struct vframe_s *vf)
 
        if (is_dolby_vision_enable())
                vf_with_el = has_enhanced_layer(vf);
+
+#ifdef PTS_TRACE_DEBUG
+#ifdef PTS_TRACE_START
+               if (pts_trace_his_rd < 15) {
+#endif
+                       pts_trace_his[pts_trace_his_rd] = pts_trace;
+                       pts_his[pts_trace_his_rd] = vf->pts;
+                       scr_his[pts_trace_his_rd] = timestamp_pcrscr_get();
+                       pts_trace_his_rd++;
+                       if (pts_trace_his_rd >= 16)
+                               pts_trace_his_rd = 0;
+#ifdef PTS_TRACE_START
+               }
+#endif
+               pts_trace = 0;
+#endif
+
        ori_start_x_lines = 0;
        ori_end_x_lines = ((vf->type & VIDTYPE_COMPRESS) ?
                vf->compWidth : vf->width) - 1;
@@ -2526,6 +2565,24 @@ static void vsync_toggle_frame(struct vframe_s *vf)
        }
        if (cur_dispbuf != &vf_local)
                video_keeper_new_frame_notify();
+
+       if ((vf != &vf_local) && (vf) && !vsync_pts_aligned) {
+#ifdef PTS_TRACE_DEBUG
+               pr_info("####timestamp_pcrscr_get() = 0x%x, vf->pts = 0x%x, vsync_pts_inc = %d\n",
+                       timestamp_pcrscr_get(), vf->pts, vsync_pts_inc);
+#endif
+               if ((abs(timestamp_pcrscr_get() - vf->pts) <= (vsync_pts_inc))
+                         && ((int)(timestamp_pcrscr_get() - vf->pts) >= 0)) {
+                       vsync_pts_align =  vsync_pts_inc / 4 -
+                               (timestamp_pcrscr_get() - vf->pts);
+                       vsync_pts_aligned = true;
+#ifdef PTS_TRACE_DEBUG
+                       pts_trace_his_rd = 0;
+                       pr_info("####vsync_pts_align set to %d\n",
+                               vsync_pts_align);
+#endif
+               }
+       }
 }
 static inline void vd1_path_select(bool afbc)
 {
@@ -3658,14 +3715,19 @@ static inline bool duration_expire(struct vframe_s *cur_vf,
 #define VPTS_RESET_THRO
 
 static inline bool vpts_expire(struct vframe_s *cur_vf,
-                              struct vframe_s *next_vf)
+                              struct vframe_s *next_vf,
+                              int toggled_cnt)
 {
-       u32 pts = next_vf->pts;
+       u32 pts;
 #ifdef VIDEO_PTS_CHASE
        u32 vid_pts, scr_pts;
 #endif
        u32 systime;
        u32 adjust_pts, org_vpts;
+       bool expired;
+
+       if (next_vf == NULL)
+               return false;
 
        if (debug_flag & DEBUG_FLAG_TOGGLE_FRAME_PER_VSYNC)
                return true;
@@ -3712,6 +3774,7 @@ static inline bool vpts_expire(struct vframe_s *cur_vf,
                return true;
 
        systime = timestamp_pcrscr_get();
+       pts = next_vf->pts;
 
        if (((pts == 0) && (cur_dispbuf != &vf_local))
            || (freerun_mode == FREERUN_DUR)) {
@@ -3902,7 +3965,27 @@ static inline bool vpts_expire(struct vframe_s *cur_vf,
                }
        }
 
-       return (int)(timestamp_pcrscr_get() - pts) >= 0;
+       expired = (int)(timestamp_pcrscr_get() + vsync_pts_align - pts) >= 0;
+
+#ifdef PTS_THROTTLE
+       if (expired && next_vf && next_vf->next_vf_pts_valid &&
+               (vsync_slow_factor == 1) &&
+               next_vf->next_vf_pts &&
+               (toggled_cnt > 0) &&
+               ((int)(timestamp_pcrscr_get() + vsync_pts_inc +
+               vsync_pts_align - next_vf->next_vf_pts) < 0)) {
+               expired = false;
+       } else if (!expired && next_vf && next_vf->next_vf_pts_valid &&
+               (vsync_slow_factor == 1) &&
+               next_vf->next_vf_pts &&
+               (toggled_cnt == 0) &&
+               ((int)(timestamp_pcrscr_get() + vsync_pts_inc +
+               vsync_pts_align - next_vf->next_vf_pts) >= 0)) {
+               expired = true;
+       }
+#endif
+
+               return expired;
 #endif
 }
 
@@ -4396,9 +4479,7 @@ static irqreturn_t vsync_isr_in(int irq, void *dev_id)
        bool show_nosync = false;
        u32 vpp_misc_save, vpp_misc_set;
        int first_set = 0;
-#ifdef CONFIG_AM_VIDEO_LOG
        int toggle_cnt;
-#endif
        struct vframe_s *toggle_vf = NULL;
        struct vframe_s *toggle_frame = NULL;
        int video1_off_req = 0;
@@ -4485,9 +4566,7 @@ static irqreturn_t vsync_isr_in(int irq, void *dev_id)
                        old_vmode = new_vmode;
                }
        }
-#ifdef CONFIG_AM_VIDEO_LOG
        toggle_cnt = 0;
-#endif
        vsync_count++;
        timer_count++;
 
@@ -4752,7 +4831,7 @@ static irqreturn_t vsync_isr_in(int irq, void *dev_id)
        }
 
        while (vf) {
-               if (vpts_expire(cur_dispbuf, vf) || show_nosync) {
+               if (vpts_expire(cur_dispbuf, vf, toggle_cnt) || show_nosync) {
                        amlog_mask(LOG_MASK_TIMESTAMP,
                        "vpts = 0x%x, c.dur=0x%x, n.pts=0x%x, scr = 0x%x\n",
                                   timestamp_vpts_get(),
@@ -4971,9 +5050,7 @@ static irqreturn_t vsync_isr_in(int irq, void *dev_id)
                        break;
                }
 
-#ifdef CONFIG_AM_VIDEO_LOG
                toggle_cnt++;
-#endif
        }
 
 #ifdef INTERLACE_FIELD_MATCH_PROCESS
@@ -5580,6 +5657,9 @@ SET_FILTER:
        }
 
  exit:
+#ifdef PTS_TRACE_DEBUG
+               pts_trace++;
+#endif
        vpp_misc_save = READ_VCBUS_REG(VPP_MISC + cur_dev->vpp_off);
        vpp_misc_set = vpp_misc_save;
 #ifdef CONFIG_AMLOGIC_MEDIA_ENHANCEMENT_VECM
@@ -6058,6 +6138,8 @@ static void video_vf_unreg_provider(void)
        vsync_pts_112 = 0;
        vsync_pts_125 = 0;
        vsync_freerun = 0;
+       vsync_pts_align = 0;
+       vsync_pts_aligned = false;
        video_prot.video_started = 0;
        spin_unlock_irqrestore(&lock, flags);
 
@@ -7380,6 +7462,33 @@ static ssize_t video_seek_flag_store(struct class *cla,
        return count;
 }
 
+#ifdef PTS_TRACE_DEBUG
+static ssize_t pts_trace_show(struct class *cla,
+                       struct class_attribute *attr, char *buf)
+{
+       return sprintf(buf, "%d %d %d %d %d %d %d %d\n"
+                               "%d %d %d %d %d %d %d %d\n"
+                               "%0x %0x %0x %0x %0x %0x %0x %0x\n"
+                               "%0x %0x %0x %0x %0x %0x %0x %0x\n"
+                               "%0x %0x %0x %0x %0x %0x %0x %0x\n"
+                               "%0x %0x %0x %0x %0x %0x %0x %0x\n",
+               pts_trace_his[0], pts_trace_his[1], pts_trace_his[2],
+               pts_trace_his[3], pts_trace_his[4], pts_trace_his[5],
+               pts_trace_his[6], pts_trace_his[7], pts_trace_his[8],
+               pts_trace_his[9], pts_trace_his[10], pts_trace_his[11],
+               pts_trace_his[12], pts_trace_his[13], pts_trace_his[14],
+               pts_trace_his[15],
+               pts_his[0], pts_his[1], pts_his[2], pts_his[3],
+               pts_his[4], pts_his[5], pts_his[6], pts_his[7],
+               pts_his[8], pts_his[9], pts_his[10], pts_his[11],
+               pts_his[12], pts_his[13], pts_his[14], pts_his[15],
+               scr_his[0], scr_his[1], scr_his[2], scr_his[3],
+               scr_his[4], scr_his[5], scr_his[6], scr_his[7],
+               scr_his[8], scr_his[9], scr_his[10], scr_his[11],
+               scr_his[12], scr_his[13], scr_his[14], scr_his[15]);
+}
+#endif
+
 static ssize_t video_brightness_show(struct class *cla,
                                     struct class_attribute *attr, char *buf)
 {
@@ -8781,6 +8890,9 @@ static struct class_attribute amvideo_class_attrs[] = {
               free_cma_buffer_store),
 #ifdef CONFIG_AM_VOUT
        __ATTR_RO(device_resolution),
+#endif
+#ifdef PTS_TRACE_DEBUG
+       __ATTR_RO(pts_trace),
 #endif
        __ATTR_RO(frame_addr),
        __ATTR_RO(frame_canvas_width),

diff --git a/include/linux/amlogic/media/frame_sync/tsync.h b/include/linux/amlogic/media/frame_sync/tsync.h
index eaf9b1597ac65e46a5b709deb07c2ddc53cb830e..2194d6a18609ab115c3accc898a6996ba43fccec 100644 (file)
--- a/include/linux/amlogic/media/frame_sync/tsync.h
+++ b/include/linux/amlogic/media/frame_sync/tsync.h
@@ -155,6 +155,8 @@ extern int tsync_set_av_threshold_min(int min);
 
 extern int tsync_set_av_threshold_max(int max);
 
+extern void set_pts_realign(void);
+
 static inline u32 tsync_vpts_discontinuity_margin(void)
 {
        return tsync_get_av_threshold_min();

diff --git a/include/linux/amlogic/media/vfm/vframe.h b/include/linux/amlogic/media/vfm/vframe.h
index b56fe5a89b0b9c26993276b712ac0e0370c55cee..9e53f1e485c9a3f6a7f506c10ec96a59a9bfb1f4 100644 (file)
--- a/include/linux/amlogic/media/vfm/vframe.h
+++ b/include/linux/amlogic/media/vfm/vframe.h
@@ -252,6 +252,8 @@ struct vframe_s {
        u32 duration_pulldown;
        u32 pts;
        u64 pts_us64;
+       bool next_vf_pts_valid;
+       u32 next_vf_pts;
        u32 disp_pts;
        u64 disp_pts_us64;
        u32 flag;