BACKPORT: sched/fair: Use util_est in LB and WU paths

When the scheduler looks at the CPU utilization, the current PELT value
for a CPU is returned straight away. In certain scenarios this can have
undesired side effects on task placement.

For example, since the task utilization is decayed at wakeup time, when
a long sleeping big task is enqueued it does not add immediately a
significant contribution to the target CPU.
As a result we generate a race condition where other tasks can be placed
on the same CPU while it is still considered relatively empty.

In order to reduce this kind of race conditions, this patch introduces the
required support to integrate the usage of the CPU's estimated utilization
in the wakeup path, via cpu_util_wake(), as well as in the load-balance
path, via cpu_util() which is used by update_sg_lb_stats().

The estimated utilization of a CPU is defined to be the maximum between
its PELT's utilization and the sum of the estimated utilization (at
previous dequeue time) of all the tasks currently RUNNABLE on that CPU.
This allows to properly represent the spare capacity of a CPU which, for
example, has just got a big task running since a long sleep period.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Morten Rasmussen <morten.rasmussen@arm.com>
Cc: Paul Turner <pjt@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rafael J . Wysocki <rafael.j.wysocki@intel.com>
Cc: Steve Muckle <smuckle@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Todd Kjos <tkjos@android.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Link: http://lkml.kernel.org/r/20180309095245.11071-3-patrick.bellasi@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
[
 backport from upstream:
 commit f9be3e5 ("sched/fair: Use util_est in LB and WU paths")

 This provides also schedutil integration, since:
    sugov_get_util()
       boosted_cpu_util()
          cpu_util_freq()
             cpu_util()

 thus, not requiring to backport:
 commit a07630b8b2c1 ("sched/cpufreq/schedutil: Use util_est for OPP selection")

 Support for energy_diff is also provided, since:
    calc_sg_energy()
       find_new_capacity()
          group_max_util()
             cpu_util_wake()
 and:
       group_norm_util()
          cpu_util_wake()

 Where both cpu_util() and cpu_util_wake() already consider the estimated
 utlilization in case of PELT being in use.
]
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Change-Id: I2be201cf7bb0b1449b14b4da64844067dbdb5eb4

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 56f080f3b8a5d316f60db637c85b995bbef860f4..89b6dac144d522984468a2a191e45b30d8de827f 100644 (file)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3635,6 +3635,11 @@ static inline unsigned long _task_util_est(struct task_struct *p)
 
 static inline unsigned long task_util_est(struct task_struct *p)
 {
+#ifdef CONFIG_SCHED_WALT
+       if (likely(!walt_disabled && sysctl_sched_use_walt_task_util))
+               return (p->ravg.demand /
+                       (walt_ravg_window >> SCHED_CAPACITY_SHIFT));
+#endif
        return max(task_util(p), _task_util_est(p));
 }
 
@@ -5767,11 +5772,13 @@ struct energy_env {
        struct sched_group      *sg;
 };
 
-/*
- * cpu_util returns the amount of capacity of a CPU that is used by CFS
- * tasks. The unit of the return value must be the one of capacity so we can
- * compare the utilization with the capacity of the CPU that is available for
- * CFS task (ie cpu_capacity).
+/**
+ * Amount of capacity of a CPU that is (estimated to be) used by CFS tasks
+ * @cpu: the CPU to get the utilization of
+ *
+ * The unit of the return value must be the one of capacity so we can compare
+ * the utilization with the capacity of the CPU that is available for CFS task
+ * (ie cpu_capacity).
  *
  * cfs_rq.avg.util_avg is the sum of running time of runnable tasks plus the
  * recent utilization of currently non-runnable tasks on a CPU. It represents
@@ -5782,6 +5789,14 @@ struct energy_env {
  * current capacity (capacity_curr <= capacity_orig) of the CPU because it is
  * the running time on this CPU scaled by capacity_curr.
  *
+ * The estimated utilization of a CPU is defined to be the maximum between its
+ * cfs_rq.avg.util_avg and the sum of the estimated utilization of the tasks
+ * currently RUNNABLE on that CPU.
+ * This allows to properly represent the expected utilization of a CPU which
+ * has just got a big task running since a long sleep period. At the same time
+ * however it preserves the benefits of the "blocked utilization" in
+ * describing the potential for other tasks waking up on the same CPU.
+ *
  * Nevertheless, cfs_rq.avg.util_avg can be higher than capacity_curr or even
  * higher than capacity_orig because of unfortunate rounding in
  * cfs.avg.util_avg or just after migrating tasks and new task wakeups until
@@ -5792,6 +5807,8 @@ struct energy_env {
  * available capacity. We allow utilization to overshoot capacity_curr (but not
  * capacity_orig) as it useful for predicting the capacity required after task
  * migrations (scheduler-driven DVFS).
+ *
+ * Return: the (estimated) utilization for the specified CPU
  */
 static inline unsigned long cpu_util(int cpu)
 {
@@ -5813,6 +5830,9 @@ static inline unsigned long cpu_util(int cpu)
        cfs_rq = &cpu_rq(cpu)->cfs;
        util = READ_ONCE(cfs_rq->avg.util_avg);
 
+       if (sched_feat(UTIL_EST))
+               util = max(util, READ_ONCE(cfs_rq->avg.util_est.enqueued));
+
        return min_t(unsigned long, util, capacity_orig_of(cpu));
 }
 
@@ -5864,6 +5884,35 @@ static unsigned long cpu_util_wake(int cpu, struct task_struct *p)
        /* Discount task's blocked util from CPU's util */
        util -= min_t(unsigned int, util, task_util(p));
 
+       /*
+        * Covered cases:
+        *
+        * a) if *p is the only task sleeping on this CPU, then:
+        *      cpu_util (== task_util) > util_est (== 0)
+        *    and thus we return:
+        *      cpu_util_wake = (cpu_util - task_util) = 0
+        *
+        * b) if other tasks are SLEEPING on this CPU, which is now exiting
+        *    IDLE, then:
+        *      cpu_util >= task_util
+        *      cpu_util > util_est (== 0)
+        *    and thus we discount *p's blocked utilization to return:
+        *      cpu_util_wake = (cpu_util - task_util) >= 0
+        *
+        * c) if other tasks are RUNNABLE on that CPU and
+        *      util_est > cpu_util
+        *    then we use util_est since it returns a more restrictive
+        *    estimation of the spare capacity on that CPU, by just
+        *    considering the expected utilization of tasks already
+        *    runnable on that CPU.
+        *
+        * Cases a) and b) are covered by the above code, while case c) is
+        * covered by the following code when estimated utilization is
+        * enabled.
+        */
+       if (sched_feat(UTIL_EST))
+               util = max(util, READ_ONCE(cfs_rq->avg.util_est.enqueued));
+
        /*
         * Utilization (estimated) can exceed the CPU capacity, thus let's
         * clamp to the maximum CPU capacity to ensure consistency with
@@ -6500,7 +6549,7 @@ schedtune_task_margin(struct task_struct *task)
        if (boost == 0)
                return 0;
 
-       util = task_util(task);
+       util = task_util_est(task);
        margin = schedtune_margin(util, boost);
 
        return margin;
@@ -6536,7 +6585,7 @@ boosted_cpu_util(int cpu)
 static inline unsigned long
 boosted_task_util(struct task_struct *task)
 {
-       unsigned long util = task_util(task);
+       unsigned long util = task_util_est(task);
        long margin = schedtune_task_margin(task);
 
        trace_sched_boost_task(task, util, margin);
@@ -7145,7 +7194,7 @@ static inline int find_best_target(struct task_struct *p, int *backup_cpu,
                         * accounting. However, the blocked utilization may be zero.
                         */
                        wake_util = cpu_util_wake(i, p);
-                       new_util = wake_util + task_util(p);
+                       new_util = wake_util + task_util_est(p);
 
                        /*
                         * Ensure minimum capacity to grant the required boost.
@@ -7542,7 +7591,7 @@ static inline struct energy_env *get_eenv(struct task_struct *p, int prev_cpu)
         * during energy calculation, but unboosted task
         * util for group utilization calculations
         */
-       eenv->util_delta = task_util(p);
+       eenv->util_delta = task_util_est(p);
        eenv->util_delta_boosted = boosted_task_util(p);
 
        cpumask_and(&cpumask_possible_cpus, &p->cpus_allowed, cpu_online_mask);
@@ -7597,9 +7646,12 @@ static int find_energy_efficient_cpu(struct sched_domain *sd,
                        if (cpu_iter == prev_cpu)
                                continue;
 
+                       /*
+                        * Consider only CPUs where the task is expected to
+                        * fit without making the CPU overutilized.
+                        */
                        spare = capacity_spare_wake(cpu_iter, p);
-
-                       if (spare * 1024 < capacity_margin * task_util(p))
+                       if (spare * 1024 < capacity_margin * task_util_est(p))
                                continue;
 
                        /* Add CPU candidate */
@@ -7701,7 +7753,7 @@ static inline int wake_energy(struct task_struct *p, int prev_cpu,
         * the heuristics we use there in selecting candidate
         * CPUs.
         */
-       if (unlikely(!sched_feat(FIND_BEST_TARGET) && !task_util(p)))
+       if (unlikely(!sched_feat(FIND_BEST_TARGET) && !task_util_est(p)))
                return false;
 
        if(!sched_feat(EAS_PREFER_IDLE)){