FROMLIST: arm64: erratum: Work around Falkor erratum #E1003 in trampoline code

We rely on an atomic swizzling of TTBR1 when transitioning from the entry
trampoline to the kernel proper on an exception. We can't rely on this
atomicity in the face of Falkor erratum #E1003, so on affected cores we
can issue a TLB invalidation to invalidate the walk cache prior to
jumping into the kernel. There is still the possibility of a TLB conflict
here due to conflicting walk cache entries prior to the invalidation, but
this doesn't appear to be the case on these CPUs in practice.

Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Laura Abbott <labbott@redhat.com>
Tested-by: Shanker Donthineni <shankerd@codeaurora.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
(cherry picked from git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
 commit d1777e686ad10ba7c594304429c6045fb79255a1)

Change-Id: Ia6c7ffd47745c179738250afa01cb8bf8594b235
[ghackmann@google.com: replace runtime alternative_if with a
 compile-time check for Code Aurora's out-of-tree CONFIG_ARCH_MSM8996.
 Kryo needs this workaround too, and 4.4 doesn't have any of the
 upstream Falkor errata infrastructure needed to detect this at boot time.]
Signed-off-by: Greg Hackmann <ghackmann@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index bf52151f1e20d967d87bb0b38181d42a23b2c94d..5be224cd01e7d67a8ab7ec994a67d22e5670b62a 100644 (file)
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -953,6 +953,16 @@ __ni_sys_trace:
        sub     \tmp, \tmp, #(SWAPPER_DIR_SIZE + RESERVED_TTBR0_SIZE)
        bic     \tmp, \tmp, #USER_ASID_FLAG
        msr     ttbr1_el1, \tmp
+#ifdef CONFIG_ARCH_MSM8996
+       /* ASID already in \tmp[63:48] */
+       movk    \tmp, #:abs_g2_nc:(TRAMP_VALIAS >> 12)
+       movk    \tmp, #:abs_g1_nc:(TRAMP_VALIAS >> 12)
+       /* 2MB boundary containing the vectors, so we nobble the walk cache */
+       movk    \tmp, #:abs_g0_nc:((TRAMP_VALIAS & ~(SZ_2M - 1)) >> 12)
+       isb
+       tlbi    vae1, \tmp
+       dsb     nsh
+#endif /* CONFIG_ARCH_MSM8996 */
        .endm
 
        .macro tramp_unmap_kernel, tmp