KVM: add explicit barrier to kvm_vcpu_kick

kvm_vcpu_kick() must issue a general memory barrier prior to reading
vcpu->mode in order to ensure correctness of the mutual-exclusion
memory barrier pattern used with vcpu->requests.  While the cmpxchg
called from kvm_vcpu_kick():

 kvm_vcpu_kick
   kvm_arch_vcpu_should_kick
     kvm_vcpu_exiting_guest_mode
       cmpxchg

implies general memory barriers before and after the operation, that
implication is only valid when cmpxchg succeeds.  We need an explicit
barrier for when it fails, otherwise a VCPU thread on its entry path
that reads zero for vcpu->requests does not exclude the possibility
the requesting thread sees !IN_GUEST_MODE when it reads vcpu->mode.

kvm_make_all_cpus_request already had a barrier, so we remove it, as
now it would be redundant.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0936c3e2e51c13a743fa73972b9df44aaa97884e..69fcee26f4dae2e36072c856becc2020de9dd83e 100644 (file)
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6853,7 +6853,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
        /*
         * 1) We should set ->mode before checking ->requests.  Please see
-        * the comment in kvm_make_all_cpus_request.
+        * the comment in kvm_vcpu_exiting_guest_mode().
         *
         * 2) For APICv, we should set ->mode before checking PIR.ON.  This
         * pairs with the memory barrier implicit in pi_test_and_set_on

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index a805ddcb7eb050a124f2143cd0c995acf58883e4..84c5396564f76400459eb3d225d3246f642ce449 100644 (file)
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -270,6 +270,12 @@ struct kvm_vcpu {
 
 static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu)
 {
+       /*
+        * The memory barrier ensures a previous write to vcpu->requests cannot
+        * be reordered with the read of vcpu->mode.  It pairs with the general
+        * memory barrier following the write of vcpu->mode in VCPU RUN.
+        */
+       smp_mb__before_atomic();
        return cmpxchg(&vcpu->mode, IN_GUEST_MODE, EXITING_GUEST_MODE);
 }
 

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 3772f7dcc72d3f725b19c4abef5cbe980506f89f..1efb076430358f1bb4234a3e994d979919319af2 100644 (file)
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -183,9 +183,6 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
                kvm_make_request(req, vcpu);
                cpu = vcpu->cpu;
 
-               /* Set ->requests bit before we read ->mode. */
-               smp_mb__after_atomic();
-
                if (!(req & KVM_REQUEST_NO_WAKEUP))
                        kvm_vcpu_wake_up(vcpu);