Commit | Line | Data |
---|---|---|
108b42b4 DH |
1 | ============================ |
2 | LINUX KERNEL MEMORY BARRIERS | |
3 | ============================ | |
4 | ||
5 | By: David Howells <dhowells@redhat.com> | |
6 | ||
7 | Contents: | |
8 | ||
9 | (*) Abstract memory access model. | |
10 | ||
11 | - Device operations. | |
12 | - Guarantees. | |
13 | ||
14 | (*) What are memory barriers? | |
15 | ||
16 | - Varieties of memory barrier. | |
17 | - What may not be assumed about memory barriers? | |
18 | - Data dependency barriers. | |
19 | - Control dependencies. | |
20 | - SMP barrier pairing. | |
21 | - Examples of memory barrier sequences. | |
22 | ||
23 | (*) Explicit kernel barriers. | |
24 | ||
25 | - Compiler barrier. | |
26 | - The CPU memory barriers. | |
27 | - MMIO write barrier. | |
28 | ||
29 | (*) Implicit kernel memory barriers. | |
30 | ||
31 | - Locking functions. | |
32 | - Interrupt disabling functions. | |
33 | - Miscellaneous functions. | |
34 | ||
35 | (*) Inter-CPU locking barrier effects. | |
36 | ||
37 | - Locks vs memory accesses. | |
38 | - Locks vs I/O accesses. | |
39 | ||
40 | (*) Where are memory barriers needed? | |
41 | ||
42 | - Interprocessor interaction. | |
43 | - Atomic operations. | |
44 | - Accessing devices. | |
45 | - Interrupts. | |
46 | ||
47 | (*) Kernel I/O barrier effects. | |
48 | ||
49 | (*) Assumed minimum execution ordering model. | |
50 | ||
51 | (*) The effects of the cpu cache. | |
52 | ||
53 | - Cache coherency. | |
54 | - Cache coherency vs DMA. | |
55 | - Cache coherency vs MMIO. | |
56 | ||
57 | (*) The things CPUs get up to. | |
58 | ||
59 | - And then there's the Alpha. | |
60 | ||
61 | (*) References. | |
62 | ||
63 | ||
64 | ============================ | |
65 | ABSTRACT MEMORY ACCESS MODEL | |
66 | ============================ | |
67 | ||
68 | Consider the following abstract model of the system: | |
69 | ||
70 | : : | |
71 | : : | |
72 | : : | |
73 | +-------+ : +--------+ : +-------+ | |
74 | | | : | | : | | | |
75 | | | : | | : | | | |
76 | | CPU 1 |<----->| Memory |<----->| CPU 2 | | |
77 | | | : | | : | | | |
78 | | | : | | : | | | |
79 | +-------+ : +--------+ : +-------+ | |
80 | ^ : ^ : ^ | |
81 | | : | : | | |
82 | | : | : | | |
83 | | : v : | | |
84 | | : +--------+ : | | |
85 | | : | | : | | |
86 | | : | | : | | |
87 | +---------->| Device |<----------+ | |
88 | : | | : | |
89 | : | | : | |
90 | : +--------+ : | |
91 | : : | |
92 | ||
93 | Each CPU executes a program that generates memory access operations. In the | |
94 | abstract CPU, memory operation ordering is very relaxed, and a CPU may actually | |
95 | perform the memory operations in any order it likes, provided program causality | |
96 | appears to be maintained. Similarly, the compiler may also arrange the | |
97 | instructions it emits in any order it likes, provided it doesn't affect the | |
98 | apparent operation of the program. | |
99 | ||
100 | So in the above diagram, the effects of the memory operations performed by a | |
101 | CPU are perceived by the rest of the system as the operations cross the | |
102 | interface between the CPU and rest of the system (the dotted lines). | |
103 | ||
104 | ||
105 | For example, consider the following sequence of events: | |
106 | ||
107 | CPU 1 CPU 2 | |
108 | =============== =============== | |
109 | { A == 1; B == 2 } | |
110 | A = 3; x = A; | |
111 | B = 4; y = B; | |
112 | ||
113 | The set of accesses as seen by the memory system in the middle can be arranged | |
114 | in 24 different combinations: | |
115 | ||
116 | STORE A=3, STORE B=4, x=LOAD A->3, y=LOAD B->4 | |
117 | STORE A=3, STORE B=4, y=LOAD B->4, x=LOAD A->3 | |
118 | STORE A=3, x=LOAD A->3, STORE B=4, y=LOAD B->4 | |
119 | STORE A=3, x=LOAD A->3, y=LOAD B->2, STORE B=4 | |
120 | STORE A=3, y=LOAD B->2, STORE B=4, x=LOAD A->3 | |
121 | STORE A=3, y=LOAD B->2, x=LOAD A->3, STORE B=4 | |
122 | STORE B=4, STORE A=3, x=LOAD A->3, y=LOAD B->4 | |
123 | STORE B=4, ... | |
124 | ... | |
125 | ||
126 | and can thus result in four different combinations of values: | |
127 | ||
128 | x == 1, y == 2 | |
129 | x == 1, y == 4 | |
130 | x == 3, y == 2 | |
131 | x == 3, y == 4 | |
132 | ||
133 | ||
134 | Furthermore, the stores committed by a CPU to the memory system may not be | |
135 | perceived by the loads made by another CPU in the same order as the stores were | |
136 | committed. | |
137 | ||
138 | ||
139 | As a further example, consider this sequence of events: | |
140 | ||
141 | CPU 1 CPU 2 | |
142 | =============== =============== | |
143 | { A == 1, B == 2, C = 3, P == &A, Q == &C } | |
144 | B = 4; Q = P; | |
145 | P = &B D = *Q; | |
146 | ||
147 | There is an obvious data dependency here, as the value loaded into D depends on | |
148 | the address retrieved from P by CPU 2. At the end of the sequence, any of the | |
149 | following results are possible: | |
150 | ||
151 | (Q == &A) and (D == 1) | |
152 | (Q == &B) and (D == 2) | |
153 | (Q == &B) and (D == 4) | |
154 | ||
155 | Note that CPU 2 will never try and load C into D because the CPU will load P | |
156 | into Q before issuing the load of *Q. | |
157 | ||
158 | ||
159 | DEVICE OPERATIONS | |
160 | ----------------- | |
161 | ||
162 | Some devices present their control interfaces as collections of memory | |
163 | locations, but the order in which the control registers are accessed is very | |
164 | important. For instance, imagine an ethernet card with a set of internal | |
165 | registers that are accessed through an address port register (A) and a data | |
166 | port register (D). To read internal register 5, the following code might then | |
167 | be used: | |
168 | ||
169 | *A = 5; | |
170 | x = *D; | |
171 | ||
172 | but this might show up as either of the following two sequences: | |
173 | ||
174 | STORE *A = 5, x = LOAD *D | |
175 | x = LOAD *D, STORE *A = 5 | |
176 | ||
177 | the second of which will almost certainly result in a malfunction, since it set | |
178 | the address _after_ attempting to read the register. | |
179 | ||
180 | ||
181 | GUARANTEES | |
182 | ---------- | |
183 | ||
184 | There are some minimal guarantees that may be expected of a CPU: | |
185 | ||
186 | (*) On any given CPU, dependent memory accesses will be issued in order, with | |
187 | respect to itself. This means that for: | |
188 | ||
189 | Q = P; D = *Q; | |
190 | ||
191 | the CPU will issue the following memory operations: | |
192 | ||
193 | Q = LOAD P, D = LOAD *Q | |
194 | ||
195 | and always in that order. | |
196 | ||
197 | (*) Overlapping loads and stores within a particular CPU will appear to be | |
198 | ordered within that CPU. This means that for: | |
199 | ||
200 | a = *X; *X = b; | |
201 | ||
202 | the CPU will only issue the following sequence of memory operations: | |
203 | ||
204 | a = LOAD *X, STORE *X = b | |
205 | ||
206 | And for: | |
207 | ||
208 | *X = c; d = *X; | |
209 | ||
210 | the CPU will only issue: | |
211 | ||
212 | STORE *X = c, d = LOAD *X | |
213 | ||
214 | (Loads and stores overlap if they are targetted at overlapping pieces of | |
215 | memory). | |
216 | ||
217 | And there are a number of things that _must_ or _must_not_ be assumed: | |
218 | ||
219 | (*) It _must_not_ be assumed that independent loads and stores will be issued | |
220 | in the order given. This means that for: | |
221 | ||
222 | X = *A; Y = *B; *D = Z; | |
223 | ||
224 | we may get any of the following sequences: | |
225 | ||
226 | X = LOAD *A, Y = LOAD *B, STORE *D = Z | |
227 | X = LOAD *A, STORE *D = Z, Y = LOAD *B | |
228 | Y = LOAD *B, X = LOAD *A, STORE *D = Z | |
229 | Y = LOAD *B, STORE *D = Z, X = LOAD *A | |
230 | STORE *D = Z, X = LOAD *A, Y = LOAD *B | |
231 | STORE *D = Z, Y = LOAD *B, X = LOAD *A | |
232 | ||
233 | (*) It _must_ be assumed that overlapping memory accesses may be merged or | |
234 | discarded. This means that for: | |
235 | ||
236 | X = *A; Y = *(A + 4); | |
237 | ||
238 | we may get any one of the following sequences: | |
239 | ||
240 | X = LOAD *A; Y = LOAD *(A + 4); | |
241 | Y = LOAD *(A + 4); X = LOAD *A; | |
242 | {X, Y} = LOAD {*A, *(A + 4) }; | |
243 | ||
244 | And for: | |
245 | ||
246 | *A = X; Y = *A; | |
247 | ||
248 | we may get either of: | |
249 | ||
250 | STORE *A = X; Y = LOAD *A; | |
251 | STORE *A = Y; | |
252 | ||
253 | ||
254 | ========================= | |
255 | WHAT ARE MEMORY BARRIERS? | |
256 | ========================= | |
257 | ||
258 | As can be seen above, independent memory operations are effectively performed | |
259 | in random order, but this can be a problem for CPU-CPU interaction and for I/O. | |
260 | What is required is some way of intervening to instruct the compiler and the | |
261 | CPU to restrict the order. | |
262 | ||
263 | Memory barriers are such interventions. They impose a perceived partial | |
264 | ordering between the memory operations specified on either side of the barrier. | |
265 | They request that the sequence of memory events generated appears to other | |
266 | parts of the system as if the barrier is effective on that CPU. | |
267 | ||
268 | ||
269 | VARIETIES OF MEMORY BARRIER | |
270 | --------------------------- | |
271 | ||
272 | Memory barriers come in four basic varieties: | |
273 | ||
274 | (1) Write (or store) memory barriers. | |
275 | ||
276 | A write memory barrier gives a guarantee that all the STORE operations | |
277 | specified before the barrier will appear to happen before all the STORE | |
278 | operations specified after the barrier with respect to the other | |
279 | components of the system. | |
280 | ||
281 | A write barrier is a partial ordering on stores only; it is not required | |
282 | to have any effect on loads. | |
283 | ||
284 | A CPU can be viewed as as commiting a sequence of store operations to the | |
285 | memory system as time progresses. All stores before a write barrier will | |
286 | occur in the sequence _before_ all the stores after the write barrier. | |
287 | ||
288 | [!] Note that write barriers should normally be paired with read or data | |
289 | dependency barriers; see the "SMP barrier pairing" subsection. | |
290 | ||
291 | ||
292 | (2) Data dependency barriers. | |
293 | ||
294 | A data dependency barrier is a weaker form of read barrier. In the case | |
295 | where two loads are performed such that the second depends on the result | |
296 | of the first (eg: the first load retrieves the address to which the second | |
297 | load will be directed), a data dependency barrier would be required to | |
298 | make sure that the target of the second load is updated before the address | |
299 | obtained by the first load is accessed. | |
300 | ||
301 | A data dependency barrier is a partial ordering on interdependent loads | |
302 | only; it is not required to have any effect on stores, independent loads | |
303 | or overlapping loads. | |
304 | ||
305 | As mentioned in (1), the other CPUs in the system can be viewed as | |
306 | committing sequences of stores to the memory system that the CPU being | |
307 | considered can then perceive. A data dependency barrier issued by the CPU | |
308 | under consideration guarantees that for any load preceding it, if that | |
309 | load touches one of a sequence of stores from another CPU, then by the | |
310 | time the barrier completes, the effects of all the stores prior to that | |
311 | touched by the load will be perceptible to any loads issued after the data | |
312 | dependency barrier. | |
313 | ||
314 | See the "Examples of memory barrier sequences" subsection for diagrams | |
315 | showing the ordering constraints. | |
316 | ||
317 | [!] Note that the first load really has to have a _data_ dependency and | |
318 | not a control dependency. If the address for the second load is dependent | |
319 | on the first load, but the dependency is through a conditional rather than | |
320 | actually loading the address itself, then it's a _control_ dependency and | |
321 | a full read barrier or better is required. See the "Control dependencies" | |
322 | subsection for more information. | |
323 | ||
324 | [!] Note that data dependency barriers should normally be paired with | |
325 | write barriers; see the "SMP barrier pairing" subsection. | |
326 | ||
327 | ||
328 | (3) Read (or load) memory barriers. | |
329 | ||
330 | A read barrier is a data dependency barrier plus a guarantee that all the | |
331 | LOAD operations specified before the barrier will appear to happen before | |
332 | all the LOAD operations specified after the barrier with respect to the | |
333 | other components of the system. | |
334 | ||
335 | A read barrier is a partial ordering on loads only; it is not required to | |
336 | have any effect on stores. | |
337 | ||
338 | Read memory barriers imply data dependency barriers, and so can substitute | |
339 | for them. | |
340 | ||
341 | [!] Note that read barriers should normally be paired with write barriers; | |
342 | see the "SMP barrier pairing" subsection. | |
343 | ||
344 | ||
345 | (4) General memory barriers. | |
346 | ||
347 | A general memory barrier is a combination of both a read memory barrier | |
348 | and a write memory barrier. It is a partial ordering over both loads and | |
349 | stores. | |
350 | ||
351 | General memory barriers imply both read and write memory barriers, and so | |
352 | can substitute for either. | |
353 | ||
354 | ||
355 | And a couple of implicit varieties: | |
356 | ||
357 | (5) LOCK operations. | |
358 | ||
359 | This acts as a one-way permeable barrier. It guarantees that all memory | |
360 | operations after the LOCK operation will appear to happen after the LOCK | |
361 | operation with respect to the other components of the system. | |
362 | ||
363 | Memory operations that occur before a LOCK operation may appear to happen | |
364 | after it completes. | |
365 | ||
366 | A LOCK operation should almost always be paired with an UNLOCK operation. | |
367 | ||
368 | ||
369 | (6) UNLOCK operations. | |
370 | ||
371 | This also acts as a one-way permeable barrier. It guarantees that all | |
372 | memory operations before the UNLOCK operation will appear to happen before | |
373 | the UNLOCK operation with respect to the other components of the system. | |
374 | ||
375 | Memory operations that occur after an UNLOCK operation may appear to | |
376 | happen before it completes. | |
377 | ||
378 | LOCK and UNLOCK operations are guaranteed to appear with respect to each | |
379 | other strictly in the order specified. | |
380 | ||
381 | The use of LOCK and UNLOCK operations generally precludes the need for | |
382 | other sorts of memory barrier (but note the exceptions mentioned in the | |
383 | subsection "MMIO write barrier"). | |
384 | ||
385 | ||
386 | Memory barriers are only required where there's a possibility of interaction | |
387 | between two CPUs or between a CPU and a device. If it can be guaranteed that | |
388 | there won't be any such interaction in any particular piece of code, then | |
389 | memory barriers are unnecessary in that piece of code. | |
390 | ||
391 | ||
392 | Note that these are the _minimum_ guarantees. Different architectures may give | |
393 | more substantial guarantees, but they may _not_ be relied upon outside of arch | |
394 | specific code. | |
395 | ||
396 | ||
397 | WHAT MAY NOT BE ASSUMED ABOUT MEMORY BARRIERS? | |
398 | ---------------------------------------------- | |
399 | ||
400 | There are certain things that the Linux kernel memory barriers do not guarantee: | |
401 | ||
402 | (*) There is no guarantee that any of the memory accesses specified before a | |
403 | memory barrier will be _complete_ by the completion of a memory barrier | |
404 | instruction; the barrier can be considered to draw a line in that CPU's | |
405 | access queue that accesses of the appropriate type may not cross. | |
406 | ||
407 | (*) There is no guarantee that issuing a memory barrier on one CPU will have | |
408 | any direct effect on another CPU or any other hardware in the system. The | |
409 | indirect effect will be the order in which the second CPU sees the effects | |
410 | of the first CPU's accesses occur, but see the next point: | |
411 | ||
412 | (*) There is no guarantee that the a CPU will see the correct order of effects | |
413 | from a second CPU's accesses, even _if_ the second CPU uses a memory | |
414 | barrier, unless the first CPU _also_ uses a matching memory barrier (see | |
415 | the subsection on "SMP Barrier Pairing"). | |
416 | ||
417 | (*) There is no guarantee that some intervening piece of off-the-CPU | |
418 | hardware[*] will not reorder the memory accesses. CPU cache coherency | |
419 | mechanisms should propagate the indirect effects of a memory barrier | |
420 | between CPUs, but might not do so in order. | |
421 | ||
422 | [*] For information on bus mastering DMA and coherency please read: | |
423 | ||
424 | Documentation/pci.txt | |
425 | Documentation/DMA-mapping.txt | |
426 | Documentation/DMA-API.txt | |
427 | ||
428 | ||
429 | DATA DEPENDENCY BARRIERS | |
430 | ------------------------ | |
431 | ||
432 | The usage requirements of data dependency barriers are a little subtle, and | |
433 | it's not always obvious that they're needed. To illustrate, consider the | |
434 | following sequence of events: | |
435 | ||
436 | CPU 1 CPU 2 | |
437 | =============== =============== | |
438 | { A == 1, B == 2, C = 3, P == &A, Q == &C } | |
439 | B = 4; | |
440 | <write barrier> | |
441 | P = &B | |
442 | Q = P; | |
443 | D = *Q; | |
444 | ||
445 | There's a clear data dependency here, and it would seem that by the end of the | |
446 | sequence, Q must be either &A or &B, and that: | |
447 | ||
448 | (Q == &A) implies (D == 1) | |
449 | (Q == &B) implies (D == 4) | |
450 | ||
451 | But! CPU 2's perception of P may be updated _before_ its perception of B, thus | |
452 | leading to the following situation: | |
453 | ||
454 | (Q == &B) and (D == 2) ???? | |
455 | ||
456 | Whilst this may seem like a failure of coherency or causality maintenance, it | |
457 | isn't, and this behaviour can be observed on certain real CPUs (such as the DEC | |
458 | Alpha). | |
459 | ||
460 | To deal with this, a data dependency barrier must be inserted between the | |
461 | address load and the data load: | |
462 | ||
463 | CPU 1 CPU 2 | |
464 | =============== =============== | |
465 | { A == 1, B == 2, C = 3, P == &A, Q == &C } | |
466 | B = 4; | |
467 | <write barrier> | |
468 | P = &B | |
469 | Q = P; | |
470 | <data dependency barrier> | |
471 | D = *Q; | |
472 | ||
473 | This enforces the occurrence of one of the two implications, and prevents the | |
474 | third possibility from arising. | |
475 | ||
476 | [!] Note that this extremely counterintuitive situation arises most easily on | |
477 | machines with split caches, so that, for example, one cache bank processes | |
478 | even-numbered cache lines and the other bank processes odd-numbered cache | |
479 | lines. The pointer P might be stored in an odd-numbered cache line, and the | |
480 | variable B might be stored in an even-numbered cache line. Then, if the | |
481 | even-numbered bank of the reading CPU's cache is extremely busy while the | |
482 | odd-numbered bank is idle, one can see the new value of the pointer P (&B), | |
483 | but the old value of the variable B (1). | |
484 | ||
485 | ||
486 | Another example of where data dependency barriers might by required is where a | |
487 | number is read from memory and then used to calculate the index for an array | |
488 | access: | |
489 | ||
490 | CPU 1 CPU 2 | |
491 | =============== =============== | |
492 | { M[0] == 1, M[1] == 2, M[3] = 3, P == 0, Q == 3 } | |
493 | M[1] = 4; | |
494 | <write barrier> | |
495 | P = 1 | |
496 | Q = P; | |
497 | <data dependency barrier> | |
498 | D = M[Q]; | |
499 | ||
500 | ||
501 | The data dependency barrier is very important to the RCU system, for example. | |
502 | See rcu_dereference() in include/linux/rcupdate.h. This permits the current | |
503 | target of an RCU'd pointer to be replaced with a new modified target, without | |
504 | the replacement target appearing to be incompletely initialised. | |
505 | ||
506 | See also the subsection on "Cache Coherency" for a more thorough example. | |
507 | ||
508 | ||
509 | CONTROL DEPENDENCIES | |
510 | -------------------- | |
511 | ||
512 | A control dependency requires a full read memory barrier, not simply a data | |
513 | dependency barrier to make it work correctly. Consider the following bit of | |
514 | code: | |
515 | ||
516 | q = &a; | |
517 | if (p) | |
518 | q = &b; | |
519 | <data dependency barrier> | |
520 | x = *q; | |
521 | ||
522 | This will not have the desired effect because there is no actual data | |
523 | dependency, but rather a control dependency that the CPU may short-circuit by | |
524 | attempting to predict the outcome in advance. In such a case what's actually | |
525 | required is: | |
526 | ||
527 | q = &a; | |
528 | if (p) | |
529 | q = &b; | |
530 | <read barrier> | |
531 | x = *q; | |
532 | ||
533 | ||
534 | SMP BARRIER PAIRING | |
535 | ------------------- | |
536 | ||
537 | When dealing with CPU-CPU interactions, certain types of memory barrier should | |
538 | always be paired. A lack of appropriate pairing is almost certainly an error. | |
539 | ||
540 | A write barrier should always be paired with a data dependency barrier or read | |
541 | barrier, though a general barrier would also be viable. Similarly a read | |
542 | barrier or a data dependency barrier should always be paired with at least an | |
543 | write barrier, though, again, a general barrier is viable: | |
544 | ||
545 | CPU 1 CPU 2 | |
546 | =============== =============== | |
547 | a = 1; | |
548 | <write barrier> | |
549 | b = 2; x = a; | |
550 | <read barrier> | |
551 | y = b; | |
552 | ||
553 | Or: | |
554 | ||
555 | CPU 1 CPU 2 | |
556 | =============== =============================== | |
557 | a = 1; | |
558 | <write barrier> | |
559 | b = &a; x = b; | |
560 | <data dependency barrier> | |
561 | y = *x; | |
562 | ||
563 | Basically, the read barrier always has to be there, even though it can be of | |
564 | the "weaker" type. | |
565 | ||
566 | ||
567 | EXAMPLES OF MEMORY BARRIER SEQUENCES | |
568 | ------------------------------------ | |
569 | ||
570 | Firstly, write barriers act as a partial orderings on store operations. | |
571 | Consider the following sequence of events: | |
572 | ||
573 | CPU 1 | |
574 | ======================= | |
575 | STORE A = 1 | |
576 | STORE B = 2 | |
577 | STORE C = 3 | |
578 | <write barrier> | |
579 | STORE D = 4 | |
580 | STORE E = 5 | |
581 | ||
582 | This sequence of events is committed to the memory coherence system in an order | |
583 | that the rest of the system might perceive as the unordered set of { STORE A, | |
584 | STORE B, STORE C } all occuring before the unordered set of { STORE D, STORE E | |
585 | }: | |
586 | ||
587 | +-------+ : : | |
588 | | | +------+ | |
589 | | |------>| C=3 | } /\ | |
590 | | | : +------+ }----- \ -----> Events perceptible | |
591 | | | : | A=1 | } \/ to rest of system | |
592 | | | : +------+ } | |
593 | | CPU 1 | : | B=2 | } | |
594 | | | +------+ } | |
595 | | | wwwwwwwwwwwwwwww } <--- At this point the write barrier | |
596 | | | +------+ } requires all stores prior to the | |
597 | | | : | E=5 | } barrier to be committed before | |
598 | | | : +------+ } further stores may be take place. | |
599 | | |------>| D=4 | } | |
600 | | | +------+ | |
601 | +-------+ : : | |
602 | | | |
603 | | Sequence in which stores committed to memory system | |
604 | | by CPU 1 | |
605 | V | |
606 | ||
607 | ||
608 | Secondly, data dependency barriers act as a partial orderings on data-dependent | |
609 | loads. Consider the following sequence of events: | |
610 | ||
611 | CPU 1 CPU 2 | |
612 | ======================= ======================= | |
c14038c3 | 613 | { B = 7; X = 9; Y = 8; C = &Y } |
108b42b4 DH |
614 | STORE A = 1 |
615 | STORE B = 2 | |
616 | <write barrier> | |
617 | STORE C = &B LOAD X | |
618 | STORE D = 4 LOAD C (gets &B) | |
619 | LOAD *C (reads B) | |
620 | ||
621 | Without intervention, CPU 2 may perceive the events on CPU 1 in some | |
622 | effectively random order, despite the write barrier issued by CPU 1: | |
623 | ||
624 | +-------+ : : : : | |
625 | | | +------+ +-------+ | Sequence of update | |
626 | | |------>| B=2 |----- --->| Y->8 | | of perception on | |
627 | | | : +------+ \ +-------+ | CPU 2 | |
628 | | CPU 1 | : | A=1 | \ --->| C->&Y | V | |
629 | | | +------+ | +-------+ | |
630 | | | wwwwwwwwwwwwwwww | : : | |
631 | | | +------+ | : : | |
632 | | | : | C=&B |--- | : : +-------+ | |
633 | | | : +------+ \ | +-------+ | | | |
634 | | |------>| D=4 | ----------->| C->&B |------>| | | |
635 | | | +------+ | +-------+ | | | |
636 | +-------+ : : | : : | | | |
637 | | : : | | | |
638 | | : : | CPU 2 | | |
639 | | +-------+ | | | |
640 | Apparently incorrect ---> | | B->7 |------>| | | |
641 | perception of B (!) | +-------+ | | | |
642 | | : : | | | |
643 | | +-------+ | | | |
644 | The load of X holds ---> \ | X->9 |------>| | | |
645 | up the maintenance \ +-------+ | | | |
646 | of coherence of B ----->| B->2 | +-------+ | |
647 | +-------+ | |
648 | : : | |
649 | ||
650 | ||
651 | In the above example, CPU 2 perceives that B is 7, despite the load of *C | |
652 | (which would be B) coming after the the LOAD of C. | |
653 | ||
654 | If, however, a data dependency barrier were to be placed between the load of C | |
c14038c3 DH |
655 | and the load of *C (ie: B) on CPU 2: |
656 | ||
657 | CPU 1 CPU 2 | |
658 | ======================= ======================= | |
659 | { B = 7; X = 9; Y = 8; C = &Y } | |
660 | STORE A = 1 | |
661 | STORE B = 2 | |
662 | <write barrier> | |
663 | STORE C = &B LOAD X | |
664 | STORE D = 4 LOAD C (gets &B) | |
665 | <data dependency barrier> | |
666 | LOAD *C (reads B) | |
667 | ||
668 | then the following will occur: | |
108b42b4 DH |
669 | |
670 | +-------+ : : : : | |
671 | | | +------+ +-------+ | |
672 | | |------>| B=2 |----- --->| Y->8 | | |
673 | | | : +------+ \ +-------+ | |
674 | | CPU 1 | : | A=1 | \ --->| C->&Y | | |
675 | | | +------+ | +-------+ | |
676 | | | wwwwwwwwwwwwwwww | : : | |
677 | | | +------+ | : : | |
678 | | | : | C=&B |--- | : : +-------+ | |
679 | | | : +------+ \ | +-------+ | | | |
680 | | |------>| D=4 | ----------->| C->&B |------>| | | |
681 | | | +------+ | +-------+ | | | |
682 | +-------+ : : | : : | | | |
683 | | : : | | | |
684 | | : : | CPU 2 | | |
685 | | +-------+ | | | |
686 | \ | X->9 |------>| | | |
687 | \ +-------+ | | | |
688 | ----->| B->2 | | | | |
689 | +-------+ | | | |
690 | Makes sure all effects ---> ddddddddddddddddd | | | |
691 | prior to the store of C +-------+ | | | |
692 | are perceptible to | B->2 |------>| | | |
693 | successive loads +-------+ | | | |
694 | : : +-------+ | |
695 | ||
696 | ||
697 | And thirdly, a read barrier acts as a partial order on loads. Consider the | |
698 | following sequence of events: | |
699 | ||
700 | CPU 1 CPU 2 | |
701 | ======================= ======================= | |
702 | STORE A=1 | |
703 | STORE B=2 | |
704 | STORE C=3 | |
705 | <write barrier> | |
706 | STORE D=4 | |
707 | STORE E=5 | |
708 | LOAD A | |
709 | LOAD B | |
710 | LOAD C | |
711 | LOAD D | |
712 | LOAD E | |
713 | ||
714 | Without intervention, CPU 2 may then choose to perceive the events on CPU 1 in | |
715 | some effectively random order, despite the write barrier issued by CPU 1: | |
716 | ||
717 | +-------+ : : | |
718 | | | +------+ | |
719 | | |------>| C=3 | } | |
720 | | | : +------+ } | |
721 | | | : | A=1 | } | |
722 | | | : +------+ } | |
723 | | CPU 1 | : | B=2 | }--- | |
724 | | | +------+ } \ | |
725 | | | wwwwwwwwwwwww} \ | |
726 | | | +------+ } \ : : +-------+ | |
727 | | | : | E=5 | } \ +-------+ | | | |
728 | | | : +------+ } \ { | C->3 |------>| | | |
729 | | |------>| D=4 | } \ { +-------+ : | | | |
730 | | | +------+ \ { | E->5 | : | | | |
731 | +-------+ : : \ { +-------+ : | | | |
732 | Transfer -->{ | A->1 | : | CPU 2 | | |
733 | from CPU 1 { +-------+ : | | | |
734 | to CPU 2 { | D->4 | : | | | |
735 | { +-------+ : | | | |
736 | { | B->2 |------>| | | |
737 | +-------+ | | | |
738 | : : +-------+ | |
739 | ||
740 | ||
741 | If, however, a read barrier were to be placed between the load of C and the | |
742 | load of D on CPU 2, then the partial ordering imposed by CPU 1 will be | |
743 | perceived correctly by CPU 2. | |
744 | ||
745 | +-------+ : : | |
746 | | | +------+ | |
747 | | |------>| C=3 | } | |
748 | | | : +------+ } | |
749 | | | : | A=1 | }--- | |
750 | | | : +------+ } \ | |
751 | | CPU 1 | : | B=2 | } \ | |
752 | | | +------+ \ | |
753 | | | wwwwwwwwwwwwwwww \ | |
754 | | | +------+ \ : : +-------+ | |
755 | | | : | E=5 | } \ +-------+ | | | |
756 | | | : +------+ }--- \ { | C->3 |------>| | | |
757 | | |------>| D=4 | } \ \ { +-------+ : | | | |
758 | | | +------+ \ -->{ | B->2 | : | | | |
759 | +-------+ : : \ { +-------+ : | | | |
760 | \ { | A->1 | : | CPU 2 | | |
761 | \ +-------+ | | | |
762 | At this point the read ----> \ rrrrrrrrrrrrrrrrr | | | |
763 | barrier causes all effects \ +-------+ | | | |
764 | prior to the storage of C \ { | E->5 | : | | | |
765 | to be perceptible to CPU 2 -->{ +-------+ : | | | |
766 | { | D->4 |------>| | | |
767 | +-------+ | | | |
768 | : : +-------+ | |
769 | ||
770 | ||
771 | ======================== | |
772 | EXPLICIT KERNEL BARRIERS | |
773 | ======================== | |
774 | ||
775 | The Linux kernel has a variety of different barriers that act at different | |
776 | levels: | |
777 | ||
778 | (*) Compiler barrier. | |
779 | ||
780 | (*) CPU memory barriers. | |
781 | ||
782 | (*) MMIO write barrier. | |
783 | ||
784 | ||
785 | COMPILER BARRIER | |
786 | ---------------- | |
787 | ||
788 | The Linux kernel has an explicit compiler barrier function that prevents the | |
789 | compiler from moving the memory accesses either side of it to the other side: | |
790 | ||
791 | barrier(); | |
792 | ||
793 | This a general barrier - lesser varieties of compiler barrier do not exist. | |
794 | ||
795 | The compiler barrier has no direct effect on the CPU, which may then reorder | |
796 | things however it wishes. | |
797 | ||
798 | ||
799 | CPU MEMORY BARRIERS | |
800 | ------------------- | |
801 | ||
802 | The Linux kernel has eight basic CPU memory barriers: | |
803 | ||
804 | TYPE MANDATORY SMP CONDITIONAL | |
805 | =============== ======================= =========================== | |
806 | GENERAL mb() smp_mb() | |
807 | WRITE wmb() smp_wmb() | |
808 | READ rmb() smp_rmb() | |
809 | DATA DEPENDENCY read_barrier_depends() smp_read_barrier_depends() | |
810 | ||
811 | ||
812 | All CPU memory barriers unconditionally imply compiler barriers. | |
813 | ||
814 | SMP memory barriers are reduced to compiler barriers on uniprocessor compiled | |
815 | systems because it is assumed that a CPU will be appear to be self-consistent, | |
816 | and will order overlapping accesses correctly with respect to itself. | |
817 | ||
818 | [!] Note that SMP memory barriers _must_ be used to control the ordering of | |
819 | references to shared memory on SMP systems, though the use of locking instead | |
820 | is sufficient. | |
821 | ||
822 | Mandatory barriers should not be used to control SMP effects, since mandatory | |
823 | barriers unnecessarily impose overhead on UP systems. They may, however, be | |
824 | used to control MMIO effects on accesses through relaxed memory I/O windows. | |
825 | These are required even on non-SMP systems as they affect the order in which | |
826 | memory operations appear to a device by prohibiting both the compiler and the | |
827 | CPU from reordering them. | |
828 | ||
829 | ||
830 | There are some more advanced barrier functions: | |
831 | ||
832 | (*) set_mb(var, value) | |
833 | (*) set_wmb(var, value) | |
834 | ||
835 | These assign the value to the variable and then insert at least a write | |
836 | barrier after it, depending on the function. They aren't guaranteed to | |
837 | insert anything more than a compiler barrier in a UP compilation. | |
838 | ||
839 | ||
840 | (*) smp_mb__before_atomic_dec(); | |
841 | (*) smp_mb__after_atomic_dec(); | |
842 | (*) smp_mb__before_atomic_inc(); | |
843 | (*) smp_mb__after_atomic_inc(); | |
844 | ||
845 | These are for use with atomic add, subtract, increment and decrement | |
dbc8700e DH |
846 | functions that don't return a value, especially when used for reference |
847 | counting. These functions do not imply memory barriers. | |
108b42b4 DH |
848 | |
849 | As an example, consider a piece of code that marks an object as being dead | |
850 | and then decrements the object's reference count: | |
851 | ||
852 | obj->dead = 1; | |
853 | smp_mb__before_atomic_dec(); | |
854 | atomic_dec(&obj->ref_count); | |
855 | ||
856 | This makes sure that the death mark on the object is perceived to be set | |
857 | *before* the reference counter is decremented. | |
858 | ||
859 | See Documentation/atomic_ops.txt for more information. See the "Atomic | |
860 | operations" subsection for information on where to use these. | |
861 | ||
862 | ||
863 | (*) smp_mb__before_clear_bit(void); | |
864 | (*) smp_mb__after_clear_bit(void); | |
865 | ||
866 | These are for use similar to the atomic inc/dec barriers. These are | |
867 | typically used for bitwise unlocking operations, so care must be taken as | |
868 | there are no implicit memory barriers here either. | |
869 | ||
870 | Consider implementing an unlock operation of some nature by clearing a | |
871 | locking bit. The clear_bit() would then need to be barriered like this: | |
872 | ||
873 | smp_mb__before_clear_bit(); | |
874 | clear_bit( ... ); | |
875 | ||
876 | This prevents memory operations before the clear leaking to after it. See | |
877 | the subsection on "Locking Functions" with reference to UNLOCK operation | |
878 | implications. | |
879 | ||
880 | See Documentation/atomic_ops.txt for more information. See the "Atomic | |
881 | operations" subsection for information on where to use these. | |
882 | ||
883 | ||
884 | MMIO WRITE BARRIER | |
885 | ------------------ | |
886 | ||
887 | The Linux kernel also has a special barrier for use with memory-mapped I/O | |
888 | writes: | |
889 | ||
890 | mmiowb(); | |
891 | ||
892 | This is a variation on the mandatory write barrier that causes writes to weakly | |
893 | ordered I/O regions to be partially ordered. Its effects may go beyond the | |
894 | CPU->Hardware interface and actually affect the hardware at some level. | |
895 | ||
896 | See the subsection "Locks vs I/O accesses" for more information. | |
897 | ||
898 | ||
899 | =============================== | |
900 | IMPLICIT KERNEL MEMORY BARRIERS | |
901 | =============================== | |
902 | ||
903 | Some of the other functions in the linux kernel imply memory barriers, amongst | |
904 | which are locking, scheduling and memory allocation functions. | |
905 | ||
906 | This specification is a _minimum_ guarantee; any particular architecture may | |
907 | provide more substantial guarantees, but these may not be relied upon outside | |
908 | of arch specific code. | |
909 | ||
910 | ||
911 | LOCKING FUNCTIONS | |
912 | ----------------- | |
913 | ||
914 | The Linux kernel has a number of locking constructs: | |
915 | ||
916 | (*) spin locks | |
917 | (*) R/W spin locks | |
918 | (*) mutexes | |
919 | (*) semaphores | |
920 | (*) R/W semaphores | |
921 | (*) RCU | |
922 | ||
923 | In all cases there are variants on "LOCK" operations and "UNLOCK" operations | |
924 | for each construct. These operations all imply certain barriers: | |
925 | ||
926 | (1) LOCK operation implication: | |
927 | ||
928 | Memory operations issued after the LOCK will be completed after the LOCK | |
929 | operation has completed. | |
930 | ||
931 | Memory operations issued before the LOCK may be completed after the LOCK | |
932 | operation has completed. | |
933 | ||
934 | (2) UNLOCK operation implication: | |
935 | ||
936 | Memory operations issued before the UNLOCK will be completed before the | |
937 | UNLOCK operation has completed. | |
938 | ||
939 | Memory operations issued after the UNLOCK may be completed before the | |
940 | UNLOCK operation has completed. | |
941 | ||
942 | (3) LOCK vs LOCK implication: | |
943 | ||
944 | All LOCK operations issued before another LOCK operation will be completed | |
945 | before that LOCK operation. | |
946 | ||
947 | (4) LOCK vs UNLOCK implication: | |
948 | ||
949 | All LOCK operations issued before an UNLOCK operation will be completed | |
950 | before the UNLOCK operation. | |
951 | ||
952 | All UNLOCK operations issued before a LOCK operation will be completed | |
953 | before the LOCK operation. | |
954 | ||
955 | (5) Failed conditional LOCK implication: | |
956 | ||
957 | Certain variants of the LOCK operation may fail, either due to being | |
958 | unable to get the lock immediately, or due to receiving an unblocked | |
959 | signal whilst asleep waiting for the lock to become available. Failed | |
960 | locks do not imply any sort of barrier. | |
961 | ||
962 | Therefore, from (1), (2) and (4) an UNLOCK followed by an unconditional LOCK is | |
963 | equivalent to a full barrier, but a LOCK followed by an UNLOCK is not. | |
964 | ||
965 | [!] Note: one of the consequence of LOCKs and UNLOCKs being only one-way | |
966 | barriers is that the effects instructions outside of a critical section may | |
967 | seep into the inside of the critical section. | |
968 | ||
969 | Locks and semaphores may not provide any guarantee of ordering on UP compiled | |
970 | systems, and so cannot be counted on in such a situation to actually achieve | |
971 | anything at all - especially with respect to I/O accesses - unless combined | |
972 | with interrupt disabling operations. | |
973 | ||
974 | See also the section on "Inter-CPU locking barrier effects". | |
975 | ||
976 | ||
977 | As an example, consider the following: | |
978 | ||
979 | *A = a; | |
980 | *B = b; | |
981 | LOCK | |
982 | *C = c; | |
983 | *D = d; | |
984 | UNLOCK | |
985 | *E = e; | |
986 | *F = f; | |
987 | ||
988 | The following sequence of events is acceptable: | |
989 | ||
990 | LOCK, {*F,*A}, *E, {*C,*D}, *B, UNLOCK | |
991 | ||
992 | [+] Note that {*F,*A} indicates a combined access. | |
993 | ||
994 | But none of the following are: | |
995 | ||
996 | {*F,*A}, *B, LOCK, *C, *D, UNLOCK, *E | |
997 | *A, *B, *C, LOCK, *D, UNLOCK, *E, *F | |
998 | *A, *B, LOCK, *C, UNLOCK, *D, *E, *F | |
999 | *B, LOCK, *C, *D, UNLOCK, {*F,*A}, *E | |
1000 | ||
1001 | ||
1002 | ||
1003 | INTERRUPT DISABLING FUNCTIONS | |
1004 | ----------------------------- | |
1005 | ||
1006 | Functions that disable interrupts (LOCK equivalent) and enable interrupts | |
1007 | (UNLOCK equivalent) will act as compiler barriers only. So if memory or I/O | |
1008 | barriers are required in such a situation, they must be provided from some | |
1009 | other means. | |
1010 | ||
1011 | ||
1012 | MISCELLANEOUS FUNCTIONS | |
1013 | ----------------------- | |
1014 | ||
1015 | Other functions that imply barriers: | |
1016 | ||
1017 | (*) schedule() and similar imply full memory barriers. | |
1018 | ||
1019 | (*) Memory allocation and release functions imply full memory barriers. | |
1020 | ||
1021 | ||
1022 | ================================= | |
1023 | INTER-CPU LOCKING BARRIER EFFECTS | |
1024 | ================================= | |
1025 | ||
1026 | On SMP systems locking primitives give a more substantial form of barrier: one | |
1027 | that does affect memory access ordering on other CPUs, within the context of | |
1028 | conflict on any particular lock. | |
1029 | ||
1030 | ||
1031 | LOCKS VS MEMORY ACCESSES | |
1032 | ------------------------ | |
1033 | ||
1034 | Consider the following: the system has a pair of spinlocks (N) and (Q), and | |
1035 | three CPUs; then should the following sequence of events occur: | |
1036 | ||
1037 | CPU 1 CPU 2 | |
1038 | =============================== =============================== | |
1039 | *A = a; *E = e; | |
1040 | LOCK M LOCK Q | |
1041 | *B = b; *F = f; | |
1042 | *C = c; *G = g; | |
1043 | UNLOCK M UNLOCK Q | |
1044 | *D = d; *H = h; | |
1045 | ||
1046 | Then there is no guarantee as to what order CPU #3 will see the accesses to *A | |
1047 | through *H occur in, other than the constraints imposed by the separate locks | |
1048 | on the separate CPUs. It might, for example, see: | |
1049 | ||
1050 | *E, LOCK M, LOCK Q, *G, *C, *F, *A, *B, UNLOCK Q, *D, *H, UNLOCK M | |
1051 | ||
1052 | But it won't see any of: | |
1053 | ||
1054 | *B, *C or *D preceding LOCK M | |
1055 | *A, *B or *C following UNLOCK M | |
1056 | *F, *G or *H preceding LOCK Q | |
1057 | *E, *F or *G following UNLOCK Q | |
1058 | ||
1059 | ||
1060 | However, if the following occurs: | |
1061 | ||
1062 | CPU 1 CPU 2 | |
1063 | =============================== =============================== | |
1064 | *A = a; | |
1065 | LOCK M [1] | |
1066 | *B = b; | |
1067 | *C = c; | |
1068 | UNLOCK M [1] | |
1069 | *D = d; *E = e; | |
1070 | LOCK M [2] | |
1071 | *F = f; | |
1072 | *G = g; | |
1073 | UNLOCK M [2] | |
1074 | *H = h; | |
1075 | ||
1076 | CPU #3 might see: | |
1077 | ||
1078 | *E, LOCK M [1], *C, *B, *A, UNLOCK M [1], | |
1079 | LOCK M [2], *H, *F, *G, UNLOCK M [2], *D | |
1080 | ||
1081 | But assuming CPU #1 gets the lock first, it won't see any of: | |
1082 | ||
1083 | *B, *C, *D, *F, *G or *H preceding LOCK M [1] | |
1084 | *A, *B or *C following UNLOCK M [1] | |
1085 | *F, *G or *H preceding LOCK M [2] | |
1086 | *A, *B, *C, *E, *F or *G following UNLOCK M [2] | |
1087 | ||
1088 | ||
1089 | LOCKS VS I/O ACCESSES | |
1090 | --------------------- | |
1091 | ||
1092 | Under certain circumstances (especially involving NUMA), I/O accesses within | |
1093 | two spinlocked sections on two different CPUs may be seen as interleaved by the | |
1094 | PCI bridge, because the PCI bridge does not necessarily participate in the | |
1095 | cache-coherence protocol, and is therefore incapable of issuing the required | |
1096 | read memory barriers. | |
1097 | ||
1098 | For example: | |
1099 | ||
1100 | CPU 1 CPU 2 | |
1101 | =============================== =============================== | |
1102 | spin_lock(Q) | |
1103 | writel(0, ADDR) | |
1104 | writel(1, DATA); | |
1105 | spin_unlock(Q); | |
1106 | spin_lock(Q); | |
1107 | writel(4, ADDR); | |
1108 | writel(5, DATA); | |
1109 | spin_unlock(Q); | |
1110 | ||
1111 | may be seen by the PCI bridge as follows: | |
1112 | ||
1113 | STORE *ADDR = 0, STORE *ADDR = 4, STORE *DATA = 1, STORE *DATA = 5 | |
1114 | ||
1115 | which would probably cause the hardware to malfunction. | |
1116 | ||
1117 | ||
1118 | What is necessary here is to intervene with an mmiowb() before dropping the | |
1119 | spinlock, for example: | |
1120 | ||
1121 | CPU 1 CPU 2 | |
1122 | =============================== =============================== | |
1123 | spin_lock(Q) | |
1124 | writel(0, ADDR) | |
1125 | writel(1, DATA); | |
1126 | mmiowb(); | |
1127 | spin_unlock(Q); | |
1128 | spin_lock(Q); | |
1129 | writel(4, ADDR); | |
1130 | writel(5, DATA); | |
1131 | mmiowb(); | |
1132 | spin_unlock(Q); | |
1133 | ||
1134 | this will ensure that the two stores issued on CPU #1 appear at the PCI bridge | |
1135 | before either of the stores issued on CPU #2. | |
1136 | ||
1137 | ||
1138 | Furthermore, following a store by a load to the same device obviates the need | |
1139 | for an mmiowb(), because the load forces the store to complete before the load | |
1140 | is performed: | |
1141 | ||
1142 | CPU 1 CPU 2 | |
1143 | =============================== =============================== | |
1144 | spin_lock(Q) | |
1145 | writel(0, ADDR) | |
1146 | a = readl(DATA); | |
1147 | spin_unlock(Q); | |
1148 | spin_lock(Q); | |
1149 | writel(4, ADDR); | |
1150 | b = readl(DATA); | |
1151 | spin_unlock(Q); | |
1152 | ||
1153 | ||
1154 | See Documentation/DocBook/deviceiobook.tmpl for more information. | |
1155 | ||
1156 | ||
1157 | ================================= | |
1158 | WHERE ARE MEMORY BARRIERS NEEDED? | |
1159 | ================================= | |
1160 | ||
1161 | Under normal operation, memory operation reordering is generally not going to | |
1162 | be a problem as a single-threaded linear piece of code will still appear to | |
1163 | work correctly, even if it's in an SMP kernel. There are, however, three | |
1164 | circumstances in which reordering definitely _could_ be a problem: | |
1165 | ||
1166 | (*) Interprocessor interaction. | |
1167 | ||
1168 | (*) Atomic operations. | |
1169 | ||
1170 | (*) Accessing devices (I/O). | |
1171 | ||
1172 | (*) Interrupts. | |
1173 | ||
1174 | ||
1175 | INTERPROCESSOR INTERACTION | |
1176 | -------------------------- | |
1177 | ||
1178 | When there's a system with more than one processor, more than one CPU in the | |
1179 | system may be working on the same data set at the same time. This can cause | |
1180 | synchronisation problems, and the usual way of dealing with them is to use | |
1181 | locks. Locks, however, are quite expensive, and so it may be preferable to | |
1182 | operate without the use of a lock if at all possible. In such a case | |
1183 | operations that affect both CPUs may have to be carefully ordered to prevent | |
1184 | a malfunction. | |
1185 | ||
1186 | Consider, for example, the R/W semaphore slow path. Here a waiting process is | |
1187 | queued on the semaphore, by virtue of it having a piece of its stack linked to | |
1188 | the semaphore's list of waiting processes: | |
1189 | ||
1190 | struct rw_semaphore { | |
1191 | ... | |
1192 | spinlock_t lock; | |
1193 | struct list_head waiters; | |
1194 | }; | |
1195 | ||
1196 | struct rwsem_waiter { | |
1197 | struct list_head list; | |
1198 | struct task_struct *task; | |
1199 | }; | |
1200 | ||
1201 | To wake up a particular waiter, the up_read() or up_write() functions have to: | |
1202 | ||
1203 | (1) read the next pointer from this waiter's record to know as to where the | |
1204 | next waiter record is; | |
1205 | ||
1206 | (4) read the pointer to the waiter's task structure; | |
1207 | ||
1208 | (3) clear the task pointer to tell the waiter it has been given the semaphore; | |
1209 | ||
1210 | (4) call wake_up_process() on the task; and | |
1211 | ||
1212 | (5) release the reference held on the waiter's task struct. | |
1213 | ||
1214 | In otherwords, it has to perform this sequence of events: | |
1215 | ||
1216 | LOAD waiter->list.next; | |
1217 | LOAD waiter->task; | |
1218 | STORE waiter->task; | |
1219 | CALL wakeup | |
1220 | RELEASE task | |
1221 | ||
1222 | and if any of these steps occur out of order, then the whole thing may | |
1223 | malfunction. | |
1224 | ||
1225 | Once it has queued itself and dropped the semaphore lock, the waiter does not | |
1226 | get the lock again; it instead just waits for its task pointer to be cleared | |
1227 | before proceeding. Since the record is on the waiter's stack, this means that | |
1228 | if the task pointer is cleared _before_ the next pointer in the list is read, | |
1229 | another CPU might start processing the waiter and might clobber the waiter's | |
1230 | stack before the up*() function has a chance to read the next pointer. | |
1231 | ||
1232 | Consider then what might happen to the above sequence of events: | |
1233 | ||
1234 | CPU 1 CPU 2 | |
1235 | =============================== =============================== | |
1236 | down_xxx() | |
1237 | Queue waiter | |
1238 | Sleep | |
1239 | up_yyy() | |
1240 | LOAD waiter->task; | |
1241 | STORE waiter->task; | |
1242 | Woken up by other event | |
1243 | <preempt> | |
1244 | Resume processing | |
1245 | down_xxx() returns | |
1246 | call foo() | |
1247 | foo() clobbers *waiter | |
1248 | </preempt> | |
1249 | LOAD waiter->list.next; | |
1250 | --- OOPS --- | |
1251 | ||
1252 | This could be dealt with using the semaphore lock, but then the down_xxx() | |
1253 | function has to needlessly get the spinlock again after being woken up. | |
1254 | ||
1255 | The way to deal with this is to insert a general SMP memory barrier: | |
1256 | ||
1257 | LOAD waiter->list.next; | |
1258 | LOAD waiter->task; | |
1259 | smp_mb(); | |
1260 | STORE waiter->task; | |
1261 | CALL wakeup | |
1262 | RELEASE task | |
1263 | ||
1264 | In this case, the barrier makes a guarantee that all memory accesses before the | |
1265 | barrier will appear to happen before all the memory accesses after the barrier | |
1266 | with respect to the other CPUs on the system. It does _not_ guarantee that all | |
1267 | the memory accesses before the barrier will be complete by the time the barrier | |
1268 | instruction itself is complete. | |
1269 | ||
1270 | On a UP system - where this wouldn't be a problem - the smp_mb() is just a | |
1271 | compiler barrier, thus making sure the compiler emits the instructions in the | |
1272 | right order without actually intervening in the CPU. Since there there's only | |
1273 | one CPU, that CPU's dependency ordering logic will take care of everything | |
1274 | else. | |
1275 | ||
1276 | ||
1277 | ATOMIC OPERATIONS | |
1278 | ----------------- | |
1279 | ||
dbc8700e DH |
1280 | Whilst they are technically interprocessor interaction considerations, atomic |
1281 | operations are noted specially as some of them imply full memory barriers and | |
1282 | some don't, but they're very heavily relied on as a group throughout the | |
1283 | kernel. | |
1284 | ||
1285 | Any atomic operation that modifies some state in memory and returns information | |
1286 | about the state (old or new) implies an SMP-conditional general memory barrier | |
1287 | (smp_mb()) on each side of the actual operation. These include: | |
108b42b4 DH |
1288 | |
1289 | xchg(); | |
1290 | cmpxchg(); | |
108b42b4 DH |
1291 | atomic_cmpxchg(); |
1292 | atomic_inc_return(); | |
1293 | atomic_dec_return(); | |
1294 | atomic_add_return(); | |
1295 | atomic_sub_return(); | |
1296 | atomic_inc_and_test(); | |
1297 | atomic_dec_and_test(); | |
1298 | atomic_sub_and_test(); | |
1299 | atomic_add_negative(); | |
1300 | atomic_add_unless(); | |
dbc8700e DH |
1301 | test_and_set_bit(); |
1302 | test_and_clear_bit(); | |
1303 | test_and_change_bit(); | |
1304 | ||
1305 | These are used for such things as implementing LOCK-class and UNLOCK-class | |
1306 | operations and adjusting reference counters towards object destruction, and as | |
1307 | such the implicit memory barrier effects are necessary. | |
108b42b4 | 1308 | |
108b42b4 | 1309 | |
dbc8700e DH |
1310 | The following operation are potential problems as they do _not_ imply memory |
1311 | barriers, but might be used for implementing such things as UNLOCK-class | |
1312 | operations: | |
108b42b4 | 1313 | |
dbc8700e | 1314 | atomic_set(); |
108b42b4 DH |
1315 | set_bit(); |
1316 | clear_bit(); | |
1317 | change_bit(); | |
dbc8700e DH |
1318 | |
1319 | With these the appropriate explicit memory barrier should be used if necessary | |
1320 | (smp_mb__before_clear_bit() for instance). | |
108b42b4 DH |
1321 | |
1322 | ||
dbc8700e DH |
1323 | The following also do _not_ imply memory barriers, and so may require explicit |
1324 | memory barriers under some circumstances (smp_mb__before_atomic_dec() for | |
1325 | instance)): | |
108b42b4 DH |
1326 | |
1327 | atomic_add(); | |
1328 | atomic_sub(); | |
1329 | atomic_inc(); | |
1330 | atomic_dec(); | |
1331 | ||
1332 | If they're used for statistics generation, then they probably don't need memory | |
1333 | barriers, unless there's a coupling between statistical data. | |
1334 | ||
1335 | If they're used for reference counting on an object to control its lifetime, | |
1336 | they probably don't need memory barriers because either the reference count | |
1337 | will be adjusted inside a locked section, or the caller will already hold | |
1338 | sufficient references to make the lock, and thus a memory barrier unnecessary. | |
1339 | ||
1340 | If they're used for constructing a lock of some description, then they probably | |
1341 | do need memory barriers as a lock primitive generally has to do things in a | |
1342 | specific order. | |
1343 | ||
1344 | ||
1345 | Basically, each usage case has to be carefully considered as to whether memory | |
dbc8700e DH |
1346 | barriers are needed or not. |
1347 | ||
1348 | [!] Note that special memory barrier primitives are available for these | |
1349 | situations because on some CPUs the atomic instructions used imply full memory | |
1350 | barriers, and so barrier instructions are superfluous in conjunction with them, | |
1351 | and in such cases the special barrier primitives will be no-ops. | |
108b42b4 DH |
1352 | |
1353 | See Documentation/atomic_ops.txt for more information. | |
1354 | ||
1355 | ||
1356 | ACCESSING DEVICES | |
1357 | ----------------- | |
1358 | ||
1359 | Many devices can be memory mapped, and so appear to the CPU as if they're just | |
1360 | a set of memory locations. To control such a device, the driver usually has to | |
1361 | make the right memory accesses in exactly the right order. | |
1362 | ||
1363 | However, having a clever CPU or a clever compiler creates a potential problem | |
1364 | in that the carefully sequenced accesses in the driver code won't reach the | |
1365 | device in the requisite order if the CPU or the compiler thinks it is more | |
1366 | efficient to reorder, combine or merge accesses - something that would cause | |
1367 | the device to malfunction. | |
1368 | ||
1369 | Inside of the Linux kernel, I/O should be done through the appropriate accessor | |
1370 | routines - such as inb() or writel() - which know how to make such accesses | |
1371 | appropriately sequential. Whilst this, for the most part, renders the explicit | |
1372 | use of memory barriers unnecessary, there are a couple of situations where they | |
1373 | might be needed: | |
1374 | ||
1375 | (1) On some systems, I/O stores are not strongly ordered across all CPUs, and | |
1376 | so for _all_ general drivers locks should be used and mmiowb() must be | |
1377 | issued prior to unlocking the critical section. | |
1378 | ||
1379 | (2) If the accessor functions are used to refer to an I/O memory window with | |
1380 | relaxed memory access properties, then _mandatory_ memory barriers are | |
1381 | required to enforce ordering. | |
1382 | ||
1383 | See Documentation/DocBook/deviceiobook.tmpl for more information. | |
1384 | ||
1385 | ||
1386 | INTERRUPTS | |
1387 | ---------- | |
1388 | ||
1389 | A driver may be interrupted by its own interrupt service routine, and thus the | |
1390 | two parts of the driver may interfere with each other's attempts to control or | |
1391 | access the device. | |
1392 | ||
1393 | This may be alleviated - at least in part - by disabling local interrupts (a | |
1394 | form of locking), such that the critical operations are all contained within | |
1395 | the interrupt-disabled section in the driver. Whilst the driver's interrupt | |
1396 | routine is executing, the driver's core may not run on the same CPU, and its | |
1397 | interrupt is not permitted to happen again until the current interrupt has been | |
1398 | handled, thus the interrupt handler does not need to lock against that. | |
1399 | ||
1400 | However, consider a driver that was talking to an ethernet card that sports an | |
1401 | address register and a data register. If that driver's core talks to the card | |
1402 | under interrupt-disablement and then the driver's interrupt handler is invoked: | |
1403 | ||
1404 | LOCAL IRQ DISABLE | |
1405 | writew(ADDR, 3); | |
1406 | writew(DATA, y); | |
1407 | LOCAL IRQ ENABLE | |
1408 | <interrupt> | |
1409 | writew(ADDR, 4); | |
1410 | q = readw(DATA); | |
1411 | </interrupt> | |
1412 | ||
1413 | The store to the data register might happen after the second store to the | |
1414 | address register if ordering rules are sufficiently relaxed: | |
1415 | ||
1416 | STORE *ADDR = 3, STORE *ADDR = 4, STORE *DATA = y, q = LOAD *DATA | |
1417 | ||
1418 | ||
1419 | If ordering rules are relaxed, it must be assumed that accesses done inside an | |
1420 | interrupt disabled section may leak outside of it and may interleave with | |
1421 | accesses performed in an interrupt - and vice versa - unless implicit or | |
1422 | explicit barriers are used. | |
1423 | ||
1424 | Normally this won't be a problem because the I/O accesses done inside such | |
1425 | sections will include synchronous load operations on strictly ordered I/O | |
1426 | registers that form implicit I/O barriers. If this isn't sufficient then an | |
1427 | mmiowb() may need to be used explicitly. | |
1428 | ||
1429 | ||
1430 | A similar situation may occur between an interrupt routine and two routines | |
1431 | running on separate CPUs that communicate with each other. If such a case is | |
1432 | likely, then interrupt-disabling locks should be used to guarantee ordering. | |
1433 | ||
1434 | ||
1435 | ========================== | |
1436 | KERNEL I/O BARRIER EFFECTS | |
1437 | ========================== | |
1438 | ||
1439 | When accessing I/O memory, drivers should use the appropriate accessor | |
1440 | functions: | |
1441 | ||
1442 | (*) inX(), outX(): | |
1443 | ||
1444 | These are intended to talk to I/O space rather than memory space, but | |
1445 | that's primarily a CPU-specific concept. The i386 and x86_64 processors do | |
1446 | indeed have special I/O space access cycles and instructions, but many | |
1447 | CPUs don't have such a concept. | |
1448 | ||
1449 | The PCI bus, amongst others, defines an I/O space concept - which on such | |
1450 | CPUs as i386 and x86_64 cpus readily maps to the CPU's concept of I/O | |
1451 | space. However, it may also mapped as a virtual I/O space in the CPU's | |
1452 | memory map, particularly on those CPUs that don't support alternate | |
1453 | I/O spaces. | |
1454 | ||
1455 | Accesses to this space may be fully synchronous (as on i386), but | |
1456 | intermediary bridges (such as the PCI host bridge) may not fully honour | |
1457 | that. | |
1458 | ||
1459 | They are guaranteed to be fully ordered with respect to each other. | |
1460 | ||
1461 | They are not guaranteed to be fully ordered with respect to other types of | |
1462 | memory and I/O operation. | |
1463 | ||
1464 | (*) readX(), writeX(): | |
1465 | ||
1466 | Whether these are guaranteed to be fully ordered and uncombined with | |
1467 | respect to each other on the issuing CPU depends on the characteristics | |
1468 | defined for the memory window through which they're accessing. On later | |
1469 | i386 architecture machines, for example, this is controlled by way of the | |
1470 | MTRR registers. | |
1471 | ||
1472 | Ordinarily, these will be guaranteed to be fully ordered and uncombined,, | |
1473 | provided they're not accessing a prefetchable device. | |
1474 | ||
1475 | However, intermediary hardware (such as a PCI bridge) may indulge in | |
1476 | deferral if it so wishes; to flush a store, a load from the same location | |
1477 | is preferred[*], but a load from the same device or from configuration | |
1478 | space should suffice for PCI. | |
1479 | ||
1480 | [*] NOTE! attempting to load from the same location as was written to may | |
1481 | cause a malfunction - consider the 16550 Rx/Tx serial registers for | |
1482 | example. | |
1483 | ||
1484 | Used with prefetchable I/O memory, an mmiowb() barrier may be required to | |
1485 | force stores to be ordered. | |
1486 | ||
1487 | Please refer to the PCI specification for more information on interactions | |
1488 | between PCI transactions. | |
1489 | ||
1490 | (*) readX_relaxed() | |
1491 | ||
1492 | These are similar to readX(), but are not guaranteed to be ordered in any | |
1493 | way. Be aware that there is no I/O read barrier available. | |
1494 | ||
1495 | (*) ioreadX(), iowriteX() | |
1496 | ||
1497 | These will perform as appropriate for the type of access they're actually | |
1498 | doing, be it inX()/outX() or readX()/writeX(). | |
1499 | ||
1500 | ||
1501 | ======================================== | |
1502 | ASSUMED MINIMUM EXECUTION ORDERING MODEL | |
1503 | ======================================== | |
1504 | ||
1505 | It has to be assumed that the conceptual CPU is weakly-ordered but that it will | |
1506 | maintain the appearance of program causality with respect to itself. Some CPUs | |
1507 | (such as i386 or x86_64) are more constrained than others (such as powerpc or | |
1508 | frv), and so the most relaxed case (namely DEC Alpha) must be assumed outside | |
1509 | of arch-specific code. | |
1510 | ||
1511 | This means that it must be considered that the CPU will execute its instruction | |
1512 | stream in any order it feels like - or even in parallel - provided that if an | |
1513 | instruction in the stream depends on the an earlier instruction, then that | |
1514 | earlier instruction must be sufficiently complete[*] before the later | |
1515 | instruction may proceed; in other words: provided that the appearance of | |
1516 | causality is maintained. | |
1517 | ||
1518 | [*] Some instructions have more than one effect - such as changing the | |
1519 | condition codes, changing registers or changing memory - and different | |
1520 | instructions may depend on different effects. | |
1521 | ||
1522 | A CPU may also discard any instruction sequence that winds up having no | |
1523 | ultimate effect. For example, if two adjacent instructions both load an | |
1524 | immediate value into the same register, the first may be discarded. | |
1525 | ||
1526 | ||
1527 | Similarly, it has to be assumed that compiler might reorder the instruction | |
1528 | stream in any way it sees fit, again provided the appearance of causality is | |
1529 | maintained. | |
1530 | ||
1531 | ||
1532 | ============================ | |
1533 | THE EFFECTS OF THE CPU CACHE | |
1534 | ============================ | |
1535 | ||
1536 | The way cached memory operations are perceived across the system is affected to | |
1537 | a certain extent by the caches that lie between CPUs and memory, and by the | |
1538 | memory coherence system that maintains the consistency of state in the system. | |
1539 | ||
1540 | As far as the way a CPU interacts with another part of the system through the | |
1541 | caches goes, the memory system has to include the CPU's caches, and memory | |
1542 | barriers for the most part act at the interface between the CPU and its cache | |
1543 | (memory barriers logically act on the dotted line in the following diagram): | |
1544 | ||
1545 | <--- CPU ---> : <----------- Memory -----------> | |
1546 | : | |
1547 | +--------+ +--------+ : +--------+ +-----------+ | |
1548 | | | | | : | | | | +--------+ | |
1549 | | CPU | | Memory | : | CPU | | | | | | |
1550 | | Core |--->| Access |----->| Cache |<-->| | | | | |
1551 | | | | Queue | : | | | |--->| Memory | | |
1552 | | | | | : | | | | | | | |
1553 | +--------+ +--------+ : +--------+ | | | | | |
1554 | : | Cache | +--------+ | |
1555 | : | Coherency | | |
1556 | : | Mechanism | +--------+ | |
1557 | +--------+ +--------+ : +--------+ | | | | | |
1558 | | | | | : | | | | | | | |
1559 | | CPU | | Memory | : | CPU | | |--->| Device | | |
1560 | | Core |--->| Access |----->| Cache |<-->| | | | | |
1561 | | | | Queue | : | | | | | | | |
1562 | | | | | : | | | | +--------+ | |
1563 | +--------+ +--------+ : +--------+ +-----------+ | |
1564 | : | |
1565 | : | |
1566 | ||
1567 | Although any particular load or store may not actually appear outside of the | |
1568 | CPU that issued it since it may have been satisfied within the CPU's own cache, | |
1569 | it will still appear as if the full memory access had taken place as far as the | |
1570 | other CPUs are concerned since the cache coherency mechanisms will migrate the | |
1571 | cacheline over to the accessing CPU and propagate the effects upon conflict. | |
1572 | ||
1573 | The CPU core may execute instructions in any order it deems fit, provided the | |
1574 | expected program causality appears to be maintained. Some of the instructions | |
1575 | generate load and store operations which then go into the queue of memory | |
1576 | accesses to be performed. The core may place these in the queue in any order | |
1577 | it wishes, and continue execution until it is forced to wait for an instruction | |
1578 | to complete. | |
1579 | ||
1580 | What memory barriers are concerned with is controlling the order in which | |
1581 | accesses cross from the CPU side of things to the memory side of things, and | |
1582 | the order in which the effects are perceived to happen by the other observers | |
1583 | in the system. | |
1584 | ||
1585 | [!] Memory barriers are _not_ needed within a given CPU, as CPUs always see | |
1586 | their own loads and stores as if they had happened in program order. | |
1587 | ||
1588 | [!] MMIO or other device accesses may bypass the cache system. This depends on | |
1589 | the properties of the memory window through which devices are accessed and/or | |
1590 | the use of any special device communication instructions the CPU may have. | |
1591 | ||
1592 | ||
1593 | CACHE COHERENCY | |
1594 | --------------- | |
1595 | ||
1596 | Life isn't quite as simple as it may appear above, however: for while the | |
1597 | caches are expected to be coherent, there's no guarantee that that coherency | |
1598 | will be ordered. This means that whilst changes made on one CPU will | |
1599 | eventually become visible on all CPUs, there's no guarantee that they will | |
1600 | become apparent in the same order on those other CPUs. | |
1601 | ||
1602 | ||
1603 | Consider dealing with a system that has pair of CPUs (1 & 2), each of which has | |
1604 | a pair of parallel data caches (CPU 1 has A/B, and CPU 2 has C/D): | |
1605 | ||
1606 | : | |
1607 | : +--------+ | |
1608 | : +---------+ | | | |
1609 | +--------+ : +--->| Cache A |<------->| | | |
1610 | | | : | +---------+ | | | |
1611 | | CPU 1 |<---+ | | | |
1612 | | | : | +---------+ | | | |
1613 | +--------+ : +--->| Cache B |<------->| | | |
1614 | : +---------+ | | | |
1615 | : | Memory | | |
1616 | : +---------+ | System | | |
1617 | +--------+ : +--->| Cache C |<------->| | | |
1618 | | | : | +---------+ | | | |
1619 | | CPU 2 |<---+ | | | |
1620 | | | : | +---------+ | | | |
1621 | +--------+ : +--->| Cache D |<------->| | | |
1622 | : +---------+ | | | |
1623 | : +--------+ | |
1624 | : | |
1625 | ||
1626 | Imagine the system has the following properties: | |
1627 | ||
1628 | (*) an odd-numbered cache line may be in cache A, cache C or it may still be | |
1629 | resident in memory; | |
1630 | ||
1631 | (*) an even-numbered cache line may be in cache B, cache D or it may still be | |
1632 | resident in memory; | |
1633 | ||
1634 | (*) whilst the CPU core is interrogating one cache, the other cache may be | |
1635 | making use of the bus to access the rest of the system - perhaps to | |
1636 | displace a dirty cacheline or to do a speculative load; | |
1637 | ||
1638 | (*) each cache has a queue of operations that need to be applied to that cache | |
1639 | to maintain coherency with the rest of the system; | |
1640 | ||
1641 | (*) the coherency queue is not flushed by normal loads to lines already | |
1642 | present in the cache, even though the contents of the queue may | |
1643 | potentially effect those loads. | |
1644 | ||
1645 | Imagine, then, that two writes are made on the first CPU, with a write barrier | |
1646 | between them to guarantee that they will appear to reach that CPU's caches in | |
1647 | the requisite order: | |
1648 | ||
1649 | CPU 1 CPU 2 COMMENT | |
1650 | =============== =============== ======================================= | |
1651 | u == 0, v == 1 and p == &u, q == &u | |
1652 | v = 2; | |
1653 | smp_wmb(); Make sure change to v visible before | |
1654 | change to p | |
1655 | <A:modify v=2> v is now in cache A exclusively | |
1656 | p = &v; | |
1657 | <B:modify p=&v> p is now in cache B exclusively | |
1658 | ||
1659 | The write memory barrier forces the other CPUs in the system to perceive that | |
1660 | the local CPU's caches have apparently been updated in the correct order. But | |
1661 | now imagine that the second CPU that wants to read those values: | |
1662 | ||
1663 | CPU 1 CPU 2 COMMENT | |
1664 | =============== =============== ======================================= | |
1665 | ... | |
1666 | q = p; | |
1667 | x = *q; | |
1668 | ||
1669 | The above pair of reads may then fail to happen in expected order, as the | |
1670 | cacheline holding p may get updated in one of the second CPU's caches whilst | |
1671 | the update to the cacheline holding v is delayed in the other of the second | |
1672 | CPU's caches by some other cache event: | |
1673 | ||
1674 | CPU 1 CPU 2 COMMENT | |
1675 | =============== =============== ======================================= | |
1676 | u == 0, v == 1 and p == &u, q == &u | |
1677 | v = 2; | |
1678 | smp_wmb(); | |
1679 | <A:modify v=2> <C:busy> | |
1680 | <C:queue v=2> | |
1681 | p = &b; q = p; | |
1682 | <D:request p> | |
1683 | <B:modify p=&v> <D:commit p=&v> | |
1684 | <D:read p> | |
1685 | x = *q; | |
1686 | <C:read *q> Reads from v before v updated in cache | |
1687 | <C:unbusy> | |
1688 | <C:commit v=2> | |
1689 | ||
1690 | Basically, whilst both cachelines will be updated on CPU 2 eventually, there's | |
1691 | no guarantee that, without intervention, the order of update will be the same | |
1692 | as that committed on CPU 1. | |
1693 | ||
1694 | ||
1695 | To intervene, we need to interpolate a data dependency barrier or a read | |
1696 | barrier between the loads. This will force the cache to commit its coherency | |
1697 | queue before processing any further requests: | |
1698 | ||
1699 | CPU 1 CPU 2 COMMENT | |
1700 | =============== =============== ======================================= | |
1701 | u == 0, v == 1 and p == &u, q == &u | |
1702 | v = 2; | |
1703 | smp_wmb(); | |
1704 | <A:modify v=2> <C:busy> | |
1705 | <C:queue v=2> | |
1706 | p = &b; q = p; | |
1707 | <D:request p> | |
1708 | <B:modify p=&v> <D:commit p=&v> | |
1709 | <D:read p> | |
1710 | smp_read_barrier_depends() | |
1711 | <C:unbusy> | |
1712 | <C:commit v=2> | |
1713 | x = *q; | |
1714 | <C:read *q> Reads from v after v updated in cache | |
1715 | ||
1716 | ||
1717 | This sort of problem can be encountered on DEC Alpha processors as they have a | |
1718 | split cache that improves performance by making better use of the data bus. | |
1719 | Whilst most CPUs do imply a data dependency barrier on the read when a memory | |
1720 | access depends on a read, not all do, so it may not be relied on. | |
1721 | ||
1722 | Other CPUs may also have split caches, but must coordinate between the various | |
1723 | cachelets for normal memory accesss. The semantics of the Alpha removes the | |
1724 | need for coordination in absence of memory barriers. | |
1725 | ||
1726 | ||
1727 | CACHE COHERENCY VS DMA | |
1728 | ---------------------- | |
1729 | ||
1730 | Not all systems maintain cache coherency with respect to devices doing DMA. In | |
1731 | such cases, a device attempting DMA may obtain stale data from RAM because | |
1732 | dirty cache lines may be resident in the caches of various CPUs, and may not | |
1733 | have been written back to RAM yet. To deal with this, the appropriate part of | |
1734 | the kernel must flush the overlapping bits of cache on each CPU (and maybe | |
1735 | invalidate them as well). | |
1736 | ||
1737 | In addition, the data DMA'd to RAM by a device may be overwritten by dirty | |
1738 | cache lines being written back to RAM from a CPU's cache after the device has | |
1739 | installed its own data, or cache lines simply present in a CPUs cache may | |
1740 | simply obscure the fact that RAM has been updated, until at such time as the | |
1741 | cacheline is discarded from the CPU's cache and reloaded. To deal with this, | |
1742 | the appropriate part of the kernel must invalidate the overlapping bits of the | |
1743 | cache on each CPU. | |
1744 | ||
1745 | See Documentation/cachetlb.txt for more information on cache management. | |
1746 | ||
1747 | ||
1748 | CACHE COHERENCY VS MMIO | |
1749 | ----------------------- | |
1750 | ||
1751 | Memory mapped I/O usually takes place through memory locations that are part of | |
1752 | a window in the CPU's memory space that have different properties assigned than | |
1753 | the usual RAM directed window. | |
1754 | ||
1755 | Amongst these properties is usually the fact that such accesses bypass the | |
1756 | caching entirely and go directly to the device buses. This means MMIO accesses | |
1757 | may, in effect, overtake accesses to cached memory that were emitted earlier. | |
1758 | A memory barrier isn't sufficient in such a case, but rather the cache must be | |
1759 | flushed between the cached memory write and the MMIO access if the two are in | |
1760 | any way dependent. | |
1761 | ||
1762 | ||
1763 | ========================= | |
1764 | THE THINGS CPUS GET UP TO | |
1765 | ========================= | |
1766 | ||
1767 | A programmer might take it for granted that the CPU will perform memory | |
1768 | operations in exactly the order specified, so that if a CPU is, for example, | |
1769 | given the following piece of code to execute: | |
1770 | ||
1771 | a = *A; | |
1772 | *B = b; | |
1773 | c = *C; | |
1774 | d = *D; | |
1775 | *E = e; | |
1776 | ||
1777 | They would then expect that the CPU will complete the memory operation for each | |
1778 | instruction before moving on to the next one, leading to a definite sequence of | |
1779 | operations as seen by external observers in the system: | |
1780 | ||
1781 | LOAD *A, STORE *B, LOAD *C, LOAD *D, STORE *E. | |
1782 | ||
1783 | ||
1784 | Reality is, of course, much messier. With many CPUs and compilers, the above | |
1785 | assumption doesn't hold because: | |
1786 | ||
1787 | (*) loads are more likely to need to be completed immediately to permit | |
1788 | execution progress, whereas stores can often be deferred without a | |
1789 | problem; | |
1790 | ||
1791 | (*) loads may be done speculatively, and the result discarded should it prove | |
1792 | to have been unnecessary; | |
1793 | ||
1794 | (*) loads may be done speculatively, leading to the result having being | |
1795 | fetched at the wrong time in the expected sequence of events; | |
1796 | ||
1797 | (*) the order of the memory accesses may be rearranged to promote better use | |
1798 | of the CPU buses and caches; | |
1799 | ||
1800 | (*) loads and stores may be combined to improve performance when talking to | |
1801 | memory or I/O hardware that can do batched accesses of adjacent locations, | |
1802 | thus cutting down on transaction setup costs (memory and PCI devices may | |
1803 | both be able to do this); and | |
1804 | ||
1805 | (*) the CPU's data cache may affect the ordering, and whilst cache-coherency | |
1806 | mechanisms may alleviate this - once the store has actually hit the cache | |
1807 | - there's no guarantee that the coherency management will be propagated in | |
1808 | order to other CPUs. | |
1809 | ||
1810 | So what another CPU, say, might actually observe from the above piece of code | |
1811 | is: | |
1812 | ||
1813 | LOAD *A, ..., LOAD {*C,*D}, STORE *E, STORE *B | |
1814 | ||
1815 | (Where "LOAD {*C,*D}" is a combined load) | |
1816 | ||
1817 | ||
1818 | However, it is guaranteed that a CPU will be self-consistent: it will see its | |
1819 | _own_ accesses appear to be correctly ordered, without the need for a memory | |
1820 | barrier. For instance with the following code: | |
1821 | ||
1822 | U = *A; | |
1823 | *A = V; | |
1824 | *A = W; | |
1825 | X = *A; | |
1826 | *A = Y; | |
1827 | Z = *A; | |
1828 | ||
1829 | and assuming no intervention by an external influence, it can be assumed that | |
1830 | the final result will appear to be: | |
1831 | ||
1832 | U == the original value of *A | |
1833 | X == W | |
1834 | Z == Y | |
1835 | *A == Y | |
1836 | ||
1837 | The code above may cause the CPU to generate the full sequence of memory | |
1838 | accesses: | |
1839 | ||
1840 | U=LOAD *A, STORE *A=V, STORE *A=W, X=LOAD *A, STORE *A=Y, Z=LOAD *A | |
1841 | ||
1842 | in that order, but, without intervention, the sequence may have almost any | |
1843 | combination of elements combined or discarded, provided the program's view of | |
1844 | the world remains consistent. | |
1845 | ||
1846 | The compiler may also combine, discard or defer elements of the sequence before | |
1847 | the CPU even sees them. | |
1848 | ||
1849 | For instance: | |
1850 | ||
1851 | *A = V; | |
1852 | *A = W; | |
1853 | ||
1854 | may be reduced to: | |
1855 | ||
1856 | *A = W; | |
1857 | ||
1858 | since, without a write barrier, it can be assumed that the effect of the | |
1859 | storage of V to *A is lost. Similarly: | |
1860 | ||
1861 | *A = Y; | |
1862 | Z = *A; | |
1863 | ||
1864 | may, without a memory barrier, be reduced to: | |
1865 | ||
1866 | *A = Y; | |
1867 | Z = Y; | |
1868 | ||
1869 | and the LOAD operation never appear outside of the CPU. | |
1870 | ||
1871 | ||
1872 | AND THEN THERE'S THE ALPHA | |
1873 | -------------------------- | |
1874 | ||
1875 | The DEC Alpha CPU is one of the most relaxed CPUs there is. Not only that, | |
1876 | some versions of the Alpha CPU have a split data cache, permitting them to have | |
1877 | two semantically related cache lines updating at separate times. This is where | |
1878 | the data dependency barrier really becomes necessary as this synchronises both | |
1879 | caches with the memory coherence system, thus making it seem like pointer | |
1880 | changes vs new data occur in the right order. | |
1881 | ||
1882 | The Alpha defines the Linux's kernel's memory barrier model. | |
1883 | ||
1884 | See the subsection on "Cache Coherency" above. | |
1885 | ||
1886 | ||
1887 | ========== | |
1888 | REFERENCES | |
1889 | ========== | |
1890 | ||
1891 | Alpha AXP Architecture Reference Manual, Second Edition (Sites & Witek, | |
1892 | Digital Press) | |
1893 | Chapter 5.2: Physical Address Space Characteristics | |
1894 | Chapter 5.4: Caches and Write Buffers | |
1895 | Chapter 5.5: Data Sharing | |
1896 | Chapter 5.6: Read/Write Ordering | |
1897 | ||
1898 | AMD64 Architecture Programmer's Manual Volume 2: System Programming | |
1899 | Chapter 7.1: Memory-Access Ordering | |
1900 | Chapter 7.4: Buffering and Combining Memory Writes | |
1901 | ||
1902 | IA-32 Intel Architecture Software Developer's Manual, Volume 3: | |
1903 | System Programming Guide | |
1904 | Chapter 7.1: Locked Atomic Operations | |
1905 | Chapter 7.2: Memory Ordering | |
1906 | Chapter 7.4: Serializing Instructions | |
1907 | ||
1908 | The SPARC Architecture Manual, Version 9 | |
1909 | Chapter 8: Memory Models | |
1910 | Appendix D: Formal Specification of the Memory Models | |
1911 | Appendix J: Programming with the Memory Models | |
1912 | ||
1913 | UltraSPARC Programmer Reference Manual | |
1914 | Chapter 5: Memory Accesses and Cacheability | |
1915 | Chapter 15: Sparc-V9 Memory Models | |
1916 | ||
1917 | UltraSPARC III Cu User's Manual | |
1918 | Chapter 9: Memory Models | |
1919 | ||
1920 | UltraSPARC IIIi Processor User's Manual | |
1921 | Chapter 8: Memory Models | |
1922 | ||
1923 | UltraSPARC Architecture 2005 | |
1924 | Chapter 9: Memory | |
1925 | Appendix D: Formal Specifications of the Memory Models | |
1926 | ||
1927 | UltraSPARC T1 Supplement to the UltraSPARC Architecture 2005 | |
1928 | Chapter 8: Memory Models | |
1929 | Appendix F: Caches and Cache Coherency | |
1930 | ||
1931 | Solaris Internals, Core Kernel Architecture, p63-68: | |
1932 | Chapter 3.3: Hardware Considerations for Locks and | |
1933 | Synchronization | |
1934 | ||
1935 | Unix Systems for Modern Architectures, Symmetric Multiprocessing and Caching | |
1936 | for Kernel Programmers: | |
1937 | Chapter 13: Other Memory Models | |
1938 | ||
1939 | Intel Itanium Architecture Software Developer's Manual: Volume 1: | |
1940 | Section 2.6: Speculation | |
1941 | Section 4.4: Memory Access |