Commit | Line | Data |
---|---|---|
5a225791 TH |
1 | I/O Barriers |
2 | ============ | |
3 | Tejun Heo <htejun@gmail.com>, July 22 2005 | |
4 | ||
5 | I/O barrier requests are used to guarantee ordering around the barrier | |
6 | requests. Unless you're crazy enough to use disk drives for | |
7 | implementing synchronization constructs (wow, sounds interesting...), | |
8 | the ordering is meaningful only for write requests for things like | |
9 | journal checkpoints. All requests queued before a barrier request | |
10 | must be finished (made it to the physical medium) before the barrier | |
11 | request is started, and all requests queued after the barrier request | |
12 | must be started only after the barrier request is finished (again, | |
13 | made it to the physical medium). | |
14 | ||
15 | In other words, I/O barrier requests have the following two properties. | |
16 | ||
17 | 1. Request ordering | |
18 | ||
19 | Requests cannot pass the barrier request. Preceding requests are | |
20 | processed before the barrier and following requests after. | |
21 | ||
22 | Depending on what features a drive supports, this can be done in one | |
23 | of the following three ways. | |
24 | ||
25 | i. For devices which have queue depth greater than 1 (TCQ devices) and | |
26 | support ordered tags, block layer can just issue the barrier as an | |
27 | ordered request and the lower level driver, controller and drive | |
6c28f2c0 | 28 | itself are responsible for making sure that the ordering constraint is |
5a225791 TH |
29 | met. Most modern SCSI controllers/drives should support this. |
30 | ||
31 | NOTE: SCSI ordered tag isn't currently used due to limitation in the | |
32 | SCSI midlayer, see the following random notes section. | |
33 | ||
34 | ii. For devices which have queue depth greater than 1 but don't | |
35 | support ordered tags, block layer ensures that the requests preceding | |
36 | a barrier request finishes before issuing the barrier request. Also, | |
37 | it defers requests following the barrier until the barrier request is | |
38 | finished. Older SCSI controllers/drives and SATA drives fall in this | |
39 | category. | |
40 | ||
41 | iii. Devices which have queue depth of 1. This is a degenerate case | |
42 | of ii. Just keeping issue order suffices. Ancient SCSI | |
43 | controllers/drives and IDE drives are in this category. | |
44 | ||
992caacf | 45 | 2. Forced flushing to physical medium |
5a225791 TH |
46 | |
47 | Again, if you're not gonna do synchronization with disk drives (dang, | |
48 | it sounds even more appealing now!), the reason you use I/O barriers | |
49 | is mainly to protect filesystem integrity when power failure or some | |
50 | other events abruptly stop the drive from operating and possibly make | |
51 | the drive lose data in its cache. So, I/O barriers need to guarantee | |
52 | that requests actually get written to non-volatile medium in order. | |
53 | ||
54 | There are four cases, | |
55 | ||
56 | i. No write-back cache. Keeping requests ordered is enough. | |
57 | ||
58 | ii. Write-back cache but no flush operation. There's no way to | |
a2ffd275 | 59 | guarantee physical-medium commit order. This kind of devices can't to |
5a225791 TH |
60 | I/O barriers. |
61 | ||
62 | iii. Write-back cache and flush operation but no FUA (forced unit | |
63 | access). We need two cache flushes - before and after the barrier | |
64 | request. | |
65 | ||
66 | iv. Write-back cache, flush operation and FUA. We still need one | |
67 | flush to make sure requests preceding a barrier are written to medium, | |
68 | but post-barrier flush can be avoided by using FUA write on the | |
69 | barrier itself. | |
70 | ||
71 | ||
72 | How to support barrier requests in drivers | |
73 | ------------------------------------------ | |
74 | ||
75 | All barrier handling is done inside block layer proper. All low level | |
76 | drivers have to are implementing its prepare_flush_fn and using one | |
77 | the following two functions to indicate what barrier type it supports | |
78 | and how to prepare flush requests. Note that the term 'ordered' is | |
79 | used to indicate the whole sequence of performing barrier requests | |
80 | including draining and flushing. | |
81 | ||
165125e1 | 82 | typedef void (prepare_flush_fn)(struct request_queue *q, struct request *rq); |
5a225791 | 83 | |
165125e1 | 84 | int blk_queue_ordered(struct request_queue *q, unsigned ordered, |
c0613c1c | 85 | prepare_flush_fn *prepare_flush_fn); |
5a225791 TH |
86 | |
87 | @q : the queue in question | |
88 | @ordered : the ordered mode the driver/device supports | |
89 | @prepare_flush_fn : this function should prepare @rq such that it | |
90 | flushes cache to physical medium when executed | |
5a225791 TH |
91 | |
92 | For example, SCSI disk driver's prepare_flush_fn looks like the | |
93 | following. | |
94 | ||
165125e1 | 95 | static void sd_prepare_flush(struct request_queue *q, struct request *rq) |
5a225791 TH |
96 | { |
97 | memset(rq->cmd, 0, sizeof(rq->cmd)); | |
c0613c1c | 98 | rq->cmd_type = REQ_TYPE_BLOCK_PC; |
5a225791 TH |
99 | rq->timeout = SD_TIMEOUT; |
100 | rq->cmd[0] = SYNCHRONIZE_CACHE; | |
c0613c1c | 101 | rq->cmd_len = 10; |
5a225791 TH |
102 | } |
103 | ||
104 | The following seven ordered modes are supported. The following table | |
105 | shows which mode should be used depending on what features a | |
106 | device/driver supports. In the leftmost column of table, | |
107 | QUEUE_ORDERED_ prefix is omitted from the mode names to save space. | |
108 | ||
109 | The table is followed by description of each mode. Note that in the | |
110 | descriptions of QUEUE_ORDERED_DRAIN*, '=>' is used whereas '->' is | |
111 | used for QUEUE_ORDERED_TAG* descriptions. '=>' indicates that the | |
112 | preceding step must be complete before proceeding to the next step. | |
113 | '->' indicates that the next step can start as soon as the previous | |
114 | step is issued. | |
115 | ||
116 | write-back cache ordered tag flush FUA | |
117 | ----------------------------------------------------------------------- | |
118 | NONE yes/no N/A no N/A | |
119 | DRAIN no no N/A N/A | |
120 | DRAIN_FLUSH yes no yes no | |
121 | DRAIN_FUA yes no yes yes | |
122 | TAG no yes N/A N/A | |
123 | TAG_FLUSH yes yes yes no | |
124 | TAG_FUA yes yes yes yes | |
125 | ||
126 | ||
127 | QUEUE_ORDERED_NONE | |
128 | I/O barriers are not needed and/or supported. | |
129 | ||
130 | Sequence: N/A | |
131 | ||
132 | QUEUE_ORDERED_DRAIN | |
133 | Requests are ordered by draining the request queue and cache | |
134 | flushing isn't needed. | |
135 | ||
136 | Sequence: drain => barrier | |
137 | ||
138 | QUEUE_ORDERED_DRAIN_FLUSH | |
139 | Requests are ordered by draining the request queue and both | |
140 | pre-barrier and post-barrier cache flushings are needed. | |
141 | ||
142 | Sequence: drain => preflush => barrier => postflush | |
143 | ||
144 | QUEUE_ORDERED_DRAIN_FUA | |
145 | Requests are ordered by draining the request queue and | |
146 | pre-barrier cache flushing is needed. By using FUA on barrier | |
147 | request, post-barrier flushing can be skipped. | |
148 | ||
149 | Sequence: drain => preflush => barrier | |
150 | ||
151 | QUEUE_ORDERED_TAG | |
152 | Requests are ordered by ordered tag and cache flushing isn't | |
153 | needed. | |
154 | ||
155 | Sequence: barrier | |
156 | ||
157 | QUEUE_ORDERED_TAG_FLUSH | |
158 | Requests are ordered by ordered tag and both pre-barrier and | |
159 | post-barrier cache flushings are needed. | |
160 | ||
161 | Sequence: preflush -> barrier -> postflush | |
162 | ||
163 | QUEUE_ORDERED_TAG_FUA | |
164 | Requests are ordered by ordered tag and pre-barrier cache | |
165 | flushing is needed. By using FUA on barrier request, | |
166 | post-barrier flushing can be skipped. | |
167 | ||
168 | Sequence: preflush -> barrier | |
169 | ||
170 | ||
171 | Random notes/caveats | |
172 | -------------------- | |
173 | ||
174 | * SCSI layer currently can't use TAG ordering even if the drive, | |
175 | controller and driver support it. The problem is that SCSI midlayer | |
176 | request dispatch function is not atomic. It releases queue lock and | |
177 | switch to SCSI host lock during issue and it's possible and likely to | |
178 | happen in time that requests change their relative positions. Once | |
179 | this problem is solved, TAG ordering can be enabled. | |
180 | ||
181 | * Currently, no matter which ordered mode is used, there can be only | |
182 | one barrier request in progress. All I/O barriers are held off by | |
183 | block layer until the previous I/O barrier is complete. This doesn't | |
184 | make any difference for DRAIN ordered devices, but, for TAG ordered | |
185 | devices with very high command latency, passing multiple I/O barriers | |
186 | to low level *might* be helpful if they are very frequent. Well, this | |
187 | certainly is a non-issue. I'm writing this just to make clear that no | |
188 | two I/O barrier is ever passed to low-level driver. | |
189 | ||
190 | * Completion order. Requests in ordered sequence are issued in order | |
191 | but not required to finish in order. Barrier implementation can | |
192 | handle out-of-order completion of ordered sequence. IOW, the requests | |
193 | MUST be processed in order but the hardware/software completion paths | |
194 | are allowed to reorder completion notifications - eg. current SCSI | |
195 | midlayer doesn't preserve completion order during error handling. | |
196 | ||
197 | * Requeueing order. Low-level drivers are free to requeue any request | |
198 | after they removed it from the request queue with | |
199 | blkdev_dequeue_request(). As barrier sequence should be kept in order | |
200 | when requeued, generic elevator code takes care of putting requests in | |
201 | order around barrier. See blk_ordered_req_seq() and | |
202 | ELEVATOR_INSERT_REQUEUE handling in __elv_add_request() for details. | |
203 | ||
204 | Note that block drivers must not requeue preceding requests while | |
205 | completing latter requests in an ordered sequence. Currently, no | |
206 | error checking is done against this. | |
207 | ||
208 | * Error handling. Currently, block layer will report error to upper | |
209 | layer if any of requests in an ordered sequence fails. Unfortunately, | |
210 | this doesn't seem to be enough. Look at the following request flow. | |
211 | QUEUE_ORDERED_TAG_FLUSH is in use. | |
212 | ||
213 | [0] [1] [2] [3] [pre] [barrier] [post] < [4] [5] [6] ... > | |
214 | still in elevator | |
215 | ||
216 | Let's say request [2], [3] are write requests to update file system | |
217 | metadata (journal or whatever) and [barrier] is used to mark that | |
218 | those updates are valid. Consider the following sequence. | |
219 | ||
220 | i. Requests [0] ~ [post] leaves the request queue and enters | |
221 | low-level driver. | |
222 | ii. After a while, unfortunately, something goes wrong and the | |
223 | drive fails [2]. Note that any of [0], [1] and [3] could have | |
224 | completed by this time, but [pre] couldn't have been finished | |
225 | as the drive must process it in order and it failed before | |
226 | processing that command. | |
227 | iii. Error handling kicks in and determines that the error is | |
228 | unrecoverable and fails [2], and resumes operation. | |
229 | iv. [pre] [barrier] [post] gets processed. | |
230 | v. *BOOM* power fails | |
231 | ||
232 | The problem here is that the barrier request is *supposed* to indicate | |
233 | that filesystem update requests [2] and [3] made it safely to the | |
234 | physical medium and, if the machine crashes after the barrier is | |
235 | written, filesystem recovery code can depend on that. Sadly, that | |
236 | isn't true in this case anymore. IOW, the success of a I/O barrier | |
237 | should also be dependent on success of some of the preceding requests, | |
238 | where only upper layer (filesystem) knows what 'some' is. | |
239 | ||
240 | This can be solved by implementing a way to tell the block layer which | |
241 | requests affect the success of the following barrier request and | |
242 | making lower lever drivers to resume operation on error only after | |
243 | block layer tells it to do so. | |
244 | ||
245 | As the probability of this happening is very low and the drive should | |
246 | be faulty, implementing the fix is probably an overkill. But, still, | |
247 | it's there. | |
248 | ||
249 | * In previous drafts of barrier implementation, there was fallback | |
250 | mechanism such that, if FUA or ordered TAG fails, less fancy ordered | |
251 | mode can be selected and the failed barrier request is retried | |
252 | automatically. The rationale for this feature was that as FUA is | |
253 | pretty new in ATA world and ordered tag was never used widely, there | |
254 | could be devices which report to support those features but choke when | |
255 | actually given such requests. | |
256 | ||
257 | This was removed for two reasons 1. it's an overkill 2. it's | |
258 | impossible to implement properly when TAG ordering is used as low | |
259 | level drivers resume after an error automatically. If it's ever | |
260 | needed adding it back and modifying low level drivers accordingly | |
261 | shouldn't be difficult. |