Commit | Line | Data |
---|---|---|
f2836352 JT |
1 | Guidance for writing policies |
2 | ============================= | |
3 | ||
4 | Try to keep transactionality out of it. The core is careful to | |
5 | avoid asking about anything that is migrating. This is a pain, but | |
6 | makes it easier to write the policies. | |
7 | ||
8 | Mappings are loaded into the policy at construction time. | |
9 | ||
10 | Every bio that is mapped by the target is referred to the policy. | |
11 | The policy can return a simple HIT or MISS or issue a migration. | |
12 | ||
13 | Currently there's no way for the policy to issue background work, | |
14 | e.g. to start writing back dirty blocks that are going to be evicte | |
15 | soon. | |
16 | ||
17 | Because we map bios, rather than requests it's easy for the policy | |
18 | to get fooled by many small bios. For this reason the core target | |
19 | issues periodic ticks to the policy. It's suggested that the policy | |
20 | doesn't update states (eg, hit counts) for a block more than once | |
21 | for each tick. The core ticks by watching bios complete, and so | |
22 | trying to see when the io scheduler has let the ios run. | |
23 | ||
24 | ||
25 | Overview of supplied cache replacement policies | |
26 | =============================================== | |
27 | ||
bccab6a0 MS |
28 | multiqueue (mq) |
29 | --------------- | |
f2836352 | 30 | |
bccab6a0 | 31 | This policy has been deprecated in favor of the smq policy (see below). |
f2836352 | 32 | |
01911c19 JT |
33 | The multiqueue policy has three sets of 16 queues: one set for entries |
34 | waiting for the cache and another two for those in the cache (a set for | |
35 | clean entries and a set for dirty entries). | |
36 | ||
f2836352 JT |
37 | Cache entries in the queues are aged based on logical time. Entry into |
38 | the cache is based on variable thresholds and queue selection is based | |
39 | on hit count on entry. The policy aims to take different cache miss | |
40 | costs into account and to adjust to varying load patterns automatically. | |
41 | ||
42 | Message and constructor argument pairs are: | |
78e03d69 JT |
43 | 'sequential_threshold <#nr_sequential_ios>' |
44 | 'random_threshold <#nr_random_ios>' | |
45 | 'read_promote_adjustment <value>' | |
46 | 'write_promote_adjustment <value>' | |
47 | 'discard_promote_adjustment <value>' | |
f2836352 JT |
48 | |
49 | The sequential threshold indicates the number of contiguous I/Os | |
f1afb36a MS |
50 | required before a stream is treated as sequential. Once a stream is |
51 | considered sequential it will bypass the cache. The random threshold | |
f2836352 JT |
52 | is the number of intervening non-contiguous I/Os that must be seen |
53 | before the stream is treated as random again. | |
54 | ||
55 | The sequential and random thresholds default to 512 and 4 respectively. | |
56 | ||
f1afb36a MS |
57 | Large, sequential I/Os are probably better left on the origin device |
58 | since spindles tend to have good sequential I/O bandwidth. The | |
59 | io_tracker counts contiguous I/Os to try to spot when the I/O is in one | |
60 | of these sequential modes. But there are use-cases for wanting to | |
61 | promote sequential blocks to the cache (e.g. fast application startup). | |
62 | If sequential threshold is set to 0 the sequential I/O detection is | |
63 | disabled and sequential I/O will no longer implicitly bypass the cache. | |
64 | Setting the random threshold to 0 does _not_ disable the random I/O | |
65 | stream detection. | |
f2836352 | 66 | |
b155aa0e JT |
67 | Internally the mq policy determines a promotion threshold. If the hit |
68 | count of a block not in the cache goes above this threshold it gets | |
69 | promoted to the cache. The read, write and discard promote adjustment | |
78e03d69 JT |
70 | tunables allow you to tweak the promotion threshold by adding a small |
71 | value based on the io type. They default to 4, 8 and 1 respectively. | |
72 | If you're trying to quickly warm a new cache device you may wish to | |
73 | reduce these to encourage promotion. Remember to switch them back to | |
74 | their defaults after the cache fills though. | |
75 | ||
bccab6a0 MS |
76 | Stochastic multiqueue (smq) |
77 | --------------------------- | |
78 | ||
79 | This policy is the default. | |
80 | ||
81 | The stochastic multi-queue (smq) policy addresses some of the problems | |
82 | with the multiqueue (mq) policy. | |
83 | ||
84 | The smq policy (vs mq) offers the promise of less memory utilization, | |
85 | improved performance and increased adaptability in the face of changing | |
86 | workloads. SMQ also does not have any cumbersome tuning knobs. | |
87 | ||
88 | Users may switch from "mq" to "smq" simply by appropriately reloading a | |
89 | DM table that is using the cache target. Doing so will cause all of the | |
90 | mq policy's hints to be dropped. Also, performance of the cache may | |
91 | degrade slightly until smq recalculates the origin device's hotspots | |
92 | that should be cached. | |
93 | ||
94 | Memory usage: | |
95 | The mq policy uses a lot of memory; 88 bytes per cache block on a 64 | |
96 | bit machine. | |
97 | ||
98 | SMQ uses 28bit indexes to implement it's data structures rather than | |
99 | pointers. It avoids storing an explicit hit count for each block. It | |
100 | has a 'hotspot' queue rather than a pre cache which uses a quarter of | |
101 | the entries (each hotspot block covers a larger area than a single | |
102 | cache block). | |
103 | ||
104 | All these mean smq uses ~25bytes per cache block. Still a lot of | |
105 | memory, but a substantial improvement nontheless. | |
106 | ||
107 | Level balancing: | |
108 | MQ places entries in different levels of the multiqueue structures | |
109 | based on their hit count (~ln(hit count)). This means the bottom | |
110 | levels generally have the most entries, and the top ones have very | |
111 | few. Having unbalanced levels like this reduces the efficacy of the | |
112 | multiqueue. | |
113 | ||
114 | SMQ does not maintain a hit count, instead it swaps hit entries with | |
115 | the least recently used entry from the level above. The over all | |
116 | ordering being a side effect of this stochastic process. With this | |
117 | scheme we can decide how many entries occupy each multiqueue level, | |
118 | resulting in better promotion/demotion decisions. | |
119 | ||
120 | Adaptability: | |
121 | The MQ policy maintains a hit count for each cache block. For a | |
122 | different block to get promoted to the cache it's hit count has to | |
123 | exceed the lowest currently in the cache. This means it can take a | |
124 | long time for the cache to adapt between varying IO patterns. | |
125 | Periodically degrading the hit counts could help with this, but I | |
126 | haven't found a nice general solution. | |
127 | ||
128 | SMQ doesn't maintain hit counts, so a lot of this problem just goes | |
129 | away. In addition it tracks performance of the hotspot queue, which | |
130 | is used to decide which blocks to promote. If the hotspot queue is | |
131 | performing badly then it starts moving entries more quickly between | |
132 | levels. This lets it adapt to new IO patterns very quickly. | |
133 | ||
134 | Performance: | |
135 | Testing SMQ shows substantially better performance than MQ. | |
136 | ||
8735a813 HM |
137 | cleaner |
138 | ------- | |
139 | ||
140 | The cleaner writes back all dirty blocks in a cache to decommission it. | |
141 | ||
f2836352 JT |
142 | Examples |
143 | ======== | |
144 | ||
145 | The syntax for a table is: | |
146 | cache <metadata dev> <cache dev> <origin dev> <block size> | |
147 | <#feature_args> [<feature arg>]* | |
148 | <policy> <#policy_args> [<policy arg>]* | |
149 | ||
150 | The syntax to send a message using the dmsetup command is: | |
151 | dmsetup message <mapped device> 0 sequential_threshold 1024 | |
152 | dmsetup message <mapped device> 0 random_threshold 8 | |
153 | ||
154 | Using dmsetup: | |
155 | dmsetup create blah --table "0 268435456 cache /dev/sdb /dev/sdc \ | |
156 | /dev/sdd 512 0 mq 4 sequential_threshold 1024 random_threshold 8" | |
157 | creates a 128GB large mapped device named 'blah' with the | |
158 | sequential threshold set to 1024 and the random_threshold set to 8. |