drm/amdgpu: throttle buffer migrations at CS using a fixed MBps limit (v2)

drm/amdgpu: throttle buffer migrations at CS using a fixed MBps limit (v2)

The old mechanism used a per-submission limit that didn't take previous
submissions within the same time frame into account. It also filled VRAM
slowly when VRAM usage dropped due to a big eviction or buffer deallocation.

This new method establishes a configurable MBps limit that is obeyed when
VRAM usage is very high. When VRAM usage is not very high, it gives
the driver the freedom to fill it quickly. The result is more consistent
performance.

It can't keep the BO move rate low if lots of evictions are happening due
to VRAM fragmentation, or if a big buffer is being migrated.

The amdgpu.moverate parameter can be used to set a non-default limit.
Measurements can be done to find out which amdgpu.moverate setting gives
the best results.

Mainly APUs and cards with small VRAM will benefit from this. For F1 2015,
anything with 2 GB VRAM or less will benefit.

Some benchmark results - F1 2015 (Tonga 2GB):

Limit      MinFPS AvgFPS
Old code:  14     32.6
128 MB/s:  28     41
64 MB/s:   15.5   43
32 MB/s:   28.7   43.4
8 MB/s:    27.8   44.4
8 MB/s:    21.9   42.8 (different run)

Random drops in Min FPS can still occur (due to fragmented VRAM?), but
the average FPS is much better. 8 MB/s is probably a good limit for this
game & the current VRAM management. The random FPS drops are still to be
tackled.

v2: use a spinlock

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>