Mouse Sampling in Unity3d

(This is going to be a bit Unity3D heavy, but should point at how polling rates are a technological constraint which can affect gameplay design – these are real considerations when trying to get the best feeling game possible, and it’s even worth avoiding certain kinds of games if you can’t be sure of the frequency of your interface’s data).

I’m working on some home-brew which centers around mouse movement for both camera, and gameplay. I use the physics system to determine game objects’ proximity in a lot of cases (i.e. bullets hitting volumes), so I have to use FixedUpdate to push gameplay forward. But I also want smooth mouse based camera movement – ideally as fast as the game can render. My camera also follows around the physics object. But physics and rendering update at different rates.

As a result things seem to fall out of sync, and stutter. Messing with interpolation settings on the rigidbody just doesn’t cut it, especially when you handle physics updates yourself (setting the rigidbody physics to kinematic pretty much nullifies interpolation, I’m pretty sure – correct me if I’m wrong!).

So if things are going out of sync, what can you do to get them in sync? Well, unfortunately, we don’t have control over the mouse polling frequency, but we can tailor physics and rendering rates to get something consistent coming into our game.

Experiment – Looking at the Raw Input

I’ve done a little demo showing what’s happening to sampled inputs when setting different frame rates. Since it’s in a web-player, you may want to right-click and fullscreen to override the Unity WebPlayer’s capping of 50Hz (although it’s possible that simply setting Application.targetFrameRate may override this, even in the web player? I could be wrong there).

The two 3D graphs show what values you’re grabbing when you sample your mouse as a 2D axis, with time being the third axis:

Vector2 GetRawAxisNow()
 {
 return new Vector2(Input.GetAxisRaw("Mouse X"), Input.GetAxisRaw("Mouse Y"));
 }

The top one samples from within void Update(), whereas the bottom is sampling from within void FixedUpdate() – the physics update call on every MonoBehaviour.

The two sliders at the bottom control Application.targetFrameRate and Time.fixedDeltaTime. These are what Render Updates and Physics Update rates are effectively “capped” to, respectively. Physics Updates are guaranteed to occur, even at the risk of slowing the game down in real time, whereas render frames drop whenever they like. See this image:

Picture courtesy of Richard Fine (@superpig)

Diagram by Richard Fine (@superpig)

This means you may have multiple physics updates per frame if the rendering is running slower, or none at all if the rendering is much quicker than the physics fixed update. This is in an attempt to keep the game state running in real time, even if the game is rendering too slowly. This isn’t always the way you want to go (many arcade bullet hell games accept gameplay slowdown as a feature – the screen fills with bullets, and the resultant processing bottlenecks means you get gameplay slowdown to help compensate. Most 1vs1 fighting games attempt to have 60fps game play for 60fps input polling), but it’s worth being aware that Unity takes this decoupled approach to meet broader demands.

A Little Analysis

When you set the Application.targetFrameRate very high, you get a lot of spiking of input values – jumping from zero to… whatever was recorded by the polling, and back to zero again.

This is because  GetRawAxis is only giving you pixels moved since the last Update. If you sample twice before the input is sampled again, it has no new information for you, and comes back as zero, even if your mouse seems to be moving smoothly in real life. Unfortunately, there’s no way to change how quickly the mouse is polled in unity, as such. It’s an edge-case desire from the engine for most, but for some kinaesthetic obsessives (like myself and the entire fighting game audience), it’d be really nice to have.

If you set your Application.targetFrameRate frame rate low, your Update frequency gets capped, and sampling GetAxisRaw gives you the accumulated change in pixel position since the last update you checked. So, if your mouse moves right, 5 pixels, stops a short while, then moves left 4 pixels in-between a render frame (and each of these moves is polled), the result that Update reacts to is just 5-4=1 pixel.

You may “miss” the fact that the mouse jerked right then left. The move shows up as a slow move to the right instead of a flick, because you’re getting coarse sampling, and not the individual polled values – just the accumulated travel since the last time you polled in the respective Render or Physics update.

This may occasionally mean that you “miss” the fact that a mouse cursor passed over an area of the screen – i.e. hover over events on a button will likely be missed if your frame rate has dropped – the UI equivalent of physics “tunnelling”.

You could, if you like, do the equivalent of “swept” mouse moves to see if your mouse passes over UI objects at high speed, but this doesn’t account for the peaks of flicking movements. Starts to get awful complicated, huh?

Mouse smoothing is often a useful technique if your frame rate is faster than your mouse sampling, as it can fill in the gaps. Unity builds in their own kind (just use Input.GetAxis, as opposed to GetAxisRaw, and play around with the settings in Edit->Project Settings->Input – and I recommend putting “snap” on if you do – stops axes having too much sense of “momentum” when jerking the mouse in the opposite direction), but you might want to set up your own smoothing systems.

Changing Time.fixedDeltaTime to higher rates than Application.targetFrameRate shows what we already know from Richard’s diagram: that we’re only polling as fast as render updates occur. This is pretty annoying, frankly, because you could have multiple physics frames (i.e. gameplay) reacting to the same inputs just because your rendering has spiked. This is why you really don’t want to use Inputs inside a FixedUpdate() to trigger, say, jumps – you might get two jump impulses because the physics is acting like you’re doing a positive edge button press for two frames in a row! And yet, if your game is in any way physics based, FixedUpdate is pushing forward the more “honest” version of the game state! Best to cache off your own versions of “Input.GetKey” or “Input.GetButton” to determine positive or negative edge presses.

Conclusion

I don’t have any perfect answers to this. This is a hard problem to fix, especially without full control over polling rates and missed data. I also still have a lot to learn about this, and am aware that I’ve only been testing on one machine, here.

But, as for half way fixes and best practices:

In theory, to maintain the smoothest frame rate for mouse inputs, your best bet is to cap your frame rate by setting Application.targetFrameRate close to, or less than the mouse polling frequency.This way you won’t sample faster than the mouse can possibly give input, and thus you’ll get smoother camera motion with respect to input (if your mouse is driving camera rotation directly in Updates/LateUpdate rather than physics).

It’s kinda sensible to do this for a number of reasons – there’s not a great amount of point in a game running faster than you can possibly give meaningful inputs to it – so the rate of a continuous input polling more or less defines what that is (but for obvious reasons, you do want that rate to be high enough to feel like a continuous feedback loop).

If your frame rate exceeds mouse polling, and your mouse is continuously driving anything in the game, you’re probably going to get something kinda jerky because of all these dead polling takes. That’s when smoothing is useful… ish. It can add a sense of momentum to your mouse movements which some people hate. Any interpolation you do for in between frames is, strictly speaking, misleading, and can take away that crisp clean feel that people love from their mice.

Remember that if you’re using physics collisions in any way, then Time.fixedDeltaTime is, in effect, your gameplay turn speed. Thus, if that is out of sync with input polling, you’re going to sometimes get more frames of gameplay than are affectable by your player. If your physics updates become so heavy that it slows gameplay enough miss polling checkpoints, you’re also missing subtle input data – potentially missing entire button presses if you don’t implement your own button check code.

Another technique is to decouple the visual elements which are affected by mouse from physics elements which are affected by the mouse. The visual element plays “catchup” in terms of positioning and rotation. This is quite the rabbit hole which I can’t go into detail on here (i.e. your visualization is not a “true” representation of your game state, but it’s at least a bit smoother).

We’re slightly in the hands of Unity on this one. Since inputs are so fundamental to realtime applications of all walks, I hope that Unity give an input system overhaul some priority, as it can have a negative subconscious effect on a player’s appreciation of a game. Plus, it’s rather ancient. For the most part, this level of accuracy is not going to be missed in a lot of games, but do remember that the highest quality games do put a premium on this stuff.

One thought on “Mouse Sampling in Unity3d

  1. Nice, very interesting, was having issues with unity>flash export so even deeper conciderations, but i’ll have a play with some of your suggestions and see if they make any difference within the unity/molehill environment!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>