Creating a pointer-friendly submenu experience

By Reid Barber

We are excited to announce a beta release of submenus in the latest release of React Spectrum, React Aria, and React Aria Components! In the process of adding this feature, we found ourselves solving some unique challenges while working to make submenus user-friendly and accessible across an array of devices and input types. In doing so, we wanted to share our thought process in solving one of the challenges we faced along the way.

The Shortcut Problem#

Submenus (or nested menus) enable multi-level exploration of menus, and even with a large number of options, users should be able to quickly find their desired option. A user should be able to hover over an item to see its submenu. Then, they should be able to move their pointer to an item in the newly opened submenu. While doing so, the pointer may leave the original item entirely and hover an unrelated item in its path towards the submenu, causing the submenu to close. We need a way to know when they’re moving their pointer to that submenu, so we can keep it open until they reach the submenu.

Predicting User Intent#

We can predict the user’s intent by tracking:

  • Pointer movement direction
  • Pointer movement speed

We can do this by listening for pointermove events and analyzing the changes in the pointer’s position (also called delta).

Valid Movements#

Now let’s imagine two lines: one from the pointer to the top of the submenu, and one from the pointer to the bottom of the submenu. We now essentially have a “range” of where the user might move their pointer on its path to the submenu. Any movement outside of this range should be considered invalid, and the submenu should close if the pointer moves outside of the original menu option.

More ActionsOption 5Option 4Option 3Option 2Option 1Submenu Option 3Submenu Option 4Submenu Option 5Submenu Option 2Submenu Option 1

Arctangent#

We can measure the angle at which the pointer moved by using the 2-argument arctangent, or atan2. If we provide our pointer’s delta x and delta y values as arguments, we’ll get the angle (in radians) from the previous pointer position to the current pointer position.

Θ = atan2(y,x)(x,y)

Now we can use the atan2 function to measure the angles formed by three separate lines:

  • Θtop : Angle formed by the line from the previous pointer position to the top inside corner of the submenu
  • Θbottom : Angle formed by the line from the previous pointer position to the bottom inside corner of the submenu
  • Θpointer : Angle formed by the line from the previous pointer position to the current pointer position (delta)
ΘtopΘbottomΘpointer

If the pointer’s delta angle is between the top and bottom angles, we know the user is moving their pointer in the direction of the submenu.

Θtop > Θpointer > Θbottom

Time#

We also want to consider the pointer’s movement over time when predicting the user’s intent. A user typically increases the speed of their pointer when moving towards the submenu, then decreases it as they reach their target. The user might change their mind on the way, however, and slow down to a stop or keep browsing other options in the parent menu. If their pointer speed dips below a certain threshold before arriving at the submenu, we could assume they're no longer intending to go to the submenu. We could check the pointer’s speed throughout its journey with our knowledge of what speeds to expect at the different stages along the way. We can even take it a step further by using our knowledge of the Accot–Zhai steering law to predict the time it may take the user to reach the target based on the width of the “tunnel” to the submenu. Alternatively, we can go with a simpler solution and implement a timeout. If the pointer hasn’t moved after a certain amount of time, we can assume they are no longer intending to go to the submenu. Since users with motor impairments may take more time to move their pointer, we should lean towards using a larger timeout value. Although our timeout solution is more simple than incorporating the pointer’s speed, we found that it still results in a good user experience.

Fault Tolerance#

Since our users aren’t perfect, we want to build in some fault tolerance in case their actions are outside the scope we have defined so far, but not by enough to invalidate their intent. Here are some ways we can do that:

  • Widen our range of allowed angles: Add radians to our top and bottom angles.
  • Allow N invalid movements: Require that the user make two (or some larger constant) consecutive invalid pointer movements before closing the submenu. Users who experience tremors may involuntarily move their pointer in other directions, so it is important to include this feature.

Optimizations#

We can introduce a few performance optimizations:

  • Throttle: Most devices refresh their screens 60 times per second. This means that we don't need to do these measurements more frequently than every 16 ms (1 second / 60 = 16.66 ms). We can keep timestamps and skip checks if enough time hasn't passed. We can also experiment with doing checks even less frequently while still maintaining a good user experience. Lowering the sample rate would also provide more accurate predictions for users who experience tremors.
  • Avoid unnecessary work: We don’t need to track pointer movement if a submenu isn’t open, and we can stop tracking pointer movement once the pointer reaches the submenu. We can also check the pointer’s delta x before calculating any angle values. If the pointer is moving in the opposite direction of the submenu, there's no need to calculate and compare angles. We can also check our pointer event's pointerType property and ignore the event if it is type 'pen' or 'touch'.

Alternatives#

Let's compare our approach to a few other methods:

  • Timeout only: We could use just a timeout instead of incorporating the direction the pointer moves. The downside of this is that moving the pointer vertically between parent menu items would cause delayed interactions that could be unpleasant or unexpected for the user.
  • Delay before opening submenus: We could introduce a delay before opening each submenu, but that would introduce a similar negative user experience as the timeout-only method described above.
  • Checking if the point is within a triangle: We could define a triangle from the previous pointer position to the top inside corner of the submenu and the bottom inside of the submenu. This method is outlined by Ben Kamens in Breaking down Amazon’s mega dropdown.
  • Rendering an invisible triangle over the parent menu: We could use an absolute-positioned HTML element and use clip-path to give it a triangle shape. There is an excellent post by Andreas Eldh called Invisible Details that walks through how to implement this. Similarly, we could draw an SVG instead, as outlined by Costa Alexoglou in Better Context Menus With Safe Triangles. This method is also mentioned in Building like it's 1984: A comprehensive guide to creating intuitive context menus by Michael Villar.

Conclusion#

We hope this post has been helpful in understanding how we approached the problem of predicting user intent when using submenus. We are excited to see how you use submenus in your own projects. You can see this feature in action in the React Spectrum Menu and React Aria Menu.