ResilienceVR is a social VR app I’m developing to experiment and iterate outside of the boundaries of existing platforms.
In this article, I will explain how my first experiment works: Enabling the ability to make the avatar IK react to objects and players, despite the latency.
In social VR, users are connected together in a single virtual space, usually puppeteering a single virtual representation of themselves (an avatar).
There are currently two major ways that social VR applications transmit the virtual representation of the player to other players:
These approaches come with various drawbacks. In the latter approach, the end users would need to perform additional computations to deduce the other players’ poses.
An example of bone networking. Footage taken in Unity editor.
An example of bone networking. Footage taken in Unity editor.
Regardless, both of these approaches result in a large quantity of data that needs to be sent through the network, so additional simplifications and optimizations are needed for this to scale depending on the application’s needs.
Generally, in the worse case scenario when everyone’s avatars are in close virtual proximity in the virtual space, network usage in such applications can be estimated by the square of the number of players, as the data of any new player joining needs to be sent to all the other existing players in the instance.
Without optimizations:
This scaling issue is a main motivator for many optimization techniques, because networking non-static data can have a serious financial cost.
Users are still separated by latency, which in some cases can result in large delays between the players in the order of 200ms to even one full second.
These delays hamper the ability for the players to puppeteer their avatar in reaction to another player’s actions. Attempting to hold hands, dance together, or manipulate an object in the virtual space that is also being manipulated by another player, visually causes a mismatch due to the latency.
In order to provide a partial solution to this issue, ResilienceVR experiments with an approach that enables some degree of reactive interactions between players, but not all. Social VR has various contexts, and the one I’m particularly interested in is bonding between users. Provide a way for users to hold hands and dance together by putting hands on each other.
Holding another player. Footage taken in the VRChat Udon port.
Holding another player. Footage taken in the VRChat Udon port.
For our purposes, we’ll focus on something that is known outside of VR as “victim control”, a term that is used by stage combat in the field of acting. When acting a fight scene, the actor who is the aggressor is not actually in control of the victim. The victim preserves their full freedom of movement. This means if an actor acts grappling the victim actor’s head, the victim actor’s head can freely move around at the victim’s will, not the aggressor’s will.
In our case, the person in control of someone’s limbs is not the person who initiated the interaction.
Context is important, so this experiment does not focus on other social VR interactions where players are put in confrontation to each other, such as a hands-to-hands fighting game or gun game.
A combat-focused networking could bring additional challenges such as requiring the ability for a player to take control of another players’ limbs, as the players engaged in combat would be confrontational in this context. What would happen to your hand if someone else were to grab your elbow away? A “networked Boneworks” would be a drastically different problem to solve. This could be a subject for a future experiment in ResilienceVR.