Leapiano
Using Leap Motion to create a virtual 3D piano that can be played with real life hand gestures
Introduction
Leap Motion is a small device that allows developers to detect the user's real life hand gestures and allow detailed interactions between the user and 3D virtual space. The first thing that we decided to make was Leapiano. I play some piano for hobby and that allowed me to think of a 3D virtual piano that could be played using the user's real life hand gestures.
Concept
Due to the fact that I was only familiar with the basic skills of how to code Unity when I began making this program, I couldn't really create detailed advanced program with Leap Motion. So I had to kind of find a way to use only the basic skills to create the same advanced result.
The only three functions that I could use at that time were 'OnTriggerEnter', 'OnTriggerStay', and 'OnTriggerExit'. I had three event handlers. When an object makes contact with another object, while they are still touching, and when they finally detach from each other. I had to somehow create a fluent interaction between the piano pads and the Leap Motion hands without using any advanced methods.
The basic concept is simple :
When a part of the Leap Motion 3D hand makes contact with the one of the piano pads and triggers 'OnTriggerEnter' and 'OnTriggerStay', make the pad follow the movement of the finger. When the pad reaches certain y-axis or below, play the audio of the note. When it stops making contact and triggers 'OnTriggerExit', make the pad automatically move towards it's original place in a constant speed.
These are some detailed adjustments that I had to make to finalize the program :
First, create a maximum height for the pads. In real life, the piano pads are affected by gravity and they always stay attached to the survace of the piano. However, My leapiano doesn't have a surface. They are just floating piano pads. So in order to make them stay in one place unless touched by the user's hands, and also allow it to move down when it is pressed, I had to create the inital location of where it's going to be without any force applied, and that inital location will have to be the maximum height of the pad since we don't want it to start flying towards the sky.
1) InitalPosition
Initial Position is the position of the pad when no force is applied to it yet. In real life piano, the pads are affected by the gravity and automatically placed on the surface of the piano and stays in one place. However, in Leapiano, the there is no surface. The pads are floating in the air. If I were to just turn on the gravity, the pads would fall down to the infinity. That's why instead of using gravity, I made the a Vector variable that stores the initial position of the pad.
2) y-axis
When I first made it so that the pad follows the location of the hand when it touches the pad, I found that it moves in all direction to follow it. This was a problem beause piano pads don't move like that. They only move up and down. That's when I decided to only calculate the location using y-axis and also only allow the pad to move in the y-axis
3) Update()
Update is one of the main System functions that loops from the start of the program till it ends running. In this function in order to create a tension force like interaction, I made the pad to always try to move up in a constant speed intil it reaches the initial position. So whenever the pads are pressed down, they would go down with the Leap Motion hand, but when it is release it would rise up back to the inital position.
4) Transform
When I was moving the pads in Leapiano, I didn't really move them, but rather teleported them to certain locations in a very fast rate. Whenever the pads were moved from one place to the other, from the user's point of view it would look like it's moving fluently, but what I really did was just constantly changing the transform(location) of the object to match the destination.
C#
When I made this virtual Leap Motion piano, I did not really learn how to professionally handle Leap Motion. So I had to create the similar effect using only the basic skills that I learned through Unity tutorials.
http://unity3d.com/learn/tutorials/modules
These are som basic things that I learned through the tutorials
1)
public float yDistanceFromIntialPositionForAudio;
(In Unity C#, you use public variables to allow the developer to easily access and change the variables without actually going into full script editing mode. When you make variables public, they become visible in the setting menu of the Unity screen and you can change the values anytime you want.)
2)
void Start()
{
This is the function that runs when the code starts running.
You set and initialize variables here.
}
3)
void Update()
{
This is like run() function in Java.
This function is a loop that keeps running until the code ends.
}
4)
void OnTriggerEnter(Collider other)
{
This function is an event handler
This function runs when an object with <RigidBody> makes contact with the object this code is running on.
The Collider other is the object that made contact. You can access the object using this variable easily
}
5)
void OnTriggerStay(Collider other)
{
This function is an event handler loop
This function keeps running from the moment an object makes contact with host object until it no longer makes contact.
Collider other is used as the reference to the object that is making contact
}
6)
void OnTriggerExit(Collider other)
{
This function is an event handler
This function is called the moment the object leaves host object's body
}
Script
public class pianopad : MonoBehaviour {
Vector3 initialPosition;
bool handtouching = false;
float time;
Collider hand;
bool notePlayed = false;
public float yDistanceFromIntialPositionForAudio;
public float yDistanceFromHandPositionForPadPosition;
public float timeDelayAfterHandExits;
// Use this for initialization
void Start () {
initialPosition = transform.position;
}
// Update is called once per frame
void Update () {
if (!handtouching && Time.time > time + timeDelayAfterHandExits) {
if (transform.position.y < initialPosition.y) {
//while (transform.position.y < initialPosition.y){
if (hand) {
Debug.Log (checkHandObovePad (hand));
if (checkHandObovePad (hand)) {
Vector3 Vc = transform.position + Vector3.up;//try adding the y to the position to check the collision
if (hand.transform.position.y >= Vc.y) {
Debug.Log ("Up");
transform.position += Vector3.up;//if it is lower than the hand, do it.
} else {
Vector3 handVc = new Vector3 (transform.position.x, hand.transform.position.y - yDistanceFromHandPositionForPadPosition, transform.position.z);
transform.position = handVc;
}
} else {
Debug.Log ("Upfalse");
transform.position += Vector3.up;
}
} else {
Debug.Log ("Up");
transform.position += Vector3.up;
}
}
} else if (!hand) {
handtouching = false;
//time = Time.time;
Debug.Log ("noHand");
}
if (transform.position.y > initialPosition.y) {
transform.position = initialPosition;
}
if (transform.position.y > initialPosition.y - yDistanceFromIntialPositionForAudio) {
notePlayed = false;
}
audioPlay ();
}
void OnTriggerEnter (Collider other) {
//GetComponent<Rigidbody> ().velocity = GetComponent<Transform> ().forward * 10;
//handtouching = true;
//transform.position += Vector3.down/50;
//handtouching = true;
//transform.position += Vector3.down/50;
if (other) {
handtouching = true;
Vector3 otherVc = new Vector3 (transform.position.x, other.transform.position.y - yDistanceFromHandPositionForPadPosition, transform.position.z);
transform.position = otherVc;
}
audioPlay ();
hand = other;
Debug.Log ("OnTriggerEnter");
}
void OnTriggerStay (Collider other) {
//if (other.attachedRigidbody)
//other.attachedRigidbody.AddForce(Vector3.up * 10);
//GetComponent<Rigidbody> ().velocity = GetComponent<Transform> ().forward * 10;
//handtouching = true;
//transform.position += Vector3.down/50;
if (other) {
handtouching = true;
Vector3 otherVc = new Vector3 (transform.position.x, other.transform.position.y - yDistanceFromHandPositionForPadPosition, transform.position.z);
transform.position = otherVc;
//Debug.Log ("other");
}
//transform.position.y += Vector.down/5;
audioPlay ();
hand = other;
Debug.Log ("OnTriggerStay");
}
void OnTriggerExit (Collider other) {
//transform.position += Vector3.up/5;
handtouching = false;
time = Time.time;
hand = other;
Debug.Log ("OnTriggerExit");
}
bool checkHandObovePad (Collider theHand){
bool handAbovePad = true;
//Vector3 otherVcX = new Vector3 (transform.size.x,transform.position.y,transform.position.z);
if (theHand.transform.position.x > transform.position.x + transform.lossyScale.x) {
handAbovePad = false;
}
if (theHand.transform.position.x < transform.position.x - transform.lossyScale.x) {
handAbovePad = false;
}
if (theHand.transform.position.z > transform.position.z + transform.lossyScale.z) {
handAbovePad = false;
}
if (theHand.transform.position.z < transform.position.z - transform.lossyScale.z) {
handAbovePad = false;
}
return handAbovePad;
}
void audioPlay(){
if (!notePlayed) {
if (transform.position.y <= initialPosition.y - yDistanceFromIntialPositionForAudio) {
//GetComponent<AudioSource>().volume = GetComponent<Rigidbody>().velocity.y * GetComponent<AudioSource>().volume;
GetComponent<AudioSource>().Play ();
notePlayed = true;
Debug.Log ("Audio Played");
}
}
}
}
Problems&Discoveries
001
Unity 5
<Rigidbody>
While I was developing Leapiano, I found some problems in the Unity official site tutorials. It seems that the tutorials are not updated to match for the new Unity 5. There were some things that I had to guess and try and research seperately, in order to write the Unity 5 code.
One of the main thing that was different from the tutorials and the actual Unity 5 was 'GetComponent<Rigidbody>'. In the tutorials it says that I can access the 'rigidbody' component of the object by simply writing 'object.rigidbody'. However, in Unity 5 I found out that it had to be 'object.GetComponent<Rigidbody>' format.
There was one problem when I tried to use 'OnTriggerEnter', 'OnTriggerStay', and 'OnTriggerExit' functions. All three of them were able to get 'Collider other' as the perameter to access the object that is making the contact with the host object. However, when I tried to use these functions with Leap Motion, I didn't know which part of the hand that the Unity agnize to be the core of the object.
In a simple 3D box, the core of the object is at the centre of the box. However in an object that has a shape of a hand, where is the core of that object? Will the 'OnTriggerEnter' recognize and pass the hand object as the paremeter when the fingers touch an object, or will it ignore the fingers and only recognize the touch from the palm of the hand object?
I was very surprised and pleased when I found the solution to this problem. It seems that Leap Motion and Unity already solved this problem. The 'OnTriggerEnter' function was able to agnize the contacts between the fingers and the objects, and the center of the hand object was embeded into each part of the hand seperated by joints. Due to this convinient fact, I was able to use these three functions to create Leapiano without too much difficulties.
002
LeapMotion
OnTriggerEnter
Collider other
Hand object
003
Transform
Lag
The fact that I was not 'moving' the objects, but actually 'teleporting' certain locations, brought many problems. The major problem was lagging. The pads were programmed to move upwards when the hands stopped touching them and triggers 'OnTriggerExit'. However, due to the gap between each 'teleportation', if the hands were not touching the pad but were slightly over the pads, the pads would trigger the 'OnTriggerExit' function. The pads would then teleport to a certain position upwards and find itself touching the hands again, and teleport back downwards; and they would do this over and over again, and in the user's view, it would look like it's lagging.
I had to implement several methods and algorithms to minimize this problem.
1) Time Dilation
One method that I implemented was the Time Dilation. I made it so that the pad does not react right away when the hand stops making the contact, but waits for some time to make sure that the user intended the hand to actually stop making the contact. One thing I relized while playing with Leap Motion is that it is very precise, and because it is so precise the numbers that represent the location and the status of the hands change rapidly. There for even if the user may feel like their hands are stable in one place, the Leap Motion might think that it's still moving in a very small scale. This is why we need the time dilation. If the pads were to react everytime the Leap Motion says that the hands' position changed, it would look like it's lagging
2) Forshadowing
The second method I implied is what I call 'Forshadowing'. The problem with lagging was that when the hand is hovering slightly obove the pads, the pads basically try to go up without knowing that there is a hand. In forshadowing, the pads try going up once internally, and sees if that will meet with the hands and create the lagging problem or not. Because this is happening internally the user won't see it happening. If the pads check and find out that going up will cause the program to lag, it does not go up. It just stays in its position.
One function I made to help this method was checkHandObovePad(). This function checks if the hands' x-axis are the same as the specified pad. Why do I need this? When the hands stop making the contact with the certain pad, the pad dicides whether it should move upward or not using 'forshadowing'. However, when the pad does that, it looks at the y-axis of the hands to calculate the position of it and do the 'forshadowing'. Meaning that even if the hands moved to another pad, if they are still low enough to affect the 'forshadowing', the pad won't go back to it's original initial position as if the hands are still pressing it down. This is why I made checkHandObovePad() function. This function checks if the hands' x-axis is actually the same as the specified pad or not, and passes a boolean to the 'forshadowning'.
004
Sound
Now that I finished the look, I needed the sound. Since it was a 'piano' it needed to have sound of notes for each pads. I underestimated this problem at the beginning, but as I took my time to find the sounds, I realized that it was a mistake. It was very hard to find clear piano notes that were free. It took me a long time to find this suitable resource to finish the Leapiano.
http://www.freesound.org/people/pinkyfinger/packs/4409/
Possibilities
This project is just the beginning of expressing my interest towards VARS (Virtual Augmented Reality System). I would like to study more about the advanced functions that Leap Motion provides to developers and make many projects to encourage students in my school Inquiry Hub to develope many things using VARS (Virtual Augmented Reality System).
I want to study how to detect not only the location of the hands, but also the status of it. Like whether the user is trying to grab something, point at something, or making some kind of shape using the hands.
I also want to study more into Unity and actually learn how to use forces and real physics to interact with the 3D objects.
Presentation
After I finished making Leapiano I was able to show this to many organizations and field trips that came to our school.
Ken Spencer Award Presentation
On the day Inquiry Hub was awarded with the Ken Spencer Award, Many people came to IHub to watch the process. In the process many of our students presented their IDS's and the equipments that they got using the Ken Spencer Award money. I also presented my Face Recognition IDS and the Leap Motion with the Leapiano that I made using it.