The disclosure relates to a tracking system and a tracking method. More particularly, the disclosure relates to the tracking system able to track a hand movement effectively.
Virtual Reality (VR), Augmented Reality (AR), Substitutional Reality (SR), and/or Mixed Reality (MR) devices are developed to provide immersive experiences to users. When a user wearing a head-mounted display (HMD) device, the visions of the user will be covered by immersive contents shown on the head-mounted display device. The immersive contents show virtual backgrounds and some objects in an immersive scenario.
In some applications, the user may hold a handheld controller as an input device to interact with items in the immersive contents. In some other applications, a tracking system is configured to track hand movements and hand gestures based on a computer vision algorithm, such that users can manipulate directly by their hands. The computer vision algorithm for tracking these hand movements and hand gestures requires a lot of computation resources, and it may cause heavy power consumption.
The disclosure provides a tracking method, which includes following steps. A series of images is captured by a tracking camera. A first position of a trackable device is tracked within the images. An object is recognized around the first position in the images. In response to the object being recognized, a second position of the object is tracked in the images.
The disclosure provides a tracking system, which includes a tracking system, which includes a trackable device, a tracking camera and a processing unit. The tracking camera is configured to capture images. The processing unit is communicated with the tracking camera. The processing unit is configured to receive the images from the tracking camera. The processing unit is further configured to track a first position of a trackable device within the images. The processing unit is further configured to recognize an object around the first position in the images. In response to the object being recognized, the processing unit is further configured to track a second position of the object in the images.
The disclosure provides a non-transitory computer-readable storage medium, storing at least one instruction program executed by a processing unit to perform aforesaid tracking method.
It is to be understood that both the foregoing general description and the following detailed description are by examples, and are intended to provide further explanation of the invention as claimed.
The disclosure can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:
Reference will now be made in detail to the present embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
Reference is made to
It is noticed that, the trackable devices 140R and 140L in the disclosure are not limited to be worn on the wrists. In some other embodiments, the trackable devices 140R or 140L can be trackable rings worn on fingers of the user, or the trackable devices 140R or 140L can also be trackable groves worn on palms of the user.
It is also noticed that, the disclosure is not limited to include two trackable devices 140R and 140L worn on both hands. In some other embodiments, the tracking system 100 can include one trackable device worn on one hand of the user, for tracking movements or gesture on this particular hand.
The tracking camera 120 is configured to capture a series of images IMG within a field of view FOV of the tracking camera 120. The images IMG indicate movements of user's hand (and also the trackable devices 140R or 140L). The processing unit 160 is communicatively connected with the tracking camera 120, and the processing unit 160 is configured to receive the images IMG captured by the tracking camera 120 and perform a computer vision algorithm to track and recognize the movements of user's hands (and also the trackable devices 140R or 140L). Details about how to track and recognize the movements of user's hands will be discussed in following paragraphs.
As the embodiment shown in
In some embodiments, the processing unit 160 can be a processor, a graphic processing unit (GPU), an application specific integrated circuit (ASIC) or any similar processing components. In some embodiments, the processing unit 160 can be integrated with the head-mounted display (HMD) device 200. In other embodiments, the processing unit 160 can be disposed in a computer, a tablet computer, a server, a smartphone or any similar computing device.
Reference is further made to
Reference is further made to
As shown in
In some other embodiments, the feature pattern 142R can be a colored pattern printed on the surface of the trackable device 140R. For example, the feature pattern 142R can be a yellow lightening icon printed on a black outer case of the trackable devices 140R.
Similarly, the trackable device 140L also includes another feature pattern 142L on a surface of the trackable device 140L.
The feature patterns 142R/142L can include specific icons, drawings, figures or colors to achieve strong distinguishable features, which are optimized to be recognized by the computer vision algorithm.
As shown in
The tracking system 100 and the tracking method 300 will perform similar manners to track the trackable device 140R and the trackable device 140L. For brevity, in following embodiments, details operations for the trackable device 140R will be discussed for demonstration. The tracking system 100 and the tracking method 300 can adopt similar steps to the trackable device 140L, and details about the trackable device 140L is not discussed below.
As mentioned above, the trackable device 140R includes the feature pattern 142R. In step S321, the processing unit 160 is configured to search the image IMG1 for the feature pattern 142R based on the computer vision algorithm.
It is noticed that, the feature pattern 142R is searched over the whole image IMG1 in step S321. Compared with directly recognizing features of user's hands (e.g., recognizing fingers, palms, wrists, skin colors) from the image IMG1, recognizing feature pattern 142R is much easier task for the computer vision algorithm. Because, the feature pattern 142R can be designed with special shapes or the feature pattern 142R may emit optical signals with special tempos, such that the feature pattern 142R is easier to be recognized. However, the hand of the user may do different gestures (e.g., holding, pinching, pointing, grasping, typing, pressing or dropping gestures), such that shapes or contours of the hands appeared in the image IMG1 may change dramatically. To search the hand directly through the whole image IMG1 will be a harder task for the processing unit 160 and the computer vision algorithm.
In step S322, the processing unit determines whether the feature pattern 142R is found in the image IMG1 or not. If the feature pattern 142R is found, step S323 is executed, to track the first position P1R of the trackable device 140R according to the feature pattern 142R found in the image IMG1, as shown in
It is noticed that, in some embodiments, based on a direction and a contour of the feature pattern 142R found in the image IMG1, the tracking method 300 further detects an orientation O1R of the trackable device 140R in the space as shown in
On the other hand, if the feature pattern 142R is not detected in the image, step S324 is executed by the processing unit 160, to track the trackable device 142R according to pose data generated by an inertial measurement unit (IMU) embedded in the trackable device 142R. More details about step S324 will be discussed later.
Based on the first position P1R detected in the image IMG1, step S330 is executed by the processing unit, to recognize an object around the first position P1R in the image IMG1. In some embodiments, the object in this embodiment is one hand of the user. One function of the tracking system 100 and the tracking method 300 is to track/recognize a hard gesture of the user. The first position P1R of the trackable device 140R tracked in step S320 is helpful to decide a potential location of user's hand. Step S330 of the tracking method 300 in
In step S331, the processing unit 160 divides each of the images IMG into at least one object-related area adjacent to the first position(s) and a background area. Reference is further made to
As shown in
In step S332, the processing unit 160 is configured to search the object-related area OAR1 and the object-related area OAL1 in the image IMG1 for the object (e.g., the right hand and the left hand) based on the computer vision algorithm.
In this case, the computer vision algorithm is not performed in the background area BG of the image IMG1. Compared with searching the whole image IMG1 for the object (e.g., user's hand), searching regions in step S332 is narrowed down to the object-related area OAR1 and the object-related area OAL1, and it will increase efficiency in searching the object.
In embodiments shown in
Reference is further made to
As mentioned above, during step S323, the tracking system 300 is able to track the first position P1R and also the orientation O1R of the trackable device 140R according to the feature pattern 142R found in the image IMG1, and also track the first position P1R and also the orientation O1R of the trackable device 140R according to the feature pattern 142R found in the image IMG1. The orientation O1R of the trackable device 140R is able to indicate a direction of the right hand of the user. As shown in
As shown in
In step S332, the processing unit 160 is configured to search the object-related area OAR2 for the object (e.g., the right hand of user). Because the trackable device 140R is worn on the right wrist of the user and located at the first position P1R, according to the orientation O1R and the inverse kinematics (IK) algorithm, the object-related area OAR2 can be narrowed down to the left-top side of the first position P1R. As shown in
Similarly, in step S332, the processing unit 160 is configured to search the object-related area OAL2 for the object (e.g., the left hand of user). Because the trackable device 140L is worn on the left wrist of the user and located at the first position P1L, according to the orientation O1L and the inverse kinematics (IK) algorithm, the object-related area OAL2 can be narrowed down to the right-top side of the first position P1L.
In step S333, the processing unit determines whether the object is found in the object-related area (referring to OAR1/OAL1 in
As shown in
After the hand gesture is recognized according to the object OBJR, step S350 is executed by the processing unit 160, to perform related functions of the recognized hand gesture. For example, when the recognized hand gesture is pressing a virtual button in an immersive content shown by the head-mounted display device 200, the processing unit 160 can activate the function of the virtual button. For example, when the recognized hand gesture is typing a virtual keyboard in an immersive content shown by the head-mounted display device 200, the processing unit 160 can activate the typing function relative to the virtual keyboard.
On the other hand, when the object is not found in step S333, the tracking method 300 can returns to step S332 to search the object-related area again.
In step S330, the object OBJR has been recognized around the first position P1R in the image IMG1. Afterward, step S340 is executed by the processing unit 160 is track the second position P2R of the object OBJR in the image IMG1 as shown in
In an ideal case, the tracking system 100 and the tracking method 300 can keep tracking the second position P2R of the object OBJR in the series of images IMG captured by the tracking camera 120.
However, in some cases, the tracking system 100 and the tracking method 300 may lose track of the second position P2R of the object OBJR. For example, the object OBJR can blocked by other items (e.g., when the user place his/her hand under a table, and the hand may be blocked by a table from the tracking camera 120), or the object OBJR can be moved outside the field of view FOV of the tracking camera 120 (e.g., when users move to their hands behind their heads or place on their ears, their hands may be out of the field of view FOV shown in
As shown in
If the second position P2R of the object OBJR is currently not trackable, step S342 is executed by the processing unit 160, to determine whether the first position of the trackable device 140R is current available or not.
Reference is further made to
As shown in
Reference is further made to
As shown in
When the tracking method 300 loses track of the second position and the first position is unavailable, the tracking method 300 returns to S324, to track the trackable device according to pose data generated by an inertial measurement unit (IMU) 144R embedded in the trackable device 140R.
In some embodiments, as shown in
As shown in
Another embodiment of the disclosure includes a non-transitory computer-readable storage medium, which stores at least one instruction program executed by a processing unit 160 to perform control methods 300 as shown in
Based on aforesaid embodiments, the tracking system 100 searches the whole images for the trackable devices 140R/140L with strong feature patterns 142R/142L (easier to distinguish/recognize) to estimate the object-related areas. Then, the tracking system 100 search with the object-related areas (instead of the whole images) for recognizing the objects. Therefore, the tracking system 100 and the tracking method 300 are able to track a hand movement and a hand gesture effectively in a shorter computing time, with a higher accuracy and adaptable in various applications. The hand movement or the hand gesture recognized by the tracking system 100 can be utilized as an input signal to control an application program, to trigger a function, to turn on/off a switch, or to perform other following operations.
Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.
It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.
This application claims the priority benefit of U.S. Provisional Application Ser. No. 63/260,069, filed Aug. 9, 2021, which is herein incorporated by reference.
Number | Date | Country | |
---|---|---|---|
63260069 | Aug 2021 | US |