Patent Application
20150199033
This application claims the benefit of the filing date of Taiwan Patent Application No. 103101123, filed Jan. 13, 2014, entitled “A METHOD FOR SIMULATING A GRAPHICS TABLET BASED ON PEN SHADOW CUES,” and the contents of which is hereby incorporated by reference in its entirety.
The invention discloses a method for simulating a graphics tablet, more particularly, the invention relates to a method for simulating a graphics tablet based on pen shadow cues, the method applying to execution in a computer for simulating a graphics tablet
Conventional graphics tablets are a common input device for computers, which is usually composed of a dedicated pen and a tablet. Users draw with the pen in a specific region of the tablet to do writing, as well as emulating mouse functions. As for functions, take handwriting tablets produced by WACOM for example. When the pen (stylus) is moving above the tablet, the movement of the mouse cursor is controlled. When the pen tip touches the tablet, left click button of the mouse is signaled to the computer. In addition, there is a button attached to the pen (stylus), by pressing the button, the right click of the mouse is signaled. In addition to mouse function emulation, the pen (stylus) is equipped with pen tilt angle and pressure detection. The thickness and depth of strokes in handwriting can be rendered with corresponding software.
Besides the previously mentioned, the conventional graphics tablet is an electronic device. However, it is sophisticated in structure and complex in design, and thus usually has the following disadvantages: (1) Overweight: The conventional graphics tablet is made up of complex electronic circuits, with the expansion of the drawing areas and increase the functions, the weight will increase, making carrying a burden for users. (2) Increase in space: The conventional graphics tablet generally does not support folding functions. Therefore, increasing the drawing area causes the volume of the tablets body to be larger. As a result, a larger tablet requires more space when storing and carrying. (3) High cost: With the increasing performance and upgrades, the conventional graphics tablet constantly increases in price. Take WACOM's handwriting tablet for example, the price is above NTD 5,000. Models with larger sizes are above NTD 10,000. (4) Vulnerability to breaking: Since the conventional graphics tablet is an electronic device, there is always a risk that it may be damaged if bumped or pressed too hard, and must be handled with care. (5) Poor practicality: The actual sensing area of a handwriting tablet is much smaller than its body size, which means space is wasted.
According to the above description of a conventional graphics table, due to its internal electronic structure, it has the drawbacks including being overweight, taking too much space, high cost, vulnerability to breaking, and poor practicality, so improvements must be made in order to solve the disadvantages of the conventional graphics tablet.
The present invention provides a method for simulating a graphics tablet based on pen shadow cues, the method applying to execution in a computer for simulating a graphics tablet. The present invention uses computer vision technology to emulate the graphics tablet, capturing any quadrilateral planes and detecting the pen-like object within the FOV of a single webcam to be emulated both as a graphics tablet and a stylus pen (the pen used for the graphics tablet). The present invention can only use the single webcam to emulate the function of the graphics tablet, which can efficiently reduce the cost disadvantages conventional graphics tablets may have.
In order to achieve the above purposes, the invention provides a method for simulating a graphics tablet based on pen shadow cues, the method applying to execution in a computer for simulating a graphics tablet and comprising the steps of: capturing an image; identifying a tablet from the image according to a predetermined tablet recognition procedure; identifying a pen entering the tablet according to a predetermined pen recognition procedure; and detecting a pen shadow of the pen according to a predetermined pen shadow recognition procedure and determining the relationship of contact and separation between the pen and the pen shadow for simulating the graphics tablet, wherein the image is captured by a webcam.
The predetermined tablet recognition procedure used in the present invention comprises the following sub-steps of: turning the color space of RGB (Red/Green/Blue, RGB) into the color space of HSV (Hue/Saturation/Value, HSV), and performing a multi-valued segmentation; capturing a colored area of the tablet selected by a user; acquiring four points from the four corners of the pixel area, the pixel area being a quadrilateral shaped and the color thereof is the best approximation in the plurality of colored area of the tablet; confirming whether the quadrilateral area can be the tablet or not; and recording and storing the tablet information.
The predetermined pen recognition procedure used in the present invention comprises the following sub-steps of: clearing the area of the tablet, the shadow and the user's hand; capturing all of the recorded and stored colored area; finding the area in accordance with a characteristic of the pen; calculating a tilt angle, a pen tip coordinate, and a user's writing hand with a left hand mode or right hand mode for the pen; and recording and storing the pen information.
The predetermined pen shadow recognition procedure used in the present invention comprises the following sub-steps of: clearing the area of the tablet, the pen and the user's hand; capturing a pen tip surrounding the shadow of the pen; capturing a pen shadow of the pen; and determining the relationship of contact and separation between the pen and the pen shadow.
Compared to conventional techniques, the present invention provides a method for simulating a graphics tablet based on pen shadow cues, the method applying to execution in a computer for simulating a graphics tablet. The invention mainly uses the single webcam and detects three objects separately to emulate the function of the graphics tablet, the quadrilateral plane as the tablet, the pen-like object which comprises the shape with a long shaft as the pen, and the shadow of the pen-like object as the pen shadow. The present invention can detect the variations of the pen shadow, and the moving direction of the object at the same time, which can make it possible to detect if a pen has touched a tablet with a single camera to simulate an expensive graphics tablet with complex function. Due to the present invention's use of only a single webcam to emulate the function of the graphics tablet, the present invention can efficiently reduce the cost, and solve the disadvantages traditional graphics tablets may have, such as not being easy to carry, heavy, fragile, and any other disadvantage.
The advantage and spirit of the invention may be understood by the following recitations together with the appended drawings.
In order for the purpose, characteristics and advantages of the present invention to be more clearly and easily understood, the embodiments and appended drawings thereof are discussed in the following.
The present invention is concerning a method for simulating a graphics tablet based on pen shadow cues. More specifically, by using a single webcam to detect and recognize three objects and the method used to achieve the function. The webcam detects three objects separately, which is the quadrilateral plane as the tablet, the pen-like object which comprises the shape with a long shaft as the pen, and the shadow of the pen-like object as the pen shadow.
The present invention is applying to execution in a computer for simulating a graphics tablet, wherein the computer can be a personal computer, a notebook computer, a tablet computer, a smart handheld device or any other type of computer. Firstly, in the present embodiment, the step (S1) of the method of the invention is capturing an image, where the image is captured by the webcam. However, the present invention is not limited to the above method. In practical application, the webcam may also be equipped with a built-in digital camera (such as a notebook computer, PDA, etc.).
Firstly, in the sub-step (S21) is turning the color space of RGB (Red/Green/Blue, RGB) into the color space of HSV (Hue/Saturation/Value, HSV), and performing a multi-level segmentation. This is done because the HSV color space representation is more similar to human's color perception. Usually, the colors of the objects are similar, so we use multi-level segmentation to group areas of similar colors together to better identify objects in an image.
The sub-step (S22) includes capturing a colored area of the tablet selected by a user. When a user has selected an area in the image to represent the tablet, the pixel value processed by multi-level segmentation is recorded. Then, the invention searches for all neighboring pixels with the same value in the image, wherein the search direction is from left to right and from top to bottom. When a pixel with the same value is found, the invention detects if the left of the current pixel point p(x,y), the left pixel point p(x−1,y), and the top of the current pixel point p(x,y), the top pixel point p(x,y−1), both of them are classified. Because this approach is from left to right and from top to bottom, only the left pixel point and the top pixel point need to be detected. When the pixel point represents the color of the tablet, the pixel point will instantly be classified, so there aren't the pixel been missed for classify.
Through the method of grouping neighboring pixels of the tablet recognition procedure, all the pixels with the same color as the tablet will be found, and given the same label for its neighboring pixels, while the different colored areas will be given a different label. The results of the label are then saved into a list to boost performance of the following searches for the address of the object clicked on by the user.
P
upper-left
=p(x,y)
with min{x+y|pi(x,y)∈ Objectpixels, i=1,2, . . . , n}
P
upper-right
=p(x,y)
with max{x−y|pi(x,y)∈ Objectpixels, i=1,2, . . . , n}
P
lower-right
=p(x,y)
with max{x+y|pi(x,y)∈ Objectpixels, i=1,2, . . . , n}
P
lower-left
=p(x,y)
with min{x−y|pi(x,y)∈ Objectpixels, i=1,2, . . . , n}
Besides, the sub-step (S24) confirms whether the quadrilateral can be the tablet or not. After the four corner points of the quadrilateral are acquired, the sub-step further confirms if the selected quadrilateral meets the defined characteristics of the tablet. If it does, go to the next step; otherwise, return a message to user.
Finally, the sub-step (S25) records and stores the tablet information, saves the tablet address, coordinates of each pixel, and the original color information of the tablet or any other information of the tablet into memory to be used for pen detection in later steps.
Firstly, in sub-step (S31) for clearing the area of the tablet, the shadow and the user's hand, recording and storing the pen information (pixel addresses and their colors), and the frame for recognizing which way the user holds the pen in the FOV (field of view), the foreground can be obtained by computing the color difference under the HSV color space. Let Pnow(x,y) be the pixels of the frame which a user holds the pen into FOV, and Ptablet(x,y) be the pixels of frame which the tablet is detected. Compute the absolute value of the difference of pixels of these two frames to obtain the pixels Pdest(x,y) of the new frame and is shown as the following Eq.
P
dest(x,y)=|Pnow(x,y)−Ptablet(x,y)|
The tablet area is cleared by the above Eq. by computing the difference to obtain an image for clearing the tablet. Then, the shadows are removed by erasing the pixels with lower H values. Therefore, only the pen and the user's hand which holds the pen will remain. Furthermore, removing the areas with colors similar to the hand, so the remaining image will be the pen and the user's sleeve, and further performing a binarization to get the image cleared of the shadow and the user's hand. Finally, save the remaining areas into a list to be used for later use.
Moreover, the sub-step (S33) is for finding the area in accordance with a characteristic of the pen. After obtaining all the groups of pixels and finding the minimum quadrilateral area, an approximate quadrilateral area will be defined as the bounding box for the pen area. Let Areapixels be the set of pixels in the area, where Pi(x,y) is the pixel of Areapixels, i is the index of pixels in the area, (i=1,2, n), x and y are the coordinates in the image, and {circumflex over (p)}upper-left, {circumflex over (p)}upper-right, {circumflex over (p)}lower-right, and {circumflex over (p)}lower-left form the bounding box, are obtained by the following Eq.
{circumflex over (p)}
upper-left
=p(x, y)
with min{x|pi(x,y)∈ Areapixels, i=1,2, . . . , n}
and min{y|pi(x,y)∈ Areapixels, i=1,2, . . . , n}
{circumflex over (p)}
upper-right
=p(x, y)
with max{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and min{y|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
{circumflex over (p)}
lower-right
=p(x, y)
with max{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and max{y|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
{circumflex over (p)}
lower-left
=p(x y)
with min{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and max{y|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
The present invention adopted the method for the defining the minimum quadrilateral area which acts as the bounding box for the pen area by rotating all pixels in the area from 0 to 90 degree iteratively. In each of the iterations, the bounding-box is obtained from the above Eq. until the smallest bounding box is found. After the smallest bounding-box is obtained, it is rotated back to its original angle, then the complete smallest bounding-box which covers the pen is obtained.
In the method for computing the tilt angle of the pen, first use the two parameter arc tangent functions shown as the following Eq.
Then, using A as the center to rotate B 90 degrees is shown as the following Eq.
Then, computing the pen tilt angle, θtilt angle, between A and B is shown as the following Eq.
Finally, once all the groups of pixels are obtained, search through all the groups and find if any of them have the characteristics of a pen, and decide the location of the pen. If two or more groups correspond to the condition, choose the one closer to the bottom of the image. This is because the object below the pen is usually a tablet, and other objects might not exist.
Nib
x=(Bx−Ax)×0.5+Ax
Nib
y=(By−Ay)×0.5+Ay
Moreover, after obtaining the pen tip (Nib) 346, use it as the middle line and get the brightness from the left side and the right side of the frames processed by the difference computation. The brighter side is the side which the user holds the pen. When the user has changed the holding hand, the result is updated, and computing the brightness of the left side and the right side are shown as the following Eq.
Finally, in the sub-step (S35) for recording and storing the pen information, saving the information of the pen comprises an object shape with a long shaft, a height/width ratio greater than two, a tilt angle less than 80 degrees or any other characteristic of the pen used for pen shadow detection.
With both observations, the present invention provides a predetermined pen shadow recognition procedure to detect the relation between the pen and the tablet, as illustrated in
First, (S41) clearing the area of the tablet, the pen and the user's hand, by recording and storing the pen information (pixel addresses and their colors), and the frames which the user holds the pen into the FOV (field of view), a image for clearing the tablet can be acquired by computing the color difference under the HSV color space. Then, erasing the areas belongs to the pen according to the information from pen detection. The remaining area below the pen will be the shadow.
The purpose of the pen shadow 422 detection is for deciding whether the pen 340 has touched the tablet 420. Therefore, the only thing that needs to be known is if there is a pen shadow 422 close to the pen tip 346, and the shape of the shadow. Use Nib for pen tip 346 detection as the center, and compute the area within radius r, and get the x and y coordinates within the area as shown in the following Eq.
Because the shadow does not occlude the pen itself, the pen detected must be erased first. Assuming that the pixels in are equivalent to the pixels by the following Eq, where i has the same sequence of the sequences of pixels, and form the pixels into a pixel set Q. E is subtracted from Q to get p, where E is the pixels of a pen, as shown by the following Eq.
q
i
=p(xi, yi), i=1,2,3, . . . n
Q={q1, q2, . . . qn}
Q−E={p|p ∈ Q, p ∉ E}
Then, get the maximum brightness and minimum brightness of the set p, in order to adjust the contrast of the set. The pixels with brightness larger than 50% of the pixels are regarded as shadows. Set bi as the brightness of the pixels within the set p, i=1,2, . . . n, and B is the set of bi, as shown by the following Eq.
After obtaining the pen shadow area, use the method proposed above with the following Eq. to find the bounding-box of the pen shadow in order to get the boundaries and position of the area.
First, obtaining all groups of pixels, then finding the minimum quadrilateral area, this is an approximate quadrilateral area which acts as the bounding box for the pen area. Let Areapixels be the set of pixels in the area, pi(x, y) is the pixel of Areapixels, where i is the index of pixels in area, (i=1,2, . . . n), x and y are the coordinates in image, and the {circumflex over (p)}upper-left, {circumflex over (p)}upper-right, {circumflex over (p)}lower-right, and {circumflex over (p)}lower-left which forms the bounding box obtained by the following Eq.
{circumflex over (p)}
upper-left
=p(x, y)
with min{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and min{y|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
{circumflex over (p)}
upper-right
=p(x, y)
with max{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and min{y|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
{circumflex over (p)}
lower-right
=p(x, y)
with max{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and max{y|pi(x, y)∈Areapixels, i=1,2, . . . , n}
{circumflex over (p)}
lower-left
=p(x, y)
with min{x|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
and max{y|pi(x, y)∈ Areapixels, i=1,2, . . . , n}
The sub-step (S44) is for computing the Eq. of the distance 446 between the pen 340 and the pen shadow 422, wherein i is the index of frames, PB is a first quadrilateral area 440 of the bounding-box for the pen, SB is a second quadrilateral area 442 of the bounding-box for the pen shadow. The distance 446 D(i) is obtained by SBiTop which acts as the upper edge of the bounding-box for the pen shadow, and PBiBottom which acts as the bounding-box for the pen, and is shown by the following Eq.
D(i)=SBiTop−PBiBottom
First, set n as the index of the current frame, and D(n) as the distance between the pen and the pen shadow in the current frame. Then, calculating the difference between D (n) and D(n−1), D(n−1) and D(n−2), D(n−2) and D(n−3), and so on. Finally, computing the average of the differences, which is the average distance variation
After getting the average distance variation
Compared to conventional techniques, the present invention provides a method for simulating a graphics tablet based on pen shadow cues, the method applying to execution in a computer for simulating a graphics tablet, using computer vision technology to emulate the graphics tablet, capturing a quadrilateral plane within the FOV of a single webcam to emulate a tablet, and detecting the pen-like object which comprises the shape of a long shaft to be held by a user to emulate the stylus (the pen for graphics tablet). By detection the variations of the shadow of the object, the present invention can detect if overlapping objects in the FOV have touched each other and can detect the movement direction of the object. At the same time, the present invention can detect if a pen has touched a tablet with a single camera to simulate a graphics tablet of complex function and high cost. The present invention uses a single webcam to emulate the function of a graphics tablet, which can efficiently reduce costs, and solve the disadvantages traditional graphics tablet may have, such as not being easy to carry, heavy, fragile, and any many other disadvantages.
With the examples and explanations mentioned above, the features and spirits of the invention are hopefully well described. More importantly, the present invention is not limited to the embodiment described herein. Those skilled in the art will readily observe that numerous modifications and alterations of the device may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 103101123 | Jan 2014 | TW | national |
