This application claims the priority benefit of Taiwan Application Serial No. 110125406, filed on Jul. 9, 2021. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of the specification.
The present application relates to a gesture determining method and an electronic device for executing the gesture determining method.
A mobile device is usually controlled by touching a screen by hand or by voice. In addition to the above-mentioned two methods, more and more functions are implemented through gesture movement to control a mobile device, making it more convenient for a user to use the mobile device.
In existing methods, for motion-sensing gestures collected by sensing, each motion-sensing gesture sensed by a sensor is manually observed, and rules are written into a control component, to recognize the motion-sensing gestures according to the rules. However, when more motion-sensing gestures are to be added, the above-mentioned written rules may become excessively complex, resulting in a decrease in the accuracy of recognition according to the rules. Furthermore, in most cases, when a user turns around, stands up, sits down or performs another action to change a posture, the user may be not using a motion-sensing gesture. A highly sensitive sensor produces numerical changes based on such actions, leading to unnecessary recognition and operation.
According to the first aspect, a gesture determining method applied to an electronic device is provided. The gesture determining method includes: sensing a control gesture through at least one motion sensor, and correspondingly generating sensing data; sequentially segmenting the sensing data into a plurality of streaming windows according to a unit of time, each streaming window including a group of sensing values; determining whether a sensing value in a streaming window is greater than a critical value, and triggering subsequent gesture recognition when the sensing value is greater than the critical value; and performing a recognition operation on the streaming window by using a gesture recognition model to consecutively output a recognition result; and determining whether the recognition result meets an output condition, and outputting a predicted gesture corresponding to the recognition result when the recognition result meets the output condition.
According to the second aspect, an electronic device is provided. The electronic device senses a control gesture through at least one motion sensor and correspondingly generates sensing data. The electronic device includes a processor, signal-connected to a motion sensor and embedded with a gesture recognition model. The processor sequentially segments the sensing data into a plurality of streaming windows according to a unit of time, each streaming window including a group of sensing values; the processor determines whether a sensing value in a streaming window is greater than a critical value, and when the sensing value is greater than the critical value, performs a recognition operation on the streaming window by using the gesture recognition model to consecutively output a recognition result; and then the processor determines whether the recognition result meets an output condition, and outputs a predicted gesture corresponding to the recognition result when the recognition result meets the output condition.
In summary, the present application provides a highly accurate gesture determining method, and before a sensing value is used for a gesture recognition operation, it is first determined whether the sensing value is the beginning of a control gesture, so as to effectively avoid unnecessary operations, save system resources and energy, and provide a user with better gesture control.
For other functions of the present application and detailed content of embodiments, descriptions are provided below with reference to the accompanying drawings.
To describe the technical solutions in the embodiments of the present application or the related art more clearly, the following briefly describes the accompanying drawings required for describing the embodiments or the related art. Apparently, the accompanying drawings in the following description show only some embodiments of the present application, and a person of ordinary skill in the art may derive other drawings from the accompanying drawings without creative efforts. To describe the technical solutions of the embodiments of the present application or the existing technology more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments or the existing technology. Apparently, the accompanying drawings in the following description show only some embodiments recorded in the present application, and a person of ordinary skill in the art still derives other drawings from these accompanying drawings without creative efforts.
The positional relationship described in the following embodiments includes: up, down, left, and right, and unless otherwise specified, all are based on directions shown by components in drawings.
In the present application, a gesture recognition model using artificial intelligence (AI) is used, to output a predicted gesture determined by using the gesture recognition model according to a control gesture. The control gesture described herein is a gesture that drives an electronic device or a remote control joystick to rotate, flip, move or perform another action. A value read by a motion sensor on the electronic device or the remote control joystick is provided to the gesture recognition model for recognition.
In an embodiment, the electronic device 10 is a mobile electronic device such as a mobile phone, a personal digital assistant (PDA), a mobile multimedia player or any type of portable electronic product. The present application is not limited thereto.
In an embodiment, the gesture recognition model 18 is a convolutional neural network (CNN) model.
Based on the electronic device 10, the present application further provides a gesture determining method, applicable to the electronic device 10. Steps of the gesture determining method are described below in detail in conjunction with the electronic device 10.
Referring to both
As shown in step S14, according to a unit of time, resampled sensing data is sequentially segmented into a plurality of streaming windows 20 that overlap each other. Each streaming window 20 includes a group of sensing values (for example, readings of an X-axis, a Y-axis, and a Z-axis). Each streaming window 20 is a piece of data that is subsequently read by the gesture recognition model 18 for recognition and determining. Next, as shown in step S16, the processor 14 determines whether a sensing value in a streaming window 20 is greater than a critical value, and triggers subsequent gesture recognition when the sensing value is greater than the critical value (step S18). When the sensing value is not greater than the critical value, the subsequent gesture recognition is not triggered, and determining continues to be performed for a next streaming window 20. Step S16 is used to avoid unnecessary operations. That is, in most cases, even when a user does not use a control gesture, but simply turns around, stands up, sits down or performs another action to change a posture, the motion sensor 12 still generates a change in value. Therefore, the critical value is set. When a sensing value is greater than the critical value, it is determined that the sensing value is the beginning of the control gesture, and subsequent sensing values are all transmitted to the gesture recognition model 18 for recognition.
As shown in step S18, the processor 14 uses the gesture recognition model 18 to perform the recognition operation on the streaming window 20 to consecutively output a recognition result. A plurality of preset gestures is embedded in the gesture recognition model 18. Each recognition result includes each preset gesture and a probability value of the preset gesture. Therefore, the gesture recognition model 18 consecutively outputs probability values of all default gestures corresponding to each streaming window 20 according to the consecutive streaming windows 20.
As shown in step S20, the processor 14 determines whether the recognition result meets an output condition. The output condition is that the gesture recognition model 18 consecutively outputs at least two identical recognition results, and outputs a predicted gesture corresponding to the recognition result when the recognition result meets the output condition, as shown in step S22. When the recognition result does not meet the output condition, as shown in step S24, the preset gesture is not outputted. Because a control gesture has a plurality of consecutive streaming windows 20, the gesture recognition model 18 correspondingly generates the same quantity of recognition results. In a plurality of consecutive recognition results, a determining strategy is needed to determine whether to output the preset gesture. In an embodiment, the processor 14 uses a preset gesture corresponding to the highest probability value as a recognition result for determining whether the output condition is met. For example, when the preset gestures corresponding to the highest probability values in two consecutive recognition results are the same gesture, it means that the output condition is met. Therefore, the preset gesture is outputted as the predicted gesture, and the processor 14 executes a system operation corresponding to the predicted gesture. In contrast, when the preset gestures corresponding to the highest probability values in two consecutive recognition results are different gestures, it means that the output condition is not met, and the preset gesture is not outputted in this case.
In the present application, before the electronic device 10 uses the gesture recognition model 18 to perform gesture determining, the electronic device 10 first performs pre-training on the gesture recognition model 18. That is, a neural network is first trained with a large amount of training data to optimize all parameters in the gesture recognition model 18.
Referring to both
As shown in step S34, the processor 14 sequentially segments the gesture data 22 into a plurality of training windows that overlap each other. In an embodiment, for each piece of marked gesture data 22, a fixed-length sliding window is used to sequentially extract training windows that overlap each other. Each training window is used as one piece of training data 24. For example, there are M sampling points in the gesture data 22. The size of the sliding window is set to N sampling points. N is less than M. At least half of the gestures need to be covered. M−N+1 training windows may be taken by stepping through the M sampling points in the gesture data 22 one by one through the sliding window.
As shown in step S36, random sampling is performed on each training window to generate more training data 24. Because a quantity of training windows generated from the marked gesture data 22 is limited, to increase the amount of training, in the present application, step S36 is used to effectively increase the training data 24. Originally, M−N+1 training windows are generated from the gesture data 22. In the present application, any N sampling points selected from the M sampling points are used as a new training window to increase the training data 24. In an embodiment, as shown in
Referring to
In an embodiment, the gesture recognition model 18 adopts a structure of a CNN. Referring to both
In another embodiment, a structure of the gesture recognition model 18 in training is also shown in
In an embodiment, the electronic device 10 may be a notebook computer, a desktop computer, a mobile phone, a PDA, a mobile multimedia player, or any electronic product with a processor. The present application is not limited thereto.
Based on the embodiments of the electronic device 10 and the remote control joystick 40, the gesture determining method of the present application is also applicable to a combination of the electronic device 10 and the remote control joystick 40. Except that a user holds the remote control joystick 40 to perform the control gestures, the rest of the methods and actions are the same as those in the previous embodiments. For detailed steps and details, refer to the related descriptions of
In summary, the present application provides a highly accurate gesture determining method, and before a sensing value is used for a gesture recognition operation, it is first determined whether the sensing value is the beginning of a control gesture, so as to effectively avoid unnecessary operations, save system resources and energy, and provide a user with better gesture control.
The foregoing embodiments and/or implementations are merely preferred embodiments and/or implementations used for describing the technologies in the present application, and are not intended to limit implementation forms of the technologies in the present application. A person skilled in the art can make alterations or modifications to obtain other equivalent embodiments without departing from the scope of the technical solutions disclosed in the content of the present application. Such equivalent embodiments shall still be regarded as technologies or embodiments substantially the same as the present application.
Number | Date | Country | Kind |
---|---|---|---|
110125406 | Jul 2021 | TW | national |