The invention relates to a recognition system, and particularly to a gesture recognition system and a gesture recognition method.
For existing touch sensing panels, if a touch sensing panel is further integrated with a fingerprint sensor, it results in an increase in the overall panel thickness due to the sensing unit layer of the fingerprint sensor in addition to the touch panel and display panel of the touch sensing panel. In view of this, solutions of several embodiments will be provided below that may replace a touch sensing function of the touch panel.
The invention provides a gesture recognition system and a gesture recognition method, in which a gesture of a user can be recognized by comparing fingerprint images of the user.
According to an embodiment of the invention, a gesture recognition system of the invention includes a fingerprint sensor and a processing circuit. The processing circuit is coupled to the fingerprint sensor. The processing circuit is used to obtain a first fingerprint image and a second fingerprint image through the fingerprint sensor. The processing circuit obtains a plurality of first coordinate parameters of a plurality of first coordinate points in the first fingerprint image, and obtains a plurality of second coordinate parameters of a plurality of second coordinate points in the second fingerprint image. The first coordinate points of the first fingerprint image and the second coordinate points of the second fingerprint image respectively have a plurality of grayscale values corresponding to each other. The processing circuit determines a gesture operation according to the first coordinate parameters and the second coordinate parameters.
According to an embodiment of the invention, a gesture recognition method of the invention includes the following steps. A first fingerprint image and a second fingerprint image are obtained. A plurality of first coordinate parameters of a plurality of first coordinate points in the first fingerprint image are obtained, and a plurality of second coordinate parameters of a plurality of second coordinate points in the second fingerprint image are obtained. A gesture operation is determined according to the first coordinate parameters and the second coordinate parameters. The first coordinate points of the first fingerprint image and the second coordinate points of the second fingerprint image respectively have a plurality of grayscale values corresponding to each other.
Based on the foregoing, in the gesture recognition system and the gesture recognition method of the invention, the gesture of the user may be recognized according to the fingerprint position change in the digital fingerprint image of the user.
To make the aforementioned features and advantages of the invention more obvious and comprehensible, several embodiments accompanied with drawings are described in detail as follows.
Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and description to refer to the same or like parts.
In this embodiment, the microcontroller 132 may analyze a plurality of feature points in the first fingerprint image and the second fingerprint image. In an embodiment, the microcontroller 132 may select the feature points of a plurality of coordinate points having the same grayscale value or a plurality of grayscale values within a grayscale value range (e.g., grayscale values from 200 to 220) to perform the following analysis and determination, but the invention is not limited thereto. Therefore, in step S330, the microcontroller 132 may obtain a plurality of first coordinate parameters of a plurality of first coordinate points in the first fingerprint image. In step S340, the microcontroller 132 may obtain a plurality of second coordinate parameters of a plurality of second coordinate points in the second fingerprint image. In step S350, the microcontroller 132 may determine a gesture operation according to the first coordinate parameters and the second coordinate parameters. In this regard, since the first fingerprint image and the second fingerprint image are mapped to the same fingerprint, the microcontroller 132 of this embodiment may determine the gesture of the finger of the user on the fingerprint sensor 120 according to displacement change, angle change, or both displacement and angle change results of the coordinate parameters corresponding to the same feature point respectively in the first fingerprint image and the second fingerprint image.
It is worth noting that in an embodiment, the microcontroller 132 or the application module 140 may first determine whether the first fingerprint image passes a fingerprint verification. If the first fingerprint image does not pass the fingerprint verification, the microcontroller 132 or the application module 140 does not perform related operations. If the first fingerprint image passes the fingerprint verification, the microcontroller 132 or the application module 140 performs related operations. For example, the application module 140 may be used to perform screen unlocking or screen locking of the electronic device, for example. When the first fingerprint image passes the fingerprint verification, the application module 140 may further decide whether to perform screen unlocking or screen locking according to the determination of the gesture operation, and provide screen unlocking or screen locking operation with good security. Naturally, in the invention, it may also be determined whether the first fingerprint image and the second fingerprint image both pass the fingerprint verification to enhance the security confirmation level.
gi=m×hi Formula (1)
In step S410, the microcontroller 132 may calculate an angle parameter (θi) according to the first coordinate parameters and the second coordinate parameters. For example, the angle parameter (θi) may be obtained as shown by Formula (2) and Formula (3) below. The first coordinate parameters are (xi, yi), and the second coordinate parameters are (xi′, yi′). It is worth noting that, in an embodiment, the microcontroller 132 may be used for the following gesture operations according to the coordinate parameters corresponding to the same rotation angle having the maximum number of occurrences.
θi=tan−1((yi′−yi)/(xi′−xi)) Formula (2)
θ1≅θ2 . . . ≅θi Formula (3)
In step S420, the microcontroller 132 may output the angle parameter (θi) to the application module 140. The application module 140 may perform a rotation gesture operation according to the angle parameter (θi). In step S430, the application module 140 may determine whether a numerical range (range of exposure value) between the first fingerprint image 510 and the second fingerprint image 520 is less than or equal to a predetermined threshold, where the predetermined threshold may be, for example, 200 (digital value). In this regard, since the finger of the user is placed on the sensing region of the panel 110 corresponding to the fingerprint sensor 120 for in-situ rotation (single-point continuous pressing), the change between values (exposure value) output by the analog-to-digital converter corresponding to the first fingerprint image 510 and the second fingerprint image 520 is relatively small, which is in a range between 340 and 450, for example. In this regard, the difference between the average digital values respectively of the first fingerprint image 510 and the second fingerprint image 520 output by the analog-to-digital converter is 100, for example, and is thus less than the predetermined threshold of 200 (100<200). According to the embodiment of
Then, with reference to
In step S410, the microcontroller 132 may also calculate the angle parameter (θi) according to the first coordinate parameters and the second coordinate parameters as shown by the relational expressions of Formula (2) and Formula (3) above. In step S420, the microcontroller 132 may output the angle parameter (θi) to the application module 140. The application module 140 may perform the rotation gesture operation according to the angle parameter (θi). In step S430, the application module 140 may determine whether a numerical range between the first fingerprint image 610 and the second fingerprint image 620 is less than or equal to the predetermined threshold. In this regard, since the finger of the user is placed on the sensing region of the fingerprint sensor 120 for sliding rotation (during the process of which the finger is slightly departed from the fingerprint sensor), the change between values output by the analog-to-digital converter corresponding to the first fingerprint image 610 and the second fingerprint image 620 is relatively great, which is in a range between 300 and 1000, for example. In this regard, the difference between the average digital values respectively of the first fingerprint image 610 and the second fingerprint image 620 output by the analog-to-digital converter is 700, for example, and is thus greater than the predetermined threshold of 200 (700>200). In this regard, in step S450, the application module 140 may perform a sliding rotation gesture operation according to the angle parameter, for example, to rotate the display screen of the electronic device by 90 degrees. Therefore, with the process flows and the gesture recognition system 100 of this embodiment, the sliding rotation gesture of the user can be effectively recognized.
However, the in-situ rotation gesture operation and the sliding rotation gesture operation are not required to be limited to the determination of the numerical range (range of exposure value) of the fingerprint image performed in step S430. In other embodiments of the invention, depending on the current use scenario, the application module 140 only sequentially performs steps S410, S420, and S440 (without performing step S430) to realize the in-situ rotation gesture operation according to the angle parameter, or the application module 140 may only sequentially perform steps S410, S420, and S450 (without performing step S430) to realize the sliding rotation gesture operation according to the angle parameter.
In this embodiment, the microcontroller 132 may respectively select a plurality of coordinate points having a plurality of grayscale values within a grayscale value range in the first fingerprint image 810 and the second fingerprint image 820, to obtain, for example, a plurality of first coordinate parameters of a plurality of first coordinate points e1 to e4 and a plurality of second coordinate parameters of a plurality of second coordinate points f1 to f4 corresponding to the coordinate points of the grayscale values respectively in the first fingerprint image 810 and the second fingerprint image 820. It is worth noting that a plurality of first grayscale values and a plurality of second grayscale values corresponding to the coordinate points respectively in the first fingerprint image 810 and the second fingerprint image 820 are in a proportional relationship (e.g., satisfying Formula (1) above).
In step S710, the microcontroller 132 may calculate a displacement parameter (Δxi and/or Δyi) according to the first coordinate parameters and the second coordinate parameters. For example, the displacement parameter (Δxi and/or Δyi) may be obtained as shown by Formula (4) and Formula (5) and/or Formula (6) and Formula (7) below. The first coordinate parameters are (xi, yi), and the second coordinate parameters are (xi′, yi′). It is worth noting that, in an embodiment, the microcontroller 132 may be used for the following gesture operation according to the coordinate parameters corresponding to the same displacement having the maximum number of occurrences.
Δxi=(xi′−xi) Formula (4)
Δx1≅Δx2 . . . ≅Δxi Formula (5)
Δyi=(yi′−yi) Formula (6)
Δy1≅Δy2 . . . ≅Δyi Formula (7)
In step S720, the microcontroller 132 may output the displacement parameter (Δxi and/or Δyi) to the application module 140. In step S730, the application module 140 may determine a displacement distance and position according to the displacement parameter (Δxi and/or Δyi), and perform a movement gesture operation. For example, the application module 140 may determine a displacement distance of a display icon in a display interface, for example, the displacement distance in the display interface in the display interface in the X direction or the Y direction according to the displacement parameter (Δxi and/or Δyi), to move an icon position of the icon displayed in the display interface. Therefore, with the process flows and the gesture recognition system 100 of this embodiment, the movement gesture of the user can be effectively recognized.
A transparent material layer may be between the first light blocking layer 921 and the second light blocking layer 931. Next, a transparent material layer 940 is formed above the second light blocking layer 931, and microlens structures 951 and 952 are formed above the transparent material layer 940. The microlens structures 951 and 952 are corresponding to the thin film transistor image sensing units 911 and 912 in the light sensing direction to converge the sensing light to the thin film transistor image sensing units 911 and 912 through the openings 922, 923, 932, and 933. Besides, in this embodiment, the overall height of the structure 900 is less than 200 micrometers.
In summary of the foregoing, in the gesture recognition system and the gesture recognition method of the invention, the touch function, such as recognizing the in-situ rotation gesture, the sliding rotation gesture, and the movement gesture of the user, of the touch panel may be realized through the fingerprint sensor. In other words, in some implementation scenarios, the fingerprint sensor utilized in the gesture recognition system and the gesture recognition method of the invention may be used as substitute for the touch panel of the device itself, reducing the cost and volume of the device. Alternatively, in other implementation scenarios, in the gesture recognition system and the gesture recognition method of the invention, combined with the existing touch system, additional touch functions or touch means that may be additionally used for special conditions of use, for example, a touch gesture that needs to pass the fingerprint recognition being demanded to be obtained by the device, may be provided.
Finally, it should be noted that the above embodiments are merely used for illustrating, instead of limiting, the technical solutions of the invention. Although the invention has been described in detail with reference to the above embodiments, those of ordinary skill in the field should understand that they may still modify the technical solutions recited in the above embodiments or equivalently replace some or all of the technical features thereof. Nonetheless, the nature of the corresponding technical solutions with such modifications or replacements does not depart from the scope of the technical solutions of the embodiments of the invention.
This application is a 371 application of the International PCT application no. PCT/CN2020/124920, filed on Oct. 29, 2020, which claims the priority benefits of U.S. provisional application Ser. No. 62/953,590, filed on Dec. 26, 2019, U.S. provisional application Ser. No. 62/954,676, filed on Dec. 30, 2019, and U.S. provisional application Ser. No. 63/028,575, filed on May 22, 2020. The entirety of each of the above-mentioned patent applications is hereby incorporated by reference herein and made a part of this specification.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN2020/124920 | 10/29/2020 | WO |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2021/129134 | 7/1/2021 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
20160077620 | Choi | Mar 2016 | A1 |
20160180145 | Setterberg et al. | Jun 2016 | A1 |
20160364600 | Shah | Dec 2016 | A1 |
20160372088 | Dow et al. | Dec 2016 | A1 |
20180101715 | Lee et al. | Apr 2018 | A1 |
20180113548 | Lee et al. | Apr 2018 | A1 |
20200302146 | Bjerre | Sep 2020 | A1 |
Number | Date | Country |
---|---|---|
104331656 | Feb 2015 | CN |
105528592 | Apr 2016 | CN |
105630325 | Jun 2016 | CN |
107122760 | Sep 2017 | CN |
108182046 | Jun 2018 | CN |
108962159 | Dec 2018 | CN |
201624348 | Jul 2016 | TW |
I620099 | Apr 2018 | TW |
201820206 | Jun 2018 | TW |
Entry |
---|
“International Search Report (Form PCT/ISA/210) of PCT/CN2020/124920,” dated Jan. 27, 2021, with English translation thereof, pp. 1-4. |
Number | Date | Country | |
---|---|---|---|
20220343688 A1 | Oct 2022 | US |
Number | Date | Country | |
---|---|---|---|
63028575 | May 2020 | US | |
62954676 | Dec 2019 | US | |
62953590 | Dec 2019 | US |