Patent Application
20250211661
The present disclosure relates to foldable electronic devices, and, more particularly, to processing sensor data in foldable electronic devices.
Foldable electronic devices typically include a top lid and a bottom lid. Two examples of foldable electronic devices are laptop computers and foldable smartphones. In the example of foldable smartphones, a screen may be implemented across both the top lid and the bottom lid. A camera or other types of image sensors may be implemented in the top lid. In a laptop computer, a keyboard is typically implemented in the bottom lid. A display is typically mounted in the top lid.
In some cases, sensors may be implemented in both the bottom lid and the top lid of foldable electronic devices. However, there may be various challenges associated with processing signals from the multiple sensors. It would be beneficial to improve the processing of signals in foldable electronic devices.
All of the subject matter discussed in the Background section is not necessarily prior art and should not be assumed to be prior art merely as a result of its discussion in the Background section. Along these lines, any recognition of problems in the prior art discussed in the Background section or associated with such subject matter should not be treated as prior art unless expressly stated to be prior art. Instead, the discussion of any subject matter in the Background section should be treated as part of the inventor's approach to the particular problem, which, in and of itself, may also be inventive.
Embodiments of the present disclosure provide improved utilization of sensor data from multiple inertial measurement units (IMUs) in a foldable electronic device. A foldable electronic device includes a first lid and a second lid rotatably coupled together via a hinge. A first IMU is implemented in the first lid and generates first sensor data. A second IMU is implemented in the second lid and generates second sensor data. As a user utilizes the foldable electronic device, the angle of rotation between the first lid and the second lid may vary. Embodiments of the present disclosure detect the angle of rotation between the first and second lids and then apply a rotation matrix to the second sensor data to generate rotated second sensor data. Embodiments of the present disclosure combine the first sensor data with the rotated second sensor data in order to generate combined sensor data. The combined sensor data has a greater accuracy than either of the first or second sensor data.
In one embodiment, a method includes generating first sensor data with a first inertial measurement unit in a first lid of a foldable electronic device, generating second sensor data with a second inertial measurement unit in a second lid of the foldable electronic device, and detecting a rotation angle between the first lid and the second lid based on the first and second sensor data. The method includes generating rotated second sensor data by adjusting the second sensor data based on the rotation angle and generating combined sensor data by combining the first sensor data and the rotated second sensor data.
In one embodiment, an electronic device includes a first lid including a first inertial measurement unit configured to generate first inertial sensor data and a second lid including a second inertial measurement unit configured to generate second inertial sensor data. The electronic device includes a sensor processing unit configured to detect a rotation angle between the first lid and the second lid based on the first sensor data and the second sensor data, generate rotated second sensor data by adjusting the second sensor data based on the rotation angle, and generate combined sensor data by combining the first sensor data and the rotated second sensor data.
In one embodiment, a foldable smartphone includes a first lid including a first inertial measurement unit configured to generate first inertial sensor data, a hinge coupled to the first lid, and a second lid rotatably coupled to the first lid by the hinge and including a second inertial measurement unit configured to generate second inertial sensor data. The foldable smartphone includes a sensor processing unit including a lid angle module configured to detect an angle of rotation between the first and second lids based on the first and second sensor data, a rotation module configured to generate rotated second sensor data by adjusting the second sensor data based on the angle of rotation, and a combiner module configured to generate combined sensor data from the first sensor data and the rotated second sensor data.
In the following description, certain specific details are set forth in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known systems, components, and circuitry associated with integrated circuits have not been shown or described in detail, to avoid unnecessarily obscuring descriptions of the embodiments.
Unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open, inclusive sense, that is as “including, but not limited to.” Further, the terms “first,” “second,” and similar indicators of sequence are to be construed as interchangeable unless the context clearly dictates otherwise.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its broadest sense, that is as meaning “and/or” unless the content clearly dictates otherwise.
The first lid 102 and the second lid 104 of the foldable electronic device 100 enable folding and unfolding of the electronic device 100. More particularly, the top lid 102 and the bottom lid 104 are coupled to a hinge 113. The hinge 113 enables the top lid 102 and the bottom lid 104 to be rotated relative to each other. When the rotation angle between the top lid 102 and the bottom lid 104 is 0°, the first lid 102 and the second lid 104 are pressed together. As the rotation angle increases from 0°, the first lid 102 and the second lid 104 are rotated away from each other.
Though not shown in
The foldable electronic device 100 may include various software applications that utilize inertial sensor data from the IMUs 106. Examples of such software applications can include map applications that track the orientation or traveling direction of the foldable electronic device 100, games that track the orientation or motion of the foldable electronic device 100, fall detection applications, and many other types of applications. The accuracy of such applications depends, in part, on the accuracy of the sensor data from the IMUs 106.
The IMUs 106 each include one or more inertial sensors. In one embodiment, each IMU 106 includes a three-axis accelerometer and a three-axis gyroscope. The accelerometer senses acceleration in each of the three axes and generates sensor signals indicative of accelerations on each axis. The gyroscope senses angular velocity around each of the three axes and generates sensor signals indicative of the angular velocities on each axis.
Each IMU 106 can include digital processing circuitry. In particular, the inertial sensors may initially generate analog sensor signals indicative of the accelerations or angular velocities. The digital processing circuitry of each IMU receives the analog sensor signals and processes the analog sensor signals to generate digital sensor data. Accordingly, each IMU 106 can include one or more analog-to-digital converters (ADC) to convert the analog signals to digital signals. Each IMU 106 can include additional signal processing circuitry to process or otherwise condition the sensor data for further analysis, as will be described in more detail below. In one embodiment, each IMU 106 includes one or more inertial sensors and an application specific integrated circuit (ASIC). The inertial sensors and the ASIC may be on separate integrated circuit dies. Alternatively, the inertial sensors and the ASIC may be implemented on a single system on chip integrated circuit die. Each IMU 106 can include additional components or circuitry without departing from the scope of the present disclosure.
The foldable electronic device 100 includes a sensor processing unit 108. The sensor processing unit 108 is coupled to the first IMU 106a and the second IMU 106b. The sensor processing unit 108 receives first sensor data from the IMU 106a and second sensor data from the IMU 106b. Each set of sensor data can include accelerometer and gyroscope data as described above. Although the sensor processing unit 108 is shown as separate from the IMU 106a and the IMU 106b, in practice, the sensor processing unit 108 may be part of either the first IMU 106a or the second IMU 106b.
The sensor processing unit 108 receives the first and second sensor data from the IMUs 106a and 106b and generates combined sensor data. The combined sensor data is more accurate than either the first sensor data or the second sensor data. The sensor processing unit 108 can then provide the combined sensor data to other components or applications of the foldable electronic device 100. The other applications or components can then utilize the combined sensor data to more effectively or accurately perform their functions.
One possible solution to generate the combined sensor data is to perform a simple averaging of the first sensor data and the second sensor data. If s1 represents the first sensor data, s2 represents the second sensor data, and sc represents the combined sensor data, then the combined sensor data sc can be generated by performing a simple averaging of the first and second sensor data in the following manner:
If the noise level of the first sensor data s1, measured by the standard deviation σ1, is approximately the same as the noise level of the second sensor data s2, measured by the standard deviation σ2, then the noise level of the combined sensor data will be less than the noise level of both the first and second sensor data, as set forth in the following relationship:
However, if the noise levels are not approximately the same, then the noise level of the combined sensor data may be worse than the noise level of either the first or second sensor data alone. At the same time, statistically, using the combined sensor data provides more accurate data than just using the first or the second sensor data alone.
Another possible solution to generate the combined sensor data is to perform a weighted average according to the maximal ratio combining technique. In this case, a weighting value σ22 based on the noise level of the second sensor data is applied to the first sensor data s1 and a weighting value σ12 based on the noise level of the first sensor data is applied to the second sensor data s2. The combined sensor data sc is generated as a weighted average in the following manner:
Generating the combined sensor data with a weighted average can provide improved accuracy (less noise) than both the first sensor data and the second sensor data even when the first and second sensor data have differing noise levels. The overall noise of the combined sensor data from the weighted average is given by the following relationship:
However, the drawback of the weighted average is that the noise level of each IMU 106 may need to be known beforehand. Accordingly, it is beneficial to ascertain the noise values in the factory.
The sensor processing unit 108 of
In one embodiment, it is beneficial for the first IMU 106a and the second IMU 106b to be relatively close to the hinge 113. Furthermore, it is beneficial for the first IMU 106a and the second IMU 106b to be at a same distance from the hinge 113. Accordingly, in one embodiment, the first IMU 106a and the second IMU 106b are at a same distance from the hinge 113. This can help increase the accuracy of the combined sensor data.
The sensor processing unit 108 includes a lid angle module 120. Lid angle module 120 receives the first sensor data from the IMU 106a and the second sensor data from the IMU 106b. The lid angle module 120 calculates the rotation angle ϕ based on the first sensor data and the second sensor data. The rotation angle ϕ corresponds to the angle between the first lid 102 and the second lid 104. The lid angle module 120 also calculates a first accuracy value based on the first sensor data and the second sensor data. The lid angle module 120 can include circuitry, memory, software instructions, and other resources for calculating, measuring, or looking up the rotation angle ϕ and calculating the first accuracy based on the first sensor data and the second sensor data.
The sensor processing unit 108 includes a rotation module 122. The rotation module 120 is coupled to the lid angle module 120. The rotation module 120 receives the second sensor data from the IMU 106b and the rotation angle ϕ from the lid angle module 120. The rotation module 122 generates rotated second sensor data from the second sensor data by adjusting the second sensor data based on the rotation angle q.
In one embodiment, the rotation module 122 generates the rotated second sensor data by aligning the second sensor data with the first sensor data. Accordingly, the rotation module 122 can apply a rotation to each axis of the second sensor data based on the rotation angle ϕ in order to generate the rotated second sensor data.
In one embodiment, the rotation module 122 generates the rotated second sensor data by applying a rotation matrix to the second sensor data. More particularly, the rotation module 122 multiplies the second sensor data by a rotation matrix. The rotation module 122 generates a rotation matrix R based on the rotation angle ϕ. More particularly, the rotation module 122 identifies an angle θ=π−ϕ (or θ=180°−ϕ) and generates the rotation matrix R based on the angle θ. In one embodiment, the rotation matrix R is given as follows:
In one embodiment, applying the rotation matrix R to the second sensor data corresponds to multiplying each sample of the sensor data by the rotation matrix R. In an example in which the second sensor data includes the three axes of accelerometer data and three axes of gyroscope data, each sample of the second sensor data will include a vector including an acceleration value for each of the three axes of the accelerometer of the second IMU 106b. Each sample of the second sensor data will also include a vector including an angular velocity value for each of the three axes of the gyroscope of the second IMU 106b. Both the acceleration vector and the angular velocity vector are multiplied by the rotation matrix R in order to generate the rotated second sensor data. Accordingly, each sample of the rotated second sensor data includes a rotated accelerometer vector and a rotated gyroscope vector.
As described previously, applying a rotation matrix R to the second sensor data corresponds to aligning the axes of the second IMU 106b with the axes of the first IMU 106a. The variances utilized for the maximal ratio combining technique, indicative of the noise level of each sensor axis, are the variances of the rotated IMU, which are computed at runtime for every processed sample as follows:
where σx
The combiner module 126 receives the rotated second sensor data and the first sensor data. The combiner module 126 generates combined sensor data by combining the first sensor data with the rotated second sensor data. The combiner module 126 can generate the combined sensor data based on a simple average of the first sensor data and the rotated second sensor data as described previously, a weighted average (according to the maximal ratio combining technique) of the first sensor data and the rotated second sensor data as described previously, or another suitable manner. The combiner module 126 can include processing resources, memory resources, software resources to generate the combined sensor data.
In one embodiment, the combiner module 126 generates a second accuracy value. The second accuracy value corresponds to an overall accuracy of the combined sensor data. The second accuracy value can be generated based on the first accuracy value describing the accuracy of the angle of rotation. In one embodiment, the combiner module 126 receives the first accuracy value from the lid angle module 120. The combiner module 126 compares the first accuracy value to a threshold value. If the first accuracy value is lower than the threshold value, then the combiner module 126 does not combine the first sensor data with the rotated second sensor data. Instead, the combiner module 126 may output the first sensor data and/or the rotated second sensor data rather than combined sensor data. Accordingly, if the first accuracy value is lower than the threshold value, the combiner module 126 stops generating combined sensor data. If the first accuracy value is greater than or equal to the threshold value, then the combiner module 126 generates the combined sensor data as described above.
In one embodiment, it is beneficial to implement sensitivity and offset calibration for both accelerometer and gyroscope of each IMU 106. Accelerometer sensitivity and offset calibration can be executed at runtime via 6-points tumble calibration or 9-points calibration, or the like. Gyroscope sensitivity calibration can be implemented in factory. Gyroscope offset calibration can be implemented at runtime (e.g., when the device is detected being steady, the gyroscope offset is computed). Sensor synchronization plays a key role while moving the device in order to minimize the difference in the sampling time. Two IMUs can be synchronized providing an external synchronization signal as input (e.g., STMicroelectronics IMUs ODR-triggered mode or Data Enable mode). Sensor placement plays a role while moving the device in order to minimize the difference in the sensed physical quantity. Accordingly, it is beneficial to place the sensors as close to each other as possible and symmetrical with respect to the hinge 113 of the device.
In one embodiment, the sensor processing unit 108 includes an angle rate detector 124. The angular rate detector 124 receives the rotation angle ϕ from the lid angle module 120. The angular rate detector 124 detects the rate of change of the rotation angle ϕ and generates an angle change value σ indicating the rate of change of the rotation angle ϕ.
Knowing the rate of change of the rotation angle ϕ can be beneficial to the combiner module 126. This is because if the angle ϕ is changing rapidly, then the accuracy of the combined acceleration data may be compromised. This occurs for example, when a user is rapidly opening or closing the foldable electronic device. In the moment of this opening or closing, the combined sensor data may not be accurate. The angle change value can be calculated based on a standard deviation, a variance, or an approximation of either of these quantities.
In one embodiment, the combiner module 126 receives the angle change value σ from the angle rate detector 124. The combiner module 126 compares the angle change value σ to a threshold value. If the angle change value σ is greater than or equal to the threshold value, then the combiner module 126 does not combine the first sensor data with the rotated second sensor data. Instead, the combiner module 126 may output the first sensor data and/or the rotated second sensor data rather than combined sensor data. Accordingly, if the angle change value σ is greater than or equal to the threshold value, the combiner module 126 stops generating combined sensor data. If the angle change value σ is less than the threshold value, then the combiner module 126 generates the combined sensor data as described above.
The foldable electronic device 100 includes a plurality of applications 130. The applications 130 may correspond to software applications that utilize processing, memory, and the software resources of the foldable electronic device 100 and that receive the combined sensor data from the sensor processing unit 108. Alternatively, the applications 130 can include hardware circuits that receive the combined sensor data from the sensor processing unit 108 and perform a function or generate data based on the combined sensor data. Examples of applications 130 can include games, map applications, orientation applications, fall sensors, collision sensors, driving monitoring applications, or other types of applications. Because the applications 130 utilize the combined sensor data based on the rotation angle ϕ, the applications 130 can operate more effectively because the combined sensor data is more accurate.
The foldable smartphone 400 includes a hinge 113. The first lid 102 and the second lid 104 can rotate relative to each other around the hinge 113. In the view of
The foldable smartphone 400 includes a camera 112 implemented in the first lid 102. The camera 112 can include a plurality of image sensors including visible light sensors, infrared light sensors, ultraviolet light sensors, or other types of sensors.
The laptop computer 500 includes a hinge 113. The first lid 102 and the second lid 104 can rotate relative to each other around the hinge 113. As can be seen in
The laptop computer 500 includes a camera 112 implemented in the first lid 102. The camera 112 can include a plurality of image sensors including visible light sensors, infrared light sensors, ultraviolet light sensors, or other types of sensors.
In one embodiment, a method includes generating first sensor data with a first inertial measurement unit in a first lid of a foldable electronic device, generating second sensor data with a second inertial measurement unit in a second lid of the foldable electronic device, and detecting a rotation angle between the first lid and the second lid based on the first and second sensor data. The method includes generating rotated second sensor data by adjusting the second sensor data based on the rotation angle and generating combined sensor data by combining the first sensor data and the rotated second sensor data.
In one embodiment, the adjusting the second sensor data includes applying to the second sensor data a rotation matrix based on the rotation angle.
In one embodiment, the method includes generating a first accuracy value based on the first and second sensor data and generating the combined sensor data based on the first accuracy value.
In one embodiment, the method includes generating an angle change value indicating how fast the rotation angle is changing.
In one embodiment, the method includes generating the combined sensor data if the angle change value is less than a threshold value and stopping generation of the combined sensor data if the angle change value is greater than or equal to the threshold value.
In one embodiment, the method includes generating a second accuracy value for the combined sensor based on the angle change value.
In one embodiment, the first sensor data includes first accelerometer data and first gyroscope data, while the second sensor data includes second accelerometer data and second gyroscope data.
In one embodiment, the combined sensor data includes combined accelerometer data and combined gyroscope data.
In one embodiment, generating combined sensor data includes averaging the first sensor data and the rotated second sensor data.
In one embodiment, generating combined sensor data includes applying a first weight to the first sensor data based on a noise level of the rotated second sensor data, applying a second weight to the rotated second sensor data based on a noise level of the first sensor data, and generating a weighted average of the first sensor data and the rotated second sensor data based on the first weight and the second weight.
In one embodiment, an electronic device includes a first lid including a first inertial measurement unit configured to generate first inertial sensor data and a second lid including a second inertial measurement unit configured to generate second inertial sensor data. The electronic device includes a sensor processing unit configured to detect a rotation angle between the first lid and the second lid based on the first sensor data and the second sensor data, generate rotated second sensor data by adjusting the second sensor data based on the rotation angle, and generate combined sensor data by combining the first sensor data and the rotated second sensor data.
In one embodiment, the electronic device includes a hinge rotatably coupling the first lid to the second lid.
In one embodiment, the sensor processing unit is configured to generate an accuracy value based on the first and second sensor data and generate the combined sensor data based on the first accuracy value.
In one embodiment, the sensor processing unit is configured to generate an angle change value indicating how fast the rotation angle is changing.
In one embodiment, the sensor processing unit is configured to generate the combined sensor data if the angle change value is less than a threshold value and stop generation of the combined sensor data if the angle change value is greater than or equal to the threshold value.
In one embodiment, the second sensor data includes applying to the second sensor data a rotation matrix based on the rotation angle.
In one embodiment, a foldable smartphone includes a first lid including a first inertial measurement unit configured to generate first inertial sensor data, a hinge coupled to the first lid, and a second lid rotatably coupled to the first lid by the hinge and including a second inertial measurement unit configured to generate second inertial sensor data. The foldable smartphone includes a sensor processing unit including a lid angle module configured to detect an angle of rotation between the first and second lids based on the first and second sensor data, a rotation module configured to generate rotated second sensor data by adjusting the second sensor data based on the angle of rotation, and a combiner module configured to generate combined sensor data from the first sensor data and the rotated second sensor data.
In one embodiment, the sensor processing unit includes angle variation detector configured to generate an angle change value indicating a rate of change of the angle of rotation.
In one embodiment, the combiner module is configured to receive the angle change value and to generate the combined sensor data based on the angle change value.
In one embodiment, the combiner circuit is configured to generate an accuracy value of the combined sensor data.
The various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.
