Patent Application
20190050995
This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2017-154275, filed on Aug. 9, 2017; the entire contents of which are incorporated herein by reference.
Embodiments described herein relate generally to a camera motion estimation device, a camera motion estimation method, and a computer program product.
The position and posture of a camera mounted on a moving object or the like are changed according to a motion of the moving object. A technique of estimating the position and posture of a camera mounted on a moving object or the like, using an image captured by the camera, is conventionally known. For example, a technique of estimating a motion of a camera, using the epipolar constraint, from a correspondence relationship between feature points included in a plurality of images, is conventionally known.
However, in the conventional techniques, improvement of estimation accuracy of the camera motion is difficult in a case where many similar patterns exist in the images. For example, in a scene where many objects exist, such as walls that are difficult to cut out as one object in an image, improvement of the estimation accuracy of the camera motion is difficult.
According to an embodiment, a camera motion estimation device includes a memory and one or more hardware processors configured to function as a first acquisition unit, a second acquisition unit, and an estimation unit. The first acquisition unit is configured to acquire feature point correspondence information indicating correspondence between feature points included in time series images. The second acquisition unit is configured to acquire pixel flow information indicating a locus of a pixel included in the time series image. The estimation unit is configured to estimate a motion of a camera that has captured the time series images, using the pixel flow information and the feature point correspondence information.
Hereinafter, a camera motion estimation device, a camera motion estimation method, and a computer program product of an embodiment will be described with reference to the appended drawings.
A camera motion estimation device according to the embodiment is mounted on, for example, a moving object.
Example of Moving Object
The moving object 10 includes a camera motion estimation device 20, an output unit 10A, a camera 10B, a sensor 10C, a power control unit 10G, and a power unit 10H.
The moving object 10 may be arbitrary. The moving object 10 is a vehicle, a bogie, a robot, or the like. The vehicle is a motorcycle, an automobile, a bicycle or the like. Further, the moving object 10 may be a moving object that travels via a driving operation by a person, or a moving object that can automatically travel (autonomously travel) without a driving operation by a person, for example.
The camera motion estimation device 20 is realized by, for example, a dedicated or general purpose computer.
Note that the camera motion estimation device 20 is not limited to the mounted on the moving object 10. The camera motion estimation device 20 may be mounted on a stationary object. A stationary object is a non-movable object such as an object fixed to the ground, for example. The stationary object fixed to the ground is a guard rail, a pole, a parked vehicle, or a road sign. Further, for example, the stationary object is an object in a state of being stationary with respect to the ground. In addition, the camera motion estimation device 20 may be mounted on a cloud server that executes processing on a cloud system.
The power unit 10H is a drive device mounted on the moving object 10. The power unit 10H is an engine, a motor, a wheel, or the like.
The power control unit 10G controls the power unit 10H. The power unit 10H is driven under the control of the power control unit 10G.
The output unit 10A outputs information. In the embodiment, the output unit 10A outputs estimation result information indicating an estimation result of the motion of the camera 10B estimated by the camera motion estimation device 20.
The output unit 10A has a communication function transmit the estimation result information, a display function to display the estimation result information, a sound output function to output a sound indicating the estimation result information, and the like. The output unit 10A includes at least one of a communication unit 10D, a display 10E, and a speaker 10F. Note that, in the embodiment, it will be described a configuration in which the output it 10A has the communication unit 10D, the display 10E, and the speaker 10F as an example.
The communication unit 10D transmits the estimation result information to another device. For example, the communication unit 10D transmits the estimation result information to another device via a communication line. The display 10E displays information about an estimation result. The display 10E is a liquid crystal display (LCD), a projection device, a light, or the like. The speaker 10F outputs a sound indicating the information about the estimation result.
The camera 10B is a monocular camera, a stereo camera, an infrared camera, or the like. The camera 10B captures time series images around the moving object 10. The time series image may be monochrome or color. The camera 10B captures the time series images by capturing the vicinity of the moving object 10 in chronological order. The vicinity of the moving object 10 is, for example, an area within a predetermined range from the moving object 10. This range is, for example, a range that can be captured by the camera 10B.
In the embodiment, a case where the camera 10B is installed to include the front of the moving object 10 as a capturing direction will be described as an example. That is, in the embodiment, the camera 10B captures the front of the moving object 10 in chronological order. Further, in the embodiment, it will be described a case in which the camera 10B is a monocular color camera as an example.
The sensor 10C is a sensor that measures measurement information. The measurement information includes, for example, the position and posture of the moving object 10 on which the camera 10B is mounted, the speed of the moving object 10, and the steering angle of a steering wheel of the moving object 10. The sensor 10C is an inertial measurement unit (IMU), a speed sensor, a steering angle sensor, and the like. The IMU measures measurement information including triaxial acceleration and triaxial angular velocity of the moving object 10. That is, the IMU can measure the position and posture of the moving object 10 on which the camera 10B is mounted. The speed sensor measures the speed from an amount of rotation of a tire. The steering angle sensor measures the steering angle of the steering wheel of the moving object 10.
Next, it will be described an example of a functional configuration of the moving object 10 of the embodiment in detail.
Example of Functional Configuration
The moving object 10 includes the camera motion estimation device 20, the output unit 10A, the camera 10B, the sensor 10C, the power control unit 10G, and the power unit 10H. The camera motion estimation device 20 includes a processor 20A and a storage unit 20B. The output unit 10A includes the communication unit 10D, the display 10E, and the speaker 10F.
The processor 20A, the storage unit 20B, the output unit 10A, the camera 10B, the sensor 10C, and the power control unit 10G are connected via a bus 10I. The power unit 10H is connected to the power control unit 10G.
Note that the output unit 10A (the communication unit 10D, the display 10E and the speaker 10F), the camera 10B, the sensor 10C, the power control unit 10G, and the storage unit 20B may be connected via a network. A communication method of the network used for the connection may be a wired method or a wireless method. Further, the network used for the connection may be realized by a combination of the wired method and the wireless method.
The storage unit 20B stores information. The storage unit 20B is a semiconductor memory device, a hard disk, an optical disk, or the like. The semiconductor memory device is, for example, a random access memory (RAM) and a flash memory. Note that the storage unit 20B may be a storage device provided outside the camera motion estimation device 20. Further, the storage unit 20B may be a storage medium. Specifically, the storage medium may store or temporarily store a program and various types of information downloaded via a local area network (LAN) or the Internet. Further, the storage unit 20B may be configured with a plurality of storage media.
The processor 20A includes a first acquisition unit 20C, a second acquisition unit 20D, a determination unit 20E, an estimation unit 20E, and an output control unit 20G. The first acquisition unit 20C, the second acquisition unit 20D, the determination unit 20E, the estimation unit 20F, and the output control unit 20G are realized by one or a plurality of processors, for example.
The processor 20A may be realized by causing a processor such as a central processing unit (CPU) to execute a program, that is, by software. Further, for example, the processor 20A may be a processor such as a dedicated integrated circuit (IC), that is, by hardware. Further, for example, the processor 20A may be realized by using software and hardware in combination.
The term “processor” used in an embodiment includes, for example, a CPU, a graphical processing unit (GPU), an application specific integrated circuit (ASIC), and a programmable logic device. The programmable logic device includes, for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA).
The processor realizes the processor 20A by reading and executing the program stored in the storage unit 20B. Note that the program may be directly incorporated into a circuit of the processor instead of being stored in the storage unit 20B. In this case, the processor realizes the processor 20A by reading and executing the program incorporated in the circuit.
The first acquisition unit 20C detects a pixel haying large change in luminance from surrounding pixels, as a feature point, for each time series image. Then, the first acquisition unit 20C acquires feature point correspondence information indicating correspondence between feature points included in time series images.
Example of Correspondence of Feature Point
Here, it will be described an example of processing of making the feature points correspond to each other. First, the first acquisition unit 20C calculates a feature vector F having pixels around the detected feature point as components. Note that the component of the feature vector F may be information indicating a feature of the feature point, rather than pixel arrangement itself. The information indicating a feature of the feature points is, for example, information abstracting and expressing the difference in luminance of a surrounding area of the feature point with 0 or 1.
Next, the first acquisition unit 20C causes a feature vector Ft−1 of the feature point 101 included in the image captured at the time t−1 to correspond to the feature point 201 at the time t having a feature vector Ft closest in distance to the feature vector Ft−1. Next, the first acquisition unit 20C inputs feature point correspondence information indicating the correspondence between the feature point 101 included in the image captured at the time t−1 and the feature point 201 included in the image captured at the time t to the determination unit 20E. The feature point correspondence information includes, for example, information in which coordinates of the feature point 101 included in the image captured at the time t−1 and coordinates of the feature point 201 included in the image captured at the time are associated with each other.
Note that, in the example of
Further, the first acquisition unit 20C may acquire feature point correspondence information including candidates of a plurality of feature points 201 corresponding to the feature point 101 in order of the feature vector Ft having closer in distance to the feature vector Ft−1. For example, the first acquisition unit 20C may acquire the feature point correspondence information in which the feature point 201 having the feature vector Ft closest in distance between the feature vectors Ft−1 and Ft and the feature point 201 having the feature vector Ft second closest in distance between the feature vectors Ft−1 and Ft are caused to correspond to the feature point 101 having the feature vector Ft−1.
Further, in a case of causing the candidates of the plurality of feature points 201 to correspond to the feature point 101, the first acquisition unit 20C may include likelihoods of the candidates of the plurality of feature points 201 in the feature point correspondence information. The likelihood is, for example, is a reciprocal of the distance between the feature vectors Ft−1 and Ft.
Further, for example, the likelihood is a ratio of the distance between the feature vector Ft closest in distance between the feature vectors Ft−1 and Ft and the feature vector Ft second closest in distance between the feature vectors Ft−1 and Ft. That is, the likelihood of the feature point 201 having the feature vector Ft can be considered to be higher, as the feature vector Ft closest in distance between the feature vectors Ft−1 and Ft and the feature vector Ft second closest in distance between the feature vectors Ft−1 and Ft are more different.
Note that a value indicating the likelihood may indicate that the likelihood is larger as the value is larger or may indicate that the likelihood is smaller as the value is smaller. In an embodiment, it will be described a case in which the likelihood is larger as the value is larger. Not that, in a case where the likelihood is smaller as the value is smaller by using the distance between the feature vectors Ft−1 and Ft as it is for the likelihood, the sign of inequality of a threshold of determination processing described below is reversed.
Returning to
Example of Flow of Pixels
Here, it will be describe described two examples of processing of acquiring pixel flow information.
The first method of acquiring pixel flow information is a method using a neural network. An input of the neural network is, for example, time series two images (the image captured at the time t−1 and the image captured at the current time t). Further, an output of the neural network is loci of pixels from positions of pixels in the image at the time t−1 to positions of pixels corresponding to them in the image at the current time t.
In the method using the neural network, a filter learns to output a flow, thereby to calculate flows of all the pixels. To be specific, in the neural network, representative portions are extracted from a filtered image, which is obtained by filtering an image, and the image size is reduced and then the reduced image is further filtered. The representative portions extracted from the filtered image are determined from information obtained through learning by the neural network. Filtering the reduced image enables filtering a relatively large region in the image even if the filter size is not changed, whereby use of global information becomes possible. In the neural network, after the filtering processing and the reduction processing are performed a specified number of times or more, the enlargement processing and the filtering processing are performed to get the size of the image back to the original size.
Here, it has been described the example in which the result of filtering is an image. However, the result of filtering may be expressed as filter data in which arrangement is simply determined. That is, in a case of using the neural network, the second acquisition unit 20D repeats processing of generating filter data by applying filter to an image, the filter having learned to output flows (loci) of pixels, and processing of reducing and increasing the filter data, a specified number of times or more, and then performs processing of returning the reduced and increased filter data to the number of locus data by the number of pixels of the original image.
The second method of acquiring pixel flow information is a method of using the estimation result information of the motion of the camera 10B obtained in the past and a three-dimensional position of an object.
Next, the second acquisition unit 20D assumes a motion model from the positions and postures of the camera 10B at the time t−2 and the time t−1 to estimate the position and posture of the camera 10B at the current time t. The motion model is, for example, a uniformly accelerated motion and a uniform motion.
Next, the second acquisition unit 20D projects the three-dimensional position of the object 200 on the image captured at the position and posture of the camera 10B at the current time t. With the above operation, the second acquisition unit 20D can acquire the flow (locus) of the pixel indicating the image of the object 200.
Note that the second acquisition unit 20D may estimate the position and posture of the camera 10B at the current time t, using the sensor 10C.
Returning to
Here, the reason why the feature point moving vector 211 and the pixel moving vector 212 are shifted as illustrated in
Next, it will be described an example of a method of determining similarity by the determination unit 20E.
Use of Vector Direction
The determination unit 20E determines that the similarity is larger as an angle θ between the feature point moving vector 211 and the pixel moving vector 212 is smaller Further, for example, the determination unit 20E determines that the feature point moving vector 211 and the pixel moving vector 212 are similar, when the angle θ between the feature point moving vector 211 and the pixel moving vector 212 is equal to or smaller than a threshold a. The determination unit 20E determines that the feature point moving vector 211 and the pixel moving vector 212 are not similar, when the angle θ between the feature point moving vector 211 and the pixel moving vector 212 is larger than the threshold a.
Note that the determination unit 20E may set similarity 1 to the case where the vectors are similar and set similarity 0 to the case where the vectors are not and express the similarity by binary values.
Use of Vector Length
For example, the determination unit 20E determines that the feature point moving vector 211 and the pixel moving vector 212 are similar when a length l of a vector 213 indicating a difference between the feature point moving vector 211 and the pixel moving vector 212 is smaller than a product of a length Lp of the feature point moving vector 211 and a distance ratio constant b (0<b≤1). The determination unit 20E determines that the feature point moving vector 211 and the pixel moving vector 212 are not similar when the length l of the vector 213 indicating the difference is equal to or larger than the product of the length Lp of the feature point moving vector 211 and the distance ratio constant b.
Note that the determination unit 20E may determine whether the feature point moving vector 211 and the pixel moving vector 212 are similar, using a length LD of the pixel moving vector 212, instead of the length Lp of the feature point moving vector 211. Further, the determination unit 20E may determine whether the feature point moving vector 211 and the pixel moving vector 212 are similar, using the length Lp of the feature point moving vector 211 and the length LD of the pixel moving vector 212.
Further, the determination unit 20E may determine whether the feature point moving vector 211 and the pixel moving vector 212 are similar, using both the direction and the length of the vectors. In that case, the determination unit 20E may determine that the feature point moving vector 211 and the pixel moving vector 212 are similar when all of a plurality of conditions (θ<a, l<bLp, and l<bLD) are satisfied. Alternatively, for example, the determination unit 20E may determine that the feature point moving vector 211 and the pixel moving vector 212 are similar when any one of the plurality of conditions (θ<a, l<bLp, and l<bLD) is satisfied.
Further, in a case where candidates of a plurality of the feature points 201 having different likelihoods is included in the feature point correspondence information, the determination unit 20E determines the similarity in order from the candidate of the feature point 201 having a larger likelihood. Then, when the similarity is determined to be less than a threshold, the determination unit 20E determines the similarity of the candidate of the feature point 201 having the next largest likelihood. For example, in a case where candidates of two feature points 201 having different likelihoods are included in the feature point correspondence information, the determination unit 20E first determines the similarity of the candidate of the feature point 201 having a larger likelihood. Then, when the similarity is determined to be less than the threshold, the determination unit 20E determines the similarity of the candidate of the feature point 201 having a smaller likelihood. As a result, the candidate of the feature point 201 having the likelihood that is not the maximum can also be used as the feature point 201 corresponding to the feature point 101, as long as the similarity is equal to or larger than the threshold. That is, even the candidate of the feature point 201 determined not to be similar according to a conventional method using only the feature point correspondence information can be used as the feature point 201 corresponding to the feature point 101, as long as the similarity considering the pixel flow information is equal to or larger than the threshold.
Note that the determination unit 20E may determine the similarity not only between the time t−1 and the time t but also by combinations of a plurality of times, such as between the time t−1 and the time t, and between the time t−2 and the time t.
Note that the feature point correspondence information of the combinations of a plurality of times to be used for determination of the similarity may be acquired by the first acquisition unit 20C, for example. Further, for example, the feature point correspondence information of the combinations of a plurality of times to be used for determination of the similarity may be obtained by storing the determination results (correspondence of the feature points), which have been determined to be similar between adjacent times by the determination unit 20E, and associating the plurality of determination results. For example, the determination unit 20E may obtain the determination result between the time t−2 and the time t by associating the determination result between the time t−2 and the time t−1 and the determination result between the time t−1 and the time t in a case where the feature point of the time t−1 is the same.
Similarly, the pixel flow information of the combinations of a plurality of times to be used for determination of the similarity may be acquired by the second acquisition unit 20D, for example. Further, for example, the pixel flow information of the combinations of a plurality of times to be used for determination of the similarity may be obtained by storing the pixel flow information between adjacent times used for determination by the determination unit 20E, and associating the pixel flow information between a plurality of the adjacent times. For example, the determination unit 20E may obtain the pixel flow information between the time t−2 and the time t by associating the pixel flow information between the time t−2 and the time t−1 and the pixel flow information between the time t−1 and the time t by the same pixel at the time t−1.
Further, the determination unit 20E may determine similarity between a vector indicating a sum of a predetermined number of the feature point moving vectors 211 obtained between time series images, and a vector indicating a sum of a predetermined number of the pixel moving vectors 212 obtained between time series images. For example, in a case where the predetermined number is two, the determination unit 20E may determine the similarity, using a vector indicating a sum of the feature point moving vectors 211 the time t−2 and the time t−1 and the feature point moving vectors 211 at the time t−1 and the time t, and a vector indicating a sum of the pixel moving vectors 212 at the time t−2 and the time t−1, and the pixel moving vectors 212 at the time t−1 and the time t.
Returning to
Note that a specific method of estimating the motion of the camera 10B may be arbitrary. In a case where a plurality of feature point correspondences and the three-dimensional position of the feature point have been obtained, for example, the estimation unit 20F estimates a motion such that a difference between the plurality of feature point correspondences and a projection position of a feature point calculated from an initial position of a motion estimated from a past motion of the camera 10B becomes smallest, and outputs a motion at the point of time when the difference converges as an estimation result. Further, for example, in a case where the three-dimensional position of the feature point has not been obtained, the estimation unit 20F estimates the motion of the camera 10B, using the epipolar constraint.
In a case where the number of the feature points included in the feature point correspondence information is equal to or larger than a number threshold, the estimation unit 20F may estimate the motion of the camera 10B that has captured the time series images from the correspondence of the feature points included in the feature point correspondence information without using the pixel flow information. Note that the processing of determining whether the number of the feature points is equal to or larger than the number threshold is executed by the above-described determination unit 20E, for example.
Further, in a case where likelihood of the feature point is included in the feature point correspondence information, and in a case where the number of the feature points having likelihood that is equal to or larger than a likelihood threshold is equal to or larger than the number threshold, the estimation unit 20F may estimate the motion of the camera 10B that has captured the time series images from the correspondence of the feature points included in the feature point correspondence information without using the pixel flow information. Note that the processing of determining whether the number of the feature points having likelihood that is equal to or larger than the likelihood threshold is equal to or larger than the number threshold is executed by the above-described determination unit 20E, for example.
Note that, in a case where likelihoods of the feature points are included in the feature point correspondence information, and in a case where the likelihood is equal to or larger than the likelihood threshold for each feature point, the estimation unit 20F may estimate the motion of the camera 10B that has captured the time series images from the correspondence of the feature points included in the feature point correspondence information without using the pixel flow information. Note that the processing of determining whether the likelihood is equal to or larger than the likelihood threshold is executed by the above-described determination unit 20E, for example.
The output control unit 20G outputs the estimation result information indicating the motion of the camera 10B estimated by the estimation unit 20F to the output unit 10A and the power control unit 10G.
Note that information indicating whether the motion of the camera 10B is estimated using the pixel flow information and the feature point correspondence information may be included in the estimation result information. Thus, the output unit 10A can output information indicating a state where the motion of the camera 10B is estimated from only the feature point correspondence information, or indicating a state where the motion of the camera 10B is estimated from the pixel flow information and the feature point correspondence information.
The power control unit 10G controls the power unit 10H according to the estimation result information received from the output control unit 20G. For example, the power control unit 10G may generate a power control signal for controlling the power unit 10H and control the power unit 10H according to the estimation result information. The power control signal is a control signal for controlling a drive unit that performs driving regarding traveling of the moving object 10 in the power unit 10H. To be specific, the power control unit 10G may control power to assist an avoidance operation to avoid the object 200 as an obstacle from the position of the moving object 10 on which the camera 10B is mounted, the position being identified from the estimation result information, and the three-dimensional position of the object 200 used for estimation of the motion of the camera 10B. The control of power to assist an avoidance operation is, for example, power control using a value for avoidance operation larger than a reference value, for increase/decrease value of the steering angle, power control supposed to apply the brake, and the like.
Note that the above-described functional configuration in
Next, it will be described an example of determination processing of the embodiment with reference to a flowchart.
First, the determination unit 20E determines whether the likelihood of the feature point 201 included in the image captured at the time t is less than the threshold (step S1).
When the likelihood of the feature point 201 is less than the threshold (step S1, Yes), the determination unit 20E calculates the feature point moving vector 211 and the pixel moving vector 212 described above (step S2). Next, the determination unit 20E determines whether the similarity between the feature point moving vector 211 and the pixel moving vector 212 calculated in the processing of step S2 is equal to or larger than the threshold (step S3).
When the similarity of the feature point 201 is not equal to or larger than the threshold (step S3, No), the estimation unit 20F does not use the feature point 201 determined not to be equal to or larger than the threshold in the processing of step S3, for estimation of the motion of the camera 10B (step S5).
When the likelihood of the feature point 201 is not less than the threshold (step S1, No), or when the similarity of the feature point 201 is equal to or larger than the threshold (step S3, Yes), the estimation unit 20F estimates the motion of the camera 10B, using the feature point 201 (step S4).
Example of Effects
Finally, an example of a hardware configuration of the camera motion estimation device 20 according to the embodiment will be described.
Example of Hardware Configuration
Note that the display device 304, the input device 305, and the communication device 306 may not be provided. For example, when the camera motion estimation device 20 is connected to another device, a display function, an input function, and a communication function of the another device may be used.
The control device 301 executes a program read from the auxiliary storage device 303 to the main storage device 302. The control device 301 is one or more processors such as a CPU. The main storage device 302 is a memory such as a read only memory (ROM) and a RAM. The auxiliary storage device 303 is a memory card, a hard disk drive (HDD), and the like.
The display device 304 displays information. The display device 304 is, for example, a liquid crystal display. The input device 305 receives an input of information. The input device 305 is, for example, a hardware key and the like. Note that the display device 304 and the input device 305 may be a liquid crystal touch panel or the like that serves both a display play function and an input function. The communication device 306 communicates with other devices.
The program executed by the camera motion estimation device 20 is stored in a file in an installable or executable format in a computer-readable storage medium such as a CD-ROM, a memory card, a CD-R, or a digital versatile disc (DVD) and provided as a computer program product.
Further, the program executed by the camera motion estimation device 20 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by the camera motion estimation device 20 may be provided via a network such as the Internet without being downloaded.
Further, the program executed by the camera motion estimation device 20 may be incorporated in ROM or the like in advance and provided.
The program executed by the camera motion estimation device 20 has a module configuration including functions realizable by the program, of the functions of the camera motion estimation device 20.
The functions realized by the program are loaded onto the main storage device 302 by the control device 301 reading and executing the program from the storage medium such as the auxiliary storage device 303. That is, the functions realized by the program are generated on the main storage device 302.
A part of the functions of the camera motion estimation device 20 may be realized by hardware such as an IC. The IC is, for example, a processor that executes dedicated processing.
In a case where the functions are realized using a plurality of processors, each processor may realize one of the functions or may realize two or more of the functions.
As described above, in the camera motion estimation device 20 of the embodiment, the first acquisition unit 20C acquires the feature point correspondence information indicating the correspondence between the feature points included in time series images. The second acquisition unit 20D acquires the pixel flow information indicating the locus of the pixel included in the time series images. Then, the determination unit 20E and the estimation unit 20F estimate the motion of the camera that has captured the time series images, using the pixel flow information and the feature point correspondence information.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiment described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiment described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2017-154275 | Aug 2017 | JP | national |
