Patent Application
20220385803
Pursuant to 35 U.S.C. ยง 119 and the Paris Convention Treaty, this application claims foreign priority to Chinese Patent Application No. 202110578350.6 filed May 26, 2021, the contents of which, including any intervening amendments thereto, are incorporated herein by reference. Inquiries from the public to applicants or assignees concerning this document or the related applications should be directed to: Matthias Scholl P. C., Attn.: Dr. Matthias Scholl Esq., 245 First Street, 18th Floor, Cambridge, Mass. 02142.
The disclosure relates to the field of image acquisition, and more particularly, to an image data acquisition device and an image annotation method for an unmanned vending machine.
Computer vision technology for an unmanned vending machine relies on a powerful database. Conventionally, an image data acquisition process includes: 1) manually moving beverages in or out of an unmanned vending machine; 2) capturing two top images showing changes of locations and types of the beverages before and after a product is removed from the unmanned vending machine; and 3) manually annotating the types and number of the products in the two images. The conventional data acquisition process is tedious, inefficient, time consuming, thus increasing labor cost.
To solve the aforesaid problems, the disclosure provides an image data acquisition device for an unmanned vending machine; the image data acquisition device is designed to simulate the process of moving products into or out of the unmanned vending machine, acquire images from cameras, and create an image database.
The image data acquisition device comprises a camera, an elevating system, and a control processor. The camera covers a photographing scene, and is configured to capture images of a plurality of objects in the photographing scene. The elevating system is connected to the camera and moves the plurality of objects in and out of the photographing scene, and moves the plurality of objects in the photographing scene to meet a preset placement condition. The preset placement condition refers to a position combination of the plurality of objects in the photographing scene. The control processor is configured to control the elevating system to move the plurality of objects in the photographing scene in a preset order to meet the preset placement condition, to control the camera to capture an image of the plurality of objects meeting the preset placement condition, and to label the image of the plurality of objects through image annotation.
In a class embodiment of the disclosure, the elevating system comprises a base and at least one elevator; and the at least one elevator comprises a transmission part and a support bracket. The support bracket is fixedly disposed on the base; and the transmission part is fixedly disposed on the base and the support bracket. The transmission part comprises a motor, a threaded rod, a directional rod, and a transmission bracket. The directional rod is fixedly disposed on the base and the support bracket, and is configured to control one of the plurality of objects to move along an axial direction; the axial direction is an extension direction of the directional rod. The threaded rod is disposed along the axial direction; one end of the threaded rod is connected to the motor; and another end of the threaded rod is fixedly connected to the support bracket. The transmission bracket is in a threaded connection to the threaded rod. The transmission bracket is slidably connected to the directional rod; and one of the plurality of objects is detachably fixed on the transmission bracket.
In a class embodiment of the disclosure, the elevating system comprises a plurality of elevators each comprising a transmission part for carrying one of the plurality of objects.
In a class embodiment of the disclosure, the camera comprises a camera lens, a camera bracket, and a backdrop board. The camera lens is disposed on the camera bracket; the backdrop board provides a background used to take a picture and is at least one in number; the backdrop board is connected to the elevating system and comprises at least one through hole; and the elevating system drives the plurality of objects to pass through the at least one through hole.
In a class embodiment of the disclosure, the backdrop board comprises at least one cover.
In a class embodiment of the disclosure, the at least one cover is horizontally connected to the backdrop board through a hinge, and is foldable relative to the backdrop board towards the photographing scene.
In a class embodiment of the disclosure, the backdrop board is flat and level with the ground, and a maximum included angle between the backdrop board and the at least one cover is not greater than 90 degrees.
In a class embodiment of the disclosure, the camera lens is disposed above the photographing scene to capture a top image of the plurality of objects in the photographing scene.
In a class embodiment of the disclosure, the camera further comprises a light source controlled by the control processor.
The disclosure further comprises an image annotation method for use in the image data acquisition device of the disclosure, the image annotation method comprising:
The following advantages are associated with the image data acquisition device and the image annotation method thereof:
1. The image data acquisition device simulates the process of moving objects into or out of the unmanned vending machine, and controls automatic image acquisition, which is time saving and reduces labor cost, thus improving the image acquisition efficiency. About five images are captured every minute in manual mode of conventional methods, which is a labor-intensive and error-prone process; and the image data acquisition device captures ten images per minute without losing quality.
2. Automatic annotation is used in conjunction with manual annotation to improve the efficiency and accuracy of image data annotation. A small number of the first images are manually annotated and used to train a binary classification model for differentiation of the second bounding boxes in the remaining second images; the binary classification model is trained over the location and preset order of each object in the first images captured by the image data acquisition device, and annotates each object in the remaining second images to create an annotation file; and the annotation file is verified manually; about five images are annotated manually every minute; and the image annotation method of the disclosure labels and verifies 50 images every minute, which is 10 times faster than the manual annotation.
In the drawings, the following reference numbers are used: 1. Elevating system; 2. Camera; 21. Backdrop board; 22. Camera lens; 23. Camera bracket; 11. Transmission part; 12. Base; 13. Support bracket; 111. Motor; 112. Threaded rod; 113. Directional rod; 114. Transmission bracket; 115. Axial direction; 212. Cover; and 213. Hinge.
To further illustrate the disclosure, embodiments detailing an image data acquisition device for an unmanned vending machine are described below. It should be noted that the following embodiments are intended to describe and not to limit the disclosure.
As shown in
The camera 2 comprises a camera lens 22, a camera bracket 23, and at least one backdrop board 21. The camera lens 22 is disposed on the camera bracket 23; and the at least one backdrop board provides a background used to take a picture. The at least one backdrop board 21 is connected to the elevating system 1, and comprises at least one through hole and at least one cover 212; the elevating system 1 drives the plurality of objects to pass through the at least one through hole. The at least one cover is disposed on the at least one through hole and is foldable relative to the at least one backdrop board 21. Specifically, the at least one backdrop board 21 further comprises a hinge 213 horizontally connected to the first at least one backdrop board 21; the at least one cover 212 is connected to the at least one backdrop board 21 through the hinge 213, and is foldable relative to the hinge 213 towards the photographing scene. In the example, the camera lens 22 is the model number G200 (1080P).
In the example, the elevating system 1 is disposed below the camera 2; the at least one backdrop board 21 is flat and level with the ground; when the plurality of objects passes through one of the plurality of through holes, the corresponding one of the plurality of covers is foldable relative to the at least one backdrop board towards the photographing scene; when the plurality of objects is moved out of the photographing scene, the at least one cover 212 falls to the at least one backdrop board 21 due to gravity, without need to any control circuits; and a maximum included angle between the at least one cover and the at least one backdrop board is not greater than 90 degree.
In an alternative preferred embodiment of the disclosure, the camera lens 22 is disposed above the photographing scene to capture a top image of the plurality of objects in the photographing scene. In an alternative preferred embodiment of the disclosure, the camera 2 further comprises a light source controlled by the control processor.
An automatic image acquisition process of beverages in the image data acquisition device of the disclosure comprises:
Several bottles of beverages are provided, placed in a grid with 3 rows and 3 columns, and fixed on eight transmission brackets 114, respectively; eight transmission brackets 114 are disposed at the top of the elevating system 1 to ensure the eight bottles of beverages are detachably fixed in the photographing scene; and the eight transmission brackets 114 is controlled by eight motors 111, respectively; the control processor controls the operation of the eight motors 111 in a preset order to ensure the eight transmission parts 11 sequentially move the eight bottles of beverages in or out of the photographing scene, respectively; and the camera 2 captures an image of a combination of the eight bottles of beverages in the photographing scene. In the example, the camera captures 28=256 images of different combinations of the eight bottles of beverages. As the eight bottles of beverages are moved with the eight transmission parts 11, the control processor creates an annotation file that has information regarding type and location of each object in each first image.
An image annotation method for use in the image data acquisition device of the disclosure, the image annotation method comprising: manually labeling a first bounding box around each object in a part of first images; training a model for a one-class object detection by using the first bounding boxes; labeling a second bounding box around each object in remaining first images by using the model, thus obtaining second images; naming each object in a part of the second image according to a location of the second bounding box for each object in the part of the second images and the preset order controlled by the image data acquisition device; training a binary classification algorithm over the first bounding box and name of each object in the part of the first images, to name the second bounding boxes in remaining second images, thus creating an annotation file; and the binary classification algorithm is an object detection algorithm, such as FCOS and YOLO.
The image data acquisition device of the disclosure takes 15 minutes to place or remove the plurality of objects and collect data of 128 images, which is faster than 1.5 hours consumed by a conventional manual method. The image data acquisition device of the disclosure takes 5 minutes to manually correct the annotation file, which is faster than 45 minutes consumed by a conventional manual method.
It will be obvious to those skilled in the art that changes and modifications may be made, and therefore, the aim in the appended claims is to cover all such changes and modifications.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202110578350.6 | May 2021 | CN | national |
