Patent Grant
11838688
This application is a § 371 National Stage Application of PCT/AU2019/050524, filed May 28, 2019, which claims priority to Australian Patent Application No. 2018901876, filed May 28, 2018, which applications are herein incorporated by reference in their entireties and to which priority is claimed.
The present invention relates to a system and method for capturing images of a product from a series of fixed positions around the product.
In recent years, online shopping has quickly gained popularity due to the convenience of not having to leave ones house and the ability to make a purchase at any time of the day, or night. As a result, e-commerce has quickly become some customer's preferred method of purchase as compared to traditional brick and mortar retail stores.
In addition to the ease of purchase afforded by e-commerce, customers are able to order goods or services from stores that are not within the customer's vicinity. This provides a customer more choices of online stores and therefore the ability to choose an e-store with a lower price point. To stay competitive, retailers have to reduce prices and streamline their processes to increase or maintain profitability.
As part of the online shopping experience, a customer is unable to see, touch and try on a product. As such, it is important for a retailer to provide content which has enough information for customers to be able to make an informed decision. Further, the content shown to the customer should be appealing enough to the customer to convince them to purchase the product.
As such, some retailers invest time and money into producing media for their products, which is needed for each product to be catalogued for display on the respective e-stores. Typically, this includes a team of highly skilled photographers, stylists, studios, lighting and other production equipment which is typically slow, labour intensive and expensive. For example, current systems include disconnected lighting systems, a media capture systems and a mounting platform. Each of these systems require professionals to separately operate, to set up and calibrate these separate component parts prior to being able to capturing images for products. For example, a single product may take around 70 minutes to create three photographs and a single 360° video. Retailers with a big portfolio of products and who are constantly updating their e-stores with new products will spend a lot of time and money producing media for their products. Retailers often find themselves in need of a more efficient method of obtaining media content of products.
It is generally desirable to overcome or ameliorate one or more of the above described difficulties, or to at least provide a useful alternative.
In accordance with the present invention, there is provided a system capturing images of a product from a series of fixed positions around the product, including:
In accordance with the present invention, there is also provided a method for capturing an image of a product, performed by one or more processors in communication with data storage includes a plurality of instructions stored thereon which, when executed by the one or more processors, cause the system to perform the steps of:
In accordance with the present invention, there is also provided a system for capturing one or more images of a product, including:
Preferred embodiments of the invention are hereafter described, by way of non-limiting example only, with reference to the accompanying drawings, in which:
The system 10 shown in
In some embodiments, the mount 50 includes a turntable 55 and a lighting system 60. In other embodiments, the control system 70 includes a server 75 and a mobile computing device 80 for receiving user instruction and communicating the user instruction to the one or more processors 3028 of the control system 70. For example, mobile computing device 80 may be located posterior to the actuator 30 as particularly shown in
The components of system 10 are in communication via the network 20. The communication network 20 may include the Internet, telecommunications networks and/or local area networks. In some embodiments, network 20 is a wireless communication system such as Wi-Fi, Bluetooth or NFC. Network 20 may also include wired connections such as USB or other forms of data transfer systems.
The system 10 makes capturing images and/or video of a product to be captured simpler, faster and more convenient. Advantageously, the system 10 allows media of a product that has been placed on the mount 50 to be captured automatically i.e. without user 90 intervention, upon receipt of user instruction. The captured media may include images of the product taken from various angles and a 360-degree video, for example. In some embodiments, post processing of the captured media may be performed after the media has been captured. Advantageously, system 10 allows media of a product to be captured efficiently i.e. in much less time and manpower compared to previous known systems.
Preferably, system 10 for capturing images of a product from a series of fixed positions around the product includes one or more of the following:
Advantageously, said data storage 3024 includes a plurality of instructions stored thereon which, when executed by the one or more processors, cause the system to perform the steps of:
In some embodiments, more than one image may be captured to make a video such as a 360° video.
As particularly shown in
As particularly shown in
Preferably as particularly shown in
For example, the lighting system 60 includes one or more LED lights. Advantageously, the lighting system 60 illuminates the displayed product while the media capturing device 40 is capturing one or more images of the product by receiving lighting instruction from the control system 70.
Preferably, the actuator 30 is couplable to various types of media capturing devices.
Advantageously, the turntable 55 includes a motor for causing the turntable 55 to rotate about the axis. For example, a servo drive 512 as particularly shown in
The turntable 55 is in communication with the control system for receiving turntable instruction. Preferably, upon receipt of instructions from the control system 70, the turntable 55 rotates from a fixed position to another fixed position to allow the media capturing device 40 to capture an image of the product from the another fixed position.
For example, turntable 55, upon receipt of instruction from the control system 70, rotates the product about an axis at a fixed velocity to allow the media capturing device 40 to capture one or more images of the product from one or more fixed positions as it rotates. In another embodiment, turntable instruction includes one or more of the following:
Preferably, the lighting system 60 is in communication with the control system 70 for receiving lighting instruction. As particularly shown in
In some embodiments, the lighting wall system 650 and lights mounted on the lighting support ring 602 are in communication with the control system 70, for sending and receiving lighting instructions. For example, lighting instructions include synchronizing the lighting settings for the lights during image capture.
As particularly shown in
For example, lighting instruction includes one or more of the following:
In some embodiments, the system 10 further includes a weighing device 550 and/or scaling device 560. The weighing device 550 may be incorporated into the mount 50 so that when a product is placed on the mount 50, the weighing device 550 obtains the weight of the product. Advantageously, the weight of the product is stored in data storage 3024, for example. In some embodiments, the weight of the product is stored with the image captured by the media capturing device 40.
The scaling device 560 provides one or more dimensions of a product placed on the mount 50. This may be provided by post-processing one or more images captured by the media capturing device 40. In some embodiments, a ruler or line gauge is placed, for example on the mount 50, within the field of view of the image captured by the media capturing device 40 so as to allow one or more dimensions of the product to be calculated. For example, the one or more dimensions of the product include the length, height and/or depth of the outer bounds of the product.
In some embodiments, the lens 402 of media capturing device 40 is in communication with the control system 70 for receiving focus, aperture and zoom instructions. For example, the lens has remote focus control which is programmed in sync with robot motion.
Preferably, control system 70 includes server 75 and mobile computing device 80 as particularly shown in
As particularly shown in
Preferably, the actuator 30 is a robotic arm as particularly shown in
Advantageously, the robotic arm 30 is in communication with the control system 70, in particular the server 75, for receiving actuator instruction. For example, actuator instruction includes one or more of the following:
The control system 70 preferably performs motion planning for setting a path for the actuator 30 to move from a first position to a second position.
Advantageously, the control system 70 further includes a mobile computing device 80 for receiving user instruction and communicating the user instruction to the one or more processors 3028 of the control system 70. The mobile computing device 80 is preferably running an application 2018. In some embodiments, data associated with the user instruction is used as an input to one or more of the following:
For example, the mobile computing device 80 is a tablet computer such as an iPad or smartphone such as an iPhone. The mobile computing device 80 for receiving user input via application 2018 or displaying data such as images taken by the media capturing device 40. In an alternate embodiment, the mobile computing device 80 is an interactive kiosk including a display. In some embodiments, the display is a touch screen for receiving user input. Advantageously, the mobile computing device 80 is in communication with network 20.
As particularly shown in
In some embodiments, one or more of the actuator instruction, lighting instruction, media capture instruction and turntable instruction are sent in sync. For example, to capture a video, the actuator 30 and turntable 55 may be programmed to move with respect to each other.
For example, media capture instruction includes one or more of the following:
For example, the media capturing device 40 is a Canon 1DX Mark II. In some embodiments, the media capturing device 40 has interchangeable lens and may be fitted with a suitable lens such as a 24 mm cine lens or a cine 35 mm lens.
In some embodiments, media capture instruction may be performed in sync with other instructions such as the lighting instruction to ensure the appropriate lighting on the product is cast when a photo or video is being captured. In other embodiments, when a video is captured by the media capturing device 40, the turntable 55 may be turning with respect to the media capturing device 40 at a particular angular velocity and from a certain origin point to a specified angular displacement of the turntable 55, for example, to ensure a full 360° video of the product is captured.
In some embodiments, a 360° video is captured by actuating the media capture device 40 to begin recording a video and the actuator 30 is stationary whilst the turntable 55 rotates at an angular velocity about an axis. In other embodiments, a 360° video is captured by actuating the media capture device 40 to begin recording a video and the turntable 55 is stationary whilst the media capture device 40 coupled to the actuator 30 moves around the product. In yet another embodiment, a 360° video is captured by actuating the media capture device 40 to begin recording a video and both the turntable 55 and the media capture device 40 coupled to the actuator 30 moves with respect to one another.
In some embodiments, the server 75 communicates with the media capturing device 40 using a SDK from the camera manufacturer via connections such as Ethernet or USB or network 20. For example, the Server Application of Industrial PC 75 as particularly shown in
Advantageously, after the step of actuating the media capturing device 40 to capture an image of the product, the media capturing device 40 sends data associated with the captured image for display to the user 90 to one or more of the following:
In some embodiments, the media capture device 40 is a 3D scanner. In another embodiment, the 3D scanner 450 is used in conjunction with a camera. Advantageously, the 3D scanner 450 is coupled to a distal end of actuator 30. For example, the product placed on mount 50 is rotated about a central axis by means of the turntable 55 whilst the 3D scanner 450 captures scans of the product. Scans of the product may include data associated with the distance between the 3D scanner 450 and the product for reconstructing a 3D model of the product. Scans may also include one or more images of the product. Further, scans of the top and bottom of the product are captured may be captured, for example by actuating the actuator 30 so that a scan of the top and/or bottom of the product is captured. Advantageously, the 3D scanner 450 is configured to capture a plurality of scans of a product placed on mount 50 for creating a digital 3D model of the product.
Preferably, the control system 70 further includes a programmable logic controller 704 for causing the actuator 30 to move upon receipt of actuator instructions from the one or more processors 3028. The programmable logic controller 704 preferably further includes an application programming interface for communicating with the one or more processors 3028 of the control system 70.
In some embodiments, the media capturing device 40 is physically set up within a custom environment featuring cycloramas, and lighting which will provide optimal conditions for the background removal algorithms. In some embodiments, a calibration sequence is performed to optimize the settings for one or more devices. For example, depending on the natural light available in the space, lighting settings may differ from day to day. As such, a number of test images may be captured by the media capture device 40 with various sets of lighting instruction to the lighting system 60. In some embodiments, the images are displayed to the user 90, for example on the mobile computing device 80 or a display monitor, and the user 90 chooses one or more images. The lighting instruction for the one or more images chosen is then used for the succeeding media capture sequences. Similar calibration sequences could be performed for the other instructions such as media capturing instructions, actuator instructions or turntable instructions. Other calibrations sequences using automatic post-processing image programs may also be employed.
In some embodiments, when the control 70 is first set up for a user 90, the server 75 is set up with:
In some embodiments, a user 90 picks up a mobile computing device 80 to perform shoots of one or more products. The user 90 logs in to the app 2018 running on mobile computing device 80 by selecting their user account and entering a password, for example. The app 2018 authenticates the user 90 by communicating with the server 75 to ensure that the user account and password entered matches an entry stored on data storage 3024. The app 2018 then generates for display 2002 a dashboard display showing one or more of the following:
The app 2018 then queries server 75 to retrieve list of available packages for user 90 and generates on display 2002 for user selection. In some embodiments, the mobile computing device 80 switches to camera mode to access the built-in camera for the user 90 to place a product barcode to be read by the camera. The product barcode auto-populates the barcode metadata field. User 90 may also enter text such as Brand, Product Name, Description, and other descriptions or classifications of the product to be captured.
Media Capturing Process 9000
For example, media capturing process 9000, as particularly shown in
In some embodiments, a live preview is provided to the user 90, for example the mobile computing device 80 receives and displays an image of the product taken with the media capturing device 40. Implementation of live preview may be camera-specific. For example, for Canon cameras, a Live Preview stream is directly captured from the devices. For example, on RED cameras, the Live Preview is achieved by taking the monitor feed out of the camera, via a connection such as serial digital interface (SDI) for example, converting it to network device interface (NDI) and utilizing the NDI Connect Webserver to convert the NDI stream to JPEG frames, which are read via a WebSocket, for example.
At step 9004, the control system 70 waits until it receives begin instruction. Begin instruction may be received from the user 90 via mobile computing device 80, for example by means of a touch screen input via an app 2018. In other embodiments, one of the components of system 10, for example, the mount 50 or media capturing device 40 may detect that a product has been placed on the mount 50 and subsequently, sending begin instructions to the control system 70 to begin to capture media. For example, begin instructions may also include instructions to the weighing device 550 and/or scaling device 560 to begin weighing and/or scaling the product.
In some embodiments, a user 90 selects a package that they want to shoot and scans a product barcode using a barcode scanner, for example a camera on a mobile computing device 80, or enters metadata into a mobile computing device 80 as part of the begin instructions.
The control system 70 then checks begin instructions and determines if instructions are for capturing media is associated with capturing a single image or video or more than one images or videos. If instructions for capturing media associated with a single image or video, the control system 70 performs process 9500 as particularly shown in
For example, data storage 3024 includes a plurality of instructions stored thereon which, when executed by the one or more processors 3028, cause the system to perform the steps of:
In some embodiments, the series of fixed positions is a single fixed position. In some embodiments, the step 9014 includes the processor sending media capture instruction to the media device 40. In other embodiments, step 9014 includes the processor sending 3D scanning instructions to 3D scanner 450, for example.
In some embodiments, step 9008 incudes the control system 70 generating instructions to locate the actuator 30 at coordinates and sending said instructions to the actuator 30. At step 9010, the actuator 30 receives instructions from the control system 70 and locates the end-point of the actuator, the end point coupled to the media capturing device 40, at the coordinates. Once the actuator 30 is located at the coordinates, at step 9012, the actuator 30 sends data to control system 70 indicating that the actuator's end point has reached the final coordinates. At step 9014, the control 70 generates instructions to capture media and sends it to media capturing device 40. At step 9016, the media capturing device 40 captures media. At step 9018, the media capturing device 50 sends data associated with the captured media to the control system 70. At step 9020, the control system 70 receives captured media from media capturing device 40 and at step 9022 saves the data associated with the captured media.
In some embodiments, instructions include capturing media associated with more than one image or video, system 10 performs the following before the step of moving the actuator 30:
In some embodiments, the step of actuating the media capturing device 40 to capture an image of the product includes the steps of:
In some embodiments, the captured media is one or more of the following:
In some embodiments, the captured media of the product is transferred to server 75 or to the in-camera storage and then immediately transferred to the server 75. In other embodiments, data associated with the product such as weight, one or more dimensions and/or 3D scans are also transferred to server 75 or in-camera storage.
In some embodiments, step 3024 includes using parameters established in initial setup by the user 90 and files are processed to the correct size, names and then exported to the selected file type(s). In some embodiments, the files may also be uploaded to a website or to the cloud-based storage provider for later use. Users other than user 90 may be authorized to access the files thus allowing multiple parties to view and use the files associated with the captured media in a timely manner.
In some embodiments, after the step of saving captured media of the product in said data storage 3024, the one or more processors 3028 post-processes the captured media.
The post-processing process may be customised to each client's individual needs. It can, for example, include one or more of the following:
In other embodiments, post-processing may include constructing a 3D model of the product based on 3D scans captured by the 3D scanner 450. In other embodiments, post-processing includes calculating one or more dimensions based on data from the scaling device 560.
In some embodiments, images saved as TIFF or PSD, will retain an editable version of the mask. In other embodiments, PNG files will retain any transparency from the background removal.
In some embodiments, post-processing steps of video files include one or more of the following:
After post-processing, the server 75 sends to the mobile computing device 80 data associated with the post-processed image for the user 90 to view. In some embodiments, the user 90 confirms that the media has been captured correctly. A user may opt to reposition the product and rerun a package to re-capture media.
Mobile Computer Device 80
As shown, the mobile computing device 80 includes the following components in electronic communication via a bus 2006:
Although the components depicted in
The display 2002 generally operates to provide a presentation of content to a user, and may be realized by any of a variety of displays (e.g., CRT, LCD, HDMI, micro-projector and OLED displays).
In general, the non-volatile data storage 2004 (also referred to as non-volatile memory) functions to store (e.g., persistently store) data and executable code.
In some embodiments for example, the non-volatile memory 2004 includes bootloader code, modem software, operating system code, file system code, and code to facilitate the implementation components, well known to those of ordinary skill in the art, which are not depicted nor described for simplicity.
In many implementations, the non-volatile memory 2004 is realized by flash memory (e.g., NAND or ONENAND memory), but it is certainly contemplated that other memory types may be utilized as well. Although it may be possible to execute the code from the non-volatile memory 2004, the executable code in the non-volatile memory 2004 is typically loaded into RAM 2008 and executed by one or more of the N processing components 210.
The N processing components 2010 in connection with RAM 2008 generally operate to execute the instructions stored in non-volatile memory 2004. As one of ordinarily skill in the art will appreciate, the N processing components 2010 may include a video processor, modem processor, DSP, graphics processing unit (GPU), and other processing components.
The transceiver component 2012 includes N transceiver chains, which may be used for communicating with external devices via wireless networks. Each of the N transceiver chains may represent a transceiver associated with a particular communication scheme. For example, each transceiver may correspond to protocols that are specific to local area networks, cellular networks (e.g., a CDMA network, a GPRS network, a UMTS networks), and other types of communication networks.
It should be recognized that
Non-transitory computer-readable medium 2004 includes both computer storage medium and communication medium including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer.
System 75
Server 75 may be embodied by a system as particularly shown in
In some embodiments, the server computer is an Industrial PC that contains a standard Windows PC and a Programmable Logic Controller (PLC). The PLC controls the actuator 30 and media capturing device 40 movements, as well as the turntable 55 and lighting system 60 (such as LED colour and brightness).
The components of the server 75 can be configured in a variety of ways. The components can be implemented entirely by software to be executed on standard computer server hardware, which may comprise one hardware unit or different computer hardware units distributed over various locations, some of which may require the communications network 20 for communication. A number of the components or parts thereof may also be implemented by application specific integrated circuits (ASICs) or field programmable gate arrays.
In the example shown in
The server 75 includes at least one or more of the following standard, commercially available, computer components, all interconnected by a bus 3035:
The server 75 includes a plurality of standard software modules, including:
Advantageously, the database 3016 forms part of the computer readable data storage 3024. Alternatively, the database 3016 is located remote from the server 75 shown in
Together, the web server 3038, scripting language 3040, and SQL modules 3042 provide the server 75 with the general ability to allow the other components of the system 10 to communicate with the server 75 and in particular to provide data to and receive data from the database 3016. It will be understood by those skilled in the art that the specific functionality provided by the server 75 to such users is provided by scripts accessible by the web server 3038, including the one or more software modules 3022 implementing the processes performed by the server 75, and also any other scripts and supporting data 3044, including markup language (e.g., HTML, XML) scripts, PHP (or ASP), and/or CGI scripts, image files, style sheets, and the like.
The boundaries between the modules and components in the software modules 3022 are exemplary, and alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into submodules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or submodule. Furthermore, the operations may be combined or the functionality of the operations may be distributed in additional operations in accordance with the invention. Alternatively, such actions may be embodied in the structure of circuitry that implements such functionality, such as the micro-code of a complex instruction set computer (CISC), firmware programmed into programmable or erasable/programmable devices, the configuration of a field-programmable gate array (FPGA), the design of a gate array or full-custom application-specific integrated circuit (ASIC), or the like.
Each of the blocks of the flow diagrams of the processes of the server 75 may be executed by a module (of software modules 3022) or a portion of a module. The processes may be embodied in a non-transient machine-readable and/or computer-readable medium for configuring a computer system to execute the method. The software modules may be stored within and/or transmitted to a computer system memory to configure the computer system to perform the functions of the module.
The server 75 normally processes information according to a program (a list of internally stored instructions such as a particular application program and/or an operating system) and produces resultant output information via input/output (I/O) devices 3030. A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. A parent process may spawn other, child processes to help perform the overall functionality of the parent process. Because the parent process specifically spawns the child processes to perform a portion of the overall functionality of the parent process, the functions performed by child processes (and grandchild processes, etc.) may sometimes be described as being performed by the parent process.
Throughout this specification, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising”, will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.
The reference to any prior art in this specification is not, and should not be taken as, an acknowledgment or any form of suggestion that the prior art forms part of the common general knowledge.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2018901876 | May 2018 | AU | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/AU2019/050524 | 5/28/2019 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2019/227143 | 12/5/2019 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 9412203 | Garcia, III et al. | Aug 2016 | B1 |
| 10122997 | Sheffield | Nov 2018 | B1 |
| 20150029359 | Nenonen | Jan 2015 | A1 |
| 20150138320 | El Daher | May 2015 | A1 |
| 20160156829 | Takamori | Jun 2016 | A1 |
| 20190073760 | Wang | Mar 2019 | A1 |
| 20190289196 | Kato | Sep 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 3421930 | Jan 2019 | EP |
| 3609669 | May 2001 | JP |
| 2002344775 | Nov 2002 | JP |
| 2013225733 | Oct 2013 | JP |
| WO 0163354 | Aug 2001 | WO |
| WO 2017146202 | Aug 2017 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20210227181 A1 | Jul 2021 | US |
