Patent Application
20180213196
The present invention relates to a method of projection mapping for projecting and mapping a video or a picture onto a three-dimensional target object, and more particularly, to a method of projection mapping that enables quick modification of the image to be projected directly on an electronic device when there are changes in the shape of the target object or in the relative position between the target object and a projection device.
Projection mapping is a projection technology, which uses a projection apparatus to project a video onto the surfaces of a physical object to present a fine artwork or an advertisement. By projecting a particularly designed video on the surfaces of a physical object, such as the walls of a building, a stage, an exhibition place or a relatively large product, for example an automobile, to create special changeful colors, patterns or other visual effects. Recently, projection mapping has become a widely welcome and popular live visual performance technology used in a verity of performing activities.
According to the currently known projection mapping technology, an angle of view of a projector relative to the physical object to be projected on, or the target object, is recorded firstly, and then a video is designed to be projected in a direction matching the recorded angle of view. Further, the position and the contour of the target object to be correspondingly presented in the designed video is calculated, and portions of the video other than the image of the target object are blacked out. Finally, the finalized video is projected from the corresponding projector onto the target object, making it looks like the video is played on the surfaces of the target object.
There are some limits to the current projection mapping technology. First, since the angle of view of the projector relative to the target object is first decided to enable the design of the video to be projected in the same direction of the angle of view, the recorded angle of view is fixed and not changeable. In the event the angle of view of the projector relative to the target object is changed, the entire video must be re-designed. Similarly, in the event there is any difference in the existing contour and position of the target object after the video is designed, the previously designed video must also be re-designed, which is of course troublesome. Second, each video is designed for projecting onto the target object in a direction matching the previously decided angle of view, so that a two-dimensional video can be correspondingly shown on the three-dimensional surfaces of the target object. In the event different videos are to be projected onto the same target object, it is also necessary to re-designed all the videos.
A primary object of the present invention is to provide a method of projection mapping, according to which, when there is any change in an orientation, a position and/or an angle of a projection device relative to a target object onto which an image is to mapped, and even any change in a contour of the target object, a video to be projected on the target object can be directly modified on an electronic device in correspondence with such change to reduce the time and cost for re-designing the video for projection mapping.
Another object of the present invention is to provide a method of projection mapping, according to which, a plurality of videos for projection mapping onto a target object in different directions can be quickly produced and projected to reduce possible projection dead angles caused by the shape of the target object.
A further object of the present invention is to provide a method of projection mapping, according to which, a plurality of videos for projection mapping onto a target object at the same time in different directions can be quickly produced and mapped onto the entire surfaces of the target object.
To achieve the above and other objects, the method of projection mapping according to a first embodiment of the present invention for mapping an image onto a target object includes a modeling step, an image capturing step, a mapping step, an adjusting step, a converting step, a projecting step, a re-capturing step and a re-mapping step.
In the modeling step, a three-dimensional (3D) model is built on or imported into an electronic device, and the 3D model has a shape substantially the same as that of the target object. In the present invention, the building of the 3D model is not particularly limited to any specific way. The 3D model can be manually built by using the currently known 3D drawing software along with pictures of the target object, or by laser scanning, ultrasonic scanning and so on, or by using pictures taken with a drone in different directions.
In the image capturing step, an image containing the 3D model viewed at a specific angle of view is captured using the electronic device.
After the image is established, the mapping step and the adjusting step are performed. In the mapping step, a projection device is used to map the image onto the target object in a direction matching the angle of view. In practical implementing the present invention, there is not any particular limit to the number and the type of the projection device.
In the adjusting step, the 3D model can be rotated, shifted and/or re-sized directly on the electronic device until the contour of the 3D model in the image projected from the projection device fully matches the contour of the target object. According to the present invention, there is not any particular limit to the way for determining whether the contour of the 3D model has already matched the contour of the target object.
In the converting step, after the contour of the 3D model has matched the contour of the target object, a video built in the electronic device or imported from other data source is converted into an attachment image. In some embodiments, the video is a pattern for projection mapping. Then, in the projecting step, the attachment image is put onto the surface of the 3D model that has been well adjusted in the adjusting step, so that the attachment image and the 3D model together form a projection model. In the present invention, there is not any particular limit to the sequence of performing other steps other than the converting step and the projecting step.
Thereafter, the re-capturing step and the re-mapping step are performed. In the re-capturing step, a projection image containing the projection model is captured at the angle of view on the electronic device. Then, in the re-mapping step, the projection image is projected by the projection device at the angle of view to map onto the target object.
To achieve the above and other objects, the method of projection mapping according to a second embodiment of the present invention for mapping an image onto a target object includes a modeling step, a converting step, a projecting step, an image capturing step, a mapping step and an adjusting step.
Unlike the first embodiment, in the second embodiment, the modeling step, the converting step and the projecting step are sequentially performed before the image capturing step, the mapping step and the adjusting step; and the re-capturing step and the re-mapping step are omitted from the method according to the second embodiment. However, even with the change in the sequence of performing the steps of the method, the modeling step and the converting step in the second embodiment are the same as those in the first embodiment.
According to the second embodiment, in the projecting step, the attachment image is put onto the surface of the 3D model to together form a projection model; in the image capturing step, an image containing the projection model viewed at a specific angle of view is captured using the electronic device; and in the mapping step, a projection device is used to project and map the image containing the projection mode onto the target object in a direction matching the angle of view.
At last, in the adjusting step, the projection model that is seen on the electronic device is rotated, shifted and/or re-sized until the projection model contained in the image projected from the projection device has a contour completely matching the contour of the target object.
To achieve the above and other objects, the method of projection mapping according to a third embodiment of the present invention for mapping an image onto a target object is established on the basis of the first or the second embodiment and further includes a re-adjusting step, in which the projection model is adjusted again when an orientation, an angle and/or a position of the projection device relative to the target object is changed, such that the projection model contained in the image projected from the projection device matches the contour of the target object again. In the present invention, there is not any particular limit to the way for detecting and sensing the changes in the position, orientation and/or viewing angle of the projection device relative to the target object.
The method of projection mapping according to the third embodiment of the present invention further includes a re-modeling step, in which the contour of the 3D model is modified in correspondence with any difference between an original contour and an after-deformation contour of the target object, such that the modified contour of the 3D model is the same as the after-deformation contour of the target object; and then all other steps are performed again until the contour of the 3D model contained in the image fully matches the contour of the target object.
According to the present invention, in the image capturing step in the above described embodiments, a background removal process can be performed on the captured image, such that only the contour and the pattern of the projection model contained in the image capture at the angle of view are presented in the image, and some unnecessary portions on the image can be omitted as much as possible to thereby enhance the quality of the image projected from the projection device. In some other embodiments, the background removal process enables the electronic device to conveniently determine the contour of the projection model and the 3D model using some programs.
According to a fourth embodiment of the present invention established on the basis of the previous embodiments, the projection device includes a first projection unit and a second projection unit, and a first angle of view and a second angle of view are used in the image capturing step. The first projection unit projects a first image at the first angle of view to map the first image onto the target object and the second projection unit projects a second image at the second angle of view to map the second image onto the target object, such that a common mapping zone, onto where both the first and the second image are mapped, is created on the target object. With the common mapping zone, it is possible to reduce projection dead angles caused by a shape of the target object that includes raised and recessed areas.
According to a fifth embodiment of the present invention established on the basis of the previous embodiments, the projection device includes a first projection unit, a second projection unit and a third projection unit, and a first angle of view, a second angle of view and a third angle of view are used in the image capturing step. The first, second and third projection units are set surrounding the target object at three different angular positions relative to the target object for projecting a first, a second and a third projection image at the first, the second and the third angle of view, respectively, to map the first, the second and the third image onto the target object; and the first, the second and the third image together cover an entire outer surface around the target object to increase the areas on the target object that are mapped by the attachment image.
In summary, all the embodiments of the method of the present invention are established to enable quick re-design of the video for projection mapping when there is any change in the orientation, position and angle of the projection device relative to the target object. The first step in the method according to all the embodiments is the modeling step, in which a 3D model is built on the electronic device. Thereafter, the video to be present on the target object is converted into the attachment image for putting on the surface of the 3D model to form the projection model. Finally, the projection image containing the projection model is captured in the direction matching the angle of view for projection mapping onto the target object via the projection device. According to the conventional projecting mapping technique, the angle of view of the video is determined first, and other subsequent processes are performed accordingly. Unlike the conventional projection mapping technique, in the method of projection mapping according to the present invention, in the event there is any change in the orientation, position and angle of the projection device relative to the target object and the video for the projection mapping must be reset, the user needs only to adjust the video in correspondence with the relative changes between the target object and the projection device and to reset the angle of view for capturing the image containing the projection model directly on the electronic device.
Further, in the event the shape of the target object is changed, the user needs only to change the shape of the 3D model in the modeling step to correspond to the changed shape of the target object, and to perform the image capturing step and other subsequent steps again without the need of producing the video again in the starting step of selecting a new angle of view, which is otherwise necessary in the conventional projection mapping technique.
Finally, when it is desired to increase the areas on the target object that can present the attachment image to reduce possible projection dead angles, the user needs only to use a plurality of the projection units of the projection device and to capture in the image capturing step a plurality of images at a plurality of angles of view, and then projects the images onto the target object at the same time.
The structure and the technical means adopted by the present invention to achieve the above and other objects can be best understood by referring to the following detailed description of the preferred embodiments and the accompanying drawings, wherein
The present invention will now be described with some preferred embodiments thereof and by referring to the accompanying drawings. For the purpose of easy to understand, elements that are the same in the preferred embodiments are denoted by the same reference numerals.
Please refer to
As shown in
It is also to be noted that, in the present invention, the electronic device 201 is not particularly limited to any specific type. The electronic device 201 can include but not limited to a desktop computer, a notebook computer, a mobile phone and a tablet computer. Further, in the present invention, the building of the 3D model 202 is also not particularly limited to any specific way. In practical implementation of the present invention, the 3D model 202 can be built by using the currently known 3D drawing software along with pictures of the target object 10, or by laser scanning, ultrasonic scanning and so on, or by using pictures taken with an all-directional camera mounted on a drone. Any other way that is not mentioned herein but can be used to create the 3D model 202 of the target object 10 on the electronic device 201 should be included in the scope of the present invention.
After the 3D model 202 is built, the method goes to the image capturing step 21. In the image capturing step 21, an image 210 containing the 3D model 202 viewed at a specific angle of view 222 is captured using the electronic device 201, i.e. the computer. The captured image 210 can be reviewed and confirmed on a screen of the computer. The angle of view 222 can be freely selected according to actual need of use. In the illustrated first embodiment, the angle of view 222 is, but not limited to, an angular direction displayed on the screen of the electronic device 201. For the purpose of convenient representation, the angle of view 222 is not shown in
In the image capturing step 21, the image 210 is further subjected to a background removal process 21a, so that only the contour of the 3D model 202 can be seen in the image 210 captured at the angle of view 222. In the present invention, there is not any particular limit to the way for the background removal process 21a. For example, the conventional way to show black contour on white background or any other technique for removing the background from an image to highlight an object all can be utilized to complete the background removal process 21a, so that the visible part in the image 210 is consistent with the 3D model 202. However, according to the present invention, the background removal process 21a is not necessarily performed in the image capturing step 21. In some other embodiments, the background removal process 21a can be performed in other steps of the method of the present invention or even be omitted. In some operable embodiments of the present invention, the computer can input an image captured by a camera unit associated with the projection device 220, so that a user can verify directly on the computer as how the projection device 220 is actually oriented to the target object 10 and to simulate the look in which the captured image 210 is projected and mapped onto the target object 10.
Please refer to
Then, please refer to
In the adjusting step 23, the 3D model 202 can be rotated, shifted and/or re-sized directly on the computer until the contour of the 3D model 202 in the image 210 projected from the projection unit 221 fully matches the contour of the target object 10. Thereafter, other subsequent steps can be performed.
It is to be noted that, in the illustrated first embodiment, the adjusting step 23 is performed on the electronic device 201 to directly rotate, shift or re-size the 3D model 202 seen on the screen of the electronic device 201 to affect the 3D model 202 in the image 210 captured at the angle of view 222 and accordingly, the 3D model 202 that is projected from the projection unit 221. That is, the rotation, shifting and re-sizing for the purpose of adjusting the contour of the 3D model 202 are not performed on the 3D model 202 that is projected from the projection unit 221. Wherein, the adjustment can be the rotation, shifting and re-sizing done on the 3D model 202 in the image 210, or can simply be any change of the position of an observation point in the captured image 10.
Also, according to the present invention, there is not any particular limit to the way for determining whether the contour of the 3D model 202 has already matched the contour of the target object 10. In the illustrated first embodiment, it is the user who directly manually determines whether the contour of the 3D model 202 in the image 210 matches the target object 10. However, in some other embodiments, the user may use a relevant program built in the computer, such as an image analysis program, to work on the image 210 projected by the projection unit 221 and directly captured by the camera unit on the projection device 220, as well as the image of the target object 10 that is observed at the location of the projection unit 221, so that the computer can automatically and instantly determine whether the contours of the 3D model 202 and the target object 10 match each other or not, and then make necessary modifications.
Please refer to
Please refer to
In the re-capturing step 26, a projection image 211 containing the projection model 243 is captured at the angle of view 222 on the electronic device 201. Then, in the re-mapping step 27, the projection image 211 is projected by the projection unit 221 at the angle of view 222 to be mapped onto the target object 10, so that the 3D model 202 in the image 210 previously projected from the projection unit 221 is changed to the projection model 242. In this manner, the attachment image 242 is presented on the target object 10.
As can be seen from the above description, in the projection model 243 contained in the last image 210 projected by the projection unit 221, the shape of the presented attachment image 242 has already been adjusted in the adjusting step 23 to match the surface shape of the 3D model 202 in the projecting step 25. Therefore, when the user intends to change the video 241 to be projected by the projection device 220, the user can directly replace the attachment image 242 on the projection model 242 with another desired video and make necessary adjustments to complete the change without the need to build a new 3D model for showing the desired video 241.
It is understood the above descriptions with reference to
In the above-described first embodiment, the converting step 24 and the projecting step 25 are performed after the adjusting step 23 and the re-capturing step 26 and the re-mapping step 27 are performed after the projecting step 25. Unlike the first embodiment, in the method according to the second embodiment of the present invention, the converting step 24 and the projecting step 25 are performed immediately after the modeling step 20 and the re-capturing step 26 and the re-mapping step 27 are omitted. Since the modeling step 20 and the converting step 24 in the second embodiment is substantially the same as those in the first embodiment, they are not repeatedly described herein. The following descriptions will directly go to the projecting step 25.
After the modeling step 20 shown in
Then, the method goes to the image-capturing step 21. In the image capturing step 21, an image 210 containing the projection model 243 viewed at a specific angle of view 222 is captured using the electronic device 201, i.e. the computer. Similarly, in the second embodiment, the angle of view 222 is, but not limited to, an angular direction displayed on the screen of the electronic device 201. However, for the purpose of convenient representation, the angle of view 222 is not shown in
Please refer to
As shown in
Please refer to
As can be seen in
Referring to
In the event the performing of the re-adjusting step 28 still fails to match the contour of the projection model 243 contained in the image 210 with the contour of the target object 10, it can be determined the non-matching is caused by some change in the contour of target object 10 that was used in the modeling step 20. That is, an original contour 203 of the target object 10 at the time the modeling step 20 was performed has changed to an after-deformation contour 204. Under this circumstance, the re-modeling step 29 must be performed. As shown in
The following are some more details about the practical application of the method of the present invention. In some other embodiments similar to the previously embodiments, when performing the image capturing step 21, the background removal process 21a is further performed before the method goes to the mapping step 22, so that the contour of the projection model 243 can be more easily determined or some unnecessary portions on the image 210 can be omitted as much as possible when mapping the image 10 onto the target object 10 to thereby enhance the quality of the image 210 projected from the projection device 220, making the image 210 present only the contour and the pattern of the projection model 243 that is captured at the angle of view 222.
In the present invention, the projection device 220 can be used in differently ways. Please refer to
To increase areas on the target object 10 that can present the projected video 241 for people to watch the attachment image 242 (not shown in
The present invention has been described with some preferred embodiments thereof and it is understood that the preferred embodiments are only illustrative and not intended to limit the present invention in any way and many changes and modifications in the described embodiments can be carried out without departing from the scope and the spirit of the invention that is intended to be limited only by the appended claims.
