Patent Application
20170124770
The invention relates to visualizing a virtual model. More specifically, the invention relates to visualizing and interacting with the virtual model.
There is an increasing trend to display the real products digitally with the help of images, videos and/or animations. A user may not be aware of existing or new features in a real consumer product. Even in real situation, when users visit a physical establishment to see a real product, say a car, the users perform known and general interaction like opening of side door, moving steering wheel etc, however seek assistance of salesman to explain particular operation, feature or seek guidance as to how to use the product for easy understanding of the product. For example, a user may want to understand airbag operation, how adjust seats, etc. in case of car. Further, almost all product manufacturer and independent product reviewers make videos or shoot videos for explaining particular operation, feature or to guide as to how to use the product. Examples of independent product reviewers are sites like cnet.com, survey sites which explains features of some real object, say features of mobile, how to operate certain functionalities in a refrigerator etc. through video shoots. A lot of money, time and effort are usually spent to make such video shoots. Further, manufacturers provide a user guide and/or a features booklet to read, and a certain fraction of users usually search for videos in web to learn and understand about a new or existing product or its features, and a lot of time is spent in the process to understand a small functionality or feature. Additionally user may be lazy to ask multiple questions. In some implementation, such as discussed in Indian patent application Nos. 2253/DEL/2012, 332/DEL/2014 and PCT application PCT/IN2013/000448, filed by the same applicants as of this application, viewing and performing user-controlled interactions with one or more 3D models representing real products is carried out to visualize, and gain active product information. However, a user might not know what sequence of steps needs to be followed to get a desired result such as getting ice crushed in a refrigerator, steps to change gears etc in understanding detailed operations or functionality very quickly and accurately. Additionally, a manufacturer may want to deliberately promote or make users aware of certain advanced or differentiating features of a product in a virtual experience, while not limiting the freedom of performing interactions with a digital virtual model of object representing real product as per user choice.
The object of the invention is to provide a cost-effective and easy to use solution for explaining/ demonstrating particular operation, feature or to guide as to how to use a real product.
The object of the invention is achieved by a method of claim 1, a system of claim 34 and a computer program product of claim 35.
According to one embodiment of the method, the method includes:
According to another embodiment of the method, wherein other part/s of virtual object is available for user controlled interactions while such operation is being performed.
According to yet another embodiment of the method, wherein the demonstration of the particular functionality comprises demonstration of multiple steps, wherein the steps are controlled by pausing the step/s and/or replaying the step/s.
According to one embodiment of the method, wherein the object comprises an electronic screen and correspondingly the 3D model comprises a virtual electronic display, interacting with the 3D model for understanding functionality to navigate to an application in the 3D model and/or understating functionality of the application by automatically demonstrating the required step in ordered manner, wherein such demonstration is shown by change in graphics and/or multimedia data on the virtual electronic display in synchronization with automatically operating the part/s of virtual 3D model.
According to another embodiment of the method, wherein two or more 3D models of two or more objects which are communicatively coupled to each other, wherein interacting with 3D model/s for understanding a particular functionality pertaining communication among the 3D model/s by automatically demonstrating steps of operation of part/s and/or movement of 3D model/s and/or change in GUI/s of virtual electronic display or multimedia data of 3D model/s in ordered manner.
According to yet another embodiment of the method, wherein interaction to understand functionality of 3D model with gesture control comprises:
According to one embodiment of the method, wherein the 3D model comprises inflatable and/or deflatable and/or folding part/s, and interacting with the part/s to understand their inflation and/or deflation and/or folding feature by automatically demonstrating the inflation and/or deflation and/or folding of the part/s in ordered manner.
According to another embodiment of the method, wherein new 3D model/s of new object/s are introduced in interactive manner and/or isolated manner with the existing 3D model for automatically demonstrating the particular functionality in an ordered manner.
According to yet another embodiment of the method, wherein demonstration of the operation is further guided by text or voice, wherein the text or voice refers to the steps involved in performance of the operation.
According to one embodiment of the method, wherein a virtual character is introduced and the voice is lisped and/or expressed with/without facial expression and/or body posture.
According to another embodiment of the method, wherein the interaction command comprises extrusive interaction and/or intrusive interactions and/or a time bound change based interaction and/or a real environment mapping based interaction and combination thereof, as per user choice and/ or as per characteristics, state and nature of the said object, wherein the time bound changes refers to representation of changes in 3D model demonstrating change in physical property of object in a span of time on using or operating of the object, and real environment mapping refers to capturing a real time environment, mapping and simulating the real time environment to create a simulated environment for interacting with the 3D model.
According to yet another embodiment of the method, wherein the interaction commands are adapted to be received before and/or during and/or after interactions for understanding particular functionality of the 3D model.
According to one embodiment of the method, wherein the extrusive interaction comprises at least one of:
According to another embodiment of the method, wherein functioning of light emitting part is shown by a video as texture on surface of said light emitting part to represent lighting as dynamic texture change.
According to yet another embodiment of the method, the intrusive interactions comprises at least one of:
According to one embodiment of the method, wherein the real environment mapping based interactions comprises at least one of:
According to another embodiment of the method, wherein the interaction comprises liquid and fumes flow based interaction for visualizing liquid and fumes flow in the 3D model with real-like texture in real-time.
According to yet another embodiment of the method, wherein the interaction comprises immersive interactions, the immersive interactions are defined as interactions where users visualize their own body performing user-controlled interactions with the virtual computer model.
According to one embodiment of the method, wherein displaying of new interaction/s to the 3D-model while previously one or more interaction has been performed or another interaction/s is being performed on the 3-D model.
According to another embodiment of the method, wherein rendering of corresponding interaction to 3D model of object in a way for displaying in a display system made of one or more electronic visual display or projection based display or combination thereof.
According to yet another embodiment of the method, wherein the display system is a wearable display or a non-wearable display or combination thereof.
According to one embodiment of the method, wherein the non-wearable display comprises electronic visual displays such as LCD, LED, Plasma, OLED, video wall, box shaped display or display made of more than one electronic visual display or projector based or combination thereof.
According to another embodiment of the method, wherein the non-wearable display comprises a pepper's ghost based display with one or more faces made up of transparent inclined foil/screen illuminated by projector/s and/or electronic display/s wherein projector and/or electronic display showing different image of same virtual object rendered with different camera angle at different faces of pepper's ghost based display giving an illusion of a virtual object placed at one places whose different sides are viewable through different face of display based on pepper's ghost technology.
According to yet another embodiment of the method, wherein the wearable display comprises head mounted display, the head mount display comprises either one or two small displays with lenses and semi-transparent mirrors embedded in a helmet, eyeglasses or visor. The display units are miniaturised and may include CRT, LCDs, Liquid crystal on silicon (LCos), or OLED or multiple micro-displays to increase total resolution and field of view.
According to one embodiment of the method, wherein the head mounted display comprises a see through head mount display or optical head-mounted display with one or two display for one or both eyes which further comprises curved mirror based display or waveguide based display.
According to another embodiment, wherein the head mounted display comprises video see through head mount display or immersive head mount display for fully 3D viewing of the 3D-model by feeding rendering of same view with two slightly different perspective to make a complete 3D viewing of the 3D-model.
According to yet another embodiment of the method, wherein the 3D model moves relative to movement of a wearer of the head-mount display in such a way to give to give an illusion of 3D model to be intact at one place while other sides of 3D model are available to be viewed and interacted by the wearer of head mount display by moving around intact 3D model.
According to one embodiment of the method, wherein the display system comprises a volumetric display to display the 3D model and interaction in three physical dimensions space, create 3-D imagery via the emission, scattering, beam splitter or through illumination from well-defined regions in three dimensional space, the volumetric 3-D displays are either auto stereoscopic or auto multiscopic to create 3-D imagery visible to an unaided eye, the volumetric display further comprises holographic and highly multiview displays displaying the 3D model by projecting a three-dimensional light field within a volume.
According to another embodiment of the method, wherein the display system comprises more than one electronic display/projection based display joined together at an angle to make an illusion of showing the 3D model inside the display system, wherein the 3D model is parted off in one or more parts, thereafter parts are skew in shape of respective display and displaying the skew parts in different displays to give an illusion of 3d model being inside display system.
According to yet another embodiment of the method, wherein the input command is received from one or more of a pointing device such as mouse; a keyboard; a gesture guided input or eye movement or voice command captured by a sensor, an infrared-based sensor; a touch input; input received by changing the positioning/orientation of accelerometer and/or gyroscope and/or magnetometer attached with wearable display or with mobile devices or with moving display; or a command to a virtual assistant.
According to one embodiment of the method, wherein command to the said virtual assistant system is a voice command or text or gesture based command, wherein virtual assistant system comprises a natural language processing component for processing of user input in form of words or sentences and artificial intelligence unit using static/dynamic answer set database to generate output in voice/text based response and/or interaction in 3D model.
The Cube can be rotated in different orientation, where change in orientation will work as rotation scene in different plane in such a way at particular orientation of cube, particular image displayed so depending on the orientation, the image is cut into one piece, two piece or three piece. These different pieces wrap themselves to fit in different display in such a way so that the cube made of such display displays a single scene which gives a feeling that the object is inside the cube. Apart from cube, even hexagonal, pentagonal, sphere shaped display with same technique can show the 3D model of the object giving feel that the 3D model is inside the display
As shown in
Communication interface 1602 functions to allow client device 1612 to communicate with other devices, access networks, and/or transport networks. Thus, communication interface 1602 may facilitate circuit-switched and/or packet-switched communication, such as POTS communication and/or IP or other packetized communication. For instance, communication interface 1602 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 1602 may take the form of a wireline interface, such as an Ethernet, Token Ring, or USB port. Communication interface 1602 may also take the form of a wireless interface, such as a Wifi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or LTE). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 102 Furthermore, communication interface 1502 may comprise multiple physical communication interfaces (e.g., a Wifi interface, a BLUETOOTH® interface, and a wide-area wireless interface).
User interface 1603 may function to allow client device 1612 to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface 1603 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, joystick, microphone, still camera and/or video camera, gesture sensor, tactile based input device. The input component also includes a pointing device such as mouse; a gesture guided input or eye movement or voice command captured by a sensor, an infrared-based sensor; a touch input; input received by changing the positioning/orientation of accelerometer and/or gyroscope and/or magnetometer attached with wearable display or with mobile devices or with moving display; or a command to a virtual assistant.
User interface 1603 may also include one or more output components such as a cut to shape display screen illuminating by projector or by itself for displaying objects, cut to shape display screen illuminating by projector or by itself for displaying virtual assistant.
User interface 1603 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and/or other similar devices, now known or later developed. In some embodiments, user interface 1603 may include software, circuitry, or another form of logic that can transmit data to and/or receive data from external user input/output devices. Additionally or alternatively, client device 112 may support remote access from another device, via communication interface 1602 or via another physical interface.
Processor 1604 may comprise one or more general-purpose processors (e.g., microprocessors) and/or one or more special purpose processors (e.g., DSPs, CPUs, FPUs, network processors, or ASICs).
Data storage 1605 may include one or more volatile and/or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor 1604. Data storage 1605 may include removable and/or non-removable components.
In general, processor 1604 may be capable of executing program instructions 1607 (e.g., compiled or non-compiled program logic and/or machine code) stored in data storage 1505 to carry out the various functions described herein. Therefore, data storage 1605 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by client device 1612, cause client device 1612 to carry out any of the methods, processes, or functions disclosed in this specification and/or the accompanying drawings. The execution of program instructions 1607 by processor 1604 may result in processor 1604 using data 1606.
By way of example, program instructions 1607 may include an operating system 1611 (e.g., an operating system kernel, device driver(s), and/or other modules) and one or more application programs 1610 installed on client device 1612 Similarly, data 1606 may include operating system data 1609 and application data 1608. Operating system data 1609 may be accessible primarily to operating system 1611, and application data 1608 may be accessible primarily to one or more of application programs 1610. Application data 1608 may be arranged in a file system that is visible to or hidden from a user of client device 1612.
Application Data 1608 includes 3D model data that includes three dimensional graphics data, texture data that includes photographs, video, interactive user controlled video, color or images, and/or audio data, and/or virtual assistant data that include video and audio.
In one embodiment as shown in
The user-controlled interactions unit 131 includes logic for making group of functioning of part/s of 3D model which will be performed in sequence or parallel order upon getting input to understand some particular operation of 3D model of the object. The user-controlled interactions unit 131 includes logic for performing extrusive and intrusive interactions, for performing liquid and fumes flow interactions, for performing addition interactions, for performing deletion interactions, for performing time-bound changes based interactions, for performing environment mapping based interactions, for performing interaction for getting un-interrupted view of internal parts using transparency-opacity effect, for performing immersive interactions, for performing inter-interactions, for performing engineering disintegration interactions with the displayed 3D model. The user control interaction unit 131 is the main logic that utilizes different sub-system 133, database 132, and according to user input generates output and a corresponding scene or user-controlled interaction response is rendered using a 3D rendering engine in real time/near real time.
The texture data 132b includes textures obtained from photographs, use of video file as texture, color or images. Texture data include texture for 3D model and its functioning surfaces such as for showing the function of digital /electronic part. 3D model can be textured using computer generated colors, brightness, hue, shades as well. It may be added in 3d model generation environment or during the rendering by using libraries for color, shades or other properties which are associated with rendering engine. For providing realistic look texture may be prepared from real photographs, images, videos. Video is used as texture in the 3D model only for that surface/s which corresponds to functioning part such as light-emitting parts in the real object. The use of video enhances reality in displaying dynamic texture changes for function part for lighting effect (one of extrusive and intrusive interactions). Multiple textures pre-calibrated on 3D model UV layouts can be stored as texture data for one/same surface in the database 132, which are called for or fetched dynamically by the user-controlled interaction unit 131 during the user-controlled interactions.
According to another embodiment for the texturing of a three-dimensional 3D model of a 3D object using photograph and/or video, the method comprising:
The user-controlled interaction support sub-system 133 includes a sound engine for producing sound as per user-controlled interaction, a motion library responsible for animation of the virtual product assistant which include rigging/animation data of 3D virtual model motion/expression or animation of one or more parts in the 3D model such rotating the wheel continually. The virtual operating sub-system for providing functionality of operation of electronic or digital parts in the displayed 3D-model/s depending on the characteristics, state and nature of displayed object. It stores the functionality of GUI look and output for different input via 3D model part/s or GUI itself or other kind of inputs and also make response for different input of GUI to make a response for part/parts of 3D model or other GUI based output an Artificial Intelligence (AI) engine for decision making and prioritizing user-controlled interactions response, a scene graph for primarily for putting more than one 3D object in scene say more than two 3D model of bikes, one bike or one 3D model etc, a terrain generator for generating surrounding, in case say 3D model is placed in some environment, lighting and shadow for generating the effect of light of 3D model, a shader for providing visual effects such as colour shades, and a physics/simulation engine for generating simulation effect, for example for showing the functioning of folding the roof of car, to show wrinkles in folding material.
According to another embodiment, for an example, “A user not only can check the laptop looks and compare specification, but can understand the functionalities of laptop just in real life scenario such as switching it to judge start-up time, which is the real start-up time for the said product, if the product would have been started in real life set-up. The digital interaction/electronic display interaction is shown on some surface of 3D model. Here as the control when reach over the GUI of digital/electronic interaction surface, the control change and it goes from the 3D model to digital/electronic interaction layer for example; drag command to virtual mobile display goes to GUI but not to the 3D model which make change in GUI possible.
According to another embodiment, during mirror effect and immersive interactions, the user-controlled interactions unit 131 uses live video input from camera, which is directly passed to message handler. The message handler further transmits the input or interaction command to the user-controlled interactions unit 131 for identification and further processing.
Initially, user input can generate a network message, an operating system message, or is direct input. The network message means a command or event generated by the user input which is sent by server software to client software in same machine or any host connected through network for an action by the client. The operating system message is a command or event generated by user input by a device handler to the client software via operating system inter process communication/message queue/or an action by the client device. In the direct input or direct messaging, the device handler and the client software are a single application, hence commands or event are directly bound to the device handler. A message Interpreter interprets the message (command/event) based upon the context and calls the appropriate handler for an action. Message handler, or event handler are logic blocks associated with an action for controls. User input can be provided using infrared based sensor, voice command based sensor, camera based sensor, or touch based screens.
According to another embodiment, Virtual assistant may be displayed in same 3d graphics environment or it may be displayed in a separate environment. It may have different virtual assistant sub system which included data base related to 3d model data of Virtual assistant, texture data, rigging & animation data, logic for movement according to output, while in case of 2d virtual assistant, image/video data and image processing based logic to generate expression. For responding it uses a voice to text convertor, a logic based on NLP and AI along with response set data and text to voice convertor, response in terms of voice/text is generated. The virtual assistant sub system may be a separate unit which in synchronise with 15 (a) is displayed and work together or this system may work together or may be inside the same system by adding its database with 3D model of product database and support logic with user controlled support sub system and user controlled unit for receiving input and generate output using libraries and logic of voice to text convertor, a logic based on NLP and AI along with response set data and text to voice convertor, response in terms of voice/text is generated and display the virtual model.
The virtual product assistant sub-system includes:
Instructions stored in a non-transitory computer readable storage system executable by the one or more processors that upon such execution cause the one or more processors to perform operations comprising:
The output is as per the query with synchronized graphics. The voice input from a microphone is transmitted to the message handler and then passes to the virtual product assistant unit.
The virtual-assistant sub-system gets configured to reply to queries with respect to current specific product displayed on the soft-copy display screen.
For example, a user may ask following queries using the microphone/text when a 3D model of bike of a particular model, say model X is displayed on the soft-copy display device and receive corresponding replies from the virtual product assistant:
Virtual assistant may be of 3d made up of using 3d graphics data, texture data, rigging & morphing/animation data to generate expression. It may be made of 2D graphics and expression is generated by image processing also it may be made of multiple pre recorded/rendered video clips.
According to another embodiment as shown in
Application Programs 1610 includes programs for performing the following steps, when executed over the processor:
Application program 1610 further includes a set of system libraries comprises functionalities for:
Rendering of corresponding interaction to 3D model of object in a way for displaying in a display system made of one or more electronic visual display or projection based display or combination thereof.
The display system can be a wearable display or a non-wearable display or combination thereof.
The non-wearable display includes electronic visual displays such as LCD, LED, Plasma, OLED, video wall, box shaped display or display made of more than one electronic visual display or projector based or combination thereof.
The non-wearable display also includes a pepper's ghost based display with one or more faces made up of transparent inclined foil/screen illuminated by projector/s and/or electronic display/s wherein projector and/or electronic display showing different image of same virtual object rendered with different camera angle at different faces of pepper's ghost based display giving an illusion of a virtual object placed at one places whose different sides are viewable through different face of display based on pepper's ghost technology.
The wearable display includes head mounted display. The head mount display includes either one or two small displays with lenses and semi-transparent mirrors embedded in a helmet, eyeglasses or visor. The display units are miniaturised and may include CRT, LCDs, Liquid crystal on silicon (LCos), or OLED or multiple micro-displays to increase total resolution and field of view.
The head mounted display also includes a see through head mount display or optical head-mounted display with one or two display for one or both eyes which further comprises curved mirror based display or waveguide based display. See through head mount display are transparent or semi transparent display which shows the 3d model in front of users eye/s while user can also see the environment around him as well.
The head mounted display also includes video see through head mount display or immersive head mount display for fully 3D viewing of the 3D-model by feeding rendering of same view with two slightly different perspective to make a complete 3D viewing of the 3D-model. Immersive head mount display shows 3d model in virtual environment which is immersive.
In one embodiment, the 3D model moves relative to movement of a wearer of the head-mount display in such a way to give to give an illusion of 3D model to be intact at one place while other sides of 3D model are available to be viewed and interacted by the wearer of head mount display by moving around intact 3D model.
The display system also includes a volumetric display to display the 3D model and interaction in three physical dimensions space, create 3-D imagery via the emission, scattering, beam splitter or through illumination from well-defined regions in three dimensional space, the volumetric 3-D displays are either auto stereoscopic or auto multiscopic to create 3-D imagery visible to an unaided eye, the volumetric display further comprises holographic and highly multiview displays displaying the 3D model by projecting a three-dimensional light field within a volume.
The input command to the said virtual assistant system is a voice command or text or gesture based command. The virtual assistant system includes a natural language processing component for processing of user input in form of words or sentences and artificial intelligence unit using static/dynamic answer set database to generate output in voice/text based response and/or interaction in 3D model.
Application program 1610 further includes a set of system libraries comprises functionalities for:
Other types of user controlled interactions are as follows:
interactions for colour change of displayed virtual model,
The displayed 3D model is preferably a life-size or greater than life-size representation of real object.
| Number | Date | Country | Kind |
|---|---|---|---|
| 429/DEL/2014 | Mar 2014 | IN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/IN2015/000130 | 3/16/2015 | WO | 00 |
