The present disclosure relates to a design review device, a design review method, and a design review program.
Design review is performed to determine whether a product can actually be manufactured and to review the design and the manufacturing processes of the product. A known device for performing design review may be a design review device that includes a head mount display (hereafter, HMD) for displaying a three-dimensional (3D) model of a product projected in a virtual space and an operator for operating the 3D model displayed in the virtual space.
Patent Literature 1 describes one example of such a design review device including an HMD that may display virtual design elements in a manner superimposed on a shoe model placed in a real environment to determine whether the shoe can be manufactured.
Patent Literature 1: Unexamined Japanese Patent Application Kokai Publication No. 2016-189213
At design review, a user wearing a non-transmissive HMD cannot easily view the real space directly and cannot easily operate an input terminal. A user wearing a semi-transmissive HMD also cannot easily operate an input terminal when the display from the HMD is superimposed on the input terminal. Another operator thus operates the input terminal. The design review involves many operators and is inefficient.
One or more aspects of the present disclosure are directed to a design review device, a design review method, and a design review program that allow efficient design review.
To achieve the above object, a design review device according to the present disclosure includes a communicator to access a data server storing data for a 3D model of a component included in a product, data for a 3D model of a tool to be used in assembling the product, and data for a review sheet including a design review result for the product, a display controller to display in a virtual space a screen for selecting at least one of the 3D model of the component or the 3D model of the tool, and a capturer to detect a user action and select, in response to the user action, the at least one of the 3D model of the component or the 3D model of the tool displayed in the virtual space.
The above aspect of the present disclosure allows efficient design review by detecting a user action and selecting, in response to the user action, at least one of a 3D model of a component or a 3D model of a tool displayed in a virtual space.
A design review device according to one or more embodiments of the present disclosure will now be described with reference to the drawings.
A design review device 100 according to the present embodiment is used for design review for an installation method used in planning and designing the specifications of products including an elevator. The purpose of the design review is to verify whether the assembly work can be actually performed or can be safely performed. In the example described below, the design review device 100 is used to verify whether a sufficient space is provided for the installation work for tightening bolts with a tool or the tool can interfere with other components.
As shown in
The data server 200 includes a hard disk drive (HDD) for storing data. The data server 200 stores a product model database (DB) 210 storing data about products, a tool model DB 220 storing data about tools, a review sheet DB 230 storing data about review sheets, a component model DB 240 storing data about components, and a review model DB 250 storing data about combinations of products for which design review has been performed previously, and associated tools and review sheets.
As shown in
As shown in
As shown in
As shown in
As shown in
The communicator 300 shown in
The glove-shaped capture device 310 in the shape of a glove includes a sensor for detecting the position and the orientation of a hand and a sensor for detecting whether each digit is extended or bent. The glove-shaped capture device 310 detects the position of a hand and the motion of each digit of a user wearing the device, and outputs a signal representing the position and the orientation of the hand and the motion of each digit to the controller 400.
The imaging device 320 captures an image of a target reference marker on a tool to detect the position and the orientation of the tool, and outputs data representing this image to the controller 400. In this example, the imaging device 320 captures an image of a target reference marker 351 on a spanner 350 as a tool shown in
To detect handwriting, the stylus capture device 330 includes an acceleration sensor 331 that detects the motion and the inclination of a stylus, and a pressing force sensor 333 that detects a force applied to the tip of the stylus pressed onto a reference board 332 as shown in
The HMD 340 is a display device worn on a head. The HMD 340 includes a right display for displaying an image viewed by a right eye, a left display for displaying an image viewed by a left eye, and a sensor for detecting the position and the orientation of the head. The HMD 340 is a non-transmissive display device. A user wearing the HMD 340 cannot directly view the real space. As controlled by the controller 400, the right display displays a virtual space image viewed by a right eye, and the left display displays a virtual space image viewed by a left eye to display the 3D image in a virtual space. The HMD 340 displays a 3D virtual space corresponding to the position and the orientation of the head of the user wearing the HMD 340.
The controller 400 shown in
The gesture capturer 410 analyzes a signal output from the glove-shaped capture device 310, captures the position and the orientation of a user's hand and the motion of each digit by integrating twice the acceleration of the hand representing the motion of the user's hand, and stores the captured data into the RAM. The motion of each digit includes the extending or bending motion of each digit. In this example, the gesture capturer 410 recognizes a motion of bending a forefinger as an action of clicking, and a motion of extending a forefinger and a middle finger and moving these fingers horizontally as an action of scrolling.
The tool capturer 420 obtains an image of a tool with a target reference marker captured by the imaging device 320, and stores the obtained image into the RAM. The tool capturer 420 analyzes the image to capture the position and the orientation of the tool. In this example, the tool capturer 420 obtains an image of the spanner 350 with the target reference marker 351 shown in
The handwriting capturer 430 shown in
The display controller 440 controls images displayed in a 3D virtual space on the HMD 340 that is worn by the user. The image includes a 3D model of a product, a 3D model of a tool, a 3D model of a component (or 3D models of components), and a format of a review sheet. The 3D model of the tool is displayed to be superimposed on the tool placed in the real space. The format of a review sheet is displayed to be superimposed on the reference board 332.
The menu controller 450 calls and selects data from the product model DB 210, the tool model DB 220, the review sheet DB 230, and the component model DB 240, and edits the data. The menu controller 450 displays a menu screen shown in
The menu controller 450 shown in
The processing performed by the menu controller 450 will now be described.
(1) Selecting a product model
In response to the user's gesture of clicking “Select product model” displayed on the menu screen, the menu controller 450 calls typical drawings showing products and the product names from the product model DB 210. The menu controller 450 then lists typical drawings showing products and the product names in the virtual space in a scrollable manner as shown in
(2) Selecting a Component
The menu controller 450 scrolls the typical drawings for the components and the component names displayed in the virtual space in response to the user action of instructing scrolling. In response to the user action of clicking to select a target component to be reviewed, the menu controller 450 selects the designated component. When the user clicks “Display details” shown in
(3) Selecting a Tool Model
In response to an action of clicking “Select tool” on the menu screen shown in
(4) Selecting a Review Sheet
In response to the user action of clicking “Select review sheet” on the menu screen shown in
The model operator 460 moves and rotates a 3D model of a component displayed in the virtual space within the range of no contact with 3D models of tools and 3D models of other components in response to a gesture of the user wearing the glove-shaped capture device 310. When the user moves his or her forefinger touching a 3D model of a component vertically and horizontally, the model operator 460 rotates the 3D model of the component displayed in the virtual space. When the user pinches a model with his or her forefinger and thumb, the model is selected as a target to be moved. When the user moves his or her hand pinching the model, the model operator 460 moves the 3D model of the component. When the user clicks a part of a 3D model of a component with his or her middle finger, the display state of the part is switched from hidden to displayed or from displayed to hidden.
The model operator 460 moves and rotates a 3D model of a tool displayed in the virtual space within the range of no contact with 3D models of other tools and 3D models of components in accordance with the position and the orientation of a tool placed in the real space and captured by the imaging device 320.
When the user manipulates a tool placed in the real space to assemble a 3D model of a component with the 3D model of the tool displayed in the virtual space, the model operator 460 moves and rotates the 3D model of the component displayed in the virtual space in response to the user action. When the 3D model of the component comes in contact with the 3D model of other components during the assembly, the model operator 460 determines a failure in the assembly due to interference between the component and another component, and reports an error. When a space in which the 3D model of the tool is moved is at a reference level or lower to cause contact with the 3D model of the component during the assembly, the model operator 460 determines that an insufficient space is provided for using the tool, and reports an error. In this example, the model operator 460 determines that an insufficient space is provided for using the tool and reports an error when the angle at which the spanner 350 as the tool is movable is less than a reference angle.
The review recorder 470 stores a review result written by the user using the stylus capture device 330 into the review sheet DB 230. The review recorder 470 determines the position at which writing has been performed with the stylus capture device 330 on a review sheet displayed to be superimposed on the reference board 332 through the touchscreen sensor included in the reference board 332, and detects the position at which the user has input a review result using the stylus capture device 330 on the review sheet. The review recorder 470 determines the language used for handwriting through pattern recognition of the handwriting captured with the stylus capture device 330, and converts the handwriting data to text data. More specifically, the language is determined through pattern recognition based on whether the handwriting pattern represents alphabetical characters, characters including kanji and hiragana, or only kanji characters. When the handwriting represents alphabetical characters, a specific language is determined based on the spelling. The review recorder 470 stores the language of the text input with the stylus capture device 330 and the review sheet including the text data into the review sheet DB 230.
The design review process performed by the design review device 100 according to the present embodiment will now be described with reference to the flowcharts shown in
The user wears the HMD 340 on the head and the glove-shaped capture device 310 on the hand. In response to an instruction for starting the design review process, the design review device 100 starts the design review process shown in
When the design review process starts, the design review device 100 displays the menu screen shown in
When the product is selected, the menu controller 450 displays typical drawings for component models included in the product and the component names shown in
When a gesture of clicking “Select tool” shown in
As shown in
A review result record process is then started (step S110). When “Select review sheet” is clicked on the menu screen shown in
The processing then returns to the design review process shown in
The design review device 100 with the above structure allows easy selection of a product, a component, and a tool through the user's gesture of scrolling and clicking a 3D model displayed in a scrollable manner. Additionally, a review result is recorded using text data obtained by converting handwriting input with the stylus capture device 330. This structure allows the user wearing the HMD 340 to easily operate the design review device 100 and efficiently perform design review. The language used in the review sheet can be determined to identify the country in which the design review has been performed.
As shown in
As shown in
As shown in
As shown in
The design review device 500 uses the service start date, the training history, and the skill information for each reviewer stored in the personnel information DB 260 and displays different review sheets for each reviewer depending on the skill, the learning level, and the experience. For example, the review sheet DB 230 stores a review sheet A describing a review procedure in detail with pictures and a review sheet B briefly describing a review procedure. Based on the learning level determined from the reviewer's years of experience, the HMD 340 displays the review sheet B for experienced reviewers who have experienced design review for at least a predetermined number of years and the review sheet A for beginners who have experienced design review for less than the predetermined number of years. The reviewer's years of experience are calculated from the service start date stored in the personnel information DB 260. The design review device 500 may use the training history or the skill information as a criterion for displaying either the review sheet A or the review sheet B.
The design review device 500 determines a product to be reviewed based on information representing a facility to which each reviewer belongs, selects candidates of relevant product models and review sheets, and displays the selected candidates on the HMD 340. The reviewer selects a product model and a review sheet from the displayed candidates. The information representing the facility and the product are associated with each other by matching the information representing the facility in the personnel information DB 260 with the information representing the product in the facility information DB 270. The design review device 500 calls a product model to be reviewed by the reviewer from the product model DB 210, a 3D model of a component included in the product from the component model DB 240, a 3D model of a tool used in assembling the product from the tool model DB 220, and a review sheet from the review sheet DB 230, and displays the obtained data on the HMD 340.
The customer spec information DB 280 and the review sheet DB 230 are associated with each other. The customer specifications are displayed on the HMD 340 as reference information every time a review sheet is displayed. The customer spec information DB 280 and the review sheet DB 230 are associated with each other based on the customer ID.
The design review device 500 with the above structure may display a review sheet in accordance with the learning level of a reviewer. Further, the design review device 500 determines a product to be reviewed by a reviewer, selects candidates of relevant product models and review sheets, and displays the selected candidates on the HMD 340. This structure allows the reviewer to easily call the product model and the review sheet.
Modifications
In the embodiments described above, the design review devices 100 and 500 each include the glove-shaped capture device 310 to capture the motions of the user's hand and digits. The design review devices 100 and 500 may include a full-body capture device for capturing the full-body motion of the user. In this case, the design review devices 100 and 500 use a tool 350 associated with a target to capture the full-body motion of the user in the actual assembly work or installation work. Based on the captured full-body motion, the presence of interference, the length of interference, and the history of interference check results for interfering components are stored into a review sheet. The structure of a component without interference is then determined based on computation using the interference check results, and candidates are displayed in a scrollable manner. A candidate is selected by the user's gesture and the selected result is stored into the component model DB 240.
The design review devices 100 and 500 including the full-body capture device may obtain log data about the motion trajectory of the user's hand or foot and a tool, and may apply the log data to review sheets. For example, the user may hold a tool in the virtual space in design review and actually attempt to tighten a bolt but cannot tighten the bolt due to interference between the tool and the product. In this case, the corresponding check item in the review sheet is marked as a failure. The log data for such check results may be used for evaluation items or check items for prototype design.
The design review devices 100 and 500 may also record an example work procedure and the motions of hands or feet performed by a skilled reviewer wearing the full-body capture device in animation. Further, notes and comments from work may be recorded on a 3D model of a product or a component with the stylus capture device 330 for use as an instruction brief. The work know-how and notes to be shared among reviewers may also be written on a 3D model of a product or a component. Additionally, the work procedure and the motions of hands or feet of unexperienced reviewers may be measured and used to visualize differences between skilled reviewers and unexperienced reviewers and use such information for education. For example, differences between skilled reviewers and unexperienced reviewers in, for example, the assembly order for each component, work posture, and eye lines during work may be displayed.
In the embodiments described above, the data server 200 may be accessible from a main facility and sub-facilities inside and outside Japan through the Internet connection. In this case, when the design review devices 100 and 500 including the communicator 300 that may access the data server 200, the glove-shaped capture device 310, the imaging device 320, the stylus capture device 330, the HMD 340, and the controller 400 are installed in the main facility and a sub-facility, design review can be performed in the main facility and the sub-facility and data stored in the data server 200 may be managed centrally. This allows review results to be updated in real time and shared between the main facility and the sub-facility. For example, when size information about a component is changed, the 3D CAD data is changed and stored in the product model DB 210 in real time. Further, the design review devices 100 and 500 may access design review devices 100 and 500 installed in other facilities including facilities located overseas through the Internet connection to share screens displaying 3D models in the virtual space and to perform design review together with the design review devices 100 and 500 installed in other facilities. This allows design review to be performed in real time at multiple facilities in collaboration with one another. Thus, when understanding the regulations or trends at an overseas sub-facility may be difficult at the main facility, design review may be performed at the sub-facility. Additionally, staff at an overseas facility and overseas customers may perform design review together efficiently to check detailed specifications for each customer. The design review devices 100 and 500 according to the present embodiment perform design review for products including elevators to facilitate the sale of the products worldwide.
The product specifications vary as the trends and preferences vary depending on countries in the world. The design review performed at multiple facilities and real-time update of the review results allow the creation of global design. For example, a country in which design review has been performed can be identified from handwriting written with the stylus capture device 330 and candidate components suitable for the country can be displayed in the virtual space. This allows products to have global design in real time. Further, a sub-screen may display a configuration tree representing the component configuration of a displayed component. The tree configuration of each component is called from the design information DB stored in the data server 200 and then displayed.
When a review result indicates that the specifications of a component are to be changed, candidates for alternative components are retrieved from the component model DB 240 based on the review result, and displayed. For example, when a review result indicates that a component A and a component B interfere with each other by 5 mm, a component having the same specifications as the component A but smaller by 5 mm is retrieved from the component model DB 240, and is then displayed in the virtual space as a candidate for an alternative component. Further, the data server 200 may include a purchase information DB storing component names, component specifications, component prices, and delivery dates in a manner associated with one another. The candidates for alternative components contained in the purchase information DB may then be displayed. Such alternative components may be displayed in the order of lower prices or in the order of shorter delivery periods.
Although the HMD 340 is non-transmissive in the embodiments described above, the HMD 340 may be semi-transmissive. In this case, a 3D model of a tool is superimposed on the tool placed in the real space. A review sheet is superimposed on the reference board 332. Further, although the HMD 340 displays the virtual space in the example described above, any display other than the HMD 340 may be used when the display can display a virtual space. A liquid crystal display may be used instead of the HMD 340.
Although the glove-shaped capture device 310 is used to detect the user's gesture representing the user action in the embodiments described above, other means may be used to detect the user's gestures. An imaging device may capture an image of the motion of the user's hand and digits, and may detect a gesture based on the captured image. The user actions include the body motion and the voice of the user, including the motion of the head, an arm, or a foot and the voice.
Although the stylus capture device 330 includes a sensor in the embodiments described above, the reference board 332 may include a touchscreen sensor and use the touchscreen sensor to detect handwriting on the reference board 332 written with a stylus including no sensor.
The design review process to be performed by the design review devices 100 and 500 and the data server 200 may be implemented by a device other than a dedicated system, and may be implemented by, for example, a general-purpose mobile information terminal or a personal computer. Programs used by the design review devices 100 and 500 and the data server 200 may be stored in a computer-readable recording medium, such as compact disc read only memory (CD-ROM), a digital versatile disc (DVD), a USB flash drive, a memory card, and a hard disk drive (HDD), and may then be distributed. When such programs are installed in a dedicated or general-purpose computer, the computer may then function as the design review devices 100 and 500 or the data server 200.
The programs described above may be stored in a storage device in another server on the Internet and may then be downloaded from the server.
The foregoing describes some example embodiments for explanatory purposes. Although the foregoing discussion has presented specific embodiments, persons skilled in the art will recognize that changes may be made in form and detail without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. This detailed description, therefore, is not to be taken in a limiting sense, and the scope of the invention is defined only by the included claims, along with the full range of equivalents to which such claims are entitled.
This application claims the benefit of Japanese Patent Application No. 2017-103675, filed on May 25, 2017, the entire disclosure of which is incorporated by reference herein.
Number | Date | Country | Kind |
---|---|---|---|
2017-103675 | May 2017 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2018/020181 | 5/25/2018 | WO | 00 |