Patent Grant
7139979
The present invention relates to the fields of software prototyping environments and graphical programming. More particularly, the invention relates to displaying operations in an application by using a graphical programming visual representation. One embodiment of the invention relates to displaying machine vision operations in an application by using a graphical programming visual representation, where the machine vision operations implement a machine vision function.
Traditionally, high level text-based programming languages have been used by programmers in writing application programs. Many different high level programming languages exist, including BASIC, C, Java, FORTRAN, Pascal, COBOL, ADA, APL, etc. Programs written in these high level languages are translated to the machine language level by translators known as compilers or interpreters. The high level programming languages in this level, as well as the assembly language level, are referred to herein as text-based programming environments.
Increasingly, computers are required to be used and programmed by those who are not highly trained in computer programming techniques. When traditional text-based programming environments are used, the user's programming skills and ability to interact with the computer system often become a limiting factor in the achievement of optimal utilization of the computer system.
There are numerous subtle complexities which a user must master before he can efficiently program a computer system in a text-based environment. The task of programming a computer system to model or implement a process often is further complicated by the fact that a sequence of mathematical formulas, steps or other procedures customarily used to conceptually model a process often does not closely correspond to the traditional text-based programming techniques used to program a computer system to model such a process. In other words, the requirement that a user program in a text-based programming environment places a level of abstraction between the user's conceptualization of the solution and the implementation of a method that accomplishes this solution in a computer program. Thus, a user often must substantially master different skills in order to both conceptualize a problem or process and then to program a computer to implement a solution to the problem or process. Since a user often is not fully proficient in techniques for programming a computer system in a text-based environment to implement his solution, the efficiency with which the computer system can be utilized often is reduced.
Examples of fields in which computer systems are employed to interact with physical systems are the fields of instrumentation, process control, industrial automation, and simulation. Computer measurement and control of devices such as instruments or industrial automation hardware has become increasingly desirable in view of the increasing complexity and variety of instruments and devices available for use. However, due to the wide variety of possible testing and control situations and environments, and also the wide array of instruments or devices available, it is often necessary for a user to develop a custom program to control a desired system.
As discussed above, computer programs used to control such systems traditionally had to be written in text-based programming languages such as, for example, assembly language, C, FORTRAN, BASIC, etc. Traditional users of these systems, however, often were not highly trained in programming techniques and, in addition, text-based programming languages were not sufficiently intuitive to allow users to use these languages without training. Therefore, implementation of such systems frequently required the involvement of a programmer to write software for control and analysis of instrumentation or industrial automation data. Thus, development and maintenance of the software elements in these systems often proved to be difficult.
U.S. Pat. Nos. 4,901,221; 4,914,568; 5,291,587; 5,301,301; and 5,301,336; among others, to Kodosky et al disclose a graphical system and method for modeling a process, i.e., a graphical programming environment which enables a user to easily and intuitively model a process. The graphical programming environment disclosed in Kodosky et al can be considered a higher and more intuitive way in which to interact with a computer. A graphically based programming environment can be represented at a level above text-based high level programming languages such as C, Basic, Java, etc.
The method disclosed in Kodosky et al allows a user to construct a diagram using a block diagram editor. The block diagram may include a plurality of interconnected icons such that the diagram created graphically displays a procedure or method for accomplishing a certain result, such as manipulating one or more input variables and/or producing one or more output variables. In response to the user constructing a diagram or graphical program using the block diagram editor, data structures and/or program instructions may be automatically constructed which characterize an execution procedure that corresponds to the displayed procedure. The graphical program may be compiled or interpreted by a computer.
Therefore, Kodosky et al teaches a graphical programming environment wherein a user places or manipulates icons and interconnects or “wires up” the icons in a block diagram using a block diagram editor to create a graphical “progam.” A graphical program for performing an instrumentation, measurement or automation function, such as measuring a Unit Under Test (UUT) or device, controlling or modeling instruments, controlling or measuring a system or process, or for modeling or simulating devices, may be referred to as a virtual instrument (VI). Thus, a user can create a computer program solely by using a graphically based programming environment. This graphically based programming environment may be used for creating virtual instrumentation systems, modeling processes, control, simulation, and numerical analysis, as well as for any type of general programming.
A graphical program may have a graphical user interface. For example, in creating a graphical program, a user may create a front panel or user interface panel. The front panel may include various graphical user interface elements or front panel objects, such as user interface controls and/or indicators, that represent or display the respective input and output that will be used by the graphical program or VI, and may include other icons which represent devices being controlled. The front panel may be comprised in a single window of user interface elements, or may comprise a plurality of individual windows each having one or more user interface elements, wherein the individual windows may optionally be tiled together. When the controls and indicators are created in the front panel, corresponding icons or terminals may be automatically created in the block diagram by the block diagram editor. Alternatively, the user can place terminal icons in the block diagram which may cause the display of corresponding front panel objects in the front panel, either at edit time or later at run time. As another example, the front panel may comprise front panel objects, e.g., the GUI, embedded in the block diagram.
During creation of the block diagram portion of the graphical program, the user may select various function nodes or icons that accomplish his desired result and connect the function nodes together. For example, the function nodes may be connected in one or more of a data flow, control flow, and/or execution flow format. The function nodes may also be connected in a “signal flow” format, which is a subset of data flow. The function nodes may be connected between the terminals of the various user interface elements, e.g., between the respective controls and indicators. Thus the user may create or assemble a graphical program, referred to as a block diagram, graphically representing the desired process. The assembled graphical program may be represented in the memory of the computer system as data structures and/or program instructions. The assembled graphical program, i.e., these data structures, may then be compiled or interpreted to produce machine language that accomplishes the desired method or process as shown in the block diagram.
Input data to a graphical program may be received from any of various sources, such as from a device, unit under test, a process being measured or controlled, another computer program, or from a file. Also, a user may input data to a graphical program or virtual instrument using a graphical user interface, e.g., a front panel as described above. The input data may propagate through the data flow block diagram or graphical program and appear as changes on the output indicators. In an instrumentation application, the front panel can be analogized to the front panel of an instrument. In an industrial automation application the front panel can be analogized to the MMI (Man Machine Interface) of a device. The user may adjust the controls on the front panel to affect the input and view the output on the respective indicators. Alternatively, the front panel may be used merely to view the input and output, or just the output, and the input may not be interactively manipulable by the user during program execution.
Thus, graphical programming has become a powerful tool available to programmers. Graphical programming development environments such as the National Instruments LabVIEW product have become very popular. Tools such as LabVIEW have greatly increased the productivity of programmers, and increasing numbers of programmers are using graphical programming environments to develop their software applications. In particular, graphical programming tools are being used for test and measurement, data acquisition, process control, man machine interface (MMI), supervisory control and data acquisition (SCADA) applications, simulation, image processing/machine vision applications, and motion control, among others.
As new techniques and computerized methods are developed for a given problem domain, specialized software applications for developing solutions to problems in that domain are often created. These specialized applications can provide an environment that is conducive to rapidly and conveniently prototyping a problem solution. Hence, these applications are also referred to herein as “prototyping environments”. A prototyping environment may integrate various capabilities to aid developers of problem solutions. For example, a prototyping environment may provide a library of operations that are specific to one or more problem domains and may enable a user to select various operations from the library for inclusion in an application. Prior art prototyping environments have used various techniques to display the operations in the application, such as using text information arranged in various ways, such as a table or tree view.
One embodiment of the invention relates to displaying operations in an application by using a graphical programming representation (or block diagram representation). A plurality of operations may be included in the application, e.g., in response to user input received to a graphical user interface of a prototyping environment. A plurality of interconnected icons may be automatically (i.e., programmatically) displayed in response to including the plurality of operations in the application. Each icon may correspond to an operation included in the application.
In one embodiment, as each operation is added to the application, a corresponding icon may be displayed and may be visually interconnected to one or more other icons that may have been previously displayed. Each icon may be designed to represent the respective operation using an intuitive picture. The plurality of interconnected icons may visually indicate the first function implemented by the plurality of operations in the application. In one embodiment, as each operation is added to the application, the respective operation may also optionally be displayed in a graphical user interface of the application development environment.
Thus, when the user adds a first operation to the application, a corresponding first icon may be displayed representing the first operation. In addition, a name or other indicia of the respective first operation may optionally be displayed in a graphical user interface of the application development environment. When the user adds a second operation to the application, a corresponding second icon may then be displayed representing the second operation. In addition, a name or other indicia of the respective second operation may be displayed in the graphical user interface of the application development environment. The second icon may be automatically connected to the first icon to indicate the sequence of operations. The above process repeats, where new icons are added to the block diagram representation (and connected with already displayed icons) as the user adds operations to the application.
As noted above, the system may display interconnecting lines between displayed icons, where each line connects two (or more) icons. The lines may have various meanings or semantics. For example, where the plurality of lines includes a first line that connects a first icon to a second icon, the first line may indicate that performing the first function includes performing the operation corresponding to the first icon before performing the operation corresponding to the second icon. In another embodiment, the first line may indicate that data produced by the operation corresponding to the first icon is used by the operation corresponding to the second icon (data flow). In various embodiments, the interconnection lines or links may visually represent one or more of data flow, control flow, and/or execution flow for the plurality of operations in the application. Thus, the plurality of interconnected icons may be referred to as a graphical programming visual representation (or a block diagram representation) of the plurality of operations in the application. The icons may be displayed in various ways, e.g., in a left-to-right manner or top-to-bottom manner, to visually represent an ordering of the operations or to visually represent other relationships among the operations.
The plurality of operations may implement a first function. In various embodiments the plurality of operations in the application may implement any of various functions or solutions. As one example, the plurality of operations may include one or more machine vision operations that implement a machine vision function, such as for acquiring and/or analyzing images. As another example, the plurality of operations may include one or more motion control operations that implement a motion control function, such as for controlling motion of a device and/or controlling a device to move an object. As another example, the plurality of operations may include one or more data acquisition (DAQ) operations that implement a DAQ function, such as for acquiring measurement data from a device.
In one embodiment, the user may perform operations on a displayed image or object (e.g., an image of an object displayed on the display), and this may cause the operations to be stored in the application. The object may also be updated on the display to reflect the operations performed. The system may display a graphical user interface with various menus, dialogs, controls or other GUI elements that the user may use to apply operations to an image. As the user performs each respective operation on the object, a corresponding icon may be displayed to visually represent the operation. Thus, as the user performs each additional operation on the object, a corresponding icon may be immediately displayed to visually represent this operation, and the newly displayed icon may be interconnected with one or more other existing icons. Thus, as noted above, the system may also display a connection between a newly displayed icon and a previously displayed icon to represent their relationship.
In an embodiment targeted toward machine vision, the system may display an image. The system may also display a graphical user interface that may be used by the user to apply various operations (e.g., image processing or machine vision operations) to the displayed image. The user may use the GUI to perform operations on the displayed image, may draw regions of interest (ROIs) on the image, etc. These actions by the user may cause visual changes to the image according to the operation performed. As the user performs machine vision operations on the displayed image, the machine vision operations may be included or stored in the application, and a corresponding icon may also be displayed to visually represent the machine vision operation. Thus, in one embodiment, as the user performs each operation to the image, 1) the operation is stored in memory, 2) the image is changed according to the operation, 3) a new icon is displayed in the block diagram representation, where the new icon represents the operation, and 4) the new icon may be connected with a pre-existing icon (assuming the new icon does not represent the first operation). The operation may also be displayed in an alternative form in the GUI, such as the textual name of the operation being displayed in a script window.
In one embodiment, each displayed icon may correspond to a node in a graphical programming development environment. In other words, for each operation provided by the prototyping environment, there may be an equivalent node in the graphical programming development environment. Each of these nodes in the graphical programming development environment may have an iconic appearance that is identical to or substantially the same as the corresponding icon.
Information representing the application may be stored, e.g., in a data structure or file. The stored information may include information specifying the plurality of operations in the application. For example, the data structure or file may store names or other identifiers that identify the operations in the application. The data structure or file may also store information regarding various properties or parameter values configured for one or more operations in the application.
In one embodiment, each operation may be executed as the operation is included in the application. In another embodiment, the entire application may be executed together. Executing the application may include executing the plurality of operations in the application to perform the first function.
Also, in one embodiment the prototyping environment may be operable to programmatically (automatically) generate a graphical program based on the application, where the graphical program is operable to perform the first function. The generated graphical program may appear the same as or substantially the same as the displayed plurality of interconnected icons. The graphical program may then be executed to perform the first function. In another embodiment, the displayed graphical programming representation (or block diagram representation) is itself an executable graphical program that the user may run, use execution highlighting with, etc.
A better understanding of the present invention can be obtained when the following detailed description of the preferred embodiment is considered in conjunction with the following drawings, in which:
While the invention is susceptible to various modifications and alternative forms specific embodiments are shown by way of example in the drawings and are herein described in detail. It should be understood however, that drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary the invention is to cover all modifications, equivalents and alternative following within the spirit and scope of the present invention as defined by the appended
Incorporation by Reference
The following references are hereby incorporated by reference in their entirety as though fully and completely set forth herein.
U.S. patent application Ser. No. 09/518,492 titled “System and Method for Programmatically Creating a Graphical Program,” filed Mar. 3, 2000.
U.S. patent application Ser. No. 09/745,023 titled “System and Method for Programmatically Generating a Graphical Program in Response to Program Information,” filed Dec. 20, 2000.
U.S. patent application Ser. No. 09/587,682 titled “System and Method for Automatically Generating a Graphical Program to Perform an Image Processing Algorithm,” filed Jun. 6, 2000.
U.S. patent application Ser. No. 09/595,003 titled “System and Method for Automatically Generating a Graphical Program to Implement a Prototype,” filed Jun. 13, 2000.
U.S. patent application Ser. No. 10/051,474 titled “System and Method for Graphically Creating a Sequence of Motion Control Operations,” filed Jan. 18, 2002.
FIG. 1—Computer System Executing a Prototyping Environment
Knowledge of computer-implemented techniques for solving problems in a wide array of fields is expanding at a rapid pace. As new techniques and computerized methods are developed for a given problem domain, specialized software programs for developing solutions to problems in that domain are often created. These specialized software programs can provide an environment that is conducive to rapidly and conveniently prototyping a problem solution (or “application”). Hence, these programs are also referred to herein as “prototyping environments”.
As used herein, the term “application” refers to the result that may be produced by such a prototyping environment. In other words, the term “application” refers to a set of operations, and possibly associated parameters, that may be selected by the user using a prototyping environment. The term “application” is intended to encompass a sequence, solution, prototype, algorithm, script, or similar concept. Thus, a prototyping environment may be used to generate an application which represents an algorithm or process designed by the user in the prototyping environment.
Prototyping environments are usually designed for ease of use and may be specialized for developing solutions for a particular problem domain. They may enable users to create a computer-implemented solution to a problem without requiring the users to utilize or understand traditional programming techniques. For example, the prototyping environment may aid the user in creating an application including a plurality of operations, without requiring the user to write programming language code. Instead of writing code, the user may interact at a higher level with the graphical user interface of the prototyping environment to create the application.
As described below, according to one embodiment of the present invention, the prototyping environment may be operable to display the operations included in the application by using a graphical programming visual representation of the operations and their relationship with each other. For example, a plurality of interconnected icons may be displayed in response to including operations in the application. Each icon may correspond to an operation in the application, so that the plurality of interconnected icons visually indicates a function or process performed by the operations. As each operation is added to the application, a corresponding icon may be immediately visually displayed in the graphical programming visual representation (or block diagram representation).
Prototyping environments may be developed for many different problem domains. Various exemplary prototyping environments are mentioned herein. One example is a motion control prototyping environment application for developing a sequence of motion control operations. A motion control sequence is also referred to herein as a motion control “application”.
Computer-based motion control involves precisely controlling the movement of a device or system. Computer-based motion control is widely used in many different types of applications, including applications in the fields of industrial automation, process control, test and measurement automation, robotics, and integrated machine vision, among others. A typical computer-based motion system includes components such as the moving mechanical device(s), a motor with feedback and motion I/O, a motor drive unit, a motion controller, and software to interact with the motion controller.
A motion control prototyping environment may be designed to enable a user to easily and efficiently develop/application a motion control sequence or application without requiring the user to perform programming, e.g., without needing to write or construct code in any programming language. For example, the environment may provide a graphical user interface (GUI) enabling the user to develop/application the motion control sequence at a high level, by selecting from and configuring a sequence of motion control operations using the GUI. According to one embodiment of the present invention, the motion control prototyping environment may be operable to display the motion control operations included in the motion control sequence by using a graphical programming visual representation of the motion control operations and their relationship with each other.
Another example of a prototyping environment described herein is a machine vision prototyping environment. A machine vision prototyping environment may enable a user to rapidly develop an application including a plurality of machine vision operations. As used herein, the term “machine vision operation” may include any of various types of operations related to image analysis, image processing, image acquisition, or other machine vision-related operations. A machine vision application may specify a sequence, script, process, or algorithm for a machine vision application. For example, the machine vision prototyping environment may include a graphical user interface enabling the user to easily apply various machine vision operations to an image and see the results, in order to develop the desired machine vision application.
In various embodiments, any of various types of machine vision operations may be supported, including image acquisition functions, filtering functions, morphology functions, histogram functions, particle analysis functions, edge detection functions, pattern matching functions, color matching functions, shape matching functions, optical character recognition (OCR), optical character verification (OCV), and 1D and 2D barcodes, etc. In one embodiment, the user may apply various machine vision operations to an image, and the operations may be included in the machine vision application. For example, each operation may be recorded as a step in a script. A script may essentially specify an algorithm; i.e., an algorithm may be defined by the plurality of steps or operations in a script. The user may create the machine vision application without specifying or writing programming language code, similarly as described above. According to one embodiment of the present invention, the machine vision prototyping environment may be operable to display the machine vision operations included in a machine vision application by using a graphical programming visual representation of the machine vision operations and their relationship with each other.
In various embodiments, the computer system 82 may execute a prototyping environment application for developing applications related to any of various other types of applications, in addition to motion control and/or machine vision. In general, a “prototyping environment” may refer to a specialized application that provides an environment that is conducive to rapidly and conveniently prototyping a problem solution, preferably without requiring the user to write code in a programming language or minimizing the amount of code the user would otherwise have to write. Other examples of prototyping environments include:
In various embodiments, the operations in any of various types of applications may be displayed by using a graphical programming visual representation. As a few examples, the application may be related to fields such as image processing, image analysis, machine vision, process control, automation, test and measurement, simulation, motion control, robotics, audio, video, graphics, telecommunications, and workflow processes, among others.
It is noted that a prototyping environment may also enable a user to develop an application which includes operations related to multiple different fields or technologies. For example, one embodiment of a prototyping environment may provide a library of operations including one or more machine vision operations, one or more motion control operations, and one or more data acquisition (DAQ) operations. In this example, the motion control prototyping environment may be referred to as a MC/MV/DAQ prototyping environment. (The abbreviation “MC/MV/DAQ” is used herein to refer to “motion control/machine vision/DAQ”.) For example, a MC/MV/DAQ prototyping environment may be used to create an application including machine vision operations for acquiring and analyzing images of an object, motion control operations for moving the object, and DAQ operations for acquiring measurement data of the object. Such an application may be useful, for example, to specify an automated inspection process performed on manufactured objects.
Referring again to
The computer system 82 may include a memory medium(s) on which one or more computer programs or software components may be stored according to one embodiment of the present invention. For example, the memory medium may store a prototyping environment application program (or portion of such an application program) such as described above. The memory medium may also store one or more applications created using the prototyping environment application. The memory medium may also store a graphical program automatically generated by the prototyping environment application based on an application, as described below. The memory medium may also store a graphical programming development environment with which a generated graphical program is associated. The memory medium may also store operating system software, as well as other software for operation of the computer system 82.
The term “memory medium” is intended to include an installation medium, e.g., a CD-ROM, floppy disks 104, or tape device; a computer system memory or random access memory such as DRAM, SRAM, EDO RAM, Rambus RAM, etc.; or a non-volatile memory such as a magnetic media, e.g., a hard drive, or optical storage. The memory medium may comprise other types of memory as well, or combinations thereof. In addition, the memory medium may be located in a first computer in which the programs are executed, or may be located in a second different computer which connects to the first computer over a network, such as the Internet. In the latter instance, the second computer may provide program instructions to the first computer for execution.
FIGS. 2A and 2B—Instrumentation and Industrial Automation Systems
In one embodiment, the host computer 82 may store a prototyping environment application operable to create an application including operations that interact with or control the one or more instruments. As described below, the prototyping environment application may be operable to display the operations in the application by using a graphical programming visual representation of the operations and their relationship with each other. In one embodiment, the host computer 82 may store an application created by the prototyping environment application and/or may execute the application to interact with or control the one or more instruments. In another embodiment, the host computer 82 may store a graphical program based on an application. For example, as described below, the graphical program may be programmatically (automatically) generated based on the application. In this instance, the host computer 82 may execute the graphical program to interact with or control the one or more instruments.
The one or more instruments may include a GPIB instrument 112 and associated GPIB interface card 122, a data acquisition board 114 and associated signal conditioning circuitry 124, a VXI instrument 116, a PXI instrument 118, a video device 132 and associated image acquisition card 134, a motion control device 136 and associated motion control interface card 138, and/or one or more computer based instrument cards 142, among other types of devices.
The GPIB instrument 112 is coupled to the computer 82 via the GPIB interface card 122 provided by the computer 82. In a similar manner, the video device 132 is coupled to the computer 82 via the image acquisition card 134, and the motion control device 136 is coupled to the computer 82 through the motion control interface card 138. The data acquisition board 114 is coupled to the computer 82, and may interface through signal conditioning circuitry 124 to the UUT. The signal conditioning circuitry 124 preferably comprises an SCM (Signal Conditioning eXtensions for Instrumentation) chassis comprising one or more SCXI modules 126.
The GPIB card 122, the image acquisition card 134, the motion control interface card 138, and the DAQ card 114 are typically plugged in to an I/O slot in the computer 82, such as a PCI bus slot, a PC Card slot, or an ISA, EISA or MicroChannel bus slot provided by the computer 82. However, these cards 122, 134, 138 and 114 are shown external to computer 82 for illustrative purposes.
The VXI chassis or instrument 116 is coupled to the computer 82 via a VXI bus, MXI bus, or other serial or parallel bus provided by the computer 82. The computer 82 preferably includes VXI interface logic, such as a VXI, MXI or GPIB interface card (not shown), which interfaces to the VXI chassis 116. The PXI chassis or instrument is preferably coupled to the computer 82 through the computer's PCI bus.
A serial instrument (not shown) may also be coupled to the computer 82 through a serial port, such as an RS-232 port, USB (Universal Serial bus) or IEEE 1394 or 1394.2 bus, provided by the computer 82. In typical instrumentation control systems an instrument will not be present of each interface type, and in fact many systems may only have one or more instruments of a single interface type, such as only GPIB instruments.
The instruments are coupled to the unit under test (UUT) or process 150, or are coupled to receive field signals, typically generated by transducers. The system 100 may be used in a data acquisition and control application, in a test and measurement application, a process control application, or a man-machine interface application.
In one embodiment, the host computer 82 may store a prototyping environment application operable to create an application including operations involved with the automation function performed by the automation system 160. As described below, the prototyping environment application may be operable to display the operations in the application by using a graphical programming visual representation of the operations and their relationship with each other. In one embodiment, the host computer 82 may store an application created by the prototyping environment application and/or may execute the application to perform the automation function. In another embodiment, the host computer 82 may store a graphical program based on an application. For example, as described below, the graphical program may be programmatically (automatically) generated based on the application. In this instance, the host computer 82 may execute the graphical program to perform the automation function.
The one or more devices may include a data acquisition board 114 and associated signal conditioning circuitry 124, a PXI instrument 118, a video device 132 and associated image acquisition card 134, a motion control device 136 and associated motion control interface card 138, a fieldbus device 170 and associated fieldbus interface card 172, a PLC (Programmable Logic Controller) 176, a serial instrument 182 and associated serial interface card 184, or a distributed data acquisition system, such as the Fieldpoint system available from National Instruments, among other types of devices.
The DAQ card 114, the PXI chassis 118, the video device 132, and the image acquisition card 136 are preferably connected to the computer 82 as described above. The serial instrument 182 is coupled to the computer 82 through a serial interface card 184, or through a serial port, such as an RS-232 port, provided by the computer 82. The PLC 176 couples to the computer 82 through a serial port, Ethernet port, or a proprietary interface. The fieldbus interface card 172 is preferably comprised in the computer 82 and interfaces through a fieldbus network to one or more fieldbus devices. Each of the DAQ card 114, the serial card 184, the fieldbus card 172, the image acquisition card 134, and the motion control card 138 are typically plugged in to an I/O slot in the computer 82 as described above. However, these cards 114, 184, 172, 134, and 138 are shown external to computer 82 for illustrative purposes. In typical industrial automation systems a device will not be present of each interface type, and in fact many systems may only have one or more devices of a single interface type, such as only PLCs. The devices are coupled to the device or process 150.
FIG. 3—Machine Vision Inspection System
As shown in
In various embodiments, the images of the objects 200 may be analyzed in any of various ways, e.g., may be analyzed for any of various kinds of characteristics or defects, using any of various machine vision or image processing techniques. As a few examples, an image of an object may be analyzed to detect: physical surface defects (scratches, etc.); one or more components located correctly on the object; a correct label on the object; a correct marking on the object; correct color information on the object, etc.
The results of the image analyses may be used to determine whether an object 200 meets desired production standards. The determination may be made based on any of various criteria, as desired for a particular application. If separate computer systems are used to analyze the images, the results from each computer system may be considered together in making this determination. If an object does not meet the desired production standards, the object may be rejected. For example, in rejecting the object, the object may be removed from the manufacturing apparatus 204, as indicated in
The computer system 82 may be a computer system of any type. In one embodiment, multiple computer systems 82 may be employed, e.g., to distribute the image processing load across multiple computer systems. In one embodiment, the computer system(s) 82 may comprise a controller or card (a “computer on a card”) housed in a PXI, VXI or other type of chassis. The chassis may further include one or more image acquisition boards which couple to one or more cameras 212.
The computer system(s) 82 may include a memory medium on which software operable to receive and analyze the images of the objects 200 is stored. In one embodiment, this software may include a machine vision application and/or a machine vision prototyping application used to create and execute the machine vision application. In another embodiment, this software may include a program that was programmatically generated based on a machine vision application, such as a graphical program or a text-based program. It is noted that in various embodiments, the analysis of the objects 200 may be performed in any of various manners, either in software, programmable logic, or hardware, or a combination thereof. For example, at least a portion of a machine vision application may be deployed on a hardware device.
Although
FIG. 4—Computer System Block Diagram
The computer may include at least one central processing unit or CPU 160 which is coupled to a processor or host bus 162. The CPU 160 may be any of various types, including an ×86 processor, e.g., a Pentium class, a PowerPC processor, a CPU from the SPARC family of RISC processors, as well as others. Main memory 166 is coupled to the host bus 162 by means of memory controller 164. In one embodiment, the main memory 166 may store a prototyping environment application for creating, configuring, and/or performing an application. The main memory 166 may also store an application created using the prototyping environment application. In another embodiment, the main memory 166 may store a program that was automatically, i.e., programmatically generated by the prototyping environment based on an application, where the program is operable to perform the application. In one embodiment, the main memory 166 may store an application development environment associated with a programmatically generated program. The main memory may also store operating system software, as well as other software for operation of the computer system.
The host bus 162 may be coupled to an expansion or input/output bus 170 by means of a bus controller 168 or bus bridge logic. The expansion bus 170 may be the PCI (Peripheral Component Interconnect) expansion bus, although other bus types can be used. The expansion bus 170 includes slots for various devices such as a data acquisition board 114 and a GPIB interface card 122 which provides a GPIB bus interface 123 to a GPIB instrument. The computer 82 further comprises a video display subsystem 180 and hard drive 182 coupled to the expansion bus 170.
A reconfigurable instrument 190 may also be connected to the computer. In various embodiments, the configurable logic may be included on an instrument or device connected to the computer through means other than an expansion slot, e.g., the instrument or device may be connected via an IEEE 1394 bus, USB, or other type of port. Also, the configurable logic may be comprised on a device such as the data acquisition board 114. In one embodiment, at least a portion of an application may execute on the reconfigurable instrument 190.
FIG. 5—Displaying Application Operations Using a Graphical Programming Representation
In 301, a graphical user interface for a prototyping environment may be displayed. The graphical user interface may provide graphical access to a set of operations available for inclusion in an application. In various embodiments, any of various operations may be provided. For example, as discussed above, the prototyping environment may be intended for creating applications or solutions for problems in any of various domains. Exemplary user interfaces for exemplary machine vision prototyping environments are described below. The GUI may include various menus, dialogs, controls or other GUI elements that the user may use to select operations and/or apply operations to a displayed object or image.
In 303, user input requesting inclusion of a plurality of operations in the application may be received to the graphical user interface. The plurality of operations may implement a first function. In one embodiment, the user is not required to specify or write any program source code to implement the application or specify the operations in the application. Instead, the application may be created graphically by interacting with the graphical user interface of the prototyping environment to include operations in the application.
As described above, in various embodiments the plurality of operations in the application may implement any of various functions or solutions. As one example, the plurality of operations may include one or more machine vision operations that implement a machine vision function, such as for acquiring and/or analyzing images. As another example, the plurality of operations may include one or more motion control operations that implement a motion control function, such as for controlling motion of a device and/or controlling a device to move an object. As another example, the plurality of operations may include one or more data acquisition (DAQ) operations that implement a DAQ function, such as for acquiring measurement data from a device.
The operations in the application may be related to one another or structured in any of various ways. For example, in one embodiment, the application may comprise a sequence of operations. In one embodiment, each operation in the sequence may be performed sequentially. In another embodiment, the user may specify conditional branches that may result in some operations being skipped or performed in different orders, e.g., depending on results of performing previous operations. The user may also specify other types of constructs, such as iteration, looping, jumps, etc.
In various embodiments, the user may perform any of various actions or may interact with the graphical user interface of the prototyping environment in any of various ways to cause operations to be included in the application. For example, in one embodiment, a plurality of buttons may be displayed, each button corresponding to a particular operation. The user may press the appropriate button to add the desired operation. In another embodiment, a library of icons may be displayed on a panel or palette, where each icon corresponds to a particular operation available for inclusion in the application. The user may select the appropriate icon to add the desired operation. In other embodiments, the user may utilize other menu, keyboard, and/or voice commands to add an operation to the application.
In one embodiment, operations may be included in the application in response to the user performing the operations on an object of interest. For example, the user may select various operations to apply to the object, and this may cause the operations to be included in or stored in the application. Thus, the application may specify the operations applied to the object. The object or an image of the object may also be updated to reflect results of applying the operations to the object. In one embodiment, the user may directly perform operations on an object, such as by using a virtual reality glove or similar device to manipulate a physical object, and the operations performed by the user may be recorded as operations in the application. Head tracking and eye tracking devices, and other types of virtual reality devices, may also be used.
In one embodiment, operations may be included in the application in response to the user performing operations on an object displayed on the screen, e.g., an image of an object. The object image may represent a physical system or device. The user may graphically manipulate or otherwise selection operations to affect the object image. For example, the user may desire to create a robotic application to assemble a portion of an automobile, and the user may graphically select and place automobile parts on the screen to create the application.
In an embodiment targeted toward machine vision, the system may display an image, and the user may perform machine vision operations on the displayed image, which may cause visual changes to the displayed image according to the operation performed. As the user performs machine vision operations on the displayed image, the machine vision operations may be included in or stored in the application.
In one embodiment each operation may be interactively executed as the operation is included in the application, e.g., may be executed automatically in response to including the operation in the application or may be executed in response to a user request. For example, this may help the user to verify that each individual operation in the application performs as intended. Also, in one embodiment, each operation may be included in the application in response to executing the operation, e.g., in response to applying the operation to an object.
In one embodiment, the graphical user interface of the prototyping environment may be updated to illustrate the effect of including and/or executing each new operation. For example, as noted above, the graphical user interface of a machine vision prototyping environment may display an image, and the image may be updated to illustrate the effect of applying a new machine vision operation included in the application to the image. As another example, the graphical user interface of a motion control prototyping environment may display one or more views of motion control performed by the application, such as a two-dimensional and/or three-dimensional view of the cumulative movement specified by the application, as well as other types of views, such as a graph indicating a velocity profile. These views may be updated to illustrate the effect of adding a new motion control operation to the motion control application. In this example, the motion control operations may not actually be executed as they are added to the motion control application, e.g., may not actually cause a motion control device to move, but rather their effects may be simulated on the graphical user interface.
In one embodiment, each operation included in an application may have various associated properties, attributes, or parameters affecting the operation. For example, an arc move motion control operation may have parameters or properties such as a radius, a start angle, and a travel angle. These parameters may initially have default values. The user may configure these parameters or properties (typically using a GUI) to customize each operation.
In the preferred embodiment, the user may configure the parameters of the operations graphically, without having to write any program code. For example, a graphical panel for configuring the operation may be displayed. This panel may be automatically displayed in response to selecting the operation or adding the operation to the application, or the panel may be displayed in response to user input requesting to configure the operation. User input for configuring the operation may be received to the graphical panel. For example, the panel may include various user interface elements for changing parameter or property values of the operation, such as numeric GUI controls, check boxes, etc.
In one embodiment, the graphical user interface of the prototyping environment may be updated to illustrate the effect of configuring the operation. For example, if the user changed a travel angle parameter of an arc move operation in a motion control application, then one or more views of the motion control performed by the application may be updated to visually reflect the new travel angle performed by the arc move operation.
In 305, a plurality of interconnected icons may be displayed in response to including the plurality of operations in the application. Each icon may correspond to an operation included in the application. The interconnection between icons may indicate sequencing (or data flow) between the operations. The plurality of interconnected icons may visually indicate the first function implemented by the plurality of operations in the application.
In one embodiment, as each operation is added to the application in 303, a corresponding icon may be added to the plurality of icons and may be visually interconnected to one or more other previously displayed icons (if any). In an embodiment in which the user applies desired operations to an object, a corresponding icon may be displayed to visually represent each operation applied to the object. Thus, as the user performs each additional operation on the object, a corresponding icon may be immediately displayed to visually represent this operation.
Thus, when the user adds a first operation to the application, a corresponding first icon may be displayed representing the first operation. In addition, a name or other indicia of the respective first operation may optionally be displayed in a graphical user interface of the application development environment. When the user adds a second operation to the application, a corresponding second icon may then be displayed representing the second operation. In addition, a name or other indicia of the respective second operation may optionally be displayed in the graphical user interface of the application development environment. The second icon may be automatically connected to the first icon to indicate the sequence of operations. When the user adds a third operation to the application, a corresponding third icon may then be displayed representing the third operation. In addition, a name or other indicia of the respective third operation may optionally be displayed in the graphical user interface of the application development environment. The third icon may be automatically connected to the second icon to indicate the sequence of operations. The above process repeats, where new icons are added to the block diagram representation (and connected with already displayed icons) as the user adds operations to the application.
Each icon may be designed to represent the respective operation using an intuitive picture. As one example, a pattern matching machine vision operation may be represented using an icon illustrating a magnifying glass to indicate that the operation involves performing a type of search on an image. In one embodiment, a name of the operation may also be displayed together with the icon.
Displaying the plurality of interconnected icons may include displaying a plurality of lines interconnecting the icons, where each line connects two icons. For example, as each new icon is added, a line may be automatically displayed to connect the new icon to a previously displayed icon. In one embodiment, the plurality of lines may indicate an execution order for one or more operations in the application. For example, where the plurality of lines includes a first line that connects a first icon to a second icon, the first line may indicate that performing the first function includes performing the operation corresponding to the first icon before performing the operation corresponding to the second icon. In another embodiment, the first line may indicate that data produced by the operation corresponding to the first icon is used by the operation corresponding to the second icon. In various embodiments, the plurality of interconnected icons may represent one or more of data flow, control flow, and/or execution flow for the plurality of operations in the application. Thus, the plurality of interconnected icons may be referred to as a graphical programming visual representation of the plurality of operations in the application.
In one embodiment, each displayed icon may correspond to a node in a graphical programming development environment. In this embodiment, the displayed icons may be referred to as “node icons”. In other words, for each operation provided by the prototyping environment, there may be an equivalent node in the graphical programming development environment. Each of these nodes in the graphical programming development environment may have an iconic appearance that is identical to or substantially the same as the corresponding icon displayed in the prototyping environment. Thus, as the user interacts with the prototyping environment to create an application, the user may become familiar with specific nodes available for use in the graphical programming development environment. This may advantageously increase the user's efficiency when subsequently using the graphical programming development environment to create a graphical program.
In one embodiment, the displayed block diagram representation (or graphical program representation) may itself be an executable graphical program. Also, as described below, in another embodiment the prototyping environment may be operable to programmatically (automatically) generate a graphical program based on an application. Nodes in the generated graphical program may appear the same as or substantially the same as the plurality of interconnected icons displayed in 305. Thus, the user may recognize the implementation of the graphical program, making it easier to understand or modify the graphical program.
It is noted that the difference between 1) the displayed block diagram representation itself being an executable graphical program and 2) the displayed block diagram representation itself not being an executable graphical program, wherein the system is operable to programmatically (automatically) generate a graphical program based on a created application, may be invisible to the user. This is especially so where the programmatically generated graphical program has the same appearance as the block diagram representation.
In 307, information representing the application may be stored, e.g., in a data structure or file. Information is preferably stored in the data structure as the user performs each operation. The stored information may include information specifying the plurality of operations in the application. For example, the data structure or file may store names or other identifiers that identify the operations in the application. The data structure or file may also store information regarding various properties or parameter values configured for one or more operations in the application.
Thus, in one embodiment, after the user provides input to the prototyping environment to add an operation to the application, the following may be performed: 1) the operation is stored in memory, 2) an image of a displayed object may be changed according to the operation, 3) a new icon may be displayed in the block diagram representation, where the new icon represents the operation, and 4) the new icon may be connected with a pre-existing icon (assuming the new icon does not represent the first operation). The operation may also be displayed in an alternative form in the GUI of the prototyping environment. For example, the textual name of the operation may be displayed in a script window.
As mentioned above, in one embodiment each operation may be interactively executed as the operation is included in the application. As shown in 309, the entire application may also be executed, e.g., after the user has included all desired operations in the application. Executing the application may include executing the plurality of operations in the application to perform the first function. In one embodiment, the application may be executed under control of the prototyping environment application. In one embodiment, executing one or more of the operations in the application may include interacting with or controlling one or more instruments or devices, such as discussed above with reference to
In one embodiment, when an operation in the application is executed, e.g., either executed individually as the operation is included in the application or executed together with other operations in the application, results of executing the operation may be displayed together with the icon that corresponds to the operation. As one example, the operation may be executable to determine a pass/fail result. This, the pass/fail result may be displayed together with the icon that corresponds to the operation. In one embodiment, if executing an operation produces a fail result, then information indicating a reason for the fail result may also be displayed together with the icon.
In one embodiment, in addition to displaying the plurality of interconnected icons that represent the operations in the application, the operations in the application may also be displayed using other techniques. For example, the prototyping environment may also be operable to display a text view indicating the plurality of operations in the application. As one example, displaying the text view may include displaying a table including text that specifies the plurality of operations in the application.
In one embodiment, when the user executes the application, the displayed icons may be highlighted to visually indicate which of the operations in the application is currently being executed. The connections between icons may also be animated, such as with “propagating bubbles”, to visually indicate flow of data or flow of execution. Thus, the plurality of interconnected icons may be animated, such as in the “execution highlighting” feature of LabVIEW, to visually indicate which operations are being performed. In one embodiment, corresponding modifications may be made to the displayed image as the application is executed. The corresponding modifications made to the displayed image may be made as a respective icon corresponding to the operation being performed is highlighted. This provides the user with a visual indication of which operation is being executed, and how this operation is affecting the image. This provides an improved debugging or feedback tool to the user.
As noted above,
FIG. 6—Programmatically Generating a Graphical Program Based on an application
As mentioned above, in one embodiment, a graphical program may be programmatically or automatically generated based on an application.
In the present application, the term “graphical program” or “block diagram” is intended to include a program comprising graphical code, e.g., two or more interconnected nodes or icons, wherein the interconnected nodes or icons may visually indicate the functionality of the program. The nodes may be connected in one or more of a data flow, control flow, and/or execution flow format. The nodes may also be connected in a “signal flow” format, which is a subset of data flow. Thus the terms “graphical program” or “block diagram” are each intended to include a program comprising a plurality of interconnected nodes or icons which visually indicate the functionality of the program.
A graphical program may also have a graphical user interface or front panel. The user interface portion may be contained in the block diagram or may be contained in one or more separate panels or windows. The user interface of a graphical program may include various graphical user interface elements or front panel objects, such as user interface controls and/or indicators, that represent or display the respective input and/or output used by the graphical program or VI, and may include other icons which represent devices being controlled. The user interface or front panel may be included in a single window of user interface elements, or may include a plurality of individual windows each having one or more user interface elements, wherein the individual windows may optionally be tiled together. As another example, the user interface or front panel may include user interface or front panel objects, e.g., the GUI, embedded in the block diagram. The user interface of a graphical program may display only output, only input, or both input and output. Further, in some embodiments the user interface or front panel of a graphical program may enable the user to interactively control or manipulate the input being provided to the graphical program.
Examples of graphical programming development environments include LabVIEW, DasyLab, and DiaDem from National Instruments, VEE from Agilent, WiT from Coreco, Vision Program Manager from PPT Vision, SoftWIRE from Measurement Computing, Simulink from the MathWorks, Sanscript from Northwoods Software, Khoros from Khoral Research, SnapMaster from HEM Data, VisSim from Visual Solutions, ObjectBench by SES (Scientific and Engineering Software), and VisiDAQ from Advantech, among others.
In 315, the graphical program may be programmatically generated based on the application. The generated graphical program may include a plurality of interconnected nodes. The plurality of interconnected nodes may be operable to perform the first function implemented by the plurality of operations in the application.
The graphical program may be automatically generated with little or no user input received during the generation process. In one embodiment, the graphical program may be programmatically generated with no user input required. In another embodiment, the user may be prompted for certain decisions during or prior to the programmatic generation, such as the type of graphical program to generate, the look and feel of a user interface for the graphical program, the number or degree of comments contained within the graphical program, etc.
In one embodiment, each node included in the graphical program may correspond to an operation from the plurality of operations in the application. As described above with reference to
In one embodiment, the plurality of interconnected nodes in the graphical program may appear identical to or substantially the same as the plurality of interconnected icons displayed in 305. Thus, the user may easily recognize and understand the function of the graphical program when the user views the graphical program.
In one embodiment, in addition to including a plurality of interconnected nodes operable to perform the first function in the graphical program, one or more additional nodes or other elements may also be programmatically included in the graphical program. For example, one or more additional nodes may be included that do not correspond to operations from the plurality of operations in the application. For example, such additional nodes or elements may perform setup functions, cleanup functions, I/O functions, or other functions for the graphical program. In one embodiment, the graphical program may include a main block diagram having a subprogram node, where the plurality of interconnected nodes operable to perform the first function are programmatically included in a block diagram of the subprogram node.
The generated graphical program may be a graphical program associated with a particular graphical programming development environment application. For example, the program may include nodes provided by the graphical programming development environment or may run under control of the graphical programming development environment.
In one embodiment, the graphical program may be automatically displayed after the graphical program has been programmatically generated. In another embodiment, the user may request to display or open the graphical program after the graphical program has been programmatically generated. The graphical program may be displayed in a separate window from the plurality of interconnected icons displayed in 305. For example, the plurality of interconnected icons may be displayed in a window of the prototyping environment, whereas the graphical program is displayed in a window of a graphical programming development environment.
In one embodiment, the prototyping environment application may include all program logic necessary for generating the graphical program. In other words, the prototyping environment application may generate the graphical program without need of any other applications or services. Alternatively, the prototyping environment may act as a client to a server program, e.g., a graphical programming development environment application, in order to request the server program to generate part or all of the graphical program. For example, the client program may interface with the server program through an application programming interface (API). The server program may reside on the same computer system as the prototyping environment application or on a different computer system.
Additional information related to programmatically generating a graphical program may be found in the above-incorporated patent applications.
In 317, the graphical program may be executed. Executing the graphical program may include executing the plurality of interconnected nodes to perform the first function. In one embodiment, executing the graphical program may include interacting with or controlling one or more instruments or devices, such as discussed above with reference to
In another embodiment, instead of programmatically generating the entire graphical program after the user has finished creating the application, the graphical program may be interactively generated as the user creates the application. For example, as each operation is selected for inclusion in the application, a corresponding node may be dynamically included in the graphical program. The corresponding node may be operable to perform the respective operation when the graphical program is executed. Similarly, if the user requests to remove an operation from the application, the corresponding node may be dynamically removed from the graphical program.
In various embodiments, the machine vision prototyping environment may be operable to load, acquire, and/or manipulate any of various types of images, including gray-level and color images. The prototyping environment may also support complex images in which pixel values have a real part and an imaginary part. The images may be obtained from any of various sources. The images may, for example, be obtained from an image file, such as a BMP, TIFF, AIPD, PNG, JPG, or GIF file, or a file formatted according to another image format. The images may also be acquired from a hardware device, such as a camera. For example, images may be acquired from a device such as the video device 132 illustrated in
The machine vision prototyping environment may support any of various machine vision operations. As used herein, the term “machine vision operation” may include any of various types of operations related to image analysis, image processing, image acquisition, or other machine vision-related operations. For example, the machine vision prototyping environment may provide access to machine vision operations or functions such as:
It is noted that the machine vision functions listed above are exemplary only and that, in various embodiments of a machine vision prototyping environment, other types of machine vision functions or operations may be supported.
As shown in the lower right corner of
The bottom left corner of
As described above, in one embodiment, in addition to displaying a plurality of interconnected icons that represent the operations in an application, a prototyping environment may also be operable to display a table including text that specifies the plurality of operations in the application.
Although the system and method of the present invention has been described in connection with the preferred embodiment, it is not intended to be limited to the specific form set forth herein, but on the contrary, it is intended to cover such alternatives, modifications, and equivalents, as can be reasonably included within the spirit and scope of the invention as defined by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4831580 | Yamada | May 1989 | A |
| 5005119 | Rumbaugh et al. | Apr 1991 | A |
| 5481712 | Silver et al. | Jan 1996 | A |
| 5481741 | McKaskle et al. | Jan 1996 | A |
| 5576946 | Bender et al. | Nov 1996 | A |
| 5623592 | Carlson et al. | Apr 1997 | A |
| 5742504 | Meyer et al. | Apr 1998 | A |
| 5911070 | Solton et al. | Jun 1999 | A |
| 5940296 | Meyer | Aug 1999 | A |
| 5946485 | Weeren et al. | Aug 1999 | A |
| 5966532 | McDonald et al. | Oct 1999 | A |
| 6053951 | McDonald et al. | Apr 2000 | A |
| 6219628 | Kodosky et al. | Apr 2001 | B1 |
| 6298474 | Blowers et al. | Oct 2001 | B1 |
| 6331864 | Coco et al. | Dec 2001 | B1 |
| 6366300 | Ohara et al. | Apr 2002 | B1 |
| 6408429 | Marrion et al. | Jun 2002 | B1 |
| 6437805 | Sojoodi et al. | Aug 2002 | B1 |
| 6453464 | Sullivan | Sep 2002 | B1 |
| 6493003 | Martinez | Dec 2002 | B1 |
| 6571133 | Mandl et al. | May 2003 | B1 |
| 6584601 | Kodosky et al. | Jun 2003 | B1 |
| 6608638 | Kodosky et al. | Aug 2003 | B1 |
| 6637022 | Weeren et al. | Oct 2003 | B1 |
| 6763515 | Vazquez et al. | Jul 2004 | B1 |
| 6784903 | Kodosky et al. | Aug 2004 | B1 |
| 6892361 | Kandogan | May 2005 | B1 |
| 6912428 | Nakai et al. | Jun 2005 | B1 |
| 6971066 | Schultz et al. | Nov 2005 | B1 |
| 7000190 | Kudukoli et al. | Feb 2006 | B1 |
| 20010020291 | Kudukoli et al. | Sep 2001 | A1 |
| 20010035879 | Washington et al. | Nov 2001 | A1 |
| 20020083413 | Kodosky et al. | Jun 2002 | A1 |
| 20020089538 | Wenzel et al. | Jul 2002 | A1 |
| 20020129333 | Chandhoke et al. | Sep 2002 | A1 |
| 20040034696 | Joffrain et al. | Feb 2004 | A1 |
| 20040034847 | Joffrain et al. | Feb 2004 | A1 |
| 20050028138 | Case et al. | Feb 2005 | A1 |
| 20050091602 | Ramamoorthy et al. | Apr 2005 | A1 |
| Number | Date | Country |
|---|---|---|
| 1077404 | Feb 2001 | EP |
| Number | Date | Country | |
|---|---|---|---|
| 20030227483 A1 | Dec 2003 | US |
