FIELD
This disclosure relates generally to tracking systems and, more particularly, to a RGB-D camera based tracking system and method thereof.
BACKGROUND
Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to the prior art by inclusion in this section.
SUMMARY
A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.
Embodiments of the disclosure related to a visual analytics system comprises a visualization assembly, the visualization assembly comprises a historical data analytic device and a real-time tracking device. The system further comprises a data analytic assembly and a computer readable medium. wherein the real-time tracking device directly retrieves data from the data storage assembly for display. The real-time tracking device further label a set of normal processes, propagate the labels, and detect the outliers from the labeled normal processes. An outlier detection configured to detect outliers from the labeled normal processes. The system further comprises a data aggregation configured to aggregate the normal processes based on temporal proximity. The system comprises a display for displaying the detected outliers. A controller communicatively coupled to the visual analytics system for control and monitoring a plurality of parts on one or more assembly lines and one or more stations is provided. The controller records a cycle time and a fault codes when the parts being processed on the stations.
BRIEF DESCRIPTION OF THE DRAWINGS
These and other features, aspects, and advantages of this disclosure will become better understood when the following detailed description of certain exemplary embodiments is read with reference to the accompanying drawings in which like characters represent like arts throughout the drawings, wherein:
FIG. 1 is a block diagram illustrating an assembly line in a manufacturing environment;
FIG. 2 illustrates a visual diagnostics graph for a real-time tracking of assembly line performance;
FIG. 3 illustrates another visual diagnostics graph having time-aware outlier-preserving visual aggregation;
FIGS. 4A and 4B illustrate examples of visual diagnostics graph processed by quantiles brush and sample brush, respectively;
FIG. 5 illustrates a block diagram illustrating a visual diagnostics (VIDX) graph; and
FIG. 6 illustrates a block diagram illustrating a visual diagnostics (VIDX) system.
DETAILED DESCRIPTION
The following description is presented to enable any person skilled in the art to make and use the described embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments. Thus, the described embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.
Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order than the described embodiment. Various additional operations may be performed and/or described operations may be omitted in additional embodiments.
FIG. 1 is a block diagram illustrating an assembly line 102 in a manufacturing environment 100. The assembly line 102 comprises a plurality of work stations S1-S10. Parts P enter from station S1 to another stations S2-S9 to be processed and assembled to form a final product at last station S10. One or more programmable logic controllers (PLCs) 104 configured to control and monitor one or more stations S1-S10, one or more assembly lines 102, or any suitable machinery on the assembly lines 102 for manufacturing automation are provided. The PLCs 104 may be communicatively coupled the stations S1-S10, assembly lines 102, or any suitable machinery, via wireless or wired communication links, to for example a central database 106.
As can be seen, the parts P move among the stations S1-S10, S3′, and S4.5 following predetermined paths of the assembly line 102. In one embodiment, at stations S1 and S7, two different types of parts P enter different subprocesses 102a, 102b, respectively. In some embodiments, at station S6 the two types of parts P from subprocesses 102a, 102b are assembled together before a final product is fully assembled. In another embodiment, at stations S3 and S3′, same procedures are performed in parallel when the parts enter the stations. After existing the station S2, parts P can be routed to either station S3 following the assembly path A1 or station S3′ following the assembly path A2. Either assembly path A1 or A2, the parts P undergo the same procedure. In some embodiment, depending on the type of the parts, different part types existing station S2 may be routed to respective stations S3 or S3′ for further processing before, the parts P are ready for assemble at station S4. In yet another embodiment, parts P enter between stations S4 and S5 may undergoes additional procedures at station S4.5.
The PLCs 104 record an event, such as timestamps, of each part P when the part moves into each station and then send the status information of the part P to the central database 106. As the part moves along the assembly path A=(sj, . . . , sk) a sequence of timestamps is created. A required time, also known as cycle time, for the part P to complete its procedures on one station and be moved onto the next as dt(p,sj)−t(p, si) where time as t(p, si) is calculated. The PLCs 104 also record other event, such as fault codes, if any error has occurred when a part is being processed on a station. The timestamps and fault codes together are referred to as a trace or process data of a corresponding part. The process data of all the parts composing a finished product can be combined. Processes with comparatively longer cycle times on one or more stations or with faults, are referred to as outliers or abnormal processes.
FIG. 2 illustrates a visual diagnostics graph 200 for a real-time tracking of assembly line performance. On graph (A) of the visual diagnostics graph 200 depicts a smooth performance of the assembly line without any abnormal delay encounter on any manufacturing processes MP1-MPn when any products P1-Pn are processed on each station S1-S10 located on an assembly line 102. Now on graph (B) of the visual diagnostics graph 200, the graph depicts two distortions occurred at different time period. The assembly line is completely stopped during a time interval at D1 which may be caused by faults, prescheduled maintenance, breaks, or other unexpected factors, for example. At time interval D2, the assembly is partially stopped to handle unprocessed products, for example. For efficient process, line segments crossing each other between two time axes are parallel to each other and have equal-sized displacement. This indicates a rhythmic and smooth processing of the products on the assembly line where no delays or interruption of operations occurs, as illustrated in graph (A) of the visual diagnostics graph 200. However, the out-of-order processes with adnormal delays occurred at two time periods D1, D2 are visually depicted by line segments stretch much longer than the other equal sized displacement line segments, as illustrated in graph (B) of the visual diagnostics graph 200. For example, at time period D1, each process experiences some delays and the interruption of operations or adnormal delays are indicated by the lengths and the slopes on the line segments. When interruption of operations or adnormal delays occurred, no part is being moved from one station to another. Now at time period D2, stations S1 and S2 stop processing while waiting for S3 or S3′ to finish handling the parts whose processing have been delayed. In one embodiment, occurrences of faults for the line segments for display can be coded in color. In some embodiments, the level or intensity of the faults or errors for the line segments for display can be coded in different. In other embodiment, the thickness of the line segments can varies depending on the occurrences of faults for display. In yet another embodiment, different length and angle of the line segments including the occurrences of faults may be displayed. In another embodiment, manufacturing process data including occurrences of faults may be displayed as any suitable visual human readable formats. An optional audible alert may trigger operators or managers for immediate attention of a particular assembly line. Operators or managers can quickly locate faults R1, R2 at time periods D1, D2 and identify the effect of the fault occurrences on the operation of the assembly line in real-time. The visual diagnostics graph 200 can be displayed on a thin client, a PLC, on a central device, or any suitable devices. Also, the visual diagnostics graph 200 can be forwarded and shared with more than one device.
FIG. 3 illustrates another visual diagnostics graph 300 having time-aware outlier-preserving visual aggregation. The graph 300 comprises normal processes 302 and outliers 304. At normal processes, the processes are aggregated based on its temporal proximity and displayed as thick bands. The outliers 304 with faults or adnormal delays overlaid on top of the aggregated normal processes and displayed as individual polylines. In one embodiment, the aggregation of the normal processes is implemented with a set of instructions such as a greedy algorithm stored in a computer readable medium which causes a computer module to perform various functions. Other suitable algorithm can be used to perform similar functions. The computer module located locally or remotely, in some embodiments, may be a scanner, a processor, or any suitable computer modules. In one embodiment, the computer module scans the processes recorded by the PLC, determines whether to either merge the scanned processes into a current group or create a new group. If the scanned processes are temporally close to the last process in the current group, the scanned processes will be merged into the current group. Otherwise, a new group is created for the scanned processes if the difference of the starting time is greater than a predetermined threshold. To further identifying the outliers or abnormal processes based on the cycle times on the work stations, two interactive methods, e.g. quantiles brush and sample brush are provided and stored in the computer readable medium. Further details of these interactive methods will be described below.
FIGS. 4A and 4B illustrate examples of visual diagnostics graph 400A, 400B processed by quantiles brush and sample brush, respectively. Quantiles brush is descriptive statistics of a variable which split a set of observations into equally sized bins. The p-quantile of a variable given a set of n samples is a value q(p), for which there are at least np samples smaller than or equal to the value q(p). Operators or managers can specify outliers among the processes based on quantiles by either selecting a pair of values (p0,p1)(p0<p1) from a range [0,1]. The corresponding quantiles (q(p0),q(p1)) for the cycle times on each station will then be calculated by any suitable computer module. Processes with cycle times lying outside the range of the corresponding quantiles (q(p0),q(p1)) on any stations are identified as outliers. In some embodiments, the range for individual stations may be adjusted depending on the applications. As illustrated in FIG. 4A, a quantile range selector 402 comprises distribution of the cycle times on each station. Outlier processes are display as individual polylines in an aggregated graph 404, similar to the aggregated graph 300 illustrated in FIG. 3. When a new set of outliers is detected, the aggregated graph 404 as well as the quantile range selector 402 are updated accordingly.
Now referring to FIG. 4B, an aggregated graph 406 similar to the aggregated graph 404 is illustrated. In this example, the second interactive method such as a samples brush is provided for detecting the outlier processes. A set of normal processes is labeled using a suitable graph-based learning algorithm such as label propagation algorithm stored in the computer readable medium. To detect the outliers, a neighborhood graph such as a k-nearest neighbor graph containing both the label and unlabeled data points is constructed. Then iteratively propagating the labels along the graph edges, starting from the labeled points. The iteration stops when the labels of the data points no longer change. As illustrated in FIG. 4B, at step 1, a k-nn graph of the processes based on the cycle times on the stations using a Euclidean distance metric is constructed. A threshold on the maximum neighborhood distances in the k-nn graph to stop labels from propagating to very dissimilar process is set at step 2. Next step 3, the normal label is propagated through the k-nn graph and gradually covers the dense regions in the data set containing the sample normal process. The remaining unlabeled processes are outliers which is displayed in the graph as individual polylines at step 4 and the normal process are aggregated.
FIG. 5 illustrates a block diagram illustrating a visual diagnostics (VIDX) graph 500 comprises a calendar based visualization 502, a timeline 504 for multi-scale temporal exploration, and a visual diagnostics graph 506. Other features such as a real-time monitoring panel in a radical graph 508, multiples of histograms 510 showing the distribution of the cycle time one each station with a quantile range selector, a station map denoting the assembly line schema 512, and an explorable 3D station model visualization 514, and expandable table of color code of the faults and the frequencies of the occurrences 516 may be provided for display on any suitable devices as well as for share with any devices.
FIG. 6 illustrates a block diagram of a visual diagnostics (VIDX) system 600 comprises a visualization assembly 602, a data analytic assembly 604, and a data storage assembly or a computer readable medium 606 communicatively coupled to each other via one or more buses. Other computer assembly may be integrated into or coupled to the system 600. The visualization assembly 602 comprises a historical data analytic device 608 and a real-time tracking device 610. The data analytic assembly 604 comprises a summary statistics device 612, an outlier detection 614, and a data aggregation 616. The data storage assembly 606 comprises a database device 618 and a 3D station models 620. The database device 618 stores the manufacturing process data and then indexed the data by timestamps to support the efficient retrieval of data that falls within a time interval. The summary statistics device 612 compute a summary statistics in the visualizations in advance in the data analytic assembly 604 and cache the results in the database device 618 for faster response time. The outlier detection 614 configured to detect outlier processes and the data aggregation 616 aggregates the normal processes based on temporal proximity. In the visualization assembly 602, data is displayed from the historical data analytic device 608 whereas the real-time tracking device 610 display data fetched directly from the data storage assembly 606. Various features 502-516 of the visual diagnostics (VIDX) graph 500 illustrated in FIG. 5 is displayed for interaction.
The embodiments described above have been shown by way of example, and it should be understood that these embodiments may be susceptible to various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the particular forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling with the sprit and scope of this disclosure.
Embodiments within the scope of the disclosure may also include non-transitory computer-readable storage media or machine-readable medium for carrying or having computer-executable instructions or data structures stored thereon. Such non-transitory computer-readable storage media or machine-readable medium may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such non-transitory computer-readable storage media or machine-readable medium can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. Combinations of the above should also be included within the scope of the non-transitory computer-readable storage media or machine-readable medium.
Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network.
Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
While the patent has been described with reference to various embodiments, it will be understood that these embodiments are illustrative and that the scope of the disclosure is not limited to them. Many variations, modifications, additions, and improvements are possible. More generally, embodiments in accordance with the patent have been described in the context or particular embodiments. Functionality may be separated or combined in blocks differently in various embodiments of the disclosure or described with different terminology. These and other variations, modifications, additions, and improvements may fall within the scope of the disclosure as defined in the claims that follow.
Patent Application
20190204804
