Patent Application
20240089418
The present invention is directed to multi-projector systems, and in particular to a method of parallel gathering of structured light patterns in multi-projector systems.
Structured light systems are known for projecting a pattern (often grids or horizontal bars) that deforms when projected on a surface, and a vision system, including at least one sensing device such as a camera, to calculate the depth and surface information of the pattern for projector calibration. In a multi-projector system, structured light is gathered from the projectors sequentially, which causes the total gather time to be directly proportional to the number of projectors, as well as the number of cameras. Current structured light systems are not capable of quickly calibrating very large multi-projector systems. For example, a fifteen second gather sequence for each projector in a three hundred projector system would require seventy five minutes to sequential gather structured light from all projectors, which is an unacceptably long ‘down time’ for customers, developers, and installers trying to troubleshoot the system.
It is an aspect of the present invention to provide a method of unambiguous parallel gathering of structured light patterns in multi-projector systems without conflating which patterns come from which projectors, thereby addressing a fundamental bottleneck of gathering images for large-scale multi-projector systems.
Projectors in a multiple-projector system can be in conflict due for example to overlaps or non-distinct patterns, etc. Although overlapping can be one type of conflict, overlapping is not always a conflict. For example, three projectors can overlap but not conflict if different primary colors are used for each projector (i.e. red/green/blue) because a camera can disambiguate the patterns from each projector. Nonetheless, overlap is the conflict over which there is least control. Although distinct patterns and colors, etc., can be assigned to help identify and lower conflicts, the overlaps themselves are predetermined by the projector arrangement.
In one aspect, a method is set forth for finding conflicts by capturing projected images using one or more sensing devices, or by using other information (installation design information, previous calibrations, etc.) and then reducing these conflicts by varying the assignment of patterns/colours, separating/disambiguating patterns/projectors within an image by various masks, colors, distinct patterns, etc. and then simultaneously gathering images based on the reduced conflicts.
As discussed below, a process is set forth to determine an overlap graph for identifying “first round” conflicts. Based on that, colors and patterns can be assigned to projectors to reduce the conflicts (e.g. two projectors may overlap, but not be in conflict because they are assigned separate colors). Then, the reduced conflict graph can be resolved into a small number of non-conflicting groups that can be gathered at the same time.
In one aspect, cameras can use masks to differentiate between projectors that do not overlap. In other aspects, the patterns used can be distinct per projector, such as by the shape of the patterns (such as patterns similar to qr-codes that are different for each projector, or one projector using squares, another using triangles, etc.), and/or distinct colors can be used per projector (i.e one that projects patterns in red, another in blue, etc.) or the timing of the patterns can be distinct (including, but not limited to, subliminal timing in the DMD bit planes) or otherwise, such that the projection from each projector can be differentiated and identified by the patterns (or partial patterns).
In some embodiments, parallel gathering of images for large-scale multi-projector systems can include a) selectively masking camera images, b) decomposing the layout of projectors into a graph of projectors that conflict due for example to overlaps or non-distinct patterns, etc., with each other and therefore cannot be gathered at the same time), c) determining the best gather groups (graph coloring) of the layout, d) separating a single camera image into an image per projector based on masks and patterns, and/or e) running gather processes in parallel across groups of projectors.
The above aspects can be attained by a method of parallel gathering of projected images projected on a surface in a multi-projector system having a plurality of projectors, comprising: creating one or more masks for limiting capture by at least one sensing device of the projected images projected by each of the plurality of projectors to portions that do not overlap with projected images projected by other ones of the plurality of projectors; using the one or more masks to determine a graph of projectors whose images conflict; creating from the graph a plurality of gather groups of projectors whose images do not conflict; and simultaneously gathering images from projectors in each of the gather groups.
These together with other aspects and advantages which will be subsequently apparent, reside in the details of construction and operation as more fully hereinafter described and claimed, reference being had to the accompanying drawings forming a part hereof, wherein like numerals refer to like parts throughout.
In this description, the term “graph” is used to denote any mathematical relationship between multiple projectors or between multiple projectors to one or more sensing devices. Examples of graphs include, but are not limited to tables, a data structures, Boolean true/false statements, matrices, lists of projectors and sensing devices, etc., to describe the mathematical relationship prior to simultaneous gather of projected images, as discussed below.
When configured in a cluster, the projectors 110A, 110B, 110C display what is perceived as a single seamless composite image composed of multiple projected images 200A, 200B, 200C on a display surface 140, or other physical objects, as shown in
At least one camera 130A is provided having a field of view 210 that captures the projected images, as shown in
Determining which projectors overlap can be done in a number of ways: In one approach, each projector 110A, 110B, 110C, etc. is selectively turned on/off to determine where the projection is seen by the camera(s) to create a list of camera/projector masks 310A, 310B, 310C, etc., as shown in
In other embodiments, additional cameras may be included, such as camera 130B in
In some embodiments, masks 310A, 310B, 310C can be enlarged to accommodate for small changes in projection between different calibration runs, as shown in
According to the method set forth herein, non-overlapping images from multiple projectors can be gathered simultaneously by one or more cameras. Moreover, the numerical limit to the number of projectors whose images can be simultaneously gathered is the number of projector image conflicts. Thus, as shown in the example of
In any projector layout of connected nodes representing projector image conflicts, the minimum gather time is given by the fewest number of gather groups. Thus, for the example of
Indeed, complex arrangements of multiple projectors can be grouped according to projector conflicts to minimize the gather time for a multi-projector system. For example, in
Since the total gather time is limited by mutual projector conflict, a layout of N conflicting projectors (known as a clique) will require >=N groups. For example, a two-dimensional array of tiled projectors will require four or fewer gather groups, regardless of the array size. For example, the 4×6 array of projectors (1,1; 1,2; 1,3 . . . 4,4) shown in
Similarly, a large dome display also requires only four or fewer gather groups, as shown in
Turning now to
At 1000, processor 150 causes projectors 110A, 110B, 110C, etc. of a multi-projector system 100 to sequentially turn on/off. The sequential images are captured at 1010 by associated cameras, such as cameras 130A and 130B and collected within processor 150 which, at 1020 creates projector/camera masks, such as masks 310, 310B, 310C, etc. As discussed above, steps 1000 and 1010 may be replaced by other steps based on a priori knowledge of where images from each of the projectors will be projected on the surface and a priori knowledge of the relationship between the at least one camera and the surface. At 1030, for each projector, processor 150 then uses the masks to create a graph of other projectors that conflict, such as shown in
Since the time it takes to gather from each projector may differ depending on model/resolution/pattern/etc., knowledge of the relative projector gather times can be used to modify the graph (e.g. to group projectors with similar speeds together).
The list can be created based on camera/projector and projector/projector conflicts, an example of which is shown in Tables 1 and 2:
Based on the lists created at 1030, at 1040 processor 150 uses a graph colouring algorithm, for example, to create gather groups for simultaneous image gathering. A number of graph colouring algorithms and data structures are known in the art that can be used to create the gather groups at 1040. For example, a greedy algorithm can be used whereby vertices (i.e. nodes) of the projector graph, as shown in
The present invention has been described with respect to the exemplary embodiments above. A person or ordinary skill in the art may conceive of other modifications and variations that are intended to be covered by the appended claims. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope of the invention.
