Patent Grant
10878585
This disclosure relates generally to a distributed system for tracking physical positions of people and objects.
Position tracking systems are used to track the physical positions of people and/or objects.
Position tracking systems are used to track the physical positions of people and/or objects in a physical space (e.g., a store). These systems typically use a sensor (e.g., a camera) to detect the presence of a person and/or object and a computer to determine the physical position of the person and/or object based on signals from the sensor. In a store setting, other types of sensors can be installed to track the movement of inventory within the store. For example, weight sensors can be installed on racks and shelves to determine when items have been removed from those racks and shelves. By tracking both the positions of persons in a store and when items have been removed from shelves, it is possible for the computer to determine which user in the store removed the item and to charge that user for the item without needing to ring up the item at a register. In other words, the person can walk into the store, take items, and leave the store without stopping for the conventional checkout process.
For larger physical spaces (e.g., convenience stores and grocery stores), additional sensors can be installed throughout the space to track the position of people and/or objects as they move about the space. For example, additional cameras can be added to track positions in the larger space and additional weight sensors can be added to track additional items and shelves. There is a limit, however, to the number of sensors that can be added before the computing capabilities of the computer are reached. As a result, the computing power of the computer limits the coverage area of the tracking system.
One way to scale these systems to handle larger spaces is to add additional computers and to divide the sensors amongst these computers such that each computer processes signals from a subset of the sensors. However, dividing the sensors amongst multiple computers introduces synchronization issues. For example, the sensors may not communicate signals to their respective computers at the same time or simultaneously. As another example, sensors may have different latencies with their respective computers and thus, it may take more time for signals from one sensor to reach a computer than signals from another sensor. As a result, the sensors and computers become desynchronized with each other and it becomes more difficult for the computers to determine, in a cohesive way, the position of persons or objects in the space and when items were removed.
This disclosure contemplates an unconventional, distributed tracking system that can scale to handle larger spaces. The system uses an array of cameras, multiple camera clients, a camera server, weight sensors, a weight server, and a central server to determine which person in the space took an item and should be charged for the item. The camera clients each process frames of videos from a different subset of cameras of the array of cameras. Each camera client determines coordinates for people detected in the frames and then timestamps these coordinates based on when the frames were received by the camera client. The camera clients then communicate the coordinates and timestamps to a camera server that is responsible for coordinating the information from the camera clients. The camera server determines, based on the coordinates and timestamps from the camera clients, the positions of people in the space. The weight server processes signals from the weight sensors to determine when items were removed from shelves in the space. The central server uses the positions of people in the space from the camera server and the determinations from the weight server of when items were removed from shelves to determine which people in the space took which items and should therefore be charged.
Generally, the camera server protects against desynchronization by assigning the coordinates from the multiple camera clients to windows of time based on the timestamps. The camera server then processes the coordinates assigned to a particular time window to determine overall coordinates for people in the space during that time window. The duration of the time window can be set to be larger than the desynchronization that is expected to occur to mitigate the effects of desynchronization. For example, if the cameras and camera clients are expected to desynchronize by a few milliseconds, then the time window can be set to last 100 milliseconds to counteract the desynchronization. In this manner, the number of cameras and camera clients can be increased to scale the system to handle any suitable space.
This disclosure also contemplates an unconventional way of wiring cameras in the array of cameras to the camera clients. The cameras are arranged as a rectangular grid above the space. Each camera in the grid is wired to a particular camera client according to certain rules. For example, no two cameras that are directly adjacent to one another in the same row or column of the grid are wired to the same camera client. As another example, cameras that are arranged along a diagonal in the grid are wired to the same camera client. In this manner, a small area of the grid should include cameras that are wired to each and every camera client in the system. As a result, even if one camera client were to go offline (e.g., maintenance, error, or crash), there would still be enough coverage from the remaining camera clients to track the positions of people in the small area. Thus, this arrangement of the cameras improves the resiliency of the system.
This disclosure further contemplates an unconventional rack and shelf design that integrates weight sensors for tracking when items have been removed from the racks and shelves. Generally, the rack includes a base, a vertical panel, and a shelf. The base forms an enclosed space in which a printed circuit board is positioned, and the base includes a drawer that opens to provide access to the enclosed space and the circuit board. The vertical panel is attached to the base, and the shelf is attached to the vertical panel. Weight sensors are positioned within the shelf. The base, panel, and shelf each define a cavity. The cavity in the shelf and the cavity in the panel are at least partially aligned. Each weight sensor communicates signals to the printed circuit board through a wire that runs from that weight sensor, through the cavity of the shelf, the cavity of the panel, and the cavity of the base, to the circuit board.
Certain embodiments include an unconventional tracking system that includes separate components (e.g., camera clients, camera servers, weight servers, and a central server) that perform different functions to track the positions of people and/or objects in a space. By spreading the functionality of the system amongst these various components, the system is capable of processing signals from more sensors (e.g., cameras and weight sensors). Due to the increase in the number of sensors, the system can track people and/or objects in a larger space. As a result, the system can be scaled to handle larger spaces (e.g., by adding additional camera clients). Certain embodiments of the tracking system are described below.
According to an embodiment, a system includes an array of cameras, a first camera client, a second camera client, a camera server, a plurality of weight sensors, a weight server, and a central server. The array of cameras is positioned above a space. Each camera of the array of cameras captures a video of a portion of the space. The space contains a person. The first camera client receives a first plurality of frames of a first video from a first camera of the array of cameras. Each frame of the first plurality of frames shows the person within the space. For a first frame of the first plurality of frames, the first camera client determines a first bounding area around the person shown in the first frame and generates a first timestamp of when the first frame was received by the first camera client. For a second frame of the first plurality of frames, the first camera client determines a second bounding area around the person shown in the second frame and generates a second timestamp of when the second frame was received by the first camera client. The second camera client is separate from the first camera client. The second camera client receives a second plurality of frames of a second video from a second camera of the array of cameras. Each frame of the second plurality of frames shows the person within the space. For a third frame of the second plurality of frames, the second camera client determines a third bounding area around the person shown in the third frame and generates a third timestamp of when the third frame was received by the second camera client. For a fourth frame of the second plurality of frames, the second camera client determines a fourth bounding area around the person shown in the fourth frame and generates a fourth timestamp of when the fourth frame was received by the second camera client.
The camera server is separate from the first and second camera clients. The camera server determines that the first timestamp falls within a first time window and in response to determining that the first timestamp falls within the first time window, assigns coordinates defining the first bounding area to the first time window. The camera server also determines that the second timestamp falls within the first time window and in response to determining that the second timestamp falls within the first time window, assigns coordinates defining the second bounding area to the first time window. The camera server further determines that the third timestamp falls within the first time window and in response to determining that the third timestamp falls within the first time window, assigns coordinates defining the third bounding area to the first time window. The camera server determines that the fourth timestamp falls within a second time window that follows the first time window and in response to determining that the fourth timestamp falls within the second time window, assigns coordinates defining the fourth bounding area to the second time window.
The camera server also determines that coordinates assigned to the first time window should be processed and in response to determining that coordinates assigned to the first time window should be processed, the camera server calculates, based at least on the coordinates defining the first bounding area and the coordinates defining the second bounding area, a combined coordinate for the person during the first time window for the first video from the first camera and calculates, based at least on the coordinates defining the third bounding area, a combined coordinate for the person during the first time window for the second video from the second camera. The camera server also determines, based at least on the combined coordinate for the person during the first time window for the first video from the first camera and the combined coordinate for the person during the first time window for the second video from the second camera, a position of the person within the space during the first time window.
The plurality of weight sensors are positioned within the space. Each weight sensor of the plurality of weight sensors produces a signal indicative of a weight experienced by that weight sensor. The weight server is separate from the first and second camera clients and the camera server. The weight server determines, based at least on a signal produced by a first weight sensor of the plurality of weight sensors, that an item positioned above the first weight sensor was removed. The central server is separate from the first and second camera clients, the camera server, and the weight server. The central server determines, based at least on the position of the person within the space during the first time window, that the person removed the item. Based at least on the determination that the first person removed the item, the person is charged for the item when the person exits the space.
According to another embodiment, a system includes an array of cameras, a first camera client, a second camera client, a camera server, a plurality of weight sensors, a weight server, and a central server. The array of cameras is positioned above a space. Each camera of the array of cameras captures a video of a portion of the space. The space contains a person. The first camera client, for each frame of a first video received from a first camera of the array of cameras, determines a bounding area around the person shown in that frame of the first video and generates a timestamp of when that frame of the first video was received by the first camera client. The second camera client, for each frame of a second video received from a second camera of the array of cameras, determines a bounding area around the person shown in that frame of the second video and generates a timestamp of when that frame of the second video was received by the second camera client.
The camera server is separate from the first and second camera clients. The camera server, for each frame of the first video, assigns, based at least on the timestamp of when that frame was received by the first camera client, coordinates defining the bounding area around the person shown in that frame to one of a plurality of time windows. For each frame of the second plurality of frames, the camera server assigns, based at least on the timestamp of when that frame was received by the second camera client, coordinates defining the bounding area around the person shown in that frame to one of the plurality of time windows. For a first time window of the plurality of time windows, the camera server calculates, based at least on the coordinates that (1) define bounding areas around the person shown in the first plurality of frames and (2) are assigned to the first time window, a combined coordinate for the person during the first time window for the first video from the first camera and calculates, based at least on the coordinates that (1) define bounding areas around the person shown in the second plurality of frames and (2) are assigned to the first time window, a combined coordinate for the person during the first time window for the second video from the second camera. The camera server determines, based at least on the combined coordinate for the person during the first time window for the first video from the first camera and the combined coordinate for the person during the first time window for the second video from the second camera, a position of the person within the space during the first time window.
The plurality of weight sensors are positioned within the space. The weight server is separate from the first and second camera clients and the camera server. The weight server determines, based at least on a signal produced by a first weight sensor of the plurality of weight sensors, that an item positioned above the first weight sensor was removed. The central server is separate from the first and second camera clients, the camera server, and the weight server. The central server determines, based at least on the position of the person within the space during the first time window, that the person removed the item.
Certain embodiments of the tracking system perform an unconventional tracking process that allows for some desynchronization amongst the components of the system (e.g., camera clients and camera server). Generally, the system processes information according to time windows. These time windows may be set to be larger than the desynchronization that is expected to exist in the system. Information that is assigned to a time window is processed together. Thus, even if some desynchronization exists amongst that information, it is nevertheless processed together within the same time window. In this manner, the tracking system can handle an increased amount of desynchronization, especially desynchronization that occurs as a result of the system being scaled to include more components so that the system can handle a larger space. As a result, the system can scale to handle larger spaces while maintaining reliability and accuracy. Certain embodiments of the tracking process are described below.
According to an embodiment, a system includes an array of cameras, a first camera client, a second camera client, and a camera server. The array of cameras is positioned above a space. Each camera of the array of cameras captures a video of a portion of the space. The space contains a person. The first camera client receives a first plurality of frames of a first video from a first camera of the array of cameras. Each frame of the first plurality of frames shows the person within the space. For a first frame of the first plurality of frames, the first camera client determines a first bounding area around the person shown in the first frame and generates a first timestamp of when the first frame was received by the first camera client. For a second frame of the first plurality of frames, the first camera client determines a second bounding area around the person shown in the second frame and generates a second timestamp of when the second frame was received by the first camera client. For a third frame of the first plurality of frames, the first camera client determines a third bounding area around the person shown in the third frame and generates a third timestamp of when the third frame was received by the first camera client.
The second camera client receives a second plurality of frames of a second video from a second camera of the array of cameras. Each frame of the second plurality of frames shows the person within the space. For a fourth frame of the second plurality of frames, the second camera client determines a fourth bounding area around the person shown in the fourth frame and generates a fourth timestamp of when the fourth frame was received by the second camera client. For a fifth frame of the second plurality of frames, the second camera client determines a fifth bounding area around the person shown in the fifth frame and generates a fifth timestamp of when the fifth frame was received by the second camera client.
The camera server is separate from the first and second camera clients. The camera server determines that the first timestamp falls within a first time window and in response to determining that the first timestamp falls within the first time window, assigns coordinates defining the first bounding area to the first time window. The camera server also determines that the second timestamp falls within the first time window and in response to determining that the second timestamp falls within the first time window, assigns coordinates defining the second bounding area to the first time window. The camera server further determines that the third timestamp falls within a second time window that follows the first time window and in response to determining that the third timestamp falls within the second time window, assigns coordinates defining the third bounding area to the second time window. The camera server also determines that the fourth timestamp falls within the first time window and in response to determining that the fourth timestamp falls within the first time window, assigns coordinates defining the fourth bounding area to the first time window. The camera server further determines that the fifth timestamp falls within the second time window and in response to determining that the fifth timestamp falls within the second time window, assigns coordinates defining the fifth bounding area to the second time window.
The camera server also determines that coordinates assigned to the first time window should be processed and in response to determining that coordinates assigned to the first time window should be processed, the camera server calculates, based at least on the coordinates defining the first bounding area and the coordinates defining the second bounding area, a combined coordinate for the person during the first time window for the first video from the first camera and calculates, based at least on the coordinates defining the fourth bounding area, a combined coordinate for the person during the first time window for the second video from the second camera. After determining that coordinates assigned to the first time window should be processed, the camera server determines that coordinates assigned to the second time window should be processed and in response to determining that coordinates assigned to the second time window should be processed, the camera servers calculates, based at least on the coordinates defining the third bounding area, a combined coordinate for the person during the second time window for the first video from the first camera and calculates, based at least on the coordinates defining the fifth bounding area, a combined coordinate for the person during the second time window for the second video from the second camera.
According to another embodiment, a system includes an array of cameras, a first camera client, a second camera client, and a camera server. The array of cameras is positioned above a space. Each camera of the array of cameras captures a video of a portion of the space. The space contains a person. The first camera client receives a first plurality of frames of a first video from a first camera of the array of cameras. Each frame of the first plurality of frames shows the person within the space. For each frame of the first plurality of frames, the first camera client determines a bounding area around the person shown in that frame and generates a timestamp of when that frame was received by the first camera client. The second camera client receives a second plurality of frames of a second video from a second camera of the array of cameras. Each frame of the second plurality of frames shows the person within the space. For each frame of the second plurality of frames, the second camera client determines a bounding area around the person shown in that frame and generates a timestamp of when that frame was received by the second camera client.
The camera server is separate from the first and second camera clients. The camera server, for each frame of the first plurality of frames, assigns, based at least on the timestamp of when that frame was received by the first camera client, coordinates defining the bounding area around the person shown in that frame to one of a plurality of time windows and for each frame of the second plurality of frames, assigns, based at least on the timestamp of when that frame was received by the second camera client, coordinates defining the bounding area around the person shown in that frame to one of the plurality of time windows.
The camera server also determines that coordinates assigned to a first time window of the plurality of time windows should be processed and in response to determining that coordinates assigned to the first time window should be processed, calculates, based at least on the coordinates that (1) define bounding areas around the person shown in the first plurality of frames and (2) are assigned to the first time window, a combined coordinate for the person during the first time window for the first video from the first camera and calculates, based at least on the coordinates that (1) define bounding areas around the person shown in the second plurality of frames and (2) are assigned to the first time window, a combined coordinate for the person during the first time window for the second video from the second camera.
Certain embodiments include an unconventional arrangement of cameras and camera clients that improve the resiliency of the camera system. Generally, the cameras are arranged in a rectangular grid that provides coverage for a physical space, and each camera is communicatively coupled to one camera client. No camera is directly adjacent in the same row or column of the grid to another camera that is communicatively coupled to the same camera client. Cameras arranged along a diagonal of the grid are communicatively coupled to the same camera client. In this manner, even if one camera client in the system were to go offline, the grid still provides sufficient coverage for the physical space. As a result, the arrangement of the cameras improves the resiliency of the system. Certain embodiments of the camera arrangement are described below.
According to an embodiment, a system includes a first camera client, a second camera client, a third camera client, and an array of cameras. The second camera client is separate from the first camera client. The third camera client is separate from the first and second camera clients. The array of cameras is positioned above a space. The cameras in the array of cameras are arranged as a rectangular grid comprising a first row, a second row, a third row, a first column, a second column, and a third column. The array includes first, second, third, fourth, fifth, and sixth cameras.
The first camera is positioned in the first row and the first column of the grid. The first camera is communicatively coupled to the first camera client. The first camera communicates a video of a first portion of the space to the first camera client. The second camera is positioned in the first row and the second column of the grid such that the second camera is directly adjacent to the first camera in the grid. The second camera is communicatively coupled to the second camera client. The second camera communicates a video of a second portion of the space to the second camera client. The third camera is positioned in the first row and the third column of the grid such that the third camera is directly adjacent to the second camera in the grid. The third camera is communicatively coupled to the third camera client. The third camera communicates a video of a third portion of the space to the third camera client. The fourth camera is positioned in the second row and the first column of the grid such that the fourth camera is directly adjacent to the first camera in the grid. The fourth camera is communicatively coupled to the second camera client. The fourth camera communicates a video of a fourth portion of the space to the second camera client. The fifth camera is positioned in the second row and the second column of the grid such that the fifth camera is directly adjacent to the fourth camera and the second camera in the grid. The fifth camera is communicatively coupled to the third camera client. The fifth camera communicates a video of a fifth portion of the space to the third camera client. The sixth camera is positioned in the third row and the first column of the grid such that the sixth camera is directly adjacent to the fourth camera in the grid. The sixth camera is communicatively coupled to the third camera client. The sixth camera communicates a video of a sixth portion of the space to the third camera client.
According to another embodiment, a system includes a plurality of camera clients and an array of cameras. The plurality of camera clients includes a number of camera clients. The array of cameras is positioned above a space. Each camera in the array of cameras communicates a video of a portion of the space to only one camera client of the plurality of camera clients. The cameras in the array of cameras are arranged such that each camera client of the plurality of camera clients is communicatively coupled to at least one camera in an N×N portion of the array. N is the number of camera clients in the plurality of camera clients minus one.
Certain embodiments include an unconventional rack for holding items. The rack includes a base and panels for holding shelves and weight sensors. The weight sensors are wired to a circuit board located in a drawer in the base. The wires run from the weight sensors through cavities and spaces defined by the shelves, panels, and base. Certain embodiments of the rack are described below.
According to an embodiment, a system includes a circuit board and a rack. The rack includes a base, a panel, a shelf, a first weight sensor, a second weight sensor, a first wire, and a second wire. The base includes a bottom surface, a first side surface, a second side surface, a third side surface, a top surface, and a drawer. The first side surface is coupled to the bottom surface of the base. The first side surface of the base extends upwards from the bottom surface of the base. The second side surface is coupled to the bottom and first side surfaces of the base. The second side surface of the base extends upwards from the bottom surface of the base. The third side surface is coupled to the bottom and second side surfaces of the base. The third side surface of the base extends upwards from the bottom surface of the base. The top surface is coupled to the first, second, and third side surfaces of the base such that the bottom and top surfaces of the base and the first, second, and third side surfaces of the base define a space. The top surface of the base defines a first opening into the space. The drawer is positioned within the space. The circuit board is positioned within the drawer.
The panel is coupled to the base and extends upwards from the base. The panel defines a second opening that extends along a width of the panel. The shelf is coupled to the panel such that the shelf is positioned vertically higher than the base and such that the shelf extends away from the panel. The shelf includes a bottom surface, a front surface that extends upwards from the bottom surface of the shelf, and a back surface that extends upwards from the bottom surface of the shelf. The back surface of the shelf is coupled to the panel. The back surface of the shelf defines a third opening. A portion of the third opening aligns with a portion of the second opening.
The first weight sensor is coupled to the bottom surface of the shelf and positioned between the front surface of the shelf and the back surface of the shelf. The second weight sensor is coupled to the bottom surface of the shelf and positioned between the front surface of the shelf and the back surface of the shelf. The first wire is coupled to the first weight sensor and the circuit board The first wire extends from the first weight sensor through the second and third openings and downwards into the space through the first opening. The second wire is coupled to the second weight sensor and the circuit board. The second wire extends from the second weight sensor through the second and third openings and downwards into the space through the first opening.
Certain embodiments may include none, some, or all of the above technical advantages discussed above. One or more other technical advantages may be readily apparent to one skilled in the art from the figures, descriptions, and claims included herein.
For a more complete understanding of the present disclosure, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:
Embodiments of the present disclosure and its advantages are best understood by referring to
Position tracking systems are used to track the physical positions of people and/or objects in a physical space (e.g., a store). These systems typically use a sensor (e.g., a camera) to detect the presence of a person and/or object and a computer to determine the physical position of the person and/or object based on signals from the sensor. In a store setting, other types of sensors can be installed to track the movement of inventory within the store. For example, weight sensors can be installed on racks and shelves to determine when items have been removed from those racks and shelves. By tracking both the positions of persons in a store and when items have been removed from shelves, it is possible for the computer to determine which user in the store removed the item and to charge that user for the item without needing to ring up the item at a register. In other words, the person can walk into the store, take items, and leave the store without stopping for the conventional checkout process.
For larger physical spaces (e.g., convenience stores and grocery stores), additional sensors can be installed throughout the space to track the position of people and/or objects as they move about the space. For example, additional cameras can be added to track positions in the larger space and additional weight sensors can be added to track additional items and shelves. There is a limit, however, to the number of sensors that can be added before the computing capabilities of the computer are reached. As a result, the computing power of the computer limits the coverage area of the tracking system.
One way to scale these systems to handle larger spaces is to add additional computers and to divide the sensors amongst these computers such that each computer processes signals from a subset of the sensors. However, dividing the sensors amongst multiple computers introduces synchronization issues. For example, the sensors may not communicate signals to their respective computers at the same time or simultaneously. As another example, sensors may have different latencies with their respective computers and thus, it may take more time for signals from one sensor to reach a computer than signals from another sensor. As a result, the sensors and computers become desynchronized with each other and it becomes more difficult for the computers to determine, in a cohesive way, the position of persons or objects in the space and when items were removed.
This disclosure contemplates an unconventional, distributed tracking system that can scale to handle larger spaces. The system uses an array of cameras, multiple camera clients, a camera server, weight sensors, a weight server, and a central server to determine which person in the space took an item and should be charged for the item. The camera clients each process frames of videos from a different subset of cameras of the array of cameras. Each camera client determines coordinates for people detected in the frames and then timestamps these coordinates based on when the frames were received by the camera client. The camera clients then communicate the coordinates and timestamps to a camera server that is responsible for coordinating the information from the camera clients. The camera server determines, based on the coordinates and timestamps from the camera clients, the positions of people in the space. The weight server processes signals from the weight sensors to determine when items were removed from shelves in the space. The central server uses the positions of people in the space from the camera server and the determinations from the weight server of when items were removed from shelves to determine which people in the space took which items and should therefore be charged. The system will be described in more detail using
Generally, the camera server protects against desynchronization by assigning the coordinates from the multiple camera clients to windows of time based on the timestamps. The camera server then processes the coordinates assigned to a particular time window to determine overall coordinates for people in the space during that time window. The duration of the time window can be set to be larger than the desynchronization that is expected to occur to mitigate the effects of desynchronization. For example, if the cameras and camera clients are expected to desynchronize by a few milliseconds, then the time window can be set to last 100 milliseconds to counteract the desynchronization. In this manner, the number of cameras and camera clients can be increased to scale the system to handle any suitable space. The cameras, camera clients, and camera server will be described in more detail using
This disclosure also contemplates an unconventional way of wiring cameras in the array of cameras to the camera clients. The cameras are arranged as a rectangular grid above the space. Each camera in the grid is wired to a particular camera client according to certain rules. For example, no two cameras that are directly adjacent to one another in the same row or column of the grid are wired to the same camera client. As another example, cameras that are arranged along a diagonal in the grid are wired to the same camera client. In this manner, a small area of the grid should include cameras that are wired to each and every camera client in the system. As a result, even if one camera client were to go offline (e.g., maintenance, error, or crash), there would still be enough coverage from the remaining camera clients to track the positions of people in the area. Thus, this arrangement of the cameras improves the resiliency of the system. The camera array will be described in more detail using
This disclosure further contemplates an unconventional rack and shelf design that integrates weight sensors for tracking when items have been removed from the racks and shelves. Generally, the rack includes a base, a vertical panel, and a shelf. The base forms an enclosed space in which a printed circuit board is positioned, and the base includes a drawer that opens to provide access to the enclosed space and the circuit board. The vertical panel is attached to the base, and the shelf is attached to the vertical panel. Weight sensors are positioned within the shelf. The base, panel, and shelf each define a cavity. The cavity in the shelf and the cavity in the panel are at least partially aligned. Each weight sensor communicates signals to the printed circuit board through a wire that runs from that weight sensor, through the cavity of the shelf, the cavity of the panel, and the cavity of the base, to the circuit board. The rack and shelf design will be described in more detail using
The system may also include a light detection and ranging (LiDAR) subsystem that will be described in more detail using
Gates 125 may include scanners 110 and 120. Scanners 110 and 120 may include a QR code scanner, a barcode scanner, or any other suitable type of scanner that can receive an electronic code embedded with information, such as information that uniquely identifies a shopper 105. Shoppers 105 may scan a personal device (e.g., a smartphone) on scanners 110 to enter store 100. When a shopper 105 scans a personal device on scanners 110, the personal device may provide scanners 110 an electronic code that uniquely identifies the shopper 105. When the shopper 105 is identified and/or authenticated, gate 125 that includes scanner 110 opens to allow the shopper 105 into store 100. Each shopper 105 may have registered an account with store 100 to receive an identification code for the personal device.
After entering store 100, shoppers 105 may move around the interior of store 100. As the shoppers 105 move throughout the space, shoppers 105 may shop for items 130 by removing items 130 from racks 115. As seen in
When shopper 105 has completed shopping for items 130, shopper 105 approaches gates 125. In certain embodiments, gates 125 will automatically open so that shopper 105 can leave store 100. In other embodiments, shopper 105 scans a personal device on scanners 120 before gates 125 will open to allow shopper 105 to exit store 100. When shopper 105 scans a personal device on scanner 120, the personal device may provide an electronic code that uniquely identifies shopper 105 to indicate that shopper 105 is leaving store 100. When shopper 105 leaves store 100, an account of shopper 105 is charged for the items 130 that shopper 105 removed from store 100.
The tracking system 132 also includes weight sensors 140 positioned on the racks 115. Weight sensors 140 can detect the weight of items 130 positioned on racks 115. When an item 130 is removed from the rack 115, the weight sensor 140 may detect a decrease in weight. The tracking system 132 may use that information to determine that a particular item 130 was removed from the rack 115.
The tracking system 132 includes a computer system 145. Computer system 145 may include multiple computers that operate together to determine which shopper 105 took which items 130 from racks 115. The components of computer system 145 and their operation will be described in more detail using
Generally, cameras 205 produce videos of portions of the interior of a space. These videos may include frames or images of shoppers 105 within the space. The camera clients 220 process the frames from the cameras 205 to detect shoppers 105 within the frames and to assign frame coordinates to those shoppers 105. The camera server 225 generally processes frame data from the camera clients 220 to determine the physical position of shoppers 105 within the space. LiDAR sensors 210 generally produce coordinates of shoppers 105 within a space. LiDAR server 230 processes these coordinates to determine the position of shoppers 105 within a space. Weight sensors 215 detect the weight of items 130 on racks 115 within the space. Weight server 235 processes these weights to determine when certain items 130 have been removed from the racks 115.
Central server 240 processes position information for shoppers 105 from camera server 225 and LiDAR server 230 and weight information from weight server 235 to determine which shopper 105 removed which items 130 from the racks 115. These shoppers 105 may then be charged for those items 130 when the shoppers 105 leave the space. The operation of these components will be described in more detail using
In one embodiment, each of the components of tracking system 132 (e.g. camera clients 220, camera server 225, LiDAR server 230, weight server 235, and central server 240) is a distinct computing device separate from the other components of tracking system 132. For example, each of these components may include its own processor, memory, and physical housing. In this manner, the components of tracking system 132 are distributed to provide additional computing power relative to a tracking system that includes only one computer.
Generally, cameras 305 in camera array 300 are arranged to form a rectangular array. In the example of
Each camera 305 of camera array 300 is communicatively coupled to a camera client 220. In the example of
Cameras 305 are communicatively to camera clients 220 using any suitable medium. For example, cameras 305 may be hardwired to camera clients 220. As another example, cameras 305 may wirelessly couple to camera clients 220 using any suitable wireless protocol (e.g., WiFi). Cameras 305 communicate captured videos through the communication medium to the camera clients 220.
Cameras 305 may be any suitable devices for capturing videos of the space. For example, cameras 305 may be three-dimensional cameras that can capture two-dimensional video of the space (e.g., x-y plane) and also detect the heights of people and/or objects in the video (e.g., z plane). As another example, cameras 305 may be two-dimensional cameras that capture two-dimensional videos of the space. Camera array 300 may include a mixture of different types of cameras 305.
Each field of view 310 is shaded differently than its neighbors to distinguish the fields of view 310. Fields of view 310A, 310C, 310I, and 310K are shaded using lines that slant downwards to the right. Fields of view 310B, 310D, 310J, and 310L are shaded using lines that slant upwards to the right. Fields of view 310E and 310G are shaded using horizontal lines, and fields of view 310F and 310H are shaded using vertical lines. The shading of each field of view 310 is meant to distinguish that field of view 310 from other, directly adjacent fields of view 310. The shading is not meant to indicate a particular characteristic of the field of view 310. In other words, even though certain fields of view 310 share the same shading, the similar shading does not indicate that these fields of view 310 share certain characteristics (e.g., size, coverage, duration, and/or shape). Fields of view 310 may share one or more of these characteristics irrespective of their individual shading.
As seen in
The overlapping fields of view 310 may be a result of the proximity of cameras 305 to each other in camera array 300. Generally, by overlapping fields of view 310, certain portions of the floor space can be captured by multiple cameras 305 of the camera array 300. As a result, even if certain cameras 305 go offline, there may still be sufficient coverage provided by the remaining cameras 305 for the tracking system 132 to operate. Additionally, the overlapping fields of view 310 may improve tracking the positions of people (e.g., shoppers 105) as they move about the space.
Camera grid 315 shows that cameras 305 are communicatively coupled to camera clients 220 according to particular rules. For example, a camera 305 that is communicatively coupled to a particular camera client 220 is not directly adjacent in the same row or the same column of camera grid 315 to another camera 305 that is communicatively coupled to the same camera client 220. As seen in
As another example, a camera 305 that is communicatively coupled to a particular camera client 220 is diagonal in camera grid 315 to another camera 305 that is communicatively coupled to the same camera client 220. As seen in
A consequence of arranging cameras 305 in this manner is that each camera client 220 is communicatively coupled to at least one camera 305 in a portion of camera grid 315. As seen in the example of
Although the previous example used a certain number of cameras 305 and a certain number of camera clients 220, the tracking system 132 may use any suitable number of cameras 305 and any suitable number of camera clients 220 to provide a desired level of overlap, scalability, and resiliency.
As shown in
During initialization, the cameras 305 of camera array 300 may power on and perform a startup sequence. For example, the components of the cameras 305 may boot up and/or warm-up. The cameras 305 may then begin capturing video footage and communicating videos 302 to their respective camera clients 220. The cameras 305 of camera array 300 may take different amounts of time to initialize. For example, certain cameras 305 may take a shorter or longer amount of time to initialize than other cameras 305 of camera array 300. Because the cameras 305 of camera array 300 do not wait for the other cameras 305 of camera array 300 to complete initialization before sending videos 302 to camera clients 220, the cameras 305 of camera array 300 may each begin sending videos 302 to camera clients 220 at different times. As a result, videos 302, and in particular, the frames of videos 302, may be desynchronized from the frames of other videos 302. In other words, the frames of these videos 302 are not being captured and sent by their respective cameras 305 simultaneously or at the same time. Consequentially, the frames of these videos 302 do not arrive at the camera clients 220 simultaneously or at the same time.
During initialization, camera clients 220 and camera server 225 power on and/or perform a bootup sequence. After booting up, camera clients 220 and camera server 225 synchronize their internal clocks 304. In the example of
Camera clients 220 may track the cameras 305 of camera array 300 that have completed initialization by tracking which cameras 305 have communicated videos 302 to camera clients 220. When camera clients 220 determine that each camera 305 of camera array 300 have begun sending videos 302 to camera clients 220, camera clients 220 may determine that camera array 300 has finished initialization. In response to that determination, camera clients 220 may begin processing the frames of the videos 302 and communicating information from those frames to camera server 225. Camera server 225 may then analyze the information from camera clients 220 to determine the physical position of people and/or objects within a space.
As seen in
Camera client 220A receives video 305 from camera 305A, and specifically a frame 320A of that video 305. Camera client 220A processes the frame 320A. As seen in frame 320A, the man is standing near the shelf and the woman is standing further away from the shelf. Camera client 220A processes frame 320A to determine bounding areas 325A and 325B around the man and the woman. In the example of
Camera client 220A determines coordinates 322 that define the bounding areas 325A and 325B within frames 320A and 320B (also referred to as “frame coordinates”). In the example of
Camera client 220A then generates frame data 330A that contains information about frame 320A. As seen in
In a corresponding manner, camera client 220B may process video 302 from camera 305B. As seen in
The coordinates 322 generated by camera clients 220A and 220B for frame data 330A and 330B may be the coordinates within a particular frame 320 and not the coordinates within the physical space. Additionally, although the same subscripts have been used for the coordinates 322 in frame data 330A and 330B, this does not mean that these coordinates 322 are the same. Rather, because cameras 305A and 305B are in different positions, it is likely that the coordinates 322 in frame 330A are different from the coordinates 322 in frame data 330B. Camera clients 220A and 220B are determining the coordinates 322 of the bounding areas 325 within the frames 320 and not within the physical space. Camera clients 220A and 220B determine these local coordinates 322 independently of each other. The subscripts indicate a sequence of coordinates 322 generated by the individual camera clients 220. For example (x1,y1) indicates the first coordinate 322 generated by camera client 220A and the first coordinate 322 generated by camera client 220B, which may be different values.
In
Similarly, camera client 220B receives frame 320D from camera 305B. Camera client 220B determines bounding areas 325E and 325F for the man and the woman, respectively. Camera client 220B then determines coordinates 322 (x5,y5) and (x6,y6) for bounding area 325E and coordinates 322 (x7,y7) and (x8,y8) for bounding area 325F. Camera client 220B generates a timestamp 324 that indicates that frame 320D was received at time t4. Camera client 220B generates frame data 330D that indicates frame 320D was generated by camera 305B and received by camera client 220B at t4. Frame data 330D indicates that the man corresponds to coordinates 322 (x5,y5) and (x6,y6) and has a height of z3 in frame 320D. Frame data 330D also indicates that the woman corresponds to coordinates 322 (x7,y7) and (x8, y8) and has a height of z4 within frame 320D. Camera clients 220A and 220B communicate frame data 330C and 330D to camera sever 225 when frame data 330C and 330D are ready.
In
Camera client 220B receives frame 320F from camera 305B. Camera client 220B determines bounding areas 3251 and 325J around the man and the woman, respectively. Camera client 220BA determines coordinates 322 (x9,y9) and (x10,y10) for bounding area 325I and coordinates 322 (x11,y11) and (x12,y12) for bounding area 325J. Camera client 220B generates a timestamp 324 indicating when frame 320F was received by camera client 220B (e.g., by using internal clock 304). Camera client 220B then generates frame data 330F indicating that frame 320F was produced by camera 305B and received by camera client 220B at t6. Frame data 330F indicates that the man corresponds to coordinates 322 (x9,y9) and (x10,y10) and has a height at z5 in frame 320F. Frame data 330F also indicates that the woman corresponds to coordinates 322 (x11,y11) and (x12,y12) and has a height at z6 in frame 320F. Camera clients 220A and 220B communicate frame data 330E and 330F to camera server 225 when ready.
In
By assigning frame data 330 to time windows 332, camera server 225 may account for desynchronization that occurs amongst the cameras 305, camera clients 220, and the camera server 225 in the camera subsystem 202. The duration of the time windows 332 can be set to be larger than the desynchronization that is expected to occur to mitigate the effects of desynchronization. For example, if the cameras 305 and camera clients 220 are expected to desynchronize by a few milliseconds, then the time window 332 can be set to last 100 milliseconds to counteract the desynchronization. In this manner, camera server 225 can mitigate the effects of desynchronization as the camera subsystem 202 is scaled to handle larger spaces by including more cameras 305 and camera clients 220. In the example of
Camera server 225 may determine whether to advance cursor 335A when frame data 330 is received from the camera client 220 corresponding to that cursor 335. If that frame data 330 has a timestamp 324 that belongs in a subsequent time window 332, then camera server 225 may advance the cursor 335 to that time window 332, thereby indicating that camera server 225 is not expecting to receive any more frame data 330 from that camera client 220 that belongs in a prior time window 332. In this manner, camera server 225 can quickly and efficiently assign frame data 330 to time windows 332 without checking every time window 332 when frame data 330 is received. For example, if camera client 2220B is faster at sending information than camera client 1220A and camera client 3220C, then cursor 335C may advance far ahead of cursors 335A and 335B. When camera server 225 receives frame data 330 from camera client 2220B, camera server 225 need not check every time window 332 beginning from time window 1332A to determine to which time window 332 that frame data 330 should be assigned. Rather, camera server 225 can start at the time window 332 to which cursor 335C points. In other words, camera server 225 need not first check whether a timestamp 324 in the frame data 330 from camera client 2220B indicates a time that falls within time window 1332A and then whether that time falls within time window 2332B. Instead, camera server 225 can first check whether that time falls within time window 3332C and ignore checking whether that time falls within time window 1332A and time window 2332B. As a result, the frame data 330 is quickly and efficiently assigned to the correct time window 332.
Camera server 225 determines that frame data 330 assigned to a particular time window 332 is ready for processing in any suitable manner. For example, camera server 225 may determine that a particular time window 332 is ready for processing when that time window 332 has frame data 330 for frames 320 from a sufficient number of cameras 305. Camera server 225 may use a threshold 338 to make this determination. When a particular time window 332 has been assigned frame data 330 for frames 320 from a number of cameras 305 that exceeds threshold 338, camera server 225 may determine that that time window 332 is ready for processing and move the information for that time window 332 to the task queue 336. For example, assume threshold 338 indicates that frame data 330 for frames 320 from ten cameras 305 of an array 300 of twelve cameras 305 need to be received before a time window 332 is ready for processing. If time window 332 contains frame data 330 for frames 320 from only eight cameras 305, then camera server 225 determines that time window 332 is not ready for processing, and as a result, time window 332 waits to be assigned frame data 330 for frames 320 from additional cameras 305. When time window 332 has received frame data 330 for frames 320 from ten or more cameras 305, camera server 225 determines that time window 332 is ready for processing and moves frame data 330 in time window 332 to task queue 336.
Camera server 225 may also determine that a particular time window 332 is ready for processing when a subsequent time window 332 has received frame data 330 for frames 320 from a number of cameras 305 exceeding threshold 338. Using the previous example, even if time window 1332A has been assigned frame data 330 for frames 320 from eight cameras, camera server 225 may nevertheless determine that time window 1332A is ready for processing when time window 2332B has been assigned frame data 330 for frames 320 from ten or more cameras 305 (e.g., from every camera 305 in camera array 300). In this scenario, camera server 225 may assume that no additional frame data 330 will be assigned to time window 1332A because frame data 330 for frames 320 from a sufficient number of cameras 305 has been assigned to a subsequent time window 2332B. In response, camera server 225 moves frame data 330 in time window 1332A to task queue 336.
Camera server 225 may also determine that a particular time window 332 is ready for processing when that time window 332 has been awaiting processing for a certain period of time. For example, if an error or bug occurs in the system and frames 320 from a number of cameras 305 are not sent or are lost, then a time window 332 may not receive frame data 330 for frames 320 from enough cameras 305. As a result, processing for that time window 332 may stall or be delayed. Camera server 225 may use a timeout or age-out beyond which a time window 332 does not wait for processing. Thus, when the time window 332 has not been processed for a certain period of time exceeding the timeout or the age-out, camera server 225 may nevertheless send the frame data 330 in that time window 332 to the task queue 336. Using the previous example, assume the timeout is 200 milliseconds. If time window 1332A has been stuck with frame data 330 from frames 320 from eight cameras 305 for over 200 milliseconds, camera server 225 may determine that time window 1332A has waited long enough for additional frame data 330 and that time window 1332A is ready for processing. In response, camera server 225 moves frame data 330 in time window 1332A to task queue 336.
In certain embodiments, when a time window 332 times out or ages out, camera server 225 may adjust threshold 338 so that future time windows 332 are less likely to time out or age out. For example, camera server 225 may lower threshold 338 when a time window 332 times out or ages out. Likewise, camera server 225 may increase threshold 338 when a subsequent time window 332 does not time out or age out. Camera server 225 may adjust threshold 338 based on the number of cameras 305 that have sent information for a particular time window 332. For example, if a particular time window 332 times out or ages out when it has frame data 330 for frames 320 from eight cameras 305, and threshold 338 is ten cameras 305, camera server 225 may reduce threshold 338 to a value closer to eight cameras. As a result, that time window 332 may then have frame data 330 for frames 320 from a sufficient number of cameras 305 and be moved to task queue 336. When a subsequent time window 332 does not time out because it has received frame data 330 for frames 320 from nine cameras 305, camera server 225 may adjust threshold 338 towards nine cameras 305. In this manner, camera server 225 may dynamically adjust the threshold 338 to prevent bugs, errors, and/or latency from causing delays in the camera subsystem 202.
In certain embodiments, camera server 225 processes time windows 332 sequentially. In other words, camera server 225 does not process a subsequent time window 332 until a prior time window 332 is ready for processing. In the example of
To process an entry of task queue 336, camera server 225 may combine or cluster the coordinates 322 of the same objects detected by the same cameras 320 to calculate combined coordinates 332 for that object. As a result of this processing, each time window 332 should include only one set of coordinates 322 per object per camera 305. After this processing, the combined coordinates 322 are placed into a result queue.
As an example, camera server 225 first processes the first entry in the task queue 336, which includes frame data 330 for frames 320A, 320B, and 320C. Frames 320A and 320C are from the same camera 320A. As a result, camera server 225 may use the frame data 330A and 330C for frames 320A and 320C to calculate a combined coordinate 322 for the people or objects detected by camera 320A. As seen in
Camera server 225 then determines combined coordinates 322 for objects detected by the cameras 305 during the second time window 332B in a similar fashion. For example, camera server 225 may use frame data 330E for frame 320E (and frame data 330 for any other frames 320 received by camera 1305A during the second time window 332B) to determine combined coordinates 322 (x17,y17), and (x18,y18) and a combined height z9 for object 1 detected by camera 1305A and combined coordinates 322 (x19,y19) and (x20,y20) and a combined height z10 for object 2 detected by camera 1305A. Camera server 225 may also use frame data 330D and 330F for frames 320D and 320F to determine combined coordinates 322 (x17,y17), and (x18,y18) and a combined height z9 for object 1 detected by camera 2305B and combined coordinates 322 (x19,y19) and (x20,y20) and a combined height z10 for object 2 detected by camera 2305B.
Camera server 225 calculates combined coordinates 322 and combined heights in any suitable manner. For example, camera server 225 may calculate combined coordinates 322 and combined heights by taking the average of the coordinates 322 and the heights of particular objects detected by the same camera 305 in a particular time window 332. Using the example in
Camera server 225 may follow a similar process to determine or to calculate the combined coordinates for object 2 detected by cameras 1305A and 2305B. Camera server 225 may calculate combined coordinates 322 (x15,y15) for camera 1305A by taking the average of coordinates 322 (x3,y3) and (x7,y7) from frame data 330A and 330C. Similarly, camera server 225 may determine the combined coordinate 322 (x16,y16) for camera 1305A by taking the average of coordinates 322 (x4,y4) and (x8,y8) from frame data 330A and 330C. Camera server 225 may determine combined height z8 for camera 1305A by taking the average of heights z2 and z4 from frame data 330A and 330C. Similarly, camera server 225 may determine combined coordinates 322 (x19,y19) for camera 2305B by taking the average of coordinates 322 (x7,y7) and (x11,y11) from frame data 330D and 330F. Likewise, camera server 225 may determine combined coordinates 322 (x20,y20) for camera 2305B by taking the average of coordinates 322 (x8,y8) and (x12,y12) from frame data 330D and 330F. Camera server 225 may determine combined height z10 for camera 2305B by taking the averages of heights z4 and z6 from frame data 330D and 330F.
Camera server 225 uses any other suitable calculation to calculate combined coordinates and combined heights. For example, camera server 225 may take a median of coordinates 322 and heights for objects detected by the same camera 305 during a time window 332. Camera server 225 may also use clustering processes to calculate the combined coordinates 322 and combined heights. For example, camera server 225 may use K-means clustering, Density-based spatial clustering of applications with noise (DBSCAN), k-medoids, gaussian mixture models, and hierarchical clustering to calculate combined coordinates 322 and combined heights.
After camera server 225 has calculated the combined coordinates 322 and combined heights, camera server 225 has determined the coordinates 322 for each object detected by each camera 305 during a time window 332. However, camera server 225 may perform additional processing to determine whether the object detected by different cameras 305 are the same object. Camera server 225 may use linking and homography to determine which objects detected by which cameras 305 are actually the same person or object in a space. Camera server 225 may then take the combined coordinates 322 for those objects from the different cameras 305 and employ homography to determine a physical location for that person or object in the physical space during a time window 332. Embodiments of this process are described in U.S. patent application Ser. No. 16/663,710 entitled, “Topview Object Tracking Using a Sensor Array”, the contents of which are incorporated by reference herein in its entirety. In this manner, camera server 225 determines the physical locations of people and/or objects within the space during particular time windows 332.
In particular embodiments, camera clients 220 may also use the same time windows 332 as camera server 225 to communicate frame data 330 in batches to camera server 225. As seen in
In certain embodiments, even if camera server 225 and camera clients 220 are not synchronized, camera server 225 can account for desynchronization that occurs (e.g., by desynchronized internal clocks 302, by latency differences between camera clients 220 to camera server 225, by processing speed differences between camera clients 220, etc.) by adjusting the timestamps 324 in frame data 330.
Camera server 225 can adjust timestamps 324 for particular cameras 305 to account for desynchronization. Generally, camera server 225 determines a delay for each camera 305 by tracking the delay for prior frames 320 from that camera 305. Camera server 225 then adjusts timestamps 324 for frame data 330 for frames 320 from that camera 305 by the determined delay. In the example of
As seen in
Camera 305A then generates frame 320B and communicates frame 320B to camera client 220A. Camera client 220A generates coordinates 322 for two people shown in frame 320B. During that process, camera 305B generates frame 320E and communicates frame 320E to camera client 220B. Camera client 220B then determines coordinates 322 for two people detected in frame 320E. Camera 305A then generates frame 320C and communicates frame 320C to camera client 220A. Camera client 220A determines coordinates 322 for two people detected in frame 320C. Importantly,
Modifications, additions, or omissions may be made to method 342 depicted in
In particular embodiments, the camera subsystem 202 may include a second camera array that operates in tandem with the first camera array 300 of the camera subsystem 202.
Each camera 305N is positioned slightly offset from a camera 305M of camera array 300. In this manner, cameras 305M capture video that is similar to the video captured by cameras 305N. In certain embodiments, cameras 305M may use different versions of software or different versions of software may be used to process video from cameras 305M relative to cameras 305N. In this manner, newer software can be run for cameras 305N to test the effectiveness of that software. The testing of that software does not interrupt the operation of the camera subsystem 202 because cameras 305M may still be using the previous software, which also acts as a baseline for comparing against the operation of the new software running on cameras 305N. For example, the accuracy of the position tracking provided by the new software can be determined and compared against the accuracy provided by the old software. If the new software is less accurate than the old software, then the old software should continue to be used.
In certain embodiments, camera server 225 can retrieve video footage from camera clients 220 or a shared memory if the camera server 225 is unable to determine the positions of people based on the frame data 330 from the camera clients 220.
In the example of
Certain embodiments of tracking system 132 include a LiDAR subsystem 204.
LiDAR array 400 is an array of LiDAR sensors 405. LiDAR array 400 may be positioned above a physical space to detect the presence and positions of people and/or objects within the space. In the example of
Each LiDAR sensor 405 detects the presence of people and/or objects within a portion of the physical space. Generally, LiDAR sensors 405 emit light pulses into the space. These light pulses are reflected back towards the LiDAR sensors 405 when the light pulses contact people and/or objects in the space. The LiDAR sensor 405 tracks characteristics of the reflected light pulses, such as the return times of the light pulses and the wavelength of the return light pulses, to detect the presence of people and/or objects within the physical space. LiDAR sensors 405 may also determine coordinates for the detected people and/or objects. LiDAR sensors 405 communicate the coordinates for the detected people and/or objects to LiDAR server 230.
LiDAR sensors 405 may be communicatively coupled to LiDAR server 230 in any suitable manner. For example, LiDAR sensors 405 may be hardwired to LiDAR server 230. As another example, LiDAR sensors 405 may wirelessly couple to LiDAR server 230 using any suitable wireless standard (e.g., WiFi). LiDAR sensors 405 communicate coordinates for detected people and/or objects through the communication medium to LiDAR server 230.
LiDAR server 230 may process the coordinates 410 assigned to a time window 332 to determine the physical position of people and/or objects within the space. In the example of
LiDAR server 230 uses these coordinates 410 to determine the physical position of people within the space. As with the camera server 225, LiDAR server 230 may determine that the coordinates 410 provided by two different LiDAR sensors 405 correspond to the same person within the physical space. In response, LiDAR server 230 may take these coordinates 410 and use homography to determine a position of the person within the physical space in a particular time window 332. In the example of
LiDAR sensor 405 determines coordinates 410 of detected people and communicates these coordinates 410 to LiDAR server 230. LiDAR sensor 405 may determine these coordinates 410 by emitting a light pulse and analyzing characteristics of the light pulse when that light pulse is reflected back to LiDAR sensor 405. For example, LiDAR sensor 405 may analyze the return time of the reflected light pulse and/or the wavelength of the reflected light pulse to determine whether a person is present in the physical space and the coordinates 410 of that person.
LiDAR server 230 analyzes the coordinates 410 from LiDAR sensor 405 to determine the positions of people within the physical space during a first time window 332 in step 416. LiDAR server 230 then communicates these positions to central server 240. LiDAR sensor 405 may subsequently determine the coordinates 410 of detected people and communicate these coordinates 410 to LiDAR server 230. LiDAR server 230 may again determine the positions of these people in a subsequent time window 332 and communicate these positions to central server 240 in step 418.
As with the camera subsystems 202, central server 240 may use these positions to determine which person removed an item 130 from the space during the particular time window 332. The operation of central server 240 will be described in more detail using
Modifications, additions, or omissions may be made to method 415 depicted in
Tracking system 132 includes a weight subsystem 206 that includes weight sensors 215 and weight server 235. Generally, weight sensors 215 detect the weights of items positioned above or near the weight sensors 215. The weight sensors 215 may be positioned on an unconventional rack 115 that holds items. Weight server 235 tracks the weights detected by weight sensors 215 to determine if and when items 130 are removed from the rack 115. The weight sensors 215, rack 115, and weight server 235 will be described in more detail using
Plates 510 form surfaces that distribute the weight of items 130 across the surfaces. Plates 510 may be made of any suitable material, such as, for example, metal and/or plastic. Items 130 may be positioned above or near plates 510 and the weight of these items 130 may be distributed across plates 510.
Load cells 505 are positioned between plates 510A and 510B. Load cells 505 produce electrical signals based on the weight experienced by the load cells 505. For example, load cells 505 may be transducers that converts an input mechanical force (e.g., weight, tension, compression, pressure, or torque) into an output electrical signal (e.g., current or voltage). As the input force increases, the output electrical signal may increase proportionally. Load cells 505 may be any suitable type of load cell (e.g., hydraulic, pneumatic, and strain gauge). Although load cells 1310 are illustrated as being cylindrical in shape, they may be any suitable size and shape that is appropriate for the particular implementation contemplated.
The signals from load cells 505 may be analyzed to determine an overall weight of items 130 positioned above or near weight sensor 500. Load cells 505 may be positioned such that the weight of items 130 positioned above or near weight sensor 500 is evenly distributed to each load cell 505. In the example of
Load cells 505 communicate electric signals that indicate a weight experienced by the load cells 505. For example, the load cells 505 may produce an electric current that varies depending on the weight or force experienced by the load cells 505. Each load cell 505 is coupled to a wire 515 that carries the electric signal. In the example of
Weight sensor 500 may be disposed in an unconventional rack 115 designed to hold items.
Surface 532E defines a cavity 534 that also allows access into the space 542. Generally, cavity 534 allows wires 520 from weight sensors 500 to extend into the space 542.
Panels 535 extend upwards from base 530. Panels 535 may be formed using any suitable materials, such as for example, wood, metal, and/or plastic. As seen in
Each shelf 540 couples to panel 535 and/or base 530. Weight sensors 500 are disposed in the shelf 540. A shelf 540 may couple to panel 535 such that the wires 520 of the weight sensors 500 disposed in the shelf 540 can run from the weight sensors 500 through a cavity 550 into space 552. These wires 520 then run down space 552 and through cavity 534 into space 542.
In certain embodiments, weight sensor 500 is positioned in shelf 540 such that weight sensor 500 detects the weight of items positioned within a particular region 555 of shelf 540. As seen in the examples of
A circuit board 565 is positioned in space 542. Circuit board 565 includes ports to which wires 520 from the weight sensors 500 of rack 525 connect. In other words, circuit board 565 connects to wires 520 from weight sensors 500 positioned on base 530 and on shelves 540. These wires 520 enter space 542 through cavity 534 and connect to circuit board 565. Circuit board 565 receives the electric signals produced by the load cells 505 of the weight sensors 500. Circuit board 565 then communicates signals to weight server 235 indicating the weights detected by the weight sensors 500. Drawer 545 may open to allow access to space 542 and to circuit board 565. For example, drawer 545 may be opened so that circuit board 565 may be serviced and/or repaired.
Circuit board 565 may communicate signals to weight server 235 through any suitable medium. For example, circuit board 565 may communicate signals to weight server 230 through an ethernet connection, a wireless connection (e.g., WiFi), a universal serial bus connection, and/or a Bluetooth connection. Circuit board 565 can automatically select a connection through which to communicate signals to weight server 235. Circuit board 565 may choose the connection based on priority. For example, if the ethernet connection is active, circuit board 565 may select the ethernet connection for communicating with weight server 235. If the ethernet connection is down and the wireless connection is active, circuit board 565 may choose the wireless connection to communicate with weight server 235. If the ethernet connection and the wireless connection are down and the universal serial bus connection is active, circuit board 565 may select the universal serial bus connection to communicate with weight server 235. If the ethernet connection, the wireless connection, and the universal serial bus connection are down and the Bluetooth connection is active, circuit board 565 may select the Bluetooth connection to communicate with weight server 235. In this manner, circuit board 565 has improved resiliency because circuit board 565 may continue to communicate with weight server 235 even if certain communication connections go down.
Circuit board 565 may receive electrical power through various connections. For example, circuit board 565 may include a power port 570 that supplies electrical power to circuit board 565. An electrical cable that plugs into an electrical outlet may couple to power port 570 to supply electrical power to circuit board 565. Circuit board 565 may also receive electrical power through the ethernet connection and/or the universal serial bus connection.
Weight server 235 may implement an internal clock 304E that is synchronized with the internal clocks 304 of other components of tracking system 132 (e.g., camera clients 220, camera server 225, and central server 240). Weight server 235 may synchronize the internal clock 304E using a clock synchronization protocol (e.g., Network Time Protocol and/or Precision Time Protocol). Weight server 235 may use clock 304E to determine the times at which signals 572 from weight sensors 500 were received and assign these signals 572 to their appropriate time windows 332.
In certain embodiments, time windows 332 in weight server 235 are aligned with time windows 332 in camera clients 220, camera server 225, and/or central server 240. For example, time window 332A in weight server 235 may have the same start time (T0) and end time (T1) as time window 332A in camera server 225 in the example of
Similar to camera server 225, weight server 235 may process the information in the time windows 332 sequentially when the time windows 332 are ready for processing. Weight server 235 may process the information in each time window 332 to determine whether an item 130 was removed during that particular time window 332. In the example of
Weight sensor 215 detects the weight experienced 582 above or around weight sensor 215 and communicates the detected weight 582 through an electric signal 572 to weight server 235. Weight server 235 may analyze the signals 572 from weight sensor 215 to determine a number 584 of items 130 that were taken during a first time window 332. Weight server 235 may communicate the determination to central server 240. Weight sensor 215 may subsequently detect a weight 586 experienced by weight sensor 215 and communicate that weight 586 to weight server 235. Weight server 235 may analyze that weight 586 to determine a number 588 of items 130 that were taken during a second time window 332. Weight server 235 may communicate that determination to central server 240. Central server 240 may track whether items 130 were taken during particular time windows 332. And if so, central server 240 may determine which person in the space took those items 130.
Modifications, additions, or omissions may be made to method 580 depicted in
According to the camera server 225, the first person is at a physical coordinate 602 (X1,Y1), and the second person is at a physical coordinate 602 (X2,Y2). Additionally, central server 240 receives from LiDAR server 230 the physical location of the two people. According to the LiDAR server 230, the first person is at coordinate 602 (X7,Y7) and the second person is at coordinate 602 (X8,Y8). Furthermore, central server 240 also receives information from weight server 235 during the first time window 332A. According to weight server 235, no items 130 were taken during the first time window 332A.
This disclosure contemplates central server 240 using any suitable process for analyzing the physical position of people from camera server 225 and LiDAR server 230. Although the coordinates 602 provided by camera server 225 and LiDAR server 230 may differ from each other, central server 240 may use any appropriate process for reconciling these differences. For example, central server 240 may use the coordinates 602 provided by camera server 225 if the coordinates 602 provided by LiDAR server 230 do not differ from the coordinates 602 provided by camera server 225 by an amount that exceeds a threshold. In this manner, the coordinates 602 provided by LiDAR sever 230 act as a check on the coordinates 602 provided by camera server 225.
During a second time window 332B, central server 240 receives from camera server 225 the physical coordinates 602 of the two people. According to camera server 225, during the second time window 332B, the first person was at coordinate 602 (X3,Y3) and the second person was at coordinate 602 (X4,Y4). During the second time window 332B, camera server 240 also receives the physical coordinates 602 of the two people from LiDAR server 230. According to the LiDAR server 230, the first person is at coordinate 602 (X9,Y9) and the second person is at coordinate 602 (X10, Y10) during the second time window 332B. Additionally, central server 240 learns from weight server 235 that no items 130 were taken during the second time window 332B.
During a third time window 332C, camera server 240 receives the physical coordinates 602 of the two people from camera server 225. According to the camera server 225, the first person is at coordinate 602 (X5,Y5) and the second person is at coordinate 602 (X6,Y6). Central server 240 also receives the physical coordinates 602 of the two people from LiDAR server 230 during the third time window 332C. According to the LiDAR server 230, the first person is at coordinate 602 (X11,Y11) and the second person is at coordinate 602 (X12,Y12) during the third time window 332C. Additionally, central server 240 learns from weight server 235 that a particular weight sensor 500 detected that two items 130 were taken during the third time window 332C.
In response to learning that a weight sensor 500 detected that two items 130 were taken, central server 240 may undergo additional analysis to determine which person took those two items 130. Central server 240 performs any suitable process for determining which person took items 130. Several of these processes are disclosed in U.S. application Ser. No. 16/663,710 entitled, “Topview Object Tracking Using a Sensor Array”, the contents of which are incorporated by reference herein.
Central server 240 then determines the distance from each person to the weight sensor 500. Central server 240 determines that the first person is a distance 1 from the weight sensor 500 and that the second person is a distance 2 from the weight sensor 500. Central server 240 then determines which person was closer to the weight sensor 500. In the example of
Central server 240 begins by receiving coordinates 602 of a first person in a space during a time window 332 in step 605. In step 610, central server 240 receives the coordinates 602 of a second person in the space during the time window 332. Central server 240 receives an indication that an item 130 was taken during the time window 332 in step 615. In response to reeving that indication, central server 240 analyzes the information to determine which person took that item 130.
In step 620, central server 240 determines that the first person was closer to the item 130 than the second person during the time window 332. Central server 240 may make this determination based on determined distances between the people and a weight sensor 500 that detected that the item 130 was removed. In step 625, central server 240 determines that the first person took the item 130 during the time window 332 in response to determining that the first person was closer to the item 130 than the second person. The first person may then be charged for the item 130 when the first person exits the store 100.
Modifications, additions, or omissions may be made to method 600 depicted in
Processor 705 is any electronic circuitry, including, but not limited to microprocessors, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and/or state machines, that communicatively couples to memory 710 and controls the operation of computer 700. Processor 705 may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. Processor 705 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. Processor 705 may include other hardware that operates software to control and process information. Processor 705 executes software stored on memory to perform any of the functions described herein. Processor 705 controls the operation and administration of computer 700 by processing information received from memory 710 and/or other computers 700. Processor 705 may be a programmable logic device, a microcontroller, a microprocessor, any suitable processing device, or any suitable combination of the preceding. Processor 705 is not limited to a single processing device and may encompass multiple processing devices.
Memory 710 may store, either permanently or temporarily, data, operational software, or other information for processor 705. Memory 710 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 710 may include random access memory (RAM), read only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or a combination of these devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in memory 710, a disk, a CD, or a flash drive. In particular embodiments, the software may include an application executable by processor 705 to perform one or more of the functions described herein.
Graphics processor 715 may be any electronic circuitry, including, but not limited to microprocessors, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and/or state machines, that receives and analyzes video data. For example, graphics processor 715 may process video data to determine the proper signals to send to a display so that the display displays an appropriate image. Graphics processor 715 may also process video data to identify certain characteristics (e.g., people or objects) within the video. Graphics processor 715 may be a component of a video card that is installed in computer 700.
Input/output ports 720 allow peripheral devices to connect to computer 700. Ports 720 may be any suitable ports, such as, parallel ports, serial ports, optical ports, video ports, network ports, etc. Peripheral devices such as keyboards, mouses, joysticks, optical tracking devices, trackpads, touchpads, etc. can connect to computer 700 through ports 720. Input and output signals are communicated between computer 700 and the peripheral devices through ports 720.
Communication interface 725 includes any suitable hardware and/or software to communicate over a network. For example, communication interface 725 may include a mode, network card, ethernet port/controller, wireless radio/controller, cellular radio/controller, and/or universal serial bus port/controller. Computer 700 may use communication interface 725 to communicate with other devices over a communication network.
Bus 730 allows components of computer 700 to communicate with one another. Computer 700 may include a bus controller 730 that manages communication over bus 730.
Although the present disclosure includes several embodiments, a myriad of changes, variations, alterations, transformations, and modifications may be suggested to one skilled in the art, and it is intended that the present disclosure encompass such changes, variations, alterations, transformations, and modifications as fall within the scope of the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5013896 | Ono et al. | May 1991 | A |
| 6725206 | Coveley | Apr 2004 | B1 |
| 7672876 | Bonner et al. | Mar 2010 | B2 |
| 7844509 | Bodin et al. | Nov 2010 | B2 |
| 7848964 | Bonner et al. | Dec 2010 | B2 |
| 9886827 | Schoner | Feb 2018 | B2 |
| 9984354 | Chinoy et al. | May 2018 | B1 |
| 9996818 | Ren et al. | Jun 2018 | B1 |
| 10055853 | Fisher et al. | Aug 2018 | B1 |
| 10064502 | Gyori et al. | Sep 2018 | B1 |
| 10127438 | Fisher et al. | Nov 2018 | B1 |
| 10133933 | Fisher et al. | Nov 2018 | B1 |
| 10134004 | Liberato, Jr. et al. | Nov 2018 | B1 |
| 10140483 | Huebner et al. | Nov 2018 | B1 |
| 10140820 | Zalewski et al. | Nov 2018 | B1 |
| 10157452 | Tighe et al. | Dec 2018 | B1 |
| 10169660 | Ren et al. | Jan 2019 | B1 |
| 10181113 | Rivalto et al. | Jan 2019 | B2 |
| 10198710 | Hahn et al. | Feb 2019 | B1 |
| 10244363 | Niranjayan et al. | Mar 2019 | B1 |
| 10250868 | Arnold et al. | Apr 2019 | B1 |
| 10262293 | Prater et al. | Apr 2019 | B1 |
| 10268983 | Kumar et al. | Apr 2019 | B2 |
| 10291862 | Liberato et al. | May 2019 | B1 |
| 10296814 | Kumar et al. | May 2019 | B1 |
| 10303133 | Dhalla et al. | May 2019 | B1 |
| 10318917 | Goldstein et al. | Jun 2019 | B1 |
| 10318919 | Bermudez Rodriguez et al. | Jun 2019 | B2 |
| 10321275 | Orlov et al. | Jun 2019 | B1 |
| 10332066 | Palaniappan et al. | Jun 2019 | B1 |
| 10339411 | Hua et al. | Jul 2019 | B1 |
| 10353982 | Kumar et al. | Jul 2019 | B1 |
| 10366306 | Raghavan et al. | Jul 2019 | B1 |
| 10368057 | Saran et al. | Jul 2019 | B1 |
| 10384869 | Shiee et al. | Aug 2019 | B1 |
| 10388019 | Hua et al. | Aug 2019 | B1 |
| 10442852 | Thiagarajan et al. | Sep 2019 | B2 |
| 10438277 | Jiang et al. | Oct 2019 | B1 |
| 10445694 | Fisher et al. | Oct 2019 | B2 |
| 10459103 | Shi et al. | Oct 2019 | B1 |
| 10466095 | O'Neill et al. | Nov 2019 | B1 |
| 10474991 | Fisher et al. | Nov 2019 | B2 |
| 10474992 | Fisher et al. | Nov 2019 | B2 |
| 10475185 | Raghavan et al. | Nov 2019 | B1 |
| 20070011099 | Sheehan | Jan 2007 | A1 |
| 20100318440 | Coveley | Dec 2010 | A1 |
| 20150312445 | Cha | Oct 2015 | A1 |
| 20190138986 | Puerini et al. | May 2019 | A1 |
| 20190147709 | Schoner | May 2019 | A1 |
| 20190156277 | Fisher | May 2019 | A1 |
| 20190156506 | Fisher | May 2019 | A1 |
| 20190244386 | Fisher et al. | Aug 2019 | A1 |
| 20190244500 | Fisher et al. | Aug 2019 | A1 |
| 20190251499 | Kumar et al. | Aug 2019 | A1 |
| 20190347611 | Fisher et al. | Nov 2019 | A1 |
| 20200013180 | Yamaura | Jan 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 1290453 | Oct 1991 | CA |
| 2201423 | Oct 1998 | CA |
| 0348484 | Jan 1990 | EP |
| 2019032304 | Feb 2019 | WO |
| Entry |
|---|
| Krishnamurthy, S. B. et al., “Scalable Position Tracking System for Tracking Position in Large Spaces,” U.S. Appl. No. 16/663,633, filed Oct. 25, 2019, 117 pages. |
| Paul, D. et al. “Custom Rack for Scalable Position Tracking System,” U.S. Appl. No. 16/633,669, filed Oct. 25, 2019, 110 pages. |
| Krishnamurthy, S. B, et al., “Tracking Positions Using a Scalable Position Tracking System,” U.S. Appl. No. 16/633,432, filed Oct. 25, 2019, 122 pages. |
| Krishnamurthy, S. B. et al., “Topview Object Tracking Using a Sensor Array,” U.S. Appl. No. 16/663,710, filed Oct. 25, 2019, 199 pages. |
| Krishnamurthy, S. B. et al., “Detecting Shelf Interactions Using a Sensor Array,” U.S. Appl. No. 16/663,766, filed Oct. 25, 2019, 205 pages. |
| Vakacharla, S. et al., “Topview Item Tracking Using a Sensor Array,” U.S. Appl. No. 16/663,451, filed Oct. 25, 2019, 204 pages. |
| Mirza, S. A. et al., “ Detecting and Identifying Misplaced Items Using a Sensor Array,” U.S. Appl. No. 16/653,794, filed Oct. 25, 2019, 200 pages. |
| Mirza, S. A. et al., “Sensor Mapping Toa Global Coordinate System,” U.S. Appl. No. 16/663,822, filed Oct. 25, 2019, 204 pages. |
| Mirza, S. A. et al., “Sensor Mapping to a Global Coordinate System Using a Marker Grid,” U.S. Appl. No. 16/663,472, filed Oct. 25, 2019, 206 pages. |
| Mirza, S. A. et al., “Shelf Position Calibration in a Global Coordinate System Using a Sensor Array,” U.S. Appl. No. 16/663,856, filed Oct. 25, 2019, 206 pages. |
| Magee, M. R. et al., “Customer-Based Video Feed,” U.S. Appl. No. 16/664,470, filed Oct. 25, 2019, 103 pages. |
| Mirza, S. A. et al.,“System and Method for Presenting a Virtual Store Shelf That Emulates a Physical Store Shelf,” U.S. Appl. No. 16/664,490, filed Oct. 25, 2019, 103 pages. |
| Nguyen, T. N. et al.,“Contour-Based Detection of Closely Spaced Objects,” U.S. Appl. No. 16/664,160, filed Oct. 25, 2019, 199 pages. |
| Mirza, S. A. et al.,“Action Detection During Image Tracking,” U.S. Appl. No. 16/663,500, filed Oct. 25, 2019, 198 pages. |
| Mirza, S. A. et al.,“Object Re-Identification During Image Tracking,” U.S. Appl. No. 16/664,219, filed Oct. 25, 2019, 201 pages. |
| Chinnam, M. M. et al.,“Vector-Based Object Re-Identification During Image Tracking,” U.S. Appl. No. 16/664,269, filed Oct. 25, 2019, 199 pages. |
| Mirza, S. A. et al.,“Image-Based Action Detection Using Contour Dilation,” U.S. Appl. No. 16/664,332, filed Oct. 25, 2019, 199 pages. |
| Mirza, S. A. et al.,“Determining Candidate Object Identities During Image Tracking,” U.S. Appl. No. 16/664,363, filed Oct. 25, 2019, 201 pages. |
| Mirza, S. A. et al.,“Object Assignment During Image Tracking,” U.S. Appl. No. 16/664,391, filed Oct. 25, 2019, 198 pages. |
| Krishnamurthy, S. B. et al.,“Auto-Exclusion Zone for Contour-Based Object Detection,” U.S. Appl. No. 16/664,426, filed Oct. 25, 2019, 199 pages. |
| Mirza, S. A. et al.,“Multi-Camera Image Tracking on a Global Plane ,” U.S. Appl. No. 16/663,533, filed Oct. 25, 2019, 198 pages. |
| Mirza, S. A. et al.,“Feedback and Training for a Machine Learning Algorithm Configured to Determine Customer Purchases During a Shopping Session at a Physical Store,” U.S. Appl. No. 16/663,564, filed Oct. 25, 2019, 97 pages. |
| Mirza, S. A. et al.,“Identifying Non-Uniform Weight Objects Using a Sensor Array,” U.S. Appl. No. 16/663,901, filed Oct. 25, 2019, 199 pages. |
| Magee, M. R. et al., “System and Method for Populating a Virtual Shopping Cart Based on Video of a Customer's Shopping Session at a Physical Store,” U.S. Appl. No. 16/663,589, filed Oct. 25, 2019, 103 pages. |
| Magee, M. R. et al., “Tool for Generating a Virtual Store That Emulates a Physical Store ,” U.S. Appl. No. 16/664,529, filed Oct. 25, 2019, 100 pages. |
| Mirza, S. A. et al.,“Sensor Mapping to a Global Coordinate System Using Homography,” U.S. Appl. No. 16/663,948, filed Oct. 25, 2019, 197 pages. |
