Patent Grant
10012729
This disclosure relates to systems that use sensor output of a ranging sensor as polar coordinates of surfaces present in an environment to identify static surfaces and moving surfaces, and track the moving surfaces within the environment.
Ranging sensors are used to detect surfaces that are present in a three-dimensional physical environment. Types of ranging sensor include, without limitation, sonic ranging sensors and light detection and ranging (LIDAR) sensors.
One or more aspects of the disclosure relates to a system for tracking moving surfaces within a three-dimensional environment using sensor output from one or more ranging sensors. Sensor output of the one or more ranging sensor includes detected ranges of surfaces at different orientations of the ranging sensors. The detected ranges and orientations are used as polar coordinates of the surfaces to determine one or more characteristics of the surfaces and to track moving surfaces.
The system includes one or more processors in communication with the one or more ranging sensors. The one or more processors are configured to execute one or more computer program components. The computer program components include one or more of a receiving component, a static surface identification component, a subject identification component, and a subject tracking component.
The receiving component is configured to receive sensor output from the one or more other ranging sensors. A given ranging sensor is rotationally mounted at a sensor location in an environment. The sensor output includes detected ranges of surfaces as a function of orientations of the ranging sensor.
The static surface identification component is configured to use the detected ranges and orientations as polar coordinates of the surfaces in the environment to determine one or more characteristics of the surfaces. The one or more characteristics determined include whether the surface is a static surface or a moving surface and whether the surface includes an obscured edge or a non-obscured edge of a volume bounded by the surface. For example, determining whether the surface is a static surface or a moving surface is based on whether the polar coordinates of the surface change over time.
The subject identification component is configured to use the detected ranges and orientations as polar coordinates of surfaces in the environment to determine characteristics of the surfaces including whether a moving surface is associated with a human being or other subject, such as an animal or a particular object. For example, if it a surface is determined to be characterized as a moving surface, the subject identification component is configured to determine whether the surface is associated with a human being such that the surface is identified as a moving human subject. Determining that the surface is associated with a human being comprises determining whether a volume bounded by the surface has a size, shape, and/or form that is indicative of a size, shape, and/or form corresponding to that of a human being, or more specifically, a human head.
The subject tracking component is configured to establish a track between locations of the moving surface that are detected over periods of time.
An environment will include one or more static surfaces and/or one or more moving surfaces. Surfaces are detected by a ranging sensor 28 that is mounted at a sensor location in the environment. The sensor output of the ranging sensor 28 includes detected ranges of surfaces of the environment as a function of orientations of the ranging sensor 28. The detected ranges and orientations are used as polar coordinates of the surfaces to determine the one or more characteristics of the surfaces. The one or more characteristics determined include whether the surface is a static surface or a moving surface, whether the surface includes an obscured edge or a non-obscured edge of a volume bounded by the surface, and whether the surface is associated with a human being. Determining whether a surface is a static surface or a moving surface is based on whether the polar coordinates of the surface change over time. This is accomplished by matching detected surface over periods of time using the polar coordinates of the surfaces, and determining whether the polar coordinates have changed. Static surfaces are also identified by determining that detected surfaces correspond to a largest detected range at a given orientation from the sensor location. Static surfaces include surfaces of a wall, a ceiling, a floor, a structural element, a support pillar in a room, or a piece of furniture.
Surfaces of moving subjects are detected using the sensor output of the ranging sensor 28. A characteristic of a moving surface is that the polar coordinates of the surface change over time. Additional determined characteristics of a moving surface include whether the moving surface is associated with a human being. Determining that the surface is associated with a human being comprises determining whether a volume bounded by the surface has a size, shape, and/or form that is indicative of a size, shape, and/or form corresponding to that of a human being.
A surface is identified by a grouping of measurements included in the sensor output. The measurements represent points on the surface. The measurements are inherently grouped based on their measured ranges and/or orientations. A group of measurements include measured ranges of a surface (e.g., distance from the sensor location) and orientations with respect to the ranging sensor during a period of time at which the group of measurement were taken. The groups of measurements are used to determine a sizes, shapes, and/or forms of volumes bounded by the surfaces.
Different surfaces are distinguished by determining a break in between the surfaces. A break in a given surface is represented by measurements representing points on the given surface and an adjacent surface. The break is identified when the measurements include measured ranges for the points on the adjacent surface that are either too far or too near to the sensor location compared to the detected range(s) of the given surface to be considered part of the given surface. Briefly, identifying breaks is used to determine whether edges of surfaces are obscured by objects/surfaces closer to the sensor location then the surface itself, or non-obscured such that an identified edge is the true edge of the surface.
The system 10 presented herein provides techniques in which the characteristics of surfaces are determined, and wherein moving subjects are identified and tracked. In some implementations, sensor output of the ranging sensor 28 is used to make a three-dimensional virtual reconstruction of the physical environment for tracking moving subjects. Sensor output related to the identified static surfaces is disregarded during processing of the sensor output for tracking identified moving surfaces to further reduce computational efforts.
The ranging sensor 28 is a sonic ranging sensor, a light detection and ranging (LIDAR) sensor, or other ranging sensor. The ranging sensor 28 performs one or more “sweeps” of the environment. A “sweep” of the environment corresponds to a period of time where one or more rotations, movements, and/or other processes are performed by ranging sensor 28. For example, a LIDAR sensor includes one or more laser emitters and detectors. The LIDAR sensor performs a “sweep” of an environment that comprises a 360 degree (or other amount) horizontal rotation of the sensor. The LIDAR sensor performs a 180 degree (or other amount) vertical rotation of the sensor at predetermined intervals of the horizontal rotation (e.g., every degree and/or fraction of a degree). In some implementations, more or less rotation is performed depending on the quantity, orientation, and/or other parameter of the emitters, detectors, and/or other components included in the LIDAR or other sensor. The ranging sensor 28 is configured to perform one or more sweeps over a predetermined period of time. For example the ranging sensor 28 may perform 2 (or other amount) sweeps per second (or other period of time).
The detected ranges of surfaces at orientations of the ranging sensor 28 are used as coordinates of the surfaces in the environment. The coordinates of the surfaces may correspond to polar coordinates, spherical coordinates, cylindrical coordinates, and/or other considerations of coordinates and/or a coordinate system suitable to define the location of surfaces in the environment with respect to the sensor location.
As an illustrative example in
Returning to
The receiving component 16 is configured to receive sensor output from the ranging sensor 28. The sensor output includes detected ranges of surfaces as a function of orientations of the ranging sensor 28. Receiving the sensor output from the ranging sensor 28 is facilitated through information exchange between the ranging sensor 28 and the computing platform 12 (e.g., receiving component 16), for example, over network 26. Information is exchanged through wired and/or wireless communications established between the ranging sensor 28 and the computing platform 12. For example, the ranging sensor 28 and/or the computing platform 12 are enabled for radio frequency communication, wired communication (e.g., USB and/or other wired connection through respective ports), and/or other communication scheme and/or protocol.
The ranging sensor 28 and/or the computing platform 12 may include one or more antennas configured for Bluetooth® enabled communications, Wi-fi, near-field communication (NFC), and/or one or more ports configured to enable wired communications. As such, it is noted that network 26 is configured to facilitate one or more of the wired and/or wireless communications presented herein. The network 26 represents a wide area network (WAN) such as the internet, a local area network (LAN), and/or a wired network.
The receiving component 16 is configured to receive sensor output from the ranging sensor 28 through active or passive communications with the ranging sensor 28. For example, the computing platform 12 pings the ranging sensor 28 for requests for sensor output (e.g., by virtue of user input and/or at predetermined intervals). As another example, the ranging sensor 28 actively communicates sensor output to the computing platform 12 (e.g., the receiving component 16) in real-time (or near-real time) as the sensor output information is generated by the ranging sensor 28 (e.g., as the ranging sensor 28 performs sweeps of the environment).
As an illustrative example, in
As a further illustrative example in
The first point 30 is a point on a first static surface 1004 (e.g., a wall). The second point 32 is a point on a second static surface 1010 (e.g., a pillar). The points 34a-f are points on a first moving surface 1012 (e.g., a moving human subject). It is noted that sensor output may include more or less measurements representing additional points on surfaces in addition to those described above. For example additional measurements may represent points on other static surfaces 1002, 1006, 1008 (e.g., a ceiling, a wall, a floor, respectively) or other moving surfaces (not shown).
The group of measurement for points 34a-f are considered a first group of measurements during a period in time the measurements were taken (e.g., during one sweep). The first group of measurements are inherently grouped based on the measured ranges and orientations included in the measurements of the points 34a-f that identify a common, continuous surface. A continuous surface is identified by a grouping of adjacent measurements that do not include breaks (e.g., discontinuities) in the measured ranges. A break in a given surface is identified when the difference between measure ranges of adjacent points on surfaces breaches a threshold. That is, measurements of adjacent points that have measured ranges that do not differ more than a threshold are grouped together as identifying a common, continuous surface.
The breaks in surfaces are used to identify edges of the surfaces. If a measurement includes a detected range of a first point on a given surface that is adjacent to at least one other point in a group of points representing a continuous surface, and the detected range of the first point is too far from the sensor location to be grouped with the points on the continuous surface (e.g., indicating a break), then a true, non-obscured edge of the continuous surface is identified as being located between the first point and the a least one other point on the continuous surface. If a measurement includes a detected range of a first point on a given surface that is adjacent to at least one other point included in a group of points representing a continuous surface, and the detected range of the first point is too near to the sensor location to be grouped with the group of points representing the continuous surface (e.g., indicating a break), then an obscured edge of the continuous surface is identified as being located between the point and at least one other adjacent point located on the continuous surface (e.g., the true edge is located behind the given surface and is not detectable by the ranging sensor).
As an illustrative example, consider a continuous surface of a static wall (or other surface and/or subject) that is positioned a fixed distance from the ranging sensor. The sensor output of the ranging sensor 28 will include measurements representing points on the wall that are inherently grouped based on the measured ranges (e.g., any two adjacent points do not differ more than a threshold amount that would indicate that the points do not lie on the same, continuous wall surface). For example, the measurements of two adjacent points (e.g., adjacent by virtue of adjacent orientations from sensor location) are compared to determine if the spatial separation between the points breaches a threshold thereby indicating a discontinuity of surfaces in between the points (e.g., one point is a point on a surface of an object between the wall and the sensor location, and another point is a point on the wall). A discontinuity will correspond to a spatial separation that breaches a threshold. For example, the threshold may be 1 millimeter, 5 millimeters, a centimeter, 5 centimeters, 10 centimeters, 50 centimeters, and/or other distance.
As further example, if a measurement of a point of a surface at a given orientation includes a measured range of 5.00 meters, and five other measurements of five other points on surfaces positioned at adjacent orientations include measured ranges of 5.01 meters, 5.02 meters, 5.03, meters, 5.04 meters, 5.05 meters, than it is determined that the points represented by this group of measurements represent a continuous surface. Conversely, if a measurement of a point includes a measured range of 5.00 meters, and five other measurements of five other points positioned at adjacent orientations include measured ranges of 5.01 meters, 5.02 meters, 3.40, meters, 3.41 meters, 5.05 meters, than it is determined, based on the discontinuity in the ranges (e.g., a relatively large spatial separation), that the points represented by this group of measurements do not represented a common, continuous surface. As described above, adjacent points representing discontinuities (large spatial separation) in the measured ranges are used to determine edges of surfaces.
By analyzing the sensor output as polar coordinates of points on surface, computational analysis to identify and distinguish between surfaces requires one-to-one comparisons between adjacent points to determine whether or not a there is a discontinuity between the adjacent points (e.g., a difference in measure range breaching a threshold). Since the detected range and orientation is inherently provided by polar coordinates of the points on surfaces (e.g., and/or spherical and/or cylindrical coordinates), processing by a computing platform (e.g., computing platform 12 in
Returning to
A size, shape, form, and/or other distinguishing characteristic of a detected surface, and/or volume bounded by the surface, is determined by using the measurements included in the group of measurements that represent the surfaces during a period of time when the measurements are taken (e.g., a sweep). The spatial separation between points located at edges of the surface is determined using the polar coordinate of the points. For example, distance formulas to determine distances between points in three-dimensions using the polar coordinates are used.
The special separation between edges is then used to determine a height, a width, a depth, a center, a surface area, a volume, and/or other characteristic of the surface and/or the volume bounded by the surface. A pitch of the surface is determined using the group of measurements. A surface normal of the surface is determined using the group of measurement. For example, surface pitch, surface normal, and/or other characteristic are determined by applying algorithms that use polar, spherical, and/or cylindrical coordinates included in the measurements of the points on the surface to determine a pitch, normal, gradient, slope, spacing, and/or other characteristic of the surface represented by the measurements. Calculating surface normal and/or surface pitch facilitates the determination of curvature and/or other characteristic of the surface.
By determining size, shape, form, and/or other distinguishing characteristic of detected surfaces, and/or volumes bounded by the surfaces over periods of time (e.g., every sweep), surfaces are matched over time. The polar coordinates of matched surfaces are compared to determine whether the surface moved or has remained static over a period of time.
In some implementations, identifying static surfaces of the environment comprises determining the surfaces in the environment that correspond to the largest detected range (e.g., radial distance from sensor location) at given orientations over a period of time (e.g., one or more sweeps of ranging sensor 28). At a given orientation (e.g., a given polar and/or azimuth angle), the ranging sensor 28 detects different ranges for surfaces over time, for example, due to subjects moving throughout the environment. By using the detected range and orientation of surfaces as polar coordinates (e.g., including radial distance), the surfaces detected at the largest ranges (e.g., largest radial distance) inherently identify the static surfaces of the environment since no surface farther than that at the largest range has been detected. Since this identification is inherent when using the sensor output as polar coordinates (e.g., by referencing the radial distance at given orientations), the processing performed by the one or more processors 14 of computing platform 12 for identifying static surfaces is performed with minimal or reduced computational effort (e.g., compared to using sensor output as Cartesian coordinates requiring complex analysis). However, such techniques for identifying static surfaces may not be suitable for crowded environments having many moving subjects (where all detected surfaces are moving), or environments wherein the static surfaces are too far from the sensor location to provide accurate detected ranges.
As an illustrative example in
As another illustrative example, consider the second measurement of the second point 32. The second point 32 corresponds to a second orientation of the ranging sensor 28 and a detected range of “R2” at a point in time the second measurement was taken. For illustrative purposes, the first orientation associated with first point 30 is an orientation that is at least horizontally adjacent (e.g., based on azimuth angle) to the second orientation associated with the second point 32. If, for example, the detected ranges R1 and R2 indicate a spatial separation of the corresponding detected surfaces that breach a threshold (e.g., 5 meters for R1 compared to 3 meters for R2, with a threshold of 0.1 meters and/or other threshold), then the surfaces represented by first point 30 and second point 32 are determined as not being a continuous, common surface, e.g., the second static surface 1010 represented by the second point 32 is not continuous with the first static surface 1004. It is also determined that an edge 1005 is located at least at the location between the first point 30 and the second point 32, and, due to the second point 32 being closer to the sensor location than the first point, the edge 1005 is a true edge of the second static surface 1010. Additional comparisons of measurement of points that indicate a break or discontinuity between first static surface 1004 and second static surface 1010 are used to identify the entirety of edge 1005.
The first group of measurements of the first group of points 34a-f are used to determine a size, shape, form, and/or other distinguishing characteristics of a volume bounded by the surface at a first period in time the measurements were taken. The surface and/or volume bounded by the surface is characterized as moving if the surface is matched with a surface detected at a subsequent period in time, wherein the matched surface has polar coordinates that are different than the polar coordinates associated with first group of measurements taken at the first period in time.
In some implementations, a surface is characterized by analyzing measured ranges at given orientations. Consider, for example, the sixth measurement of the sixth point 34d. The sixth point 34d corresponds to range and orientation in the environment with respect to the sensor location. If, after a subsequent point in time from when the sixth measurement is taken, a different range is detected at the same orientation, it is determined that the surface that the sixth point 34d was lying on at the previous period in time identifies the first moving surface 1012. The first moving surface 1012 is then matched with surfaces detected at subsequent periods in time for purposes of tracking the first moving surface 1012.
As a further illustrative example in
Returning to
Constraints on size, shape, form, and/or other distinguishing characteristic of different subjects are known and distinguishable from one another. The size, shape, form, and/or other distinguishing characteristic of detected surfaces of a dog are distinguishable from the size, shape, form, and/or other distinguishing characteristic of detected surfaces of a human being based on known constraints for size, shape, form, and/or other distinguishing characteristic for dogs and/or human beings. As another example, the size, shape, form, and/or other distinguishing characteristic of detected surfaces of a human female is distinguishable from the size, shape, form, and/or other distinguishing characteristic of detected surfaces of a human male. As yet another example, the size, shape, form, and/or other distinguishing characteristic of detected surfaces of a human carrying an object (such as a briefcase) is distinguishable from the size, shape, form, and/or other distinguishing characteristic of detected surfaces of a human not carrying anything. As still another example, the size, shape, form, and/or other distinguishing characteristic of detected surfaces of a particular type of furniture (e.g., a couch) are known and used to distinguish from other types of furniture and/or other static and/or moving object.
Information used to facilitate determining correlations between determined size, shape, and/or form of detected surfaces and known constraints on size, shape and/or form subjects are stored locally by computing platform 12 (e.g., data storage 25), retrieved from an external resources (e.g., external resource 27), and/or are made accessible to computing platform 12 in other ways. For example, known constraints for known types of subjects (e.g., a human being) are stored in data storage 25 and/or other storage location.
As an illustrative example in
As an illustrative example in
Returning to
The subject tracking component 22 is further configured to, responsive to a match not being determined, establish that a surface is no longer present in the environment or has moved outside the detectable range of the ranging sensor 28. In some implementations, surface matching is accomplished using spin images, and/or other surface matching techniques. Algorithms for verifying surface matches may be employed.
As an illustrative example in
Returning to
The external resources 27 include sources of information, hosts and/or providers outside of system 10, external entities participating with system 10, and/or other resources. In some implementations, some or all of the functionality attributed herein to external resources 27 may be provided by resources included in system 10.
The computing platform 12 includes data storage 25. Data storage 25 includes one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Data storage 25 store files, software algorithms, information determined by processor 14, information received from computing platform 12, and/or other information that enables computing platform 12 to function as described.
Processor(s) 14 is configured to provide information processing capabilities in computing platform 12. Although processor 14 is shown in
Method 100 is implemented in one or more processing devices (e.g., a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information), and/or one or more other components. The one or more processing devices include one or more devices executing some or all of the operations of method 100 in response to instructions stored electronically on a data storage medium.
Referring now to method 100 in
At an operation 104, one or more characteristics of the detected surfaces are determined using the detected ranges and orientations as polar coordinates of the surfaces in the environment. In some implementations, operation 104 is performed by a static surface identification component and/or a subject identification component the same or similar to static surface identification component 18 and/or subject identification component 20, respectively (shown in
Although the present technology has been described for illustration based on what is considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the technology is not limited to the disclosed implementations, but, on the contrary, should cover modifications and equivalent arrangements within the spirit and scope of the appended claims. It is to be understood that the present technology contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20100208234 | Kaehler | Aug 2010 | A1 |
| 20100238182 | Geisner | Sep 2010 | A1 |
| 20110230263 | Ng | Sep 2011 | A1 |
| 20120173185 | Taylor | Jul 2012 | A1 |
| 20130278501 | Bulzacki | Oct 2013 | A1 |
| 20140125966 | Phillips | May 2014 | A1 |
| 20140307927 | Folmer | Oct 2014 | A1 |
| Entry |
|---|
| Shackleton et al., “Tracking People with a 360-Degree Lidar”, 7th IEEE International Conference on Advanced Video and Signal-Based Surveillance, 2010, 7 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20160077207 A1 | Mar 2016 | US |
