Patent Grant
11235465
The present disclosure generally relates to robot motion planning, and in particular to systems and methods that facilitate motion planning of a robot storing a discretized representation of the environment on one or more processors.
Motion planning is a fundamental problem in robot control and robotics. A motion plan completely specifies a path a robot can follow from a starting state to a goal state, typically without colliding with any obstacles in an operational environment or with a reduced possibility of colliding with any obstacles in the operational environment. Challenges to motion planning involve the ability to perform motion planning at a relatively low cost and very fast speeds even as characteristics of the robot itself change. For example, such characteristics may include the volume considered occupied by the robot when the robot is holding objects of various sizes, when changing to a different end effector or when changing to a different appendage. Also, there are also challenges with respect to a limited amount of motion planning information that could be stored on processor chip circuitry.
A method of operation in a robot control system may be summarized as including: for a first robot that will operate in an environment, determining a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states are represented by respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph; for at least two or more of the edges of each of the planning graphs, generating a respective set of edge information that represents a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge; storing the plurality of planning graphs and the sets of edge information in at least one nontransitory processor-readable storage; based on at least a portion of the first robot having a first set of physical dimensions at a first time, providing the sets of edge information for a first one of the planning graphs to at least one processor; and based on at least a portion of the first robot having a second set of physical dimensions at a second time, at least one dimension in the second set of physical dimensions different than a corresponding one of the dimensions of the first set, providing the sets of edge information for a second one of the planning graphs to the at least one processor.
The first robot may have at least a first appendage that is selectively operable for movement with respect to the environment in which the first robot operates, and the first robot may have a first end effector attached to the first appendage, the first end effector selectively operable for movement between at least a first and a second end effector arrangements, and may further include determining that the first end effector attached to the first appendage is in a first end effector arrangement, wherein the first set of physical dimensions represent a set of dimensions of the first end effector in the first end effector arrangement; and wherein providing the sets of edge information for a first one of the planning graphs to at least one processor may be in response to the determination that the first end effector attached to the first appendage is in the first end effector arrangement.
The method of operation in a robot control system may further include: determining that the first end effector attached to the first appendage is in a second end effector arrangement, wherein the second set of physical dimensions represents a set of dimensions of the first end effector in the second end effector arrangement; and wherein providing the sets of edge information for a second one of the planning graphs to at least one processor is in response to the determination that the first end effector attached to the first appendage is in the second end effector arrangement.
The first robot may have at least a first appendage that is selectively operable for movement with respect to the environment in which the first robot operates, and a first end effector may be attached to the first appendage, the first end effector selectively operable for movement between at least an un-grasped arrangement and a grasped arrangement, at least one of a size or shape of a volume occupied by the first end effector in the grasped arrangement being different from at least one of a size or shape of a volume occupied by the first end effector in the un-grasped arrangement, and may further include determining that the first end effector attached to the first appendage is in the un-grasped arrangement; wherein providing the sets of edge information for a first one of the planning graphs to at least one processor may be in response to the determination that the first end effector attached to the first appendage is in the un-grasped arrangement; and determining that the first end effector attached to the first appendage may be in the grasped arrangement; wherein providing the sets of edge information for a second one of the planning graphs to at least one processor may be in response to the determination that the first end effector attached to the first appendage is in grasped arrangement. The first robot may have at least a first appendage that is selectively operable for movement with respect to the environment in which the first robot operates, and, may further include determining that the first robot has a first end effector attached to the first appendage, wherein the first set of physical dimensions may represent a set of dimensions of the first end effector attached to the first appendage; and wherein providing the sets of edge information for a first one of the planning graphs to at least one processor may be in response to the determination that the first robot has a first end effector attached to the first appendage.
The method of operation in a robot control system may further include determining that the first robot has a second end effector attached to the first appendage, the second end effector different from the first end effector in at least one of shape or size, wherein the second set of physical dimensions may represent a set of dimensions of the second end effector attached to the first appendage; and wherein providing the sets of edge information for a second one of the planning graphs to at least one processor may be in response to the determination that the first robot has a second end effector attached to the first appendage.
The first robot may be at least one of an autonomous or semi-autonomous vehicle, at least one of a size or shape of a volume occupied by the at least one of an autonomous or semi-autonomous vehicle in a first physical state may be different from at least one of a size or shape of a volume occupied by the at least one of an autonomous or semi-autonomous vehicle in the second physical state (e.g., weight may change due to consumption of fuel, hence velocity may change and distance traveled changes), and may further include determining that the at least one of an autonomous or semi-autonomous vehicle is in the first physical state; wherein providing the sets of edge information for a first one of the planning graphs to at least one processor may be in response to the determination that the at least one of an autonomous or semi-autonomous vehicle is in the first physical state; and determining that the at least one of an autonomous or semi-autonomous vehicle may be in the second physical state; wherein providing the sets of edge information for a second one of the planning graphs to at least one processor may be in response to the determination that the at least one of an autonomous or semi-autonomous vehicle is in second physical state. Providing the sets of edge information for a first one of the planning graphs to at least one processor may include applying the edge information for one of the edges to each of a plurality of circuits of the at least one processor to determine which edges collide with a unit volume occupied by an obstacle in the environment in which the robot operates. Providing the sets of edge information for a first one of the planning graphs to at least one processor may include applying the edge information for one of the edges to each of a plurality of circuits of the at least one processor in parallel. Generating a respective set of edge information that represents a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge may include generating a respective set of edge information that represents in terms of voxels a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge. Generating a respective set of edge information that represents a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge may include generating a respective set of edge information that represents in terms of units of volume, the units of volume that cover two or more voxels, a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge. Generating a respective set of edge information that represents a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge may include generating a respective set of edge information that represents in terms of rectangular prisms (parallelepiped) a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge. Generating a respective set of edge information that represents in terms of rectangular prisms (parallelepiped) a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge may include, for each of the rectangular prisms, storing a pair of three dimensional coordinates that completely define the volume of the respective rectangular prism. The determining a plurality of planning graphs and the generating a respective set of edge information may be performed during a pre-run time period. The providing the sets of edge information for a second one of the planning graphs to the at least one processor may be performed during a run time period.
A processor-based robot control system may be summarized as including: at least one processor; and at least one nontransitory processor-readable medium that stores at least one of processor-executable instructions or data which, when executed by the at least one processor, causes the at least one processor to: for a first robot that will operate in an environment, determine a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states are represented by respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph; for at least two or more of the edges of each of the planning graphs, generate a respective set of edge information that represents a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge; store the plurality of planning graphs and the sets of edge information in at least one nontransitory processor-readable storage; based on at least a portion of the first robot having a first set of physical dimensions at a first time, provide the sets of edge information for a first one of the planning graphs to at least one processor; and based on at least a portion of the first robot having a second set of physical dimensions at a second time, at least one dimension in the second set of physical dimensions different than a corresponding one of the dimensions of the first set, provide the sets of edge information for a second one of the planning graphs to the at least one processor.
The at least one of processor-executable instructions or data, when executed by the at least one processor, may further cause the at least one processor to perform any of the methods described above.
A method of operation in a robot control system may be summarized as including: for a first discretized representation of an environment in which at least a first robot will operate, the environment occupied by one or more obstacles, supplying at least a portion of the first discretized representation of the environment to at least one processor; for each edge of a first planning graph of a plurality of planning graphs stored in memory relative to the at least one processor, wherein each planning graph of the plurality of planning graphs is associated with a different set of physical dimensions of the first robot, providing a respective set of edge information to the at least one processor, the respective set of edge information which represents a volume swept by at least a portion of the first robot in transitioning between a pair of states of the first robot, the pair of states of the first robot represented by respective ones of a pair of nodes of the first planning graph, the respective pair of nodes which are coupled by a respective edge of the first planning graph, the respective edge which represents a transition between the respective pair of states of the first robot; and identifying any of the edges of the first planning graph that the corresponding transition would result in a collision between at least a portion of the robot and at least a portion of at least one of the one or more obstacles in the environment.
Providing a respective set of edge information to the at least one processor may include applying the edge information for one of the edges to each of a plurality of circuits of the at least one processor in parallel. Providing a respective set of edge information to the at least one processor may include, for each edge, applying the edge information for the respective edge to each of a plurality of circuits of the at least one processor in parallel. Providing a respective set of edge information to the at least one processor may include, for each edge, applying to circuits of the at least one processor a respective set of edge information that represents in terms of voxels a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge. Providing a respective set of edge information to the at least one processor may include, for each edge, applying to circuits of the at least one processor a respective set of edge information that represents in terms of units of volume a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge, the units of volume which each cover two or more voxels. Providing a respective set of edge information to the at least one processor t may include, for each edge, applying to circuits of the at least one processor a respective set of edge information that represents, in terms of rectangular prisms, a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge, the units of volume which each cover two or more voxels. Applying to the circuits a respective set of edge information that represents, in terms of rectangular prisms, a volume swept by at least a portion of the first robot in transitioning between the states represented by the respective nodes that are coupled by the respective edge may include, for each of the rectangular prisms, storing a pair of three dimensional coordinates that completely define the volume of the respective rectangular prism.
The method of operation in a robot control system may further include: determining that the first robot will or has changed from a first arrangement to a second arrangement, the second arrangement different from the first arrangement; for each edge of a second planning graph of the plurality of planning graphs, providing a respective set of edge information to the at least one processor, the respective set of edge information which represents a volume swept by at least a portion of the first robot in transitioning between a pair of states of the first robot, the pair of states of the first robot represented by respective ones of a pair of nodes of the second planning graph, the respective pair of nodes which are coupled by a respective edge of the second planning graph, the respective edge which represents a transition between the respective pair of states of the first robot, the second planning graph different from the first planning graph; and identifying any of the edges of the second planning graph that the corresponding transition would result in a collision between at least a portion of the robot and at least a portion of at least one of the one or more obstacles in the environment.
The first robot may include a first appendage that is selectively operable for movement with respect to the environment in which the first robot operates, and determining that the first robot will or has changed from a first arrangement to a second arrangement may include determining that a second end effector is attached or being attached to the first appendage in place of a first end effector. The first robot may include a first appendage that is selectively operable for movement with respect to the environment in which the first robot operates, and a first end effector may be attached to the first appendage, the first end effector selectively operable for movement between at least an un-grasped arrangement and a grasped arrangement, at least one of a size or shape of a volume occupied by the first end effector in the grasped arrangement being different from at least one of a size or shape of a volume occupied by the first end effector in the un-grasped arrangement, and determining that the first robot will or has changed from a first arrangement to a second arrangement may include determining that the first end effector attached to the first appendage is changing or has changed a grasp arrangement. Determining that the first end effector attached to the first appendage is changing or has changed a grasp arrangement may include determining that the first end effector is or has transitioned between an un-grasped arrangement and a grasped arrangement. The first robot may be at least one of an autonomous or semi-autonomous vehicle, at least one of a size or shape of a volume occupied by the at least one of an autonomous or semi-autonomous vehicle in a first arrangement is different from at least one of a size or shape of a volume occupied by the at least one of an autonomous or semi-autonomous vehicle in the second arrangement, and wherein determining that the first robot will or has changed from a first arrangement to a second arrangement may include determining that the at least one of an autonomous or semi-autonomous vehicle is transitioning or has transitioned between in the first and the second arrangements of the at least one of an autonomous or semi-autonomous vehicle. The providing a respective set of edge information for each edge of the first planning graph to the at least one processor may include retrieving the respective set of edge information from a nontransitory storage during a run time period, the respective set of edge information which was stored to the nontransitory storage during a pre-run time period. The at least one processor may be at least one of a field programmable gate array or application specific integrated circuit, and providing the respective set of edge information to at least one processor may include applying the edge information for one of the edges to each of a plurality of circuits of the at least one processor implemented in the at least one of a field programmable gate array or application specific integrated circuit.
The method of operation in a robot control system may further include: storing the first discretized representation of the environment in a memory on the at least one processor; and after storing the first discretized representation of the environment in a memory on the at least one processor, receiving the set of edge information by the at least one processor from a storage that is separate from the at least one processor.
A processor-based robot control system may be summarized as including: at least one processor; and at least one nontransitory processor-readable medium that stores at least one of processor-executable instructions or data which, when executed by the at least one processor, causes the at least one processor to: for a first discretized representation of an environment in which at least a first robot will operate, the environment occupied by one or more obstacles, supply at least a portion of the first discretized representation of the environment to at least one processor; for each edge of a first planning graph of a plurality of planning graphs stored in memory relative to the at least one processor, wherein each planning graph of the plurality of planning graphs is associated with a different set of physical dimensions of the first robot, provide a respective set of edge information to the at least one processor, the respective set of edge information which represents a volume swept by at least a portion of the first robot in transitioning between a pair of states of the first robot, the pair of states of the first robot represented by respective ones of a pair of nodes of the first planning graph, the respective pair of nodes which are coupled by a respective edge of the first planning graph, the respective edge which represents a transition between the respective pair of states of the first robot; and identify any of the edges of the first planning graph that the corresponding transition would result in a collision between at least a portion of the robot and at least a portion of at least one of the one or more obstacles in the environment.
The at least one of processor-executable instructions or data, when executed by the at least one processor, may further cause the at least one processor to perform any of the methods described above.
A method of operation in a system to facilitate motion planning may be summarized as including: for at least a first scenario that includes a set of a plurality of pairs of tasks and environments, for each of the pairs of tasks and environments of the at least the first scenario, for each of a plurality of iterations, generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor, at least two of the respective discretizations comprising a respective set of voxels, where the voxels of the at least two of the respective discretizations are non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the at least two of the respective discretizations is different from one another; assessing an effectiveness of the generated respective discretizations of the representation of the environment in which the robot will operate; and storing to at least one nontransitory processor-readable media at least the generated respective discretizations of the representation of the environment in which the robot will operate that is assessed to be the most effective for at least the first scenario.
Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels of at least one region in a front of the robot has a relatively small volume as compared to a respective volume of each of a plurality of the voxels in at least one region behind the robot. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels of a first region in a front of the robot has a first volume, a plurality of the voxels of a second region in a front of the robot has a second volume, the second volume different than the first volume. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels of a first region in a front of the robot has a first volume, a plurality of the voxels of a second region in the front of the robot has a second volume, and a plurality of the voxels of a third region in the front of the robot has a third volume, the third volume different than the second volume. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels of a first region in a front of the robot has a first volume, a plurality of the voxels of a second region in the front of the robot has a second volume, and a plurality of the voxels of a third region in the front of the robot has a third volume, the second volume different than the first volume, and the third volume different than the second volume. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels in a first region immediately in a front of the robot has a first volume, a plurality of the voxels in a second region in the front of the robot and spaced relatively outward from the first region with respect to the robot has a second volume, and a plurality of the voxels in a third region in the front of the robot and spaced relatively outward from the second region with respect to the robot has a third volume, the second volume different than the first volume, and the third volume different than the second volume. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which a shape of each of a plurality of the voxels in at least one region in a front of the robot is different than a shape of each of a plurality of the voxels of at least one region behind the robot. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels a first region in a front of the robot has a first shape, a plurality of the voxels a second region in a front of the robot has a second shape, the second shape different then the first shape. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels in a first region in a front of the robot has a first shape, a plurality of the voxels a second region in the front of the robot has a second shape, and a plurality of the voxels in a third region in the front of the robot has a third shape, the third shape different than the second shape. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels in a first region in a front of the robot has a first shape, a plurality of the voxels a second region in the front of the robot has a second shape, and a plurality of the voxels in a third region in the front of the robot has a third shape, the second shape different than the first shape, and the third shape different than the second shape. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which each of a plurality of the voxels in a first region immediately in a front of the robot has a first shape, a plurality of the voxels in a second region in the front of the robot and spaced relatively outward from the first region with respect to the robot has a second shape, and a plurality of the voxels in a third region in the front of the robot and spaced relatively outward from the second region with respect to the robot has a third shape, the second shape different than the first shape, and the third shape different than the second shape. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which both a shape and a volume of each of a plurality of the voxels in at least one region in a front of the robot is different than both a shape and a volume of each of a plurality of the voxels of at least one region behind the robot. Generating a respective discretization of a representation of an environment in which a robot will operate by at least one processor may include generating a first respective discretization in which both a shape and a volume of each of a plurality of the voxels in at least one region in a front of the robot is different than both a shape and a volume of each of a plurality of the voxels of at least one region behind the robot, and at least one of the shapes is non-cubic. Assessing an effectiveness of the generated respective discretizations of the representation of the environment in which the robot will operate may include determining which of generated respective discretizations of the representation of the environment in which the robot will operate is mostly likely to result in a path without collisions. Assessing an effectiveness of the generated respective discretizations of the representation of the environment in which the robot will operate may include determining which of generated respective discretizations of the representation of the environment in which the robot will operate is mostly likely to result in a path with a lowest probability of a collision. Assessing an effectiveness of the generated respective discretizations of the representation of the environment in which the robot will operate may include determining which of generated respective discretizations of the representation of the environment in which the robot will operate is mostly likely to result in a lowest cost collision free path. Storing to at least one nontransitory processor-readable media at least the generated respective discretizations of the representation of the environment in which the robot will operate that are assessed to be the most effective for at least the first scenario may include storing at least one generated respective discretization for each of a plurality of scenarios, along with an indication of which of the generated respective discretizations should be used for each of the plurality of scenarios.
A processor-based system to facilitate motion planning may be summarized as including: at least one processor; and at least one nontransitory processor-readable medium that stores at least one of processor-executable instructions or data which, when executed by the at least one processor, causes the at least one processor to: for at least a first scenario that includes a set of a plurality of pairs of tasks and environments, for each of the pairs of tasks and environments of the at least the first scenario, for each of a plurality of iterations, generate a respective discretization of a representation of an environment in which a robot will operate by at least one processor, at least two of the respective discretizations comprising a respective set of voxels, where the voxels of the at least two of the respective discretizations are non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the at least two of the respective discretizations is different from one another; assess an effectiveness of the generated respective discretizations of the representation of the environment in which the robot will operate; and store to at least one nontransitory processor-readable media at least the generated respective discretizations of the representation of the environment in which the robot will operate that is assessed to be the most effective for at least the first scenario.
The at least one of processor-executable instructions or data, when executed by the at least one processor, may further cause the at least one processor to perform any of the methods described above.
A method of operation in a system to facilitate motion planning may be summarized as including: based at least in part on an identified scenario that classifies a pair of a task which a robot will perform and an environment in which the robot will operate, determining which of a number of discretizations to use to generate a number of swept volumes that represent respective regions through which at least a portion of the robot will pass when transitioning between one state of the robot and another state of the robot; for each of a plurality of edges in a planning graph, determining a respective swept volume of the edge using the determined discretization, the planning graph comprising a plurality of nodes and a plurality of edges, each node which represents a respective one of a plurality of states of the robot, each of the edges coupling a respective pair of the nodes and representing a respective transition by the robot between the states represented by the respective nodes coupled by the respective edge; and storing to at least one nontransitory processor-readable media at least one of the determined swept volume's respective discretizations of the representation of the environment in which the robot will operate that is assessed to be the most effective for at least the identified scenario.
Determining which of a number of discretizations to use to generate a number of swept volumes may include selecting between at least two discretizations based on the identified scenario, where the voxels of the at least two of the respective discretizations may be non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the at least two of the respective discretizations may be different from one another. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels of at least one region in a front of the robot may have a relatively small volume as compared to a respective volume of each of a plurality of the voxels in at least one region behind the robot. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first volume, a plurality of the voxels in a second region in a front of the robot may have a second volume, the second volume different then the first volume. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first volume, a plurality of the voxels in a second region in the front of the robot may have a second volume, and a plurality of the voxels in a third region in the front of the robot may have a third volume, the third volume different than the second volume. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first volume, a plurality of the voxels in a second region in the front of the robot may have a second volume, and a plurality of the voxels in a third region in the front of the robot may have a third volume, the second volume different than the first volume, and the third volume different than the second volume. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region immediately in a front of the robot may have a first volume, a plurality of the voxels in a second region in the front of the robot and spaced relatively outward from the first region with respect to the robot may have a second volume, and a plurality of the voxels in a third region in the front of the robot and spaced relatively outward from the second region with respect to the robot may have a third volume, the second volume different than the first volume, and the third volume different than the second volume. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which a shape of each of a plurality of the voxels in at least one region in a front of the robot may be different than a shape of each of a plurality of the voxels of at least one region behind the robot. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first shape, and a plurality of the voxels in a second region in a front of the robot may a second shape, the second shape different then the first shape. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first shape, a plurality of the voxels in a second region in the front of the robot may have a second shape, and a plurality of the voxels in a third region in the front of the robot may have a third shape, the third shape different than the second shape. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first shape, a plurality of the voxels in a second region in the front of the robot may have a second shape, and a plurality of the voxels in a third region in the front of the robot may have a third shape, the second shape different than the first shape, and the third shape different than the second shape. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which each of a plurality of the voxels in a first region immediately in a front of the robot may have a first shape, a plurality of the voxels in a second region in the front of the robot and spaced relatively outward from the first region with respect to the robot may have a second shape, and a plurality of the voxels in a third region in the front of the robot and spaced relatively outward from the second region with respect to the robot may have a third shape, the second shape different than the first shape, and the third shape different than the second shape. Determining a respective swept volume of the edge using the determined discretization may include determining the respective swept volume of the edge using the determined discretization in which both a shape and a volume of each of a plurality of the voxels in at least one region in a front of the robot is different than both a shape and a volume of each of a plurality of the voxels of at least one region behind the robot.
The method of operation in a system to facilitate motion planning may further include receiving the identified scenario.
The method of operation in a system to facilitate motion planning may further include providing a respective set of edge information to at least one processor, the respective set of edge information which represents the respective swept volume swept by at least a portion of the first robot in transitioning between a pair of states of the robot.
The at least one processor may be at least one of a field programmable gate array or application specific integrated circuit, and may further include applying a respective set of edge information to each of a plurality of circuits of the at least one processor implemented in the at least one of a field programmable gate array or application specific integrated circuit, the respective set of edge information which represents the respective swept volume swept by at least a portion of the robot in transitioning between a pair of states of the robot.
A processor-based system to facilitate motion planning may be summarized as including: at least one processor; and at least one nontransitory processor-readable medium that stores at least one of processor-executable instructions or data which, when executed by the at least one processor, causes the at least one processor to: based at least in part on an identified scenario that classifies a pair of a task which a robot will perform and an environment in which the robot will operate, determine which of a number of discretizations to use to generate a number of swept volumes that represent respective regions through which at least a portion of the robot will pass when transitioning between one state of the robot and another state of the robot; for each of a plurality of edges in a planning graph, determine a respective swept volume of the edge using the determined discretization, the planning graph comprising a plurality of nodes and a plurality of edges, each node which represents a respective one of a plurality of states of the robot, each of the edges coupling a respective pair of the nodes and representing a respective transition by the robot between the states represented by the respective nodes coupled by the respective edge; and store to at least one nontransitory processor-readable media at least one of the determined swept volume's respective discretizations of the representation of the environment in which the robot will operate that is assessed to be the most effective for at least the identified scenario.
The at least one of processor-executable instructions or data, when executed by the at least one processor, may further cause the at least one processor to perform any of the methods described above.
A method of operation in a system to facilitate motion planning may be summarized as including: based at least in part on an identified scenario that classifies a pair of a task which a robot will perform and an environment in which the robot operates, determining which of a number of discretizations to use to generate a discretized representation of the environment, including obstacles, if any, in the environment; receiving sensor information produced by one or more sensors that sense the environment, the sensor information which represents the environment, including obstacles, if any, in the environment; and generating a discretized representation of the environment, including obstacles, if any, in the environment using the determined discretization, wherein a plurality of voxels of the determined discretization are non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the determined discretization is different from that of another one of the number of discretizations.
The method of operation in a system to facilitate motion planning may further include storing to at least one nontransitory processor-readable media the determined generated discretized representation of the environment, including obstacles, if any, in the environment.
Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using a distribution of voxel size and shape that matches a distribution of voxel size and shape used to generate a discretized representation of a swept volume. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using a distribution of voxel size and shape that matches a distribution of voxel size and shape used to generate a discretized representation of a swept volume swept by at least a portion of a robot. Determining which of a number of discretizations to use to generate a discretized representation of the environment, including obstacles, if any, in the environment, may include selecting between at least two discretizations based on the identified scenario, where the voxels of the at least two of the respective discretizations may be non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the at least two of the respective discretizations may be different from one another. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels of at least one region in a front of the robot may have a relatively small volume as compared to a respective volume of each of a plurality of the voxels in at least one region behind the robot. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first volume, and a plurality of the voxels in a second region in a front of the robot may a second volume, the second volume different then the first volume. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first volume, a plurality of the voxels in a second region in the front of the robot may have a second volume, and a plurality of the voxels in a third region in the front of the robot may have a third volume, the third volume different than the second volume. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first volume, a plurality of the voxels in a second region in the front of the robot may have a second volume, and a plurality of the voxels in a third region in the front of the robot may have a third volume, the second volume different than the first volume, and the third volume different than the second volume. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region immediately in a front of the robot may have a first volume, a plurality of the voxels in a second region in the front of the robot and spaced relatively outward from the first region with respect to the robot may have a second volume, and a plurality of the voxels in a third region in the front of the robot and spaced relatively outward from the second region with respect to the robot may have a third volume, the second volume different than the first volume, and the third volume different than the second volume. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which a shape of each of a plurality of the voxels in at least one region in a front of the robot may be different than a shape of each of a plurality of the voxels of at least one region behind the robot. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first shape, and a plurality of the voxels in a second region in a front of the robot may have a second shape, the second shape different then the first shape. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first shape, a plurality of the voxels in a second region in the front of the robot may have a second shape, and a plurality of the voxels in a third region in the front of the robot may have a third shape, the third shape different than the second shape. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region in a front of the robot may have a first shape, a plurality of the voxels in a second region in the front of the robot may have a second shape, and a plurality of the voxels in a third region in the front of the robot may have a third shape, the second shape different than the first shape, and the third shape different than the second shape. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which each of a plurality of the voxels in a first region immediately in a front of the robot may have a first shape, a plurality of the voxels in a second region in the front of the robot and spaced relatively outward from the first region with respect to the robot may have a second shape, and a plurality of the voxels in a third region in the front of the robot and spaced relatively outward from the second region with respect to the robot may have a third shape, the second shape different than the first shape, and the third shape different than the second shape. Generating a discretized representation of the environment, including obstacles, if any, in the environment may include generating the discretized representation of the environment using the determined discretization in which both a shape and a volume of each of a plurality of the voxels in at least one region in a front of the robot may be different than both a shape and a volume of each of a plurality of the voxels of at least one region behind the robot.
The method of operation in a system to facilitate motion planning may further include: receiving information that represents the task to be performed by the robot and the environment in which the robot is to operate; identifying the identified scenario based at least in part on received information.
The method of operation in a system to facilitate motion planning may further include providing the discretized representation of the environment to at least one processor.
The at least one processor may be at least one of a field programmable gate array or application specific integrated circuit, and may further include applying the discretized representation of the environment to each of a plurality of circuits implemented in the at least one of a field programmable gate array or application specific integrated circuit.
A processor-based system to facilitate motion planning may be summarized as including: at least one processor; and at least one nontransitory processor-readable medium that stores at least one of processor-executable instructions or data which, when executed by the at least one processor, causes the at least one processor to: based at least in part on an identified scenario that classifies a pair of a task which a robot will perform and an environment in which the robot operates, determine which of a number of discretizations to use to generate a discretized representation of the environment, including obstacles, if any, in the environment; receive sensor information produced by one or more sensors that sense the environment, the sensor information which represents the environment, including obstacles, if any, in the environment; and generate a discretized representation of the environment, including obstacles, if any, in the environment using the determined discretization, wherein a plurality of voxels of the determined discretization are non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the determined discretization is different from that of another one of the number of discretizations.
The at least one of processor-executable instructions or data, when executed by the at least one processor, may further cause the at least one processor to perform any of the methods described above.
A method of operation in a robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph may be summarized as including: for a first planning graph of the plurality of planning graphs, for each of a plurality of edges of the first planning graph performing collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate; updating the first planning graph based on the collision checking; performing an optimization of the updated first planning graph to identify one or more optimized results, if any, from the updated first planning graph; determining whether the one or more optimized results, if any, from the updated first planning graph meets a satisfaction condition; in response to determining that the optimized result does not meet the satisfaction condition: for each of a plurality of edges of the second planning graph performing collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate, updating the second planning graph based on the collision checking; and performing an optimization of the updated second planning graph to identify one or more optimized results, if any, from the updated second planning graph.
The method of operation in a robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph may further include determining whether the one or more optimized results, if any, from the updated second planning graph, if any, meets a satisfaction condition.
The method of operation in a robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph may, in response to determining that the one or more optimized results, if any, from the updated second planning graph meets the satisfaction condition, apply a transition identified by one of the one or more optimized results from the updated second planning graph to the robot.
The method of operation in a robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph may, in response to determining that the one or more optimized results, if any, from the updated second planning graph does not meet the satisfaction condition: for each of a plurality of edges of a third planning graph performing collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate, update the third planning graph based on the collision checking; and perform an optimization of the updated third planning graph to identify one or more optimized results, if any, from the updated third planning graph.
The method of operation in a robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph may, in response to determining that the one or more optimized results, if any, from the updated third planning graph meets the satisfaction condition: apply a transition identified by one of the one or more optimized results from the updated third planning graph to the robot.
Determining that the one or more optimized results, if any, from the updated second planning graph meets the satisfaction condition may include determining whether the optimization produces any paths. Determining that the one or more optimized results, if any, from the updated second planning graph meets the satisfaction condition may include determining whether the optimization produces any collision-free paths. Performing collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate may include, for each of the edges in the first planning graph, applying a set of edge information for the edges to each of a plurality of circuits in parallel, the circuits which each represent a respective unit volume occupied by an obstacle in the environment in which the robot operates.
The method of operation in a robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph may further include configuring at least one processor to implement a plurality of circuits that represent respective ones of each of a plurality of unit volumes occupied by one or more obstacles in the environment in which at least the first robot will operate.
A processor-based robot control system that employs a plurality of planning graphs, each planning graph respectively comprising a plurality of nodes connected by a plurality of edges, each node which represents, implicitly or explicitly, variables that characterize a respective state of the first robot, and each edge which represents a transition between a respective pair of the states of the first robot, where the respective pair of states is represented by a respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph, may be summarized as including: at least one processor; and at least one nontransitory processor-readable medium that stores at least one of processor-executable instructions or data which, when executed by the at least one processor, causes the at least one processor to: for a first planning graph of the plurality of planning graphs, for each of a plurality of edges of the first planning graph perform collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate; update the first planning graph based on the collision checking; perform an optimization of the updated first planning graph to identify one or more optimized results, if any, from the updated first planning graph; determine whether the one or more optimized results, if any, from the updated first planning graph meets a satisfaction condition; in response to determining that the optimized result does not meet the satisfaction condition: for each of a plurality of edges of the second planning graph perform collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate, update the second planning graph based on the collision checking; and perform an optimization of the updated second planning graph to identify one or more optimized results, if any, from the updated second planning graph.
The at least one of processor-executable instructions or data, when executed by the at least one processor, may further cause the at least one processor to perform any of the methods described above.
In the drawings, identical reference numbers identify similar elements or acts. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the shapes of various elements and angles are not drawn to scale, and some of these elements are arbitrarily enlarged and positioned to improve drawing legibility. Further, the particular shapes of the elements as drawn are not intended to convey any information regarding the actual shape of the particular elements, and have been solely selected for ease of recognition in the drawings.
In the following description, certain specific details are set forth in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, actuator systems, and/or communications networks have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the embodiments. In other instances, well-known computer vision methods and techniques for generating perception data and volumetric representations of one or more objects and the like have not been described in detail to avoid unnecessarily obscuring descriptions of the embodiments.
Unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open, inclusive sense, that is as “including, but not limited to.”
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its sense including “and/or” unless the content clearly dictates otherwise.
The headings and Abstract of the Disclosure provided herein are for convenience only and do not interpret the scope or meaning of the embodiments.
Referring to
Challenges to motion planning involve the ability to perform motion planning at a relatively low cost, but at very fast speeds and frequently as characteristics of the robot itself change. For example, such characteristics may include, but are not limited to, the volume considered occupied by the robot when the robot is holding objects of various sizes, when changing to a different end effector or when changing to a different appendage. Thus, it is advantageous for the robot 102 to keep up with those changes in order to perform motion. Involved with this are challenges to efficiently represent, communicate and compare the space considered to be occupied by the robot 102 and the obstacles in the environment 100. For example, as shown in
While
Although not required, many of the embodiments will be described in the general context of computer-executable instructions, such as program application modules, objects, or macros stored on computer- or processor-readable media and executed by one or more computer or processors that can perform collision assessments and motion planning operations. Such motion planning operations may include, but are not limited to, one or more of: generating one or more discretized representations of robot swept volumes associated with edges of planning graphs; determining which of a number of discretizations to use to generate discretized representations of robot sweep volumes; generating discretized representations of the environment in which the robot 102 will operate, including obstacles in the environment; determining which of a number of discretizations to use to generate a discretized representation of the environment; determining a plurality of planning graphs; storing the determined plurality of planning graphs and respective sets of edge information; generating information that represents a volume swept by at least a portion of the robot 102 in transitioning between the states represented by the nodes of the planning graphs; performing a collision assessment on edges of a planning graph; providing sets of edge information for the planning graphs; identifying one or more optimized results from planning graphs; collision checking for collisions between discretized representations of swept volumes associated with edges of planning graphs and discretized representations of obstacles in the environment in which the robot 102 will operate; determining whether the optimization produces any collision-free paths for the robot 102; and implementing a motion plan for the robot 102.
The robot 102 of
Many embodiments may combine the outputs of multiple sensors and the sensors may provide a very fine granularity voxelization. However, in order for the robot 102 to efficiently perform motion planning, the processor 212a of the robot 102 may use coarser voxels (i.e., “processor voxels”) to represent the environment and the volume in 3D space swept by the robot 102 when making transitions between various states. Thus, the system 200 may transform the output of the sensors 282 accordingly. For example, the output of the sensors 282 may use 10 bits of precision on each axis, so each voxel originating directly from the sensors 282 (i.e., a “sensor voxel”) has a 30-bit ID, and there are 230 sensor voxels. The system 200 uses (at design time and runtime) 6 bits of precision on each axis for an 18-bit processor voxel ID, and there are 218 processor voxels. Thus there are 212 sensor voxels per processor voxel. At runtime, if the system 200 determines any of the sensor voxels within a processor voxel is occupied, the system 200 considers the processor voxel to be occupied and generates the occupancy grid accordingly.
Each edge of a planning graph for the robot 102 also has some number of voxels (or boxes) corresponding to the volume in 3D space swept by the robot 102 when making the transition in the planning graph from one state to another state represented by that edge. Those voxels or boxes swept by the robot 102 when making the transition in the planning graph from one state to another state represented by that edge may be stored for each edge of the planning graph in off-chip memory devoted to the planning graph, such as in planning graph edge information memory 284. In various other embodiments, the voxels or boxes swept by the robot 102 when making the transition in the planning graph from one state to another state represented by that edge may be stored for each edge of the planning graph in other locations, such as in on-chip memory in one or more application specific integrated circuits (ASIC), for example.
In one embodiment, the collision assessment is performed by first streaming in all of the obstacle voxels (or boxes) onto a processor, such as processor 212a. For example, the obstacle voxels (or boxes) representing environment 100, including obstacle A 112 and obstacle B 104, may be streamed into processor 212a and stored on environment memory 294. The environment memory 294 is on-chip memory of processor 212a. In some embodiments, the environment memory 294 may be block RAM (BRAM) in a field programmable gate array (FPGA). In some embodiments, the BRAM may be a dedicated, configurable two-port memory unit containing several kilobits of random access memory (RAM). The FPGA contains several of these blocks. Then the edge information for each edge of the planning graph for the robot 102 is streamed from the off-chip memory devoted to the planning graph, such as from planning graph edge information memory 284. For each edge voxel (or box), when it is streamed in from the swept volume of an edge, if it collides with any of the obstacle voxels (or boxes), the processor 212a determines a collision with that edge in the planning graph.
For example, when an edge voxel is streamed in from the swept volume of edge x of the planning graph, the processor 212a may use Boolean circuitry to compare the edge voxel with all of the obstacle voxels (or boxes) stored on environment memory 294. If the system 200 determines the edge voxel collides with any of the obstacle voxels (or boxes) based on the comparison, the system 200 notes a collision with edge x. This embodiment improves on the technology of collision assessment because it enables much larger planning graphs to be used in the collision assessment as compared to other designs in which the collision assessment is performed in parallel on all the edges of the planning graph. In particular, this helps overcome the disadvantage of other designs with respect to a limited amount of planning graph information that could be stored on processor chip circuitry. Using the collision assessment method described herein, on-chip storage, such as environment memory 294, is often more than sufficient for storing all the obstacle boxes (although it may be less so with using voxels). This provides the ability to store large planning graphs and/or multiple planning graphs in less expensive, off-chip storage. For example, this provides the ability to store large planning graphs and/or multiple planning graphs in planning graph edge information memory 284 which, in some embodiments, is a less expensive type of memory, such as dynamic random access memory (DRAM).
In various embodiments, such operations may be performed entirely in hardware circuitry or as software stored in a memory storage, such as system memory 214, and executed by one or more hardware processors 212a, such as one or more microprocessors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) processors, programmed logic controllers (PLCs), electrically programmable read only memories (EEPROMs), or as a combination of hardware circuitry and software stored in the memory storage.
Also, implementation of various relevant aspects of perception, planning graph construction, collision detection, and path search are also described in International Patent Application No. PCT/US2017/036880, filed Jun. 9, 2017 entitled “MOTION PLANNING FOR AUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS,” International Patent Application Publication No. WO 2016/122840, filed Jan. 5, 2016, entitled “SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME” and U.S. Patent Application No. 62/616,783, filed Jan. 12, 2018, entitled, “APPARATUS, METHOD AND ARTICLE TO FACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVING DYNAMIC OBJECTS” and are incorporated herein by reference, in their entirety. Those skilled in the relevant art will appreciate that the illustrated embodiments, as well as other embodiments, can be practiced with other system structures and arrangements and/or other computing system structures and arrangements, including those of robots, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, personal computers (“PCs”), networked PCs, mini computers, mainframe computers, and the like. The embodiments or portions thereof (e.g., at design time and pre-runtime) can be practiced in distributed computing environments where tasks or modules are performed by remote processing devices, which are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices or media. However, where and how certain types of information are stored is important to help improve motion planning.
For example, various motion planning solutions “bake in” a roadmap (i.e., a planning graph) into the processor, and each edge in the roadmap corresponds to a non-reconfigurable Boolean circuit of the processor. The design in which the planning graph is “baked in” to the processor, poses a problem of having limited processor circuitry to store multiple or large planning graphs.
One solution provides a reconfigurable design that places the planning graph information into memory storage. With such a solution, once again there is information for each edge of the planning graph, but this information is stored in memory instead of being baked into a circuit. Each edge corresponds to a motion of the robot 102, and each motion sweeps a volume in 3D space. This swept volume collides with some number of obstacle voxels (or boxes, or however it may be decided to represent obstacles in various embodiments). With such a solution, the voxels that this edge collides with is the information that is stored in memory.
The present disclosure provides a solution that provides for dynamically switching between multiple planning graphs at runtime (i.e., reconfiguring the processor) in a more efficient manner, providing several improvements over other designs. For example, consider a robot with an end effector that can grasp objects. The robot arm collides with different parts of 3D space when holding something than when not holding something. One example is the robot arm 106 holding a big ball at the end of the arm with the end effector. The planning graph corresponding to when the robot 102 is “holding nothing” doesn't work if the robot 102 is holding something. Conversely, one could very conservatively use the “holding something” planning graph even if the robot 102 isn't holding anything, but that is both inefficient and would require the use of a single planning graph corresponding to the scenario in which the robot 102 is holding the largest possible object. Instead, in various embodiments, the system 200 builds a family of planning graphs corresponding to different possible characteristics, e.g., “holding nothing”, “holding a small sphere of size x”, “holding a rectangular prism of size y”, etc. These planning graphs may be swapped in and out from the planning graph edge information memory 284 by the processor 212a as the robot 102 picks things up and puts them down. This solution also applies to a robot whose end effector changes at times. For example, the robot 102 may be in an arrangement having a first end effector with a first set of dimensions. This first end effector may be swapped for a second end effector with a second set of dimensions. When the first end effector is swapped for a second end effector with a second set of dimensions the robot 102 will be in a different arrangement having the second end effector. Where the dimensions of the second set of dimensions are different from the dimensions of the first set of dimensions, the volume swept by the robot changes when the end effectors are swapped. Also for example, where the robot is an autonomous or partially autonomous vehicle, the vehicle may have a first set of dimensions in a first arrangement during a first period and the same vehicle may have a second set of dimensions in a second arrangement during a second period. For instance, a vehicle may be empty in a first arrangement and fully loaded in a second arrangement, changing the weight of the vehicle and potentially changing a height of the vehicle above a road, a clearance under the vehicle, or even a trajectory of a vehicle, for instance as it corners around a turn or curve. Where the dimensions of the second set of dimensions are different from the dimensions of the first set of dimensions, the volume swept by the vehicle changes between the first and the second arrangements. Likewise, where the trajectory changes between the first and the second arrangements the volume swept by the vehicle changes between the arrangements. The system 200 thus stores different planning graphs in planning graph edge information memory 284 for different end effectors.
Planning graphs take significant time and resources to build, but with the solutions provided herein, one would only have to do that once, for example, prior to runtime. Once the planning graphs are generated, they may all be stored in planning graph edge information memory 284 and it is relatively quick and efficient for the processor 212a to swap them in and out, or select which one to use based on the current characteristic of the robot 102, such as when the robot is grasping an object of a particular size.
As noted above, some pre-processing activities may be performed before runtime and thus, in some embodiments, these operations may be performed by remote processing devices, which are linked through a communications network to the robot control system 200 via network interface 260. For example, a programming phase allows preparation of the robot for the problem of interest. In such embodiments, extensive preprocessing is leveraged to avoid runtime computation. Precomputed data regarding the volume in 3D space swept by the robot 102 when making the transition in the planning graph from one state to another state represented by edges in the roadmap may be stored in planning graph edge information memory 284 and accessed by the processor 212a during runtime. The system 200 may also build a family of planning graphs prior to runtime corresponding to different possible changing dimensional characteristics of the robot that may occur during runtime. The system 200 then stores such planning graphs in planning graph edge information memory 284.
During the runtime phase, the sensors 282 send perception data to processor 212a. The perception data may be a stream of which voxels or boxes that are present in the current environment and are stored in on-chip environment memory 294. Using Boolean circuitry to compare the perception data retrieved from the environment memory 294 to the information stored in the planning graph edge information memory 284, the processor 212a calculates which motions avoid collision and selects a corresponding path in the planning graph for the robot 102 to follow. The processor 212a then runs and returns the resulting path to the actuator system 266.
The robot control system 200 may also be communicably coupled to remote systems, e.g., desktop computer, laptop computer, ultraportable computer, tablet computer, smartphone, wearable computer (not shown) that are directly communicably coupled or indirectly communicably coupled to the various components of the robot control system 200 via the network interface 260. Such remote systems may be used to program, configure, control or otherwise interface with or input data to the robot control system 200 and various components within the robot control system 200. Such a connection may be through one or more communications channels, for example, one or more wide area networks (WANs), for instance, the Internet, using Internet protocols. As noted above, pre-runtime calculations (e.g., generation of the family of planning graphs) may be performed by a system that is separate from the robot 102 or other robot, while runtime calculations may be performed on the processor 212 that is on the robot 102 since it is important for the system 200 to be able to change planning graphs to react in real time to changing physical dimensions of the robot 102.
Unless described otherwise, the construction and operation of the various blocks shown in
The robot control system 200 may include one or more processing units 212, the system memory 214, the planning graph edge information memory 284 and the system bus 216 that couples various system components including the system memory 214 and the planning graph edge information memory 284 to the processing units 212. In some embodiments, the planning graph edge information memory 284 may be, or be part of, the system memory 214. The processing units may be any logic processing unit, such as one or more central processing units (CPUs) 212a, digital signal processors (DSPs) 212b, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The system memory 214 may include read-only memory (“ROM”) 218 and random access memory (“RAM”) 220. The planning graph edge information memory 284 may include RAM, such as DRAM. A basic input/output system (“BIOS”) 222, which can form part of the ROM 218, contains basic routines that help transfer information between elements within the robot control system 200, such as during start-up.
The robot control system 200 may include a disk drive 224, which may be, for example, a hard disk drive for reading from and writing to a hard disk, a flash memory drive for reading from and writing to removable flash memory devices, an optical disk drive for reading from and writing to removable optical disks, or a magnetic disk drive for reading from and writing to magnetic disks. The robot control system 200 may also include any combination of such disk drives in various different embodiments. The disk drive 224 may communicate with the processing units 212 via the system bus 216. The disk drive 224 may include interfaces or controllers (not shown) coupled between such drives and the system bus 216, as is known by those skilled in the relevant art. The disk drive 224 and its associated computer-readable media 226 provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the robot control system 200. Those skilled in the relevant art will appreciate that other types of computer-readable media that can store data accessible by a computer may be employed, such as WORM drives, RAID drives, magnetic cassettes, digital video disks (“DVD”), Bernoulli cartridges, RAMs, ROMs, smart cards, etc.
Program modules can be stored in the system memory 214, such as an operating system 236, one or more application programs 238, other programs or modules 240 and program data 242. Application programs 238 may include instructions that cause the processor(s) 212 to perform one or more of: generating one or more discretized representations of robot swept volumes associated with edges of planning graphs; determining which of a number of discretizations to use to generate discretized representations of robot swept volumes; generating discretized representations of the environment 100 in which the robot will operate, including obstacles in the environment 100; determining which of a number of discretizations to use to generate a discretized representation of the environment 100; determining a plurality of planning graphs; storing the determined plurality of planning graphs and respective sets of edge information; generating information that represents a volume swept by at least a portion of the robot in transitioning between the states represented by the nodes of the planning graphs; performing a collision assessment on edges of a planning graph; providing sets of edge information for the planning graphs; identifying one or more optimized results from planning graphs; collision checking for collisions between discretized representations of swept volumes associated with edges of planning graphs and discretized representations of obstacles in the environment 100 in which the robot 102 will operate; determining whether the optimization produces any collision-free paths for the robot; and implementing a motion plan for the robot. Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to perform other operations of perception (via sensors 282), planning graph construction, collision detection, and path search as described herein and in the references incorporated herein by reference.
Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to generate respective sets of planning graph edge information that represent different volumes swept by the robot 102 in transitioning between states corresponding to when the robot 102 has different dimensions and store a plurality of planning graphs in planning graph edge information memory 284 corresponding to the respective sets of planning graph edge information.
Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to, for a first discretized representation of an environment 100 in which the robot 102 will operate, supply and store in environment memory 294 at least a portion of the first discretized representation of the environment 100; for each edge of the planning graph, provide and store a respective set of edge information in planning graph edge information memory 284; and identify any of the edges of the planning graph that the corresponding transition would result in a collision between at least a portion of the robot 102 and at least a portion of at least one of the one or more obstacles, such as obstacle A 112 and obstacle B 104 in the environment 100.
The term “environment” is used in the present example to refer to the robot's current workspace, including obstacles. The term “task” is used in the present example to refer to a robot task in which the robot 102 must get from Pose A to Pose B (perhaps grasping or dropping something) without colliding with obstacles in its environment. The term “scenario” is used in the present example to refer to a class of environment/task pairs. For example, a scenario could be “pick-and-place tasks in an environment with a 3-foot table and between x and y obstacles with sizes and shapes in a given range.” There may be many different task/environment pairs that fit into such criteria, depending on the locations of goals and the sizes and shapes of obstacles. The system 200 may include one or more remote processing devices, which are linked through a communications network via network interface 260. Such one or more remote processing devices may execute one or more machine-readable instruction sets that cause the system 200 to generate a respective discretization of a representation of an environment 100 in which the robot 102 will operate for pairs of tasks and environments of various different scenarios. In an example embodiment, at least two of the respective discretizations comprise a respective set of voxels. The voxels of the respective discretizations may be non-homogenous in at least one of size and shape within the respective discretization. Also, a respective distribution of the non-homogeneousness of the voxels of the respective discretizations may be different from one another. In particular, the discretizations may comprise a respective set of voxels, where the voxels of at least two of the respective discretizations are non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of at least two of the respective discretizations is different from one another. The Application programs 238 may include one or more machine-readable instruction sets that cause the processor(s) 212 to then assess an effectiveness of the generated respective discretizations of the representation of the environment 100 in which the robot will operate and store the generated respective discretizations that are assessed to be the most effective for particular scenarios.
Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to, based at on an identified scenario that classifies a pair of a task which the robot 102 will perform and the environment 100 in which the robot will operate, determine which of a number of discretizations to use to generate a number of swept volumes that represent respective regions through which the robot 102 will pass when transitioning between one state of the robot and another state of the robot 102; and for each of a plurality of edges in a planning graph, determine a respective swept volume of the edge using the determined discretization. The application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to store the determined swept volume's respective discretizations of the representation of the environment 100 in which the robot 102 will operate that is assessed to be the most effective for the identified scenario.
Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to, based at least in part on an identified scenario that classifies a pair of tasks which the robot 102 will perform and an environment 100 in which the robot operates, determine which of a number of discretizations to use to generate a discretized representation of the environment 100. Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to receive sensor information produced by one or more sensors 282 that sense the environment 100 and generate a discretized representation of the environment, including obstacles, if any, in the environment using the determined discretization. A plurality of voxels of the determined discretization may be non-homogenous in at least one of size and shape within the respective discretization, and a respective distribution of the non-homogeneousness of the voxels of the determined discretization may be different from that of another one of the number of discretizations.
Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to perform collision checking for multiple planning graphs between edges of the planning graphs and any obstacles in an environment in which the robot will operate. The processor(s) 212 may perform such collision checking on each planning graph, temporarily update the planning graph accordingly, and perform an optimization and determine whether the optimized results, if any, from the updated planning graph meet a satisfaction condition. If the satisfaction condition is not met, then the processor(s) 212 may move to the next planning graph and perform the same operation. Once a planning graph is found in which the satisfaction condition is met, the processor(s) 212 apply a transition identified by one of the optimized results from the planning graph which met the satisfaction condition to the robot 102.
Application programs 238 may additionally include one or more machine-readable instruction sets that cause the processor(s) 212 to perform various other methods described herein, including, but not limited to, those illustrated in
In various embodiments, one or more of the operations described above may be performed by one or more remote processing devices of the system 200, which are linked through a communications network via network interface 260 or by one or more processor(s) 212 that are located on the robot 102.
While shown in
The processor(s) 212 may be any logic processing units, such as one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Non-limiting examples of commercially available computer systems include, but are not limited to, the Celeron, Core, Core 2, Itanium, and Xeon families of microprocessors offered by Intel® Corporation, U.S.A.; the K8, K10, Bulldozer, and Bobcat series microprocessors offered by Advanced Micro Devices, U.S.A.; the A5, A6, and A7 series microprocessors offered by Apple Computer, U.S.A.; the Snapdragon series microprocessors offered by Qualcomm, Inc., U.S.A.; and the SPARC series microprocessors offered by Oracle Corp., U.S.A. Unless described otherwise, the construction and operation of the various blocks shown in
In the example illustrated in
Thus, to facilitate avoiding collisions, at design time (prior to runtime), one or more planning graphs are generated by the system 200 to determine which areas are occupied by the robot 102 when making various transitions from one state to another state. For example, an edge of a planning graph for the robot 102 has some number of voxels (or boxes) corresponding to the volume in 3D space swept by the robot 102 corresponding to region 302. Those voxels or boxes swept by the robot 102 corresponding to region 302 when making the corresponding transition in the planning graph may be stored as corresponding to an edge of the planning graph in off-chip memory devoted to the planning graph, such as in planning graph edge information memory 284. Then, at runtime, the obstacle voxels (or boxes) representing environment 100, including obstacle A 112 (and also obstacle B 104) are streamed into processor 212a in the form of an occupancy grid and stored on the on-chip environment memory 294. The system 200 determines which voxels are occupied (based on the occupancy grid), and determines to not use any motion that would collide with any currently occupied voxel. In particular, for each edge voxel (or box) representing a portion of the swept volume of an edge, when it is streamed in from planning graph edge information memory 284, the processor determines whether it collides with any of the obstacle voxels (or boxes) that have been stored in environment memory 294 based on the occupancy grid. If the edge voxel (or box) collides with any of the obstacle voxels (or boxes), the system 200 determines a collision with that edge in the planning graph and will determine to not use the motion of the robot 102 associated with that edge in the planning graph.
For example, prior to executing the motion of the robot arm 106 depicted in
The system 200 may also make decisions regarding how to represent the swept volume represented by each of the planning graphs as well as how to represent the obstacles in the environment 100. Such decisions may include decisions made by the processor 212a regarding how the discretization is to be performed, such as, for example, whether to represent the swept volume with voxels or boxes, the shape and size of the voxels or boxes to use, whether to use a discretization that is non-uniform in size and/or shape of the voxels or boxes used, and in which scenarios to use different shapes and/or sizes of voxels or boxes. In various embodiments, one or more of such operations described above to make decisions regarding how to represent the swept volume represented by each of the planning graphs as well as how to represent the obstacles in the environment 100 may be performed by one or more remote processing devices of the system 200, which are linked through a communications network via network interface 260 or by one or more processor(s) 212 that are located on the robot 102.
In particular, the system 200 may decide that the working space of the robot 102 will be discretized into voxels (which may later be grouped into boxes). One example implementation performs uniform quantization in each of the 3 dimensions. However, it may be advantageous to have non-cube-shaped voxels and/or voxels that are smaller/larger in different parts of the robot's workspace. For example, one embodiment uses smaller voxels (finer resolution) in the space right in front of the robot 102 and larger voxels (coarser resolution) at the extreme end of the robot's reach. Thus, various embodiments may use non-uniform quantization and/or use of non-cube-shaped voxels. The present disclosure also provides an algorithm for how the system 200 chooses the voxel sizes and/or shapes.
One example embodiment uses a training algorithm for execution prior to runtime to determine which voxel sizes and shapes may be better to use for various different scenarios. The system 200 is trained by using a given or generated large set of task/environment pairs from one or more scenarios. The system 200 then chooses the discretization that is assessed to be most effective over the large set of training samples.
For example, for any given environment/task pair, there is an optimal discretization (or multiple discretizations that are equally optimal). The system 200 may test/train on T environment/task pairs, where T is a large number, and then record the optimal discretization for each. This may result in many different discretizations, each of which is optimal for only one or a small number of environment/task pairs. After testing all T environment/task pairs, the system selects the discretization that is assessed to be most effective over the whole set of samples. This approach would also involve trying every possible discretization on every possible environment/task pair, which would be intractable.
To overcome the above problem, the system 200 performs training like that explained above, except that the system 200 considers a finite number of possible discretizations, G. For each environment/task pair, the system records which of the G discretizations provided is assessed to be most effective. After processing all T environment/task pairs, the system 200 selects the discretization that is assessed to be most effective in the most environment/task pairs.
As an example, the robot 102 may be facing a box of space in front of it. The system may determine to discretize the space such that 6-bits are used to describe the position in each dimension (i.e., there are 26 possible positions on each axis). This results in 218 voxels, each of which has a unique 18-bit ID. At design time, the system 200 precomputes the swept volume for every edge's motion, and records it as the collection of voxel IDs in that swept volume. If the swept volume even partially intersects a voxel, it is included in the collection of voxel IDs in that swept volume. This is why using a finer resolution is better. Then, at runtime, the sensors 282 tell the processor 212a what is in front of the robot 102, but at a much finer granularity than the voxels representing the precomputed swept volume, so the system 200 processes the sensor input and communicates to the processor 212a which of the voxels are currently occupied. In some embodiments, the processor 212a processes the sensor input and in other embodiments, there are separate processors, such as those of the sensors 282, which processes the sensor input and communicates to the processor 212a. The processor 212a compares the voxels or boxes currently in the environment based on the information stored in environment memory 294 to the voxels or boxes listed for each edge of the planning graph edge information stored in planning graph edge information memory 284 to determine which edges are currently in collision. In such embodiments, it is important that the same discretization is used at design time as at runtime in order to enable the comparison.
When the system 200 selects to use non-uniform discretization, the system 200 may still use 18-bit voxel IDs and 218 voxels, but the voxels need not be cubes (or they could be cubes but not of all the same size). Once again, it is important in this embodiment that the system uses the same discretization at design time as at runtime. When using a non-uniform discretization, the system 200 selects what each voxel's size and shape is, using training such as that described above, such that the system picks the voxel sizes and shapes that will be used at design time and at runtime that are assessed to be most effective. The evaluation criteria used in such training may include criteria regarding the fraction of tasks that can be solved (e.g., the system 200 can find some collision-free path to accomplish the goal of the robot 102) and the quality of the paths that are found.
Each edge of the planning graph 400 for the robot 102 has some number of voxels (or boxes) corresponding to the volume in 3D space swept by the robot 102 when making the transition in the planning graph 400 from one state to another state represented by that edge. Such information may be stored as planning graph edge information in off-chip planning graph edge information memory 284, such that the system 200 can swap out different planning graphs to accommodate changing dimensions of the robot 102. Using information received from the sensors 282 representing the current environment at runtime, the system 200 determines which voxels (or boxes) are currently occupied, and the system 200 determines to not use any motion that would collide with any currently occupied voxel (or box).
In some embodiments, the system determines to not use any motion that would collide with any occupied voxel (or box) by comparing at runtime the information received from the sensors 282 and stored in on-chip environment memory 294 representing the current environment (and obstacles therein) with the planning graph edge information stored in planning graph edge information memory 284. Based on this comparison, the system 200 determines which planning graph edges represent transitions (and thus which corresponding motions) that would result in a collision between the robot 102 and one or more obstacles in the environment 100. As an example,
For example, in order for the robot 102 to achieve a goal of moving from a state represented by node 408a to a state represented by node 408d, the robot 102 would have to avoid transitions represented by edges 410b, 410c, 410d, 410e, 410f, 410g, 410h, 410i, 410j, 410k and 410l that have been determined to represent transitions that would result in a collision between the robot 102 and one or more obstacles in the environment 100. Thus,
At 504, for a robot 102 that will operate in an environment, the system 200 determines a plurality of planning graphs. Each planning graph respectively comprises a plurality of nodes connected by a plurality of edges, each node representing, implicitly or explicitly, variables that characterize a respective state of the robot 102. Each edge represents a transition between a respective pair of the states of the robot 102. The respective pairs of states are represented by respective ones of a pair of nodes that are coupled by a respective edge in the respective planning graph.
At 506, for at least two or more of the edges of each of the planning graphs, the system 200 generates a respective set of edge information that represents a volume swept by at least a portion of the robot 102 in transitioning between the states represented by the respective nodes that are coupled by the respective edge. In at least one embodiment, the system 200 generates the respective set of edge information before runtime.
At 508, the system 200 stores the plurality of planning graphs and the sets of edge information in at least one nontransitory processor-readable storage.
At 510, based on at least a portion of the robot 102 having a first set of physical dimensions at a first time, the system 200 provides the sets of edge information for a first one of the planning graphs to at least one processor.
At 512, based on at least a portion of the robot 102 having a second set of physical dimensions at a second time, the system 200 provides the sets of edge information for a second one of the planning graphs to the at least one processor. At least one dimension in the second set of physical dimensions is different than a corresponding one of the dimensions of the first set. The method 500 of operation in a robot control system 200 to facilitate motion planning by providing edge information of different planning graphs according to particular circumstances concludes at 514.
The robot 102 may have at least a first appendage that is selectively operable for movement with respect to the environment in which the robot 102 operates. The robot 102 may also have a first end effector attached to the first appendage. The first end effector is selectively operable for movement between at least a first and a second end effector arrangement (e.g., a grasped and un-grasped arrangement). The system 200 may determine that the first end effector attached to the first appendage is in a first end effector arrangement. For example, the end effector may be in an un-grasped or open position. This is an example of a first end effector arrangement. The end effector being in a grasped position is an example of a second end effector arrangement. The first set of physical dimensions represents a set of dimensions of the first end effector in the first end effector arrangement. In particular, the first set of physical dimensions are the physical dimensions of the robot when the end effector is in an un-grasped position. In such a case, providing the sets of edge information for a first one of the planning graphs to at least one processor is in response to the determination that the first end effector attached to the first appendage is in the first end effector arrangement (i.e., in an un-grasped or open position).
Also, the system 200 may determine that the first end effector attached to the first appendage is in a second end effector arrangement. For example, the end effector may be in grasped or closed position. This is an example of a second end effector arrangement. The end effector being in a un-grasped or open position is an example of a first end effector arrangement. In a case where the first appendage is in a second end effector arrangement (i.e., in a grasped or closed position), the second set of physical dimensions represents a set of dimensions of the first end effector in the second end effector arrangement. In particular, the first set of physical dimensions are the physical dimensions of the robot when the end effector is in a grasped position. Providing the sets of edge information for a second one of the planning graphs to at least one processor is then in response to the determination that the first end effector attached to the first appendage is in the second end effector arrangement (i.e., in an un-grasped or open position).
In some embodiments, the robot 102 has at least a first appendage that is selectively operable for movement with respect to the environment in which the robot 102 operates, and a first end effector is attached to the first appendage. The first end effector is selectively operable for movement between at least an un-grasped arrangement and a grasped arrangement. At least one of a size or shape of a volume occupied by the first end effector in the grasped arrangement is different from at least one of a size or shape of a volume occupied by the first end effector in the un-grasped arrangement. In such a case, the system 200 may determine that the first end effector attached to the first appendage is in the un-grasped arrangement. Providing the sets of edge information for a first one of the planning graphs to at least one processor is then in response to the determination that the first end effector attached to the first appendage is in the un-grasped arrangement.
At 604, for a first discretized representation of an environment in which at least a robot 102 will operate, the system 200 supplies at least a portion of the first discretized representation of the environment to at least one processor. The environment is occupied by one or more obstacles.
At 606, for each edge of a first planning graph, the system 200 provides a respective set of edge information to the at least one processor. The first planning graph is one of a plurality of planning graphs stored in off-chip memory relative the processor. Each planning graph of the plurality of planning graphs is associated with a different set of physical dimensions of the first robot. The respective set of edge information represents a volume swept by at least a portion of the robot 102 in transitioning between a pair of states of the robot 102. The pair of states of the robot 102 is represented by respective ones of a pair of nodes of the first planning graph. The nodes of the respective pair of nodes are coupled by a respective edge of the first planning graph and the respective edge represents a transition between the respective pair of states of the robot 102.
At 608, the system 200 identifies any of the edges of the first planning graph for which the corresponding transition would result in a collision between at least a portion of the robot 102 and at least a portion of at least one of the one or more obstacles in the environment. The method 600 of operation in a robot control system 200 to facilitate motion planning by identifying edges of a planning graph for which the corresponding transition would result in a collision concludes at 610.
At 704, for each of a plurality of iterations, the system 200 generates a respective discretization of a representation of an environment in which a robot 102 will operate. The system 200 performs this for at least a first scenario that includes a set of a plurality of pairs of tasks and environments and for each of the pairs of tasks and environments of at least the first scenario. At least two of the respective discretizations comprise a respective set of voxels. The voxels of the at least two of the respective discretizations are non-homogenous in at least one of size and shape within the respective discretization. A respective distribution of the non-homogeneousness of the voxels of the at least two of the respective discretizations is also different from one another.
At 706, the system 200 assesses an effectiveness of the generated respective discretizations of the representation of the environment in which the robot will operate. The system 200 performs this for at least a first scenario that includes a set of a plurality of pairs of tasks and environments and for each of the pairs of tasks and environments of at least the first scenario.
At 708, the system determines whether there are further pairs of tasks and environments to process. If the system 200 determines there are further pairs of tasks and environments to process for the current scenario, the method 700 returns to 704 to continue generating the respective discretizations for the sets of pairs of tasks and environments included in the scenario and then assesses their effectiveness at 706. If the system 200 determines there are no further pairs of tasks and environments to process for the current scenario, the method 700 continues to 710.
At 710, the system 200 stores to at least one nontransitory processor-readable media at least the generated respective discretizations of the representation of the environment in which the robot will operate that is assessed to be the most effective for at least the first scenario.
At 712, the system 200 determines whether there are further scenarios to process. If the system 200 determines there are further scenarios to process, the method 700 returns to 704 to continue generating the respective discretizations for the sets of pairs of tasks and environments included in the next scenario and assesses their effectiveness at 706. If the system 200 determines there are no further scenarios to process, the method 700 of operation in a robot control system 200 to facilitate motion planning by assessing an effectiveness of generated non-uniform discretizations of a representation of the environment in which the robot 102 will operate concludes at 714. In some embodiments, there may be one planning graph associated with multiple scenarios. In such embodiments, the system 200 may generate the respective discretizations for the sets of pairs of tasks and environments associated such a planning graph.
At 804, based at least in part on an identified scenario that classifies a pair of a task which a robot 102 will perform and an environment in which the robot will operate, the system 200 determines which of a number of discretizations to use to generate a number of swept volumes that represent respective regions through which at least a portion of the robot 102 will pass when transitioning between one state of the robot 102 and another state of the robot 102. This determination may include selecting between at least two discretizations based on the identified scenario. In such a case, the voxels of the at least two of the respective discretizations are non-homogenous in at least one of size and shape within the respective discretization. Also, a respective distribution of the non-homogeneousness of the voxels of the at least two of the respective discretizations is different from one another.
At 806, for each of a plurality of edges in a planning graph, the system 200 determines a respective swept volume of the edge using the determined discretization. The planning graph comprises a plurality of nodes and a plurality of edges. Each node represents a respective one of a plurality of states of the robot. Each of the edges couples a pair of the nodes and represents a transition by the robot 102 between the states represented by the respective nodes coupled by the edge.
At 808, the system 200 stores to at least one nontransitory processor-readable media at least one of the determined swept volume's respective discretizations of the representation of the environment in which the robot 102 will operate that is assessed to be the most effective for at least the identified scenario. The method 800 of operation in a robot control system 200 to facilitate motion planning by determining which of a number of discretizations to use to generate a number of swept volumes that represent respective regions through which the robot 102 will pass concludes at 810. In some embodiments, the planning graph may be associated with multiple scenarios.
At 904, based at least in part on an identified scenario that classifies a pair of a task which a robot 102 will perform and an environment in which the robot 102 operates, the system 200 determines which of a number of discretizations to use to generate a discretized representation of the environment, including obstacles, if any, in the environment.
At 906, the system 200 receives sensor information produced by one or more sensors 282 that sense the environment. The sensor information represents the environment, including obstacles, if any, in the environment.
At 908, the system 200 generates a discretized representation of the environment, including obstacles, if any, in the environment using the determined discretization. A plurality of voxels of the determined discretization are non-homogenous in at least one of size and shape within the respective discretization. A respective distribution of the non-homogeneousness of the voxels of the determined discretization is different from that of another one of the number of discretizations. The system generates the discretized representation of the environment using a distribution of voxel size and shape that matches a distribution of voxel size and shape used to generate a discretized representation of a swept volume. The method 900 of operation in a robot control system 200 to facilitate motion planning by generating a discretized representation of the environment in which the robot 102 will operate using a determined discretization concludes at 910.
At 1004, for a first planning graph of the plurality of planning graphs, and for each of a plurality of edges of the first planning graph, the system 200 performs collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot will operate.
At 1006, the system 200 updates the first planning graph based on the collision checking.
At 1008, the system 200 performs an optimization of the updated first planning graph to identify one or more optimized results, if any, from the updated first planning graph.
At 1010, the system 200 determines whether the one or more optimized results, if any, from the updated first planning graph meets a satisfaction condition.
At 1012, if the system 200 determined that the one or more optimized results, if any, from the updated first planning graph did not meet the satisfaction condition, the method 1000 continues to 1014. If the system 200 determined that the one or more optimized results, if any, from the updated first planning graph met the satisfaction condition, the method 1000 concludes at 1020. In response to determining that the one or more optimized results, if any, from the updated first planning graph meets the satisfaction condition, the system 200 may apply a transition identified by one of the one or more optimized results from the updated first planning graph to the robot 102 (e.g., move the robot).
At 1014, in response to the system 200 determining that the one or more optimized results, if any, from the updated first planning graph did not meet the satisfaction condition, the system 200, for each of a plurality of edges of the second planning graph, performs collision checking for collisions between a discretized representation of a swept volume associated with the edge and a discretized representation of any obstacles in an environment in which the robot 102 will operate.
At 1016, the system 200 updates the second planning graph based on the collision checking.
At 1018, the system 200 performs an optimization of the updated second planning graph to identify one or more optimized results, if any, from the updated second planning graph. The method 1000 of operation in a robot control system 200 to facilitate motion planning by determining whether optimized results of planning graphs meet satisfaction conditions concludes at 1020.
The system 200 may also determine whether the one or more optimized results, if any, from the updated second planning graph meets a satisfaction condition. In response to determining that the one or more optimized results, if any, from the updated second planning graph meets the satisfaction condition, the system 200 may apply a transition identified by one of the one or more optimized results from the updated second planning graph to the robot 102 (e.g., move the robot).
Furthermore, in response to determining that the one or more optimized results, if any, from the updated second planning graph does not meet the satisfaction condition, the system may repeat the collision checking, updating the planning graph, performing the optimization on subsequent planning graphs until the system identifies a planning graph that meets the satisfaction condition. The system 200 may then apply a transition identified by one of the one or more optimized results from that identified planning graph to the robot 102. In some embodiments, determining whether optimized results from the updated planning graphs meets the satisfaction condition in such a process may include determining whether the optimization produces any collision-free paths. Such collision-free paths are paths that avoid collision with all obstacles in the environment.
The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, schematics, and examples. Insofar as such block diagrams, schematics, and examples contain one or more functions and/or operations, it will be understood by those skilled in the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In one embodiment, the present subject matter may be implemented via Boolean circuits, Application Specific Integrated Circuits (ASICs) and/or FPGAs. However, those skilled in the art will recognize that the embodiments disclosed herein, in whole or in part, can be implemented in various different implementations in standard integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers) as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of ordinary skill in the art in light of this disclosure.
Those of skill in the art will recognize that many of the methods or algorithms set out herein may employ additional acts, may omit some acts, and/or may execute acts in a different order than specified.
In addition, those skilled in the art will appreciate that the mechanisms taught herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of signal bearing media include, but are not limited to, the following: recordable type media such as hard disk drives, CD ROMs and computer memory.
The various embodiments described above can be combined to provide further embodiments. All of the commonly assigned US patent application publications, US patent applications, foreign patents, and foreign patent applications referred to in this specification and/or listed in the Application Data Sheet, including but not limited U.S. provisional patent application Ser. No. 62/626,939 filed Feb. 6, 2018; International Patent Application No. PCT/US2017/036880, filed Jun. 9, 2017 entitled “MOTION PLANNING FOR AUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS,” and International Patent Application Publication No. WO 016/122840, filed Jan. 5, 2016, entitled “SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME” are incorporated herein by reference, in their entirety. These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4862373 | Meng | Aug 1989 | A |
| 4949277 | Trovato et al. | Aug 1990 | A |
| 5347459 | Greenspan et al. | Sep 1994 | A |
| 5544282 | Chen et al. | Aug 1996 | A |
| 6049756 | Libby | Apr 2000 | A |
| 6089742 | Warmerdam et al. | Jul 2000 | A |
| 6259988 | Galkowski et al. | Jul 2001 | B1 |
| 6493607 | Bourne et al. | Dec 2002 | B1 |
| 6539294 | Kageyama | Mar 2003 | B1 |
| 7865277 | Larson et al. | Jan 2011 | B1 |
| 8315738 | Chang | Nov 2012 | B2 |
| 8666548 | Lim | Mar 2014 | B2 |
| 8825207 | Kim et al. | Sep 2014 | B2 |
| 8855812 | Kapoor | Oct 2014 | B2 |
| 8972057 | Freeman et al. | Mar 2015 | B1 |
| 9102055 | Konolige et al. | Aug 2015 | B1 |
| 9280899 | Biess et al. | Mar 2016 | B2 |
| 9632502 | Levinson et al. | Apr 2017 | B1 |
| 9687982 | Jules | Jun 2017 | B1 |
| 9687983 | Prats | Jun 2017 | B1 |
| 9731724 | Yoon | Aug 2017 | B2 |
| 9981383 | Nagarajan | May 2018 | B1 |
| 10124488 | Lee et al. | Nov 2018 | B2 |
| 10131053 | Sampedro | Nov 2018 | B1 |
| 10430641 | Gao | Oct 2019 | B2 |
| 10705528 | Wierzynski | Jul 2020 | B2 |
| 10723024 | Konidaris | Jul 2020 | B2 |
| 10782694 | Zhang et al. | Sep 2020 | B2 |
| 20050071048 | Watanabe et al. | Mar 2005 | A1 |
| 20050216181 | Estkowski et al. | Sep 2005 | A1 |
| 20060235610 | Ariyur et al. | Oct 2006 | A1 |
| 20060247852 | Kortge et al. | Nov 2006 | A1 |
| 20070112700 | Den et al. | May 2007 | A1 |
| 20080125893 | Tilove et al. | May 2008 | A1 |
| 20080234864 | Sugiura et al. | Sep 2008 | A1 |
| 20090192710 | Eidehall et al. | Jul 2009 | A1 |
| 20090326711 | Chang et al. | Dec 2009 | A1 |
| 20090326876 | Miller | Dec 2009 | A1 |
| 20100235033 | Yamamoto et al. | Sep 2010 | A1 |
| 20110066282 | Bosscher et al. | Mar 2011 | A1 |
| 20110153080 | Shapiro et al. | Jun 2011 | A1 |
| 20120010772 | Pack et al. | Jan 2012 | A1 |
| 20120083964 | Montemerlo et al. | Apr 2012 | A1 |
| 20140025201 | Ryu et al. | Jan 2014 | A1 |
| 20140025203 | Inazumi | Jan 2014 | A1 |
| 20140121833 | Lee et al. | May 2014 | A1 |
| 20140147240 | Noda et al. | May 2014 | A1 |
| 20140249741 | Levien et al. | Sep 2014 | A1 |
| 20140251702 | Berger et al. | Sep 2014 | A1 |
| 20140309916 | Bushnell | Oct 2014 | A1 |
| 20150051783 | Tamir et al. | Feb 2015 | A1 |
| 20150261899 | Atohira et al. | Sep 2015 | A1 |
| 20150266182 | Strandberg | Sep 2015 | A1 |
| 20160001775 | Wilhelm et al. | Jan 2016 | A1 |
| 20160107313 | Hoffmann et al. | Apr 2016 | A1 |
| 20160154408 | Eade et al. | Jun 2016 | A1 |
| 20160299507 | Shah et al. | Oct 2016 | A1 |
| 20160357187 | Ansari | Dec 2016 | A1 |
| 20170004406 | Aghamohammadi | Jan 2017 | A1 |
| 20170120448 | Lee et al. | May 2017 | A1 |
| 20170123419 | Levinson et al. | May 2017 | A1 |
| 20170132334 | Levinson et al. | May 2017 | A1 |
| 20170146999 | Cherepinsky et al. | May 2017 | A1 |
| 20170157769 | Aghamohammadi et al. | Jun 2017 | A1 |
| 20170168485 | Berntorp et al. | Jun 2017 | A1 |
| 20170168488 | Wierzynski et al. | Jun 2017 | A1 |
| 20170193830 | Fragoso et al. | Jul 2017 | A1 |
| 20170305015 | Krasny et al. | Oct 2017 | A1 |
| 20180001472 | Konidaris | Jan 2018 | A1 |
| 20180029233 | Lager | Feb 2018 | A1 |
| 20180074505 | Lv et al. | Mar 2018 | A1 |
| 20180136662 | Kim | May 2018 | A1 |
| 20180164430 | Kuriyama et al. | Jun 2018 | A1 |
| 20180281786 | Oyaizu et al. | Oct 2018 | A1 |
| 20190163191 | Sorin | May 2019 | A1 |
| 20190232496 | Graichen | Aug 2019 | A1 |
| 20190293443 | Kelly et al. | Sep 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 2306153 | Apr 2011 | EP |
| 3250347 | Dec 2017 | EP |
| 11296229 | Oct 1999 | JP |
| 2002073130 | Mar 2002 | JP |
| 2008065755 | Mar 2008 | JP |
| 2010061293 | Mar 2010 | JP |
| 2011075382 | Apr 2011 | JP |
| 2011249711 | Dec 2011 | JP |
| 2012190405 | Oct 2012 | JP |
| 2012243029 | Dec 2012 | JP |
| 2014184498 | Oct 2014 | JP |
| 2015517142 | Jun 2015 | JP |
| 2018505788 | Mar 2018 | JP |
| 20110026776 | Mar 2011 | KR |
| 20170018564 | Feb 2017 | KR |
| 20170050166 | May 2017 | KR |
| 201318793 | May 2013 | TW |
| 9924914 | May 1999 | WO |
| 2016122840 | Aug 2016 | WO |
| 2017214581 | Dec 2017 | WO |
| 2019183141 | Sep 2019 | WO |
| Entry |
|---|
| International Search Report and Written Opinion for PCT/US2019/016700 dated May 20, 2019, 14 pages. |
| International Search Report and Written Opinion issued in PCT Application No. PCT/US2019/045270; dated Nov. 25, 2019, 11 pages. |
| Johnson, David E., et al., “Bound Coherence for Minimum Distance Computations”, Proceedings of the 1999 IEEE International Conference on Robotics and Automation, May 1999. |
| Stilman, Mike , et al., “Manipulation Planning Among Movable Obstacles”, Proceedings of the IEEE Int. Conf, on Robotics and Automation, Apr. 2007. |
| European Search Report issued in European Application No. 17811131.6, dated Jun. 16, 2020, 5 pages. |
| First Office Action issued in Chinese No. 201680006941.3 with English translation, dated Sep. 29, 2019, 16 pages. |
| International Search Report and Written Opinion for PCT/US2019/023031 dated Aug. 31, 2019, 18 pages. |
| International Search Report and Written Opinion for PCT/US2019/064511, dated Mar. 27, 2020, 10 pages. |
| International Search Report and Written Opinion issued in PCT Application No. PCT/2020/034551, dated Aug. 31, 2020, 18 pages. |
| International Search Report and Written Opinion, dated Jun. 23, 2020 for PCT/US2020/039193, 9 pages. |
| Non-Final Office Action Issued in U.S. Appl. No. 16/308,693, dated Jun. 1, 2020, 16 pages. |
| Non-Final Office Action Issued in U.S. Appl. No. 15/546,441, dated Sep. 17, 2019, 58 Pages. |
| Office Action Issued in Japanese Application No. 2018-564836, dated Dec. 3, 2019, 3 Pages. |
| Office Action Issued in Japanese Application No. 2018-564836, dated May 19, 2020, 3 Pages. |
| Office Action with English translation Issued in Japanese Application No. 2018-564836, dated Nov. 29, 2019, 7 pages. |
| Communication Pursuant to Article 94(3) EPC, issued in European Application No. 18209405.2, dated Nov. 23, 2020, 4 pages. |
| European Search Report dated Nov. 17, 2020 for EP Application No. 16743821.7, 4 pages. |
| Non-Final Office Action issued In U.S. Appl. No. 16/308,693, dated Dec. 11, 2020, 17 pages. |
| Extended European Search Report dated Apr. 10, 2018 for EP Application No. 16743821.7, in 9 pages. |
| Extended European Search Report issued in European Application No. 17811131.6, dated Apr. 24, 2019, 16 pages. |
| Extended European Search Report issued in European Application No. 18209405.2, dated Aug. 2, 2019, 9 pages. |
| First Office Action issued in Japanese Patent Application No. 2017-557268, dated Aug. 7, 2018, 15 pages. |
| International Search Report and Written Opinion for PCT/US2019/023031 dated Aug. 14, 2019 in 19 pages. |
| International Search Report and Written Opinion issued in PCT Application No. PCT/US2016/012204; dated Mar. 21, 2016, 10 pages. |
| International Search Report and Written Opinion issued in PCT Application No. PCT/US2017/036880; dated Oct. 10, 2017, 15 pages. |
| International Search Report and Written Opinion issued in PCT/US2019/012209, dated Apr. 25, 2019, 24 pages. |
| Invitation to Pay Additional Fees and, Where Applicable, Protest Fee issued in PCT/US2017/036880, dated Aug. 14, 2017, 2 pages (no copy available). |
| or.pdf (Or | Definition of Or by Merriam-Webster, Sep. 9, 2019, https://www.merriam-webster.com/dictionary/or, pp. 1-12; Year: 2019. |
| Second Office Action issued in Japanese Patent Application No. 2017-557268, dated Feb. 26, 2019, 5 pages. |
| Atay, Nuzhet , et al., “A Motion Planning Processor on Reconfigurable Hardware”, All Computer Science and Engineering Research, Computer Science and Engineering; Report No. WUCSE-2005-46; Sep. 23, 2005. |
| Hauck, Scott, et al., “Configuration Compression for the Xilinx XC6200 FPGA”, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 18, No. 8; Aug. 1999. |
| Kavraki, L.E., et al., “Probabilistic Roadmaps for Path Planning in High-Dimensional Configuration Spaces”, IEEE Transactions on Robotics and Automation, IEEE Inc.; vol. 12, No. 4, pp. 566-580; Aug. 1, 1996. |
| Murray, Sean , et al., “Robot Motion Planning on a Chip”, Robotics: Science and Systems 2016; Jun. 22, 2016; 9 pages. |
| Murray, Sean , et al., “The microarchitecture of a real-time robot motion planning accelerator”, 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (Micro), IEEE, Oct. 15, 2016, 12 pages. |
| Rodriguez, Carlos , et al., “Planning manipulation movements of a dual-arm system considering obstacle removing”, Robotics and Autonomous Systems, Elsevier Science Publishers, vol. 62, No. 12, pp. 1816-1826; Aug. 1, 2014. |
| Non-Final Office Action Issued in U.S. Appl. No. 16/308,693, dated May 14, 2021, 16 pages. |
| Non-Final Office Action Issued in U.S. Appl. No. 16/240,086, dated Feb. 11, 2021, 79 pages. |
| European Search Report issued in European Application No. 19771537.8, dated Mar. 29, 2021, 8 pages. |
| Murray, Sean , et al., “Robot Motion Planning on a Chip”, Robotics: Science and Systems, Jan. 1, 2016, 9 pages. |
| Rodriguez, Carlos , et al., “Planning manipulation movements of a dual-arm system considering obstacle removing”, Robotics and Autonomous Systems 62 (2014), ELSEVIER, Journal homepage: www.elsevier.com/locate/robot, pp. 1816-1826. |
| European Search Report dated Jul. 23, 2021, for European Application No. 19851097.6, 15 pages. |
| Taiwanese First Office Action—Application No. 106119452 dated Jun. 18, 2021, 25 pages. |
| Corrales, J.A., et al., Safe Human-robot interaction based on dynamic sphere-swept line bounding volumes, Robotic and Computer-Integrated Manufacturing 27 (2011) 177-185, 9 pages. |
| Pobil, Angel P, et al., “A New Representation for Collision Avoidance and Detection”, Proceedings of the 1992 IEEE, XP000300485, pp. 246-251. |
| Sato, Yuichi, et al., “Efficient Collision Detection using Fast Distance-Calculation Algorithms for Convex and Non-Convex Objects”, Proceeding of the 1996 IEEE, XP-000750294, 8 pages. |
| Turrillas, Alexander Martin, “Improvement of a Multi-Body Collision Computation Framework and Its Application to Robot (Self-) Collision Avoidance”, German Aerospace Center (DLR). Master's Thesis, Jun. 1, 2015, 34 pages. |
| Final Office Action dated Aug. 2, 2021 for U.S. Appl. No. 16/240,086 in 66 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20190240835 A1 | Aug 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62626939 | Feb 2018 | US |
