Patent Grant
11417088
The present application is based on PCT filing PCT/JP2019/015241, filed Apr. 8, 2019, which claims priority to JP 2018-114695, filed Jun. 15, 2018, the entire contents of each are incorporated herein by reference.
The present disclosure relates to an information processing device, an information processing method, a program, and an information processing system.
In recent years, a system, a service, and the like that use an unmanned aerial vehicle (UAV) have been actively developed. For example, Patent Document 1 described below discloses an information processing system that can identify information regarding an object appearing in image data obtained by a UAV performing aerial image capturing, and identify the position of the object. Furthermore, Patent Document 2 discloses an autonomous aerial mobile object that tracks a moving target. Even in a case where the autonomous aerial mobile object loses sight of the target, the autonomous aerial mobile object can rediscover the target and continue tracking by predicting a migration path of the target and going ahead of the target.
Note that Patent Document 3 discloses a service robot system for performing a service to a person appearing in a bird's-eye image captured from above from a bird's eye viewpoint, by analyzing the bird's-eye image, although the service robot system is not a system or a service that uses a UAV.
However, the technologies disclosed in Patent Documents 1 to 3, or the like cannot identify a direction that a target appearing in captured image data faces.
In view of the foregoing, the present disclosure has been devised in view of the above-described situations, and the present disclosure provides an information processing device, an information processing method, a program, and an information processing system that are novel and improved, and can more appropriately identify a direction that a target appearing in captured image data faces.
According to the present disclosure, there is provided an information processing device including an acquisition unit configured to acquire captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing, and a direction identification unit configured to identify a direction that a target appearing in the captured image data faces, by analyzing the captured image data.
Furthermore, according to the present disclosure, there is provided an information processing method executed by a computer, including acquiring captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing, and identifying a direction that a target appearing in the captured image data faces, by analyzing the captured image data.
Furthermore, according to the present disclosure, there is provided a program for causing a computer to execute acquiring captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing, and identifying a direction that a target appearing in the captured image data faces, by analyzing the captured image data.
Furthermore, according to the present disclosure, there is provided an information processing system including an information processing device and a robot, in which the information processing device includes an acquisition unit configured to acquire captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing, a direction identification unit configured to identify a direction that a target appearing in the captured image data faces, by analyzing the captured image data, a position identification unit configured to identify a position of a target appearing in the captured image data, by analyzing the captured image data, and a robot control unit configured to control a movement of the robot on the basis of the position and the direction.
As described above, according to the present disclosure, it becomes possible to more appropriately identify a direction that a target appearing in captured image data faces.
Note that the above-described effect is not always limitative, and together with the above-described effect or in place of the above-described effect, any of the effects described in this specification, or other effects recognized from this specification may be caused.
Hereinafter, a preferred embodiment of the present disclosure will be described in detail with reference to the attached drawings. Note that, in this specification and the drawings, the redundant description will be omitted by allocating the same reference numerals to the components having substantially the same functional configuration.
Note that the description will be given in the following order.
1. System Configuration Example
2. Functional Configuration Example of UAV 100
3. Functional Configuration Example of Server 200
4. Functional Configuration Example of Mobile Robot 300
5. Processing Flow Example
6. Variation Regarding Identification Method of Direction That Person Faces
7. Modified Example
8. Hardware Configuration Example
9. Conclusion
First of all, a system configuration example of an information processing system according to an embodiment of the present disclosure will be described with reference to
As illustrated in
The information processing system according to the present embodiment is assumed to be a system that provides a predetermined service to a user in a predetermined facility such as an entertainment park, a sport facility, or a park, for example, but the information processing system is not limited to this. In other words, the information processing system according to the present embodiment can be operated in an arbitrary facility or location.
(1.1. UAV 100)
The UAV 100 is an unmanned air vehicle including an imaging unit that can acquire captured image data, and also serves as a device functioning as an information processing device. The UAV 100 performs flight on the basis of control information provided from the server 200. Furthermore, the UAV 100 can also perform more detailed flight control by analyzing captured image data acquired by the imaging unit mounted on the own device. For example, in a case where an obstacle appears in a travelling direction when the UAV 100 is performing flight on the basis of control information provided from the server 200, the UAV 100 may perform flight in such a manner as to avoid collision by recognizing the obstacle by analyzing the captured image data acquired by the imaging unit. In other words, the UAV 100 may change control information provided from the server 200, and generate new control information. Note that “captured image data” according to the present embodiment is concept including still image data, moving image data, or a measurement value not visualized as an image. Furthermore, “captured image data” may be acquired by an RGB camera, or may be acquired by a camera that can detect light in a wavelength band other than RGB, such as an infrared camera.
The UAV 100 captures an aerial image of a target positioned on the land surface, using the imaging unit during flight. Here, the “target” is mainly assumed to be a person (human) (hereinafter, the “target” will be sometimes referred to as a “person” or a “user”). For example, the “target” can be a user u1 or u2 in
Then, the UAV 100 can search for a specific person from above on the basis of captured image data of a person that has been input in advance. For example, at the entrance of an entertainment park in which the information processing system according to the present embodiment is operated, captured image data, attribute information, and the like of a visitor (for example, name, age, gender, address, or the like) are assumed to be acquired. By analyzing captured image data obtained by the own device by performing image capturing, the UAV 100 extracts a feature amount of a person appearing in the captured image data, and by comparing the extracted feature amount with a feature amount of a person appearing in captured image data acquired in advance at the entrance, the UAV 100 identifies a person appearing in the captured image data obtained by the own device by performing image capturing. With this configuration, for example, in a case where this service is provided only to an applicant, the UAV 100 can determine whether or not a person appearing in the captured image data obtained by the own device by performing image capturing is a person who desires the service to be provided (in this case, for example, whether or not a visitor desires the service to be provided is acquired at the entrance for each visitor). Then, in a case where a person appearing in the captured image data obtained by the UAV 100 by performing image capturing is a person who desires the service to be provided, the UAV 100 can track the person from above.
Furthermore, the UAV 100 can identify a direction that a target appearing in captured image data faces, by analyzing captured image data acquired by the imaging unit included in the own device. Here, the “direction that the target faces” is assumed to be a “direction that a person faces” as described above (hereinafter, the “direction that the target faces” will be sometimes referred to as a “direction that a person faces”). By identifying a direction that a “face” of a person faces, by the analysis of captured image data, the UAV 100 can identify the direction that the person faces (in this case, the direction that the face of the person faces and the direction that the person faces are assumed to be substantially the same, but the directions are not limited to this). More specifically, the UAV 100 can identify the direction that the person faces, on the basis of a feature of a face or a head of the person appearing in captured image data. For example, the UAV 100 can identify the direction that the person faces, on the basis of a feature of at least any one of both eyes, one eye, an eyebrow, a nose, a mouth, teeth, an ear, or hair. Here, a region of the face or the head will be sometimes referred to as a “feature region” (in other words, the feature region refers to the region of the face or the head, or a predetermined portion positioned around these regions). The details of a method of identifying the direction that the person faces (the direction that the face of the person faces), on the basis of a feature of the face of the person (or a feature region) will be described later.
Note that the UAV 100 may identify the direction that the person faces, on the basis of a feature of a region other than a face or a head of the person. The details will be described later. Furthermore, in a case where a “target” is a tangible object other than a person, the definition of the “direction that the target faces” or an identification method can vary depending on the tangible object. For example, in a case where a “target” is a living object such as a dog, a “direction that the target faces” can be identified on the basis of a direction that a face of the dog faces that is identified on the basis of a feature of the face of the dog. Furthermore, in a case where a “target” is a vehicle, a “direction that the target faces” can be identified on the basis of a feature or a travelling direction of the vehicle, or the like.
Here, in a case where image capturing is performed against light, in a case where an image of a region with a shadow is captured, in a case where image capturing is performed at a location with many obstacles, or the like, in some cases, a feature region of a face does not clearly appear or is hidden by an obstacle in captured image data. In this case, there is a possibility that the identification accuracy of a direction that a person faces declines. Nevertheless, the UAV 100 can control flight of the own device in such a manner as to eliminate a cause for disturbing the identification of a direction that a person faces (hereinafter, the flight will be sometimes referred to as “obstacle avoidance flight”). For example, in a case where a feature region of a face does not clearly appear in captured image data due to backlight image capturing, by moving in a front-back direction, a left-right direction, or an up-down direction, the UAV 100 can search for a position at which image capturing can be performed in direct light. With this configuration, the UAV 100 can acquire captured image data in which a feature region of a face clearly appears, and enhance the identification accuracy of a direction that a person faces. The details will be described later.
Furthermore, the UAV 100 can also identify a position of a target appearing in captured image data, in addition to a direction that the target faces. More specifically, the UAV 100 is equipped with a Global Navigation Satellite System (GNSS) receiver, a barometer, or the like (sensors used for identifying the position of the UAV 100 like these devices will be sometimes collectively referred to as a “positioning sensor”), and can identify a position coordinate of the own device in a three-dimensional space on the basis of information acquired by these devices. Then, the UAV 100 can identify a position coordinate of the person by recognizing a relative positional relationship between the own device and the person on the basis of the position of the person in captured image data acquired by the imaging unit.
The UAV 100 transmits information regarding the direction that the person faces (hereinafter, will also be referred to as “direction information”), and information regarding the position of the person (position coordinate. Hereinafter, will also be referred to as “position information”) that have been identified by the above-described processing, to the server 200.
Note that the UAV 100 is only required to be a device that has a basic flight function, and the type of the UAV 100 is not specifically limited. For example, the UAV 100 may be an airplane-type air vehicle (vertical take-off and landing (VTOL), or the like), a helicopter-type air vehicle, or a multicopter-type air vehicle. The VTOL has characteristics of both the airplane type and the multicopter-type.
(1.2. Server 200)
The server 200 is an information processing device that controls the UAV 100 and the mobile robot 300. For example, the server 200 can control the flight of one UAV 100 or each of two or more UAVs 100. More specifically, the server 200 can control the flight of each of the UAVs 100 by setting a round route or a flight area for each of the UAVs 100.
Furthermore, as described above, in a case where the UAV 100 transmits direction information and position information of the person to the server 200, the server 200 controls the mobile robot 300 on the basis of these pieces of information. For example, the server 200 moves the mobile robot 300 closer to the person from a certain position existing on an extended line of a direction that the person faces, which serves as a starting point (in other words, the server 200 moves the mobile robot 300 closer to the person from a front direction of the person). With this configuration, the person can naturally recognize the mobile robot 300. Then, the server 200 can provide a predetermined service to the user by stopping the mobile robot 300 at a position distant from the person by a predetermined distance, and causing the mobile robot 300 to perform a predetermined operation. The details of a control method of the mobile robot 300 that is used by the server 200 will be described later.
Note that the server 200 according to the present embodiment is assumed to be a cloud server positioned on a cloud network, but the server 200 is not limited to this. For example, the server 200 may be a server not connected to a network on the outside of the information processing system according to the present embodiment. Furthermore, the type of the server 200 is not specifically limited. For example, the server 200 may be an arbitrary device including a general-purpose computer, a personal computer (PC), or a tablet PC. Furthermore, the server 200 may connect with a proportional device in a wired or wireless manner, and the server 200 that has received a request from the proportional device may provide control information to the UAV 100.
(1.3. Mobile Robot 300)
The mobile robot 300 is a mobile object moving on the ground, and is a device that provides a predetermined service to a person. For example, the mobile robot 300 moves to a provision location of a service on the basis of migration path information provided from the server 200. More specifically, as described above, the mobile robot 300 moves closer to the person from a direction that the person faces. Furthermore, in a case where the mobile robot 300 includes various sensors such as an imaging unit that can acquire captured image data, the mobile robot 300 may perform more fine movement control by analyzing various types of sensor information. For example, in a case where an obstacle appears in the travelling direction when the mobile robot 300 is moving on the basis of the migration path information provided from the server 200, the mobile robot 300 may recognize the obstacle by analyzing various types of sensor information, and move in such a manner as to avoid collision. In other words, the mobile robot 300 may change migration path information provided from the server 200, and generate new migration path information.
Then, in a case where the mobile robot 300 reaches a provision location of a service (for example, a position distant from the person by a predetermined distance as described above), the mobile robot 300 provides a predetermined service to the person. Here, the content of the service provided by the mobile robot 300 is not specifically limited. For example, the mobile robot 300 can provide various services such as a picture taking service for capturing an image of a person, or a freight transportation service for transporting a freight in place of a person.
Note that the mobile robot 300 needs not always move on the ground. For example, the mobile robot 300 may be an air vehicle (for example, an air vehicle flying low (about several meters from the ground), or the like) similarly to the UAV 100.
Heretofore, the configuration example of the information processing system according to the present embodiment has been described. Note that the above-described configuration described with reference to
Heretofore, the system configuration example of the information processing system according to the present embodiment has been described. Subsequently, a functional configuration example of the UAV 100 will be described with reference to
As illustrated in
(Control Unit 110)
The control unit 110 is a functional configuration that controls the overall processing performed by the UAV 100. For example, on the basis of control information provided from the server 200, the control unit 110 can control the startup and stop of each functional configuration. Furthermore, the control unit 110 controls processing of transmitting position information of a person that has been output by a position identification unit 111b, and direction information of a person that has been output by a direction identification unit 111c, to the server 200 via the communication unit 120. Note that processing controlled by the control unit 110 is not limited to these. For example, the control unit 110 may control processing generally performed by various servers, a general-purpose computer, a PC, a tablet PC, or the like (for example, processing performed by an operating system (OS), or the like).
As illustrated in
(Analysis Unit 111)
The analysis unit 111 is a functional configuration that analyzes captured image data acquired by the imaging unit 130, and various types of sensor information acquired by the sensor unit 140. For example, the analysis unit 111 identifies a tangible object appearing in captured image data, by analyzing the captured image data. More specifically, the analysis unit 111 identifies a tangible object appearing in captured image data, by extracting a feature amount of the tangible object appearing in the captured image data, and comparing the feature amount with feature amounts of various tangible objects that have been acquired in advance. The analysis unit 111 can thereby recognize whether or not a tangible object appearing in captured image data is a person. Note that content of analysis performed by the analysis unit 111 or an analysis method is not limited to this.
As illustrated in
(Person Identification Unit 111a)
The person identification unit 111a is a functional configuration that identifies a person appearing in captured image data, by analyzing the captured image data acquired by the imaging unit 130 (in other words, the person identification unit 111a functions as a target identification that identifies a target appearing in captured image data). For example, captured image data of a visitor that has been acquired at the entrance of an entertainment park in which the information processing system according to the present embodiment is operated is once stored into the server 200, and the UAV 100 receives a feature amount of the person appearing in the captured image data (alternatively, may be captured image data itself), from the server 200 via the communication unit 120. Then, the person identification unit 111a extracts a feature amount of a person appearing in captured image data, by analyzing the captured image data acquired by the imaging unit 130, and comparing the feature amount with the feature amount of the person appearing in captured image data that has been provided from the server 200. The person identification unit 111a thereby identifies a person appearing in the captured image data acquired by the imaging unit 130.
Then, for example, in a case where this service is provided only to an applicant, the person identification unit 111a determines whether or not the person appearing in the captured image data acquired by the imaging unit 130 is a person who desires the service to be provided. Note that an identification method of a person that is used by the person identification unit 111a is not limited to the above-described method. For example, in a case where a person wears a tag device that can communicate with the UAV 100, the person identification unit 111a may identify each person by receiving identification information (for example, an ID or the like) of the person wearing the tag device, from the tag device via the communication unit 120.
(Position Identification Unit 111b)
The position identification unit 111b is a functional configuration that identifies the position of a person appearing in captured image data, by analyzing the captured image data acquired by the imaging unit 130. More specifically, the sensor unit 140 includes a positioning sensor such as a GNSS receiver or a barometer, and the position identification unit 111b identifies a position coordinate of the own device in a three-dimensional space by acquiring these pieces of sensor information from the sensor unit 140 and analyzing the sensor information.
Then, the position identification unit 111b identifies a position coordinate of the person by analyzing captured image data acquired by the imaging unit 130, and recognizing a relative positional relationship between the own device and the person on the basis of the position of the person in the captured image data. Here, the “position of the person” in the captured image data is assumed to be a “position of a center point of the top of the head of the person”. Furthermore, the “position coordinate of the person” identified by the position identification unit 111b is assumed to be a coordinate represented by a two-dimensional orthogonal coordinate system (for example, an x coordinate and a y coordinate) when a predetermined point (hereinafter, will be referred to as a “reference point”) is used as a reference.
Here, the “position of the person” and the “position coordinate of the person” will be specifically described with reference to
Then, the position identification unit 111b identifies a “position coordinate of the person” by representing the position of the center point of the top of the head in captured image data, by a two-dimensional orthogonal coordinate system in a case where a predetermined reference point is set as an origin. Note that the position of the reference point is not specifically limited. For example, the reference point can be set to a predetermined position in the entertainment park. Furthermore, the definition of the “position of the person” is not limited to the above-described definition. For example, the “position of the person” may be a center point of a circle encompassing a person appearing in captured image data, or may be a specific portion or the like of a person appearing in captured image data (for example, a tip of a nose, or the like). Furthermore, the definition of the “position coordinate of the person” is not limited to the above-described definition. For example, the “position coordinate of the person” may be a coordinate represented by a two-dimensional oblique coordinate system when a predetermined reference point is set as an origin.
(Direction Identification Unit 111c)
The direction identification unit 111c is a functional configuration that identifies a direction that a person appearing in captured image data faces, by analyzing the captured image data acquired by the imaging unit 130. More specifically, the direction identification unit 111c identifies a direction that a person faces, on the basis of a feature of at least any one of both eyes, one eye, an eyebrow, a nose, a mouth, teeth, an ear, or hair. For example, as illustrated in
Then, as illustrated in
(Flight Control Unit 112)
The flight control unit 112 is a functional configuration that controls the flight of the UAV 100. More specifically, in a case where the analysis unit 111 analyzes captured image data acquired by the imaging unit 130, and various types of sensor information acquired by the sensor unit 140, the flight control unit 112 generates control information for controlling the flight mechanism 160, on the basis of the analysis result, and provides the information to the flight mechanism 160. For example, in a case where an obstacle is detected in the travelling direction as a result of the analysis, the flight control unit 112 can control the flight in such a manner as to avoid the collision with the obstacle, on the basis of the shape and the operation of the detected obstacle.
Here, in a case where two or more UAVs 100 exist, the UAVs 100 each perform the identification of a person and the like (and the identification of the position of the person, the identification of a direction that the person faces, tracking of the person, and the like) in cooperation with each other. For example, as illustrated in
Furthermore, the flight control unit 112 tracks at least one person by controlling the flight of the own device on the basis of a change in relative positional relationship between the own device and a person to be tracked. For example, the flight control unit 112 calculates, for each unit time, a distance between the position of a person existing within the search range of the own device and the center point of the search range. More specifically, in
Note that, in the information processing system according to the present embodiment, the “unit time” is an interval of time at which the inside of the search range is searched, and the length of the unit time is not specifically limited. Furthermore, the unit time may be dynamically changed. For example, the unit time may be changed to a longer time as the battery of the UAV 100 is running low. Furthermore, for example, in a case where users included in the search range move in different directions, the flight control unit 112 may implement the tracking of each person more efficiently by changing a flight altitude of the UAV 100 to a higher flight altitude. Furthermore, in a case where a priority (alternatively, predetermined information corresponding to a priority) is allocated to each person, the flight control unit 112 may control the flight of the UAV 100 in such a manner as to preferentially track a person with a high priority.
Furthermore, in
Furthermore, the flight control unit 112 can control the flight in such a manner as to eliminate a cause for disturbing the identification of a direction that a person faces (in other words, can implement obstacle avoidance flight). For example, in a case where image capturing is performed against light, in a case where an image of a region with a shadow is captured, in a case where image capturing is performed at a location with many obstacles, or the like, in some cases, a feature region of a face does not clearly appear or is hidden by an obstacle in captured image data. For example, as illustrated in
(Communication Unit 120)
The communication unit 120 is a functional configuration that performs communication with the server 200. For example, the communication unit 120 receives, from the server 200, control information used for the flight control of the own device. Furthermore, the communication unit 120 transmits position information and direction information of a person that have been output by the analysis unit 111, to the server 200. Note that content of information communicated by the communication unit 120 is not specifically limited. For example, the communication unit 120 may receive, from the server 200, control information or the like for instructing predetermined processing other than flight control (for example, information cooperation processing with a different UAV 100, or the like). Furthermore, the communication unit 120 may transmit position information of the own device (for example, a position coordinate of the own device in the three-dimensional space, or the like) to the server 200.
Furthermore, the communication unit 120 may perform communication with a different UAV 100 or the mobile robot 300. For example, in a case where the UAV 100 includes all or a part of the functions of the server 200, the communication unit 120 may control the mobile robot 300 by communicating with the mobile robot 300. Furthermore, by communicating with a different UAV 100, the communication unit 120 may transmit an analysis result obtained by the analysis unit 111 of the own device, or the like to the different UAV 100. In contrast, the communication unit 120 may receive an analysis result obtained by the different UAV 100, or the like from the different UAV 100. Furthermore, a communication method used by the communication unit 120 for communication is not specifically limited. More specifically, the communication unit 120 is assumed to perform predetermined wireless communication, but the communication unit 120 may perform predetermined wired communication by being connected with a communication cable.
(Imaging Unit 130)
The imaging unit 130 is a functional configuration that functions as an acquisition unit that acquires captured image data by performing aerial image capturing. More specifically, the imaging unit 130 acquires captured image data by capturing an aerial image of a person (target) or a surrounding environment thereof. Furthermore, the imaging unit 130 includes a lens system including an image capturing lens, a diaphragm, a zoom lens, a focus lens, and the like, a drive system that causes the lens system to perform a focus operation or a zoom operation, a solid-state image sensor array that generates an image capturing signal by photoelectrically converting image capturing light obtained by the lens system, and the like. The solid-state image sensor array may be realized by, for example, a charge coupled device (CCD) sensor array or a complementary metal oxide semiconductor (CMOS) sensor array.
Then, the image capturing lens of the imaging unit 130 is installed in a state of being orientated toward a land surface direction in such a manner that aerial image capturing can be performed. Note that an installation configuration of each member (the image capturing lens or the like) of the imaging unit 130 is not limited to this, and each member of the imaging unit 130 may be installed in a configuration in which an image capturing direction can be changed.
(Sensor Unit 140)
The sensor unit 140 is a functional configuration that acquires various types of sensor information regarding the surrounding environment of the UAV 100. For example, the sensor unit 140 includes a positioning sensor (GNSS receiver, a barometer, or the like), a gyro sensor, an acceleration sensor, a geomagnetic sensor, a proximity sensor, a microphone, a temperature sensor (for example, thermosensor or the like), a humidity sensor, an illuminance sensor, a pressure sensor, an infrared sensor, or the like. Note that the types of sensors included in the sensor unit 140 are not limited to these. Furthermore, the sensor unit 140 may include a plurality of sensors for each type of sensor. Furthermore, each of these types of sensors may be included in an external device (alternatively, external system) other than the UAV 100, and various types of sensor information may be acquired from the external device.
(Storage Unit 150)
The storage unit 150 is a functional configuration that stores various types of information. For example, the storage unit 150 stores information used for analysis processing performed by the analysis unit 111 (for example, information such as captured image data, various types of sensor information, a feature amount of a person, a reference point, or a reference line), information output by the analysis unit 111 (for example, position information and direction information of a person, or the like), control information provided from the server 200, or the like. Furthermore, the storage unit 150 also stores programs, parameters, and the like that are to be used in processing performed by each functional configuration. Note that information stored by the storage unit 150 is not limited to these types of information.
(Flight Mechanism 160)
The flight mechanism 160 is a mechanism causing the UAV 100 to fly, and includes an actuator, a motor, a propeller (not illustrated), and the like, for example. The flight mechanism 160 is driven in accordance with control information provided from the flight control unit 112, and causes the UAV 100 to fly.
Heretofore, the functional configuration example of the UAV 100 has been described. Note that the above-described functional configuration described using
Heretofore, the functional configuration example of the UAV 100 has been described. Subsequently, a functional configuration example of the server 200 will be described with reference to
As illustrated in
(Control Unit 210)
The control unit 210 is a functional configuration that controls the overall processing performed by the server 200. For example, the control unit 210 can control the startup and stop of each functional configuration of the own device. Furthermore, the control unit 210 acquires, via the communication unit 220 or an input unit (not illustrated), captured image data of a visitor that has been acquired at the entrance of an entertainment park in which the information processing system according to the present embodiment is operated (and attribute information or the like of a visitor (for example, name, age, gender, address, or the like), and manages these types of information by registering the information into the storage unit 230. Furthermore, the control unit 210 also acquires, via the communication unit 220, position information and direction information of a person that have been output by the UAV 100, and manages these types of information by registering the information into the storage unit 230. Note that processing controlled by the control unit 210 is not limited to these. For example, the control unit 210 may control processing generally performed by various servers, a general-purpose computer, a PC, a tablet PC, or the like (for example, processing performed by an OS, or the like).
As illustrated in
(UAV Control Unit 211)
The UAV control unit 211 is a functional configuration that controls the overall processing performed by one UAV 100 or two or more UAVs 100. For example, the UAV control unit 211 can control the flight of each of the UAVs 100 by setting a round route or a flight area for each of the UAVs 100. More specifically, the UAV control unit 211 controls the flight of each of the UAVs 100 by setting a round route or a flight area for each of the UAVs 100, and transmitting control information generated on the basis of the set content, to each of the UAV control units 211 via the communication unit 220. Furthermore, the UAV control unit 211 may change the setting of a round route, a flight area, or the like of each of the UAVs 100 as needed. For example, the UAV control unit 211 may change the setting of a flight area of each of the UAVs 100 in such a manner as to increase the number of UAVs 100 flying in an area crowded with a larger number of people. Note that processing controlled by the UAV control unit 211 is not limited to the above-described processing. For example, the UAV control unit 211 may control information cooperation processing performed between the UAVs 100, or the like.
(Robot Control Unit 212)
The robot control unit 212 is a functional configuration that controls the overall processing performed by the mobile robot 300. For example, the robot control unit 212 controls the movement of the mobile robot 300 or the like on the basis of direction information and position information of a person that have been provided from each of the UAVs 100.
More specifically, on the basis of direction information and position information of a person, and position information (and direction information) of the mobile robot 300, the robot control unit 212 moves the mobile robot 300 closer to the person from a direction that the person faces (alternatively, a different direction determined on the basis of the direction that the person faces (for example, a traverse direction of the person, or the like)), and outputs a migration path (hereinafter, will be sometimes referred to as an “approach curve”) of the mobile robot 300 that can stop the mobile robot 300 at a position distant from the person by a predetermined distance d.
Here, an output method of an approach curve will be described more specifically with reference to
The robot control unit 212 can cause the mobile robot 300 to provide a predetermined service to a person by stopping the mobile robot 300 at a position distant from the person by the predetermined distance d, and causing the mobile robot 300 to perform a predetermined operation. Note that the “predetermined distance d” varies depending on the content of a service to be provided to a person. For example, in a case where a service provided to a person is a “picture taking service”, the “predetermined distance” can be a distance of about several meters, and in a case where a service provided to a person is a “freight transportation service”, the “predetermined distance” can be a distance of several tens of centimeters in consideration of the delivery of freight.
The robot control unit 212 transmits migration path information including information regarding the output approach curve, to the mobile robot 300 via the communication unit 220. Note that the “migration path information” may include information regarding one approach curve, or may include information regarding a plurality of approach curves connected at a certain point. Here, the “information regarding an approach curve” may be a formula representing an approach curve, or may be identification information (ID) indicating an approach curve that is prepared in advance, or the like. Furthermore, as described above, in a case where the robot control unit 212 outputs, in place of an approach curve, a plurality of coordinates to which the mobile robot 300 is to be moved, and an order of movement to the coordinates, the “migration path information” includes these pieces of information.
Furthermore, for example, in a case where a person is riding on a predetermined vehicle (hereinafter, will be referred to as “a vehicle robot”), the robot control unit 212 may also control the vehicle robot in addition to the mobile robot 300. More specifically, the robot control unit 212 may move the vehicle robot on which the person rides, or change a direction that the vehicle robot faces, in such a manner that the mobile robot 300 can easily move closer from a direction that a person faces. The robot control unit 212 can thereby more surely move the mobile robot 300 closer to the person from an appropriate direction.
(Communication Unit 220)
The communication unit 220 is a functional configuration that performs communication with the UAV 100 and the mobile robot 300. The communication with the UAV 100 will be described. The communication unit 220 transmits control information to be used for flight control of the UAV 100, to the UAV 100. Furthermore, the communication unit 220 may transmit, to the UAV 100, control information or the like for instructing predetermined processing other than flight control (for example, information cooperation processing with a different UAV 100, or the like). Furthermore, the communication unit 220 receives position information and direction information of a person from the UAV 100. Furthermore, the communication unit 220 may receive, from the UAV 100, position information of the UAV 100 (for example, a position coordinate or the like in the three-dimensional space).
Furthermore, the communication with the mobile robot 300 will be described. The communication unit 220 transmits migration path information to be used for movement control of the mobile robot 300, to the mobile robot 300. Furthermore, the communication unit 220 may transmit, to the mobile robot 300, control information or the like for instructing predetermined processing other than movement control (for example, processing regarding a service provided by the mobile robot 300, or the like). Note that content of information communicated by the communication unit 220 is not limited to the above-described content.
(Storage Unit 230)
The storage unit 230 is a functional configuration that stores various types of information. For example, the storage unit 230 stores information regarding the UAV 100 and the mobile robot 300, and the like (for example, identification information, position information, information regarding a status, or the like). Furthermore, the storage unit 230 also stores programs, parameters, and the like that are to be used in processing performed by each functional configuration. Note that information stored by the storage unit 230 is not limited to these types of information.
As illustrated in
(Person DB 231)
The person DB 231 is a functional configuration that stores information regarding a person. For example, the person DB 231 stores information regarding a visitor to an entertainment park in which the information processing system according to the present embodiment is operated. More specifically, the person DB 231 stores captured image data of a visitor (and a feature amount of a visitor appearing in the captured image data), identification information, attribute information (for example, name, age, gender, or address, or the like), action history information, or the like (for example, an entry time, history information regarding a migration path, or history information regarding content of a service provided by the mobile robot 300, or the like). Note that information stored by the person DB 231 is not limited to these types of information.
(Position/Direction DB 232)
The position/direction DB 232 is a functional configuration that stores position information and direction information of a person. More specifically, the position/direction DB 232 stores position information and direction information of a person that have been provided from the UAV 100. Then, the position/direction DB 232 updates information stored in itself every time position information and direction information of a person are updated by the UAV 100. As described above, position information of a person is assumed to be information represented by an x coordinate and a y coordinate in a case where a predetermined reference point is set as an origin, and direction information of a person is assumed to be information represented by an angle θ formed by a reference line and a parallel line of a face, but formats of these pieces of information can be appropriately changed.
Heretofore, the functional configuration example of the server 200 has been described. Note that the above-described functional configuration described using
Heretofore, the functional configuration example of the server 200 according to the present embodiment has been described. Subsequently, a functional configuration example of the mobile robot 300 according to the present embodiment will be described with reference to
As illustrated in
(Control Unit 310)
The control unit 310 is a functional configuration that comprehensively controls the overall processing performed by the mobile robot 300. For example, the control unit 310 can control the startup and stop of each functional configuration of the own device. Note that processing controlled by the control unit 310 is not limited to this. For example, the control unit 310 may control processing generally performed by various servers, a general-purpose computer, a PC, a tablet PC, or the like (for example, processing performed by an OS, or the like).
As illustrated in
(Analysis Unit 311)
The analysis unit 311 is a functional configuration that analyzes various types of sensor information acquired by the sensor unit 330. For example, the analysis unit 311 recognizes a surrounding environment of the mobile robot 300 by analyzing various types of sensor information acquired by the sensor unit 330 (captured image data in a case where the sensor unit 330 includes a function similar to the imaging unit 130 of the UAV 100). More specifically, the analysis unit 311 can recognize an obstacle existing in the travelling direction, a person to be provided with a service, or the like, by analyzing various types of sensor information. Furthermore, by analyzing various types of sensor information, the analysis unit 311 may perform the identification of a person positioned around the mobile robot 300, the identification of the position of a person, the identification of a direction that a person faces, or the like similarly to the analysis unit 111 of the UAV 100. Note that content of analysis processing performed by the analysis unit 311 is not limited to these.
(Movement Control Unit 312)
The movement control unit 312 is a functional configuration that controls the movement of the mobile robot 300. For example, in a case where the server 200 transmits migration path information to the mobile robot 300, the movement control unit 312 controls the movement of the own device on the basis of information regarding an approach curve that is included in the migration path information. More specifically, the movement control unit 312 generates control information for controlling the movement mechanism 350, on the basis of the information regarding an approach curve, and provides the information to the movement mechanism 350. The movement control unit 312 can thereby control the movement of the own device in accordance with an instruction from the server 200.
Furthermore, in a case where the analysis unit 311 analyzes various types of sensor information acquired by the sensor unit 330, the movement control unit 312 may appropriately control the movement of the mobile robot 300 on the basis of the analysis result. In other words, the analysis unit 311 may change migration path information provided from the server 200 and generate new migration path information, on the basis of the analysis result of various types of sensor information. For example, in a case where an obstacle is detected in the travelling direction as a result of the analysis, for avoiding the collision with the obstacle on the basis of the shape and the operation of the detected obstacle, the movement control unit 312 may change migration path information provided from the server 200 and generate new migration path information.
(Communication Unit 320)
The communication unit 320 is a functional configuration that performs communication with the server 200. For example, the communication unit 320 receives, from the server 200, migration path information used for movement control of the own device. Note that content of information communicated by the communication unit 320 is not specifically limited. For example, the communication unit 320 may receive, from the server 200, control information or the like for instructing predetermined processing other than movement control (for example, processing associated with a service performed by the mobile robot 300, or the like). Furthermore, the communication unit 320 may transmit position information of the own device (for example, a position coordinate in the three-dimensional space, or the like), to the server 200.
(Sensor Unit 330)
The sensor unit 330 is a functional configuration that acquires various types of sensor information regarding the surrounding environment of the mobile robot 300. The types of sensors included in the sensor unit 330 can be similar to the sensor unit 140 of the UAV 100, which have been described above, but the types of sensors are not limited to these. More specifically, the sensor unit 330 needs not always include the sensors included in the sensor unit 140, and may include a sensor other than the sensors included in the sensor unit 140. Furthermore, the sensor unit 330 may include a function similar to the imaging unit 130 of the UAV 100 that can generate captured image data.
(Storage Unit 340)
The storage unit 340 is a functional configuration that stores various types of information. For example, the storage unit 340 stores various types of sensor information acquired by the sensor unit 330, information regarding an analysis result of the analysis unit 311, migration path information provided from the server 200, and the like. Furthermore, the storage unit 340 also stores programs, parameters, and the like that are to be used in processing performed by each functional configuration. Note that information stored by the storage unit 340 is not limited to these types of information.
(Movement Mechanism 350)
The movement mechanism 350 is a configuration for moving the mobile robot 300, and includes, for example, an actuator, a motor, a wheel, or a tire, or the like (not illustrated). The movement mechanism 350 moves the mobile robot 300 by driving in accordance with control information provided from the movement control unit 312.
Heretofore, the functional configuration example of the mobile robot 300 has been described. Note that the above-described functional configuration described using
Heretofore, the functional configuration example of the mobile robot 300 has been described. Subsequently, a processing flow example according to the present embodiment will be described.
(5.1. Recognition Processing Flow of Position and Direction of Person that is Performed by UAV 100)
First of all, a recognition processing flow of the position and direction of a person that is performed by the UAV 100 will be described with reference to
In Step S1000, the flight control unit 112 of the UAV 100 starts round on the basis of control information provided from the server 200. In Step S1004, the imaging unit 130 generates captured image data by capturing an image of a person from above. In Step S1008, the analysis unit 111 recognizes that a tangible object appearing in the captured image data is a person, by analyzing the captured image data.
In Step S1012, the position identification unit 111b identifies a position coordinate of the person by analyzing the captured image data. In Step S1016, by analyzing the captured image data, the direction identification unit 111c tries to extract a feature region (for example, both eyes, a nose, a mouth, or the like) of a face of the person (in addition, by analyzing the captured image data, the person identification unit 111a tries to extract a feature region of the face of the person). In a case where the extraction of a feature region of the face of the person (or a feature amount of the face of the person) has not succeeded (Step S1020/No), in Step S1024, the flight control unit 112 implements obstacle avoidance flight. For example, the flight control unit 112 tries to extract a feature region of the face of the recognized person, by changing the position of the UAV 100 in various directions such as the front-back direction, the left-right direction, or the up-down direction (In Step S1016).
In a case where the extraction of a feature region of the face of the person (or a feature amount of the face of the person) has succeeded (Step S1020/Yes), in Step S1028, the person identification unit 111a identifies a person appearing in captured image data, by comparing the feature amount of the person that has been extracted from the captured image data, and a feature amount of a person that has been provided from the server 200 in advance (for example, a feature amount extracted from captured image data obtained at the entrance of an entertainment park). In Step S1032, the direction identification unit 111c identifies a direction of the face of the person by outputting a parallel line on the basis of the feature region of the person that has been extracted from the captured image data, and identifying an angle θ formed by the reference line and the parallel line. Note that, as illustrated in
In Step S1036, the control unit 110 integrates position information of the person that has been output by the position identification unit 111b, direction information of the person that has been output by the direction identification unit 111c, and the like. In Step S1040, by the communication unit 120 transmitting the integrated position information, direction information, and the like to the server 200, a series of processes ends.
By the processing illustrated in
(5.2. Control Processing Flow of Mobile Robot 300 that is Performed by Server 200)
Subsequently, a control processing flow of the mobile robot 300 that is performed by the server 200 will be described with reference to
In Step S1100, the communication unit 220 of the server 200 receives position information, direction information, and the like from the UAV 100. In Step S1104, the control unit 210 registers the position information and the direction information into the position/direction DB 232. In Step S1108, on the basis of the direction information and the position information of the person, and position information (and direction information) of the mobile robot 300, the robot control unit 212 moves the mobile robot 300 closer to the person from a direction that the person faces, and outputs a migration path (including an approach curve) of the mobile robot 300 that can stop the mobile robot 300 at a position distant from the person by the predetermined distance d. In Step S1112, by the communication unit 220 transmitting the migration path information to the mobile robot 300, a series of processes ends.
(5.3. Movement Processing Flow of Mobile Robot 300)
Subsequently, a movement processing flow of the mobile robot 300 will be described with reference to
In Step S1200, the communication unit 320 of the mobile robot 300 receives migration path information from the server 200. In Step S1204, the movement control unit 312 controls the movement of the own device by generating control information for controlling the movement mechanism 350, on the basis of information regarding an approach curve that is included in the migration path information, and providing the information to the movement mechanism 350. The mobile robot 300 can thereby move closer to the person from the direction that the person faces, stop at the position distant from the person by the predetermined distance d, and provide a predetermined service to the person. Note that, as in Step S1208 in
Note that the steps in the flowcharts illustrated in
(5.4. State Change of UAV 100)
The UAV 100 does not always perform the round and tracking of a person, and for example, returns to a station (standby location) when a battery is running low. In this manner, the “round”, the “tracking”, or the like that is performed by the UAV 100 will be referred to as a “state” of the UAV 100 in this specification. Subsequently, a change in state (hereinafter, will also be referred to as a “state change”) of the UAV 100 will be described with reference to
In Step S1300, the UAV 100 that has taken off from the station flies on the basis of control information provided from the server 200, and moves to a point at which round is to be started (hereinafter, will be referred to as a “round start point”) (in other words, the state of the UAV 100 is a “movement” state). In Step S1304, in a case where the UAV 100 has reached the round start point, in Step S1308, the UAV 100 starts the round (in other words, the state of the UAV 100 transitions from the “movement” state to a “round” state).
In Step S1312, in a case where the UAV 100 has discovered a person to be provided with a service, or in a case where an instruction for starting the tracking of a predetermined person has been received from the server 200, in Step S1316, the UAV 100 performs the tracking of the person (in other words, the state of the UAV 100 transitions from the “round” state to the “tracking” state).
In Step S1320, in a case where the battery of the UAV 100 is running low (for example, in a case where a remaining amount of the battery becomes a predetermined value or less), or in a case where the UAV 100 receives a return instruction from the server 200, in Step S1324, the UAV 100 returns to the station, and in Step S1328, the UAV 100 performs the charging of battery or standby (in other words, the state of the UAV 100 transitions from the “tracking” state or the “round” state to a “charging/standby” state through the “movement” state).
Then, in Step S1332, in a case where the charging of the battery is completed, and the UAV 100 receives a recovery instruction from the server 200, the UAV 100 takes off from the station and moves again to the round start point (in other words, the state of the UAV 100 transitions from the “charging/standby” state to the “round” state through the “movement” state).
Note that, in a case where the UAV 100 takes off from the station and moves again to the round start point, or the like, in a case where alternation with a different UAV 100 occurs, the different UAV 100 may take over various types of information to the UAV 100 that newly starts round. For example, the different UAV 100 may take over, directly using the respective communication units 120 or indirectly via the server 200, information regarding the identified person, position information and direction information of the person, and the like to the UAV 100 that newly starts round. At this time, it is desirable that information takeover is performed in a state in which the positions of the different UAV 100 and the UAV 100 that receives the takeover are as close as possible. For example, it is desirable that information takeover is performed in a state in which the UAV 100 that receives takeover moves to a position immediately above the different UAV 100, and then, the different UAV 100 moves (for example, returns to the station). With this configuration, it becomes easier to maintain consistency of the position of the UAV 100 that has received takeover, and information taken over from the different UAV 100. Furthermore, the UAV 100 that has received takeover needs not perform identification processing of a person or the like again. Note that the UAV 100 that has received takeover may appropriately correct the taken-over information on the basis of a positional relationship between the own device and the different UAV 100 at the time point at which takeover is performed, or the like.
Heretofore, a processing flow example of each device according to the present embodiment has been described. Subsequently, a variation regarding an identification method of a direction that a person faces will be described.
Heretofore, it has been described that the direction identification unit 111c of the UAV 100 can identify a direction that a person faces, on the basis of a feature of a face or a head of the person. However, the direction identification unit 111c may identify the direction that the person faces, on the basis of a feature of a region other than a face or a head of the person.
For example, the direction identification unit 111c may identify the direction that the person faces, on the basis of a feature of a body other than a face or a head. More specifically, as illustrated in
Furthermore, in a case where a person is moving, the direction identification unit 111c may identify a direction that a person faces on the basis of the travelling direction of the person. More specifically, because the position identification unit 111b can identify the position of a person appearing in captured image data, by analyzing the captured image data, the direction identification unit 111c can recognize the travelling direction of the person on the basis of a change in the position. Because it is generally considered that there is a high possibility that the travelling direction of the person (for example, a direction in which the person walks, or the like), and a direction that the person faces coincide with each other (alternatively, a difference between the directions becomes a predetermined value or less), the direction identification unit 111c may regard the travelling direction of the person as the direction that the person faces. Furthermore, the information processing system according to the present disclosure may output a direction that a person faces, by performing predetermined calculation using the travelling direction of the person.
Furthermore, the direction identification unit 111c may identify a direction that a person faces, on the basis of a feature of an object worn by the person (hereinafter, will be referred to as a “worn object”). More specifically, as illustrated in
Furthermore, the direction identification unit 111c may identify a direction that a person faces, on the basis of an analysis result other than an analysis result of captured image data acquired by the imaging unit 130. For example, the direction identification unit 111c may identify a direction that a person faces, on the basis of an analysis result of sensor information acquired by the sensor unit 140. For example, the direction identification unit 111c may identify a direction that a person faces, by identifying a feature region of a face by the analysis of temperature distribution information acquired by a temperature sensor (for example, thermosensor or the like) included in the sensor unit 140. Furthermore, in a case where the sensor unit 140 includes a sensor that can detect breath blown out by a person, the direction identification unit 111c may identify a direction that a person faces, on the basis of a direction in which the breath is blown out.
Note that the direction identification unit 111c may identify a direction that a person faces, on the basis of an analysis result of captured image data or sensor information that has been acquired by an imaging unit or a sensor mounted on a device other than the UAV 100. For example, the direction identification unit 111c may identify a direction that a person faces, on the basis of an analysis result of captured image data acquired by a device (may be the mobile robot 300) that can capture an image of the person, or an analysis result of sensor information acquired by a sensor (for example, a gyro sensor, an acceleration sensor, or the like) mounted on a wearable device being a worn object of the person.
Furthermore, the direction identification unit 111c may identify a direction that a person faces, by combining various methods described above. The direction identification unit 111c can thereby enhance the identification accuracy of a direction that a person faces.
Heretofore, a variation regarding a determination method of a direction that a person faces has been described. Subsequently, a modified example according to the present disclosure will be described.
(7.1. Control of Mobile Robot 300 by Gesture, Etc.)
First of all, the control of the mobile robot 300 by gesture (operation) or the like will be described. More specifically, if a person appearing in an image captured by the imaging unit 130 of the UAV 100 performs a predetermined gesture or sends a line-of-sight to the UAV 100, the UAV 100 recognizes the gesture or the line-of-sight by analyzing captured image data acquired by the imaging unit 130, and provides the recognition result to the server 200. Then, the server 200 can control the operation of the mobile robot 300 on the basis of the recognition result provided from the UAV 100. With this configuration, the person can call the mobile robot 300 and be provided with a predetermined service provided by the mobile robot 300 more easily, without performing a cumbersome operation such as manipulation of a predetermined device.
Here, the content of gesture or the like that is to be used for the control of the mobile robot 300 is not specifically limited. For example, as illustrated in
Furthermore, as illustrated in
Furthermore, as illustrated in
Furthermore, as illustrated in
Note that analysis processing of gesture or sending of a line-of-sight to the UAV 100 (operation) is assumed to be implemented by the analysis unit 111 of the UAV 100 (in other words, the analysis unit 111 functions as an operation analysis unit that analyze an operation of a target (person)), and operation control processing of the mobile robot 300 is assumed to be implemented by the robot control unit 212 of the server 200, but the configuration is not limited to this. Furthermore, the control of the mobile robot 300 may be implemented by combining the above-described gesture, or sending of a line-of-sight to the UAV 100. With this configuration, a possibility of the occurrence of a malfunction of the mobile robot 300 further declines, and the control of the mobile robot 300 can be defined more finely.
(7.2. Control of Mobile Robot 300 that is Based on Prediction Result of Congestion Situation)
Subsequently, the control of the mobile robot 300 that is based on a prediction result of a congestion situation caused by a tangible object existing around a targeted person will be described. As described above, the server 200 can prevent the mobile robot 300 from colliding with an obstacle, by outputting a migration path of the mobile robot 300 also considering position information of an obstacle (or a tangible object, including another person) existing around a targeted person. However, in a case where a congestion situation around a targeted person changes from moment to moment by the movement of the obstacle, it becomes more likely that the mobile robot 300 collides with the obstacle.
In view of the foregoing, the server 200 may control the mobile robot 300 on the basis of a prediction result of a congestion situation around a targeted person. For example, by analyzing captured image data acquired by the imaging unit 130, the UAV 100 recognizes the position of each obstacle at each time. Then, by outputting a tensor in which position information pieces of obstacles are chronologically arranged in a matrix, and analyzing the tensor using a predetermined method, the UAV 100 predicts the position of an obstacle existing around a targeted person at the time point at which a certain time elapses, and provides the prediction result to the server 200. The “predetermined method” includes inputting a tensor to a machine learning (alternatively, artificial intelligence) library that can perform chronological analysis of the tensor, for example, but the predetermined method is not limited to this. Then, the server 200 determines a migration path of the mobile robot 300 on the basis of the prediction result of the position of the obstacle that has been provided from the UAV 100.
Note that the above-described prediction processing is assumed to be implemented by the analysis unit 111 of the UAV 100, and movement control processing of the mobile robot 300 is assumed to be implemented by the robot control unit 212 of the server 200, but the configuration is not limited to this. Furthermore, by the UAV 100 performing the above-described prediction processing and provision processing of a prediction result as needed, the server 200 may change a migration path of the mobile robot 300 as needed. With this configuration, also in a case where a congestion situation around a targeted person changes from moment to moment, it becomes more likely that the mobile robot 300 can reach a target position without colliding with an obstacle.
(7.3. Control of Mobile Robot 300 in Case where Target is Moving)
Subsequently, the control of the mobile robot 300 in a case where a target is moving will be described. For example, in a case where a person being a target is moving, even if the movement of the mobile robot 300 is controlled on the basis of a position of the person and a direction that the person faces at a certain time point, when the mobile robot 300 reaches a target position, the position of the person and the direction that the person faces have already changed. Thus, the mobile robot 300 cannot provide a service to the person.
In view of the foregoing, the server 200 may control the mobile robot 300 on the basis of a movement situation of a person. For example, by analyzing captured image data acquired by the imaging unit 130, the UAV 100 recognizes whether or not a person appearing in the captured image data is moving (alternatively, in a case where the person is moving, a movement speed). Then, in a case where the person is moving (alternatively, in a case where the movement speed is faster than a predetermined value), the server 200 may avoid moving the mobile robot 300, and in a case where the person stops moving (alternatively, in a case where the movement speed becomes a predetermined value or less), the server 200 may move the mobile robot 300. With this configuration, it becomes more likely that the mobile robot 300 can reach a targeted person.
Furthermore, the server 200 may control the mobile robot 300 on the basis of a prediction result of a position of a person or a direction that the person faces. For example, by analyzing captured image data acquired by the imaging unit 130, the UAV 100 recognizes whether or not a person appearing in the captured image data is moving, and in a case where the person is moving, recognizes a movement speed, a movement direction, action details of the person, and the like (for example, eating, resting, playing an attraction in an entertainment park, or the like). Then, on the basis of these recognition results, the UAV 100 predicts the position of the person or a direction that the person faces, at a time point at which a certain time elapses, and provides a prediction result to the server 200. Then, the server 200 determines a migration path of the mobile robot 300 on the basis of the prediction result of the position of the person or a direction that the person faces that has been provided from the UAV 100.
Note that the above-described prediction processing is assumed to be implemented by the analysis unit 111 of the UAV 100, and movement control processing of the mobile robot 300 is assumed to be implemented by the robot control unit 212 of the server 200, but the configuration is not limited to this. Furthermore, the prediction method of the position of the person or a direction that the person faces is not limited to the above-described prediction method. For example, the UAV 100 may predict these using action history information of the person or the like. Furthermore, by the UAV 100 performing the above-described prediction processing and provision processing of a prediction result as needed, the server 200 may change a migration path of the mobile robot 300 as needed. With this configuration, also in a case where a person is moving, it becomes more likely that the mobile robot 300 can reach an appropriate person.
(7.4. Identification Method of Position of Entrance in Architectural Structure)
Subsequently, a method of identifying the position of an entrance in an architectural structure on the basis of an analysis result of captured image data acquired by the imaging unit 130 of the UAV 100 will be described.
In a case where the server 200 desires to move the mobile robot 300 into a certain architectural structure, the server 200 sometimes does not preliminarily recognize the position of the entrance in the architectural structure. For example, in a case where the information processing system according to the present disclosure is applied to a logistics system, and the server 200 desires to move the mobile robot 300 into a building being a delivery destination, the server 200 sometimes does not preliminarily recognize the position of the entrance in the building.
In this case, the analysis unit 111 of the UAV 100 can identify the position of the entrance in the architectural structure by analyzing captured image data acquired by the imaging unit 130. For example, as illustrated in A of
Heretofore, a modified example according to the present disclosure has been described. Subsequently, a hardware configuration example of an information processing device 900 that realizes the UAV 100, the server 200, or the mobile robot 300 will be described with reference to
The CPU 901 functions as an arithmetic processing unit and a control device, and controls the overall operations in the information processing device 900 in accordance with various programs. Furthermore, the CPU 901 may be a microprocessor. The ROM 902 stores programs, calculation parameters, and the like that are to be used in the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters appropriately changing in the execution, and the like. These components are connected to each other by the host bus 904 including a CPU bus and the like. By the cooperation of the CPU 901, the ROM 902, and the RAM 903, the functions of the control unit 110 of the UAV 100, the control unit 210 of the server 200, or the control unit 310 of the mobile robot 300 are implemented.
The host bus 904 is connected, via the bridge 905, to the external bus 906 such as a peripheral component interconnect/interface (PCI) bus. Note that the host bus 904, the bridge 905, and the external bus 906 need not be always separated, and these functions may be mounted on one bus.
The input device 908 includes an input means for the user inputting information, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and lever, an input control circuit that generates an input signal on the basis of the input performed by the user, and outputs the input signal to the CPU 901, and the like. By operating the input device 908, the user of the information processing device 900 can input various types of information to each device or instruct processing operation. The function of an input unit (not illustrated) of each device is implemented by the input device 908.
The output device 909 includes a display device such as a cathode ray tube (CRT) display device, a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, and a lamp, for example. Moreover, the output device 909 includes a voice output device such as a speaker and headphones. The output device 909 outputs reproduced content, for example. Specifically, the display device displays various types of information such as reproduced video data, by a text or an image. On the other hand, the voice output device converts reproduced voice data or the like into voice, and outputs the voice. The function of an output unit (not illustrated) of each device is implemented by the output device 909.
The storage device 910 is a data storage device. The storage device 910 may include a storage medium, a recording device that records data onto the storage medium, a reading device that reads out data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 includes a hard disk drive (HDD), for example. The storage device 910 drives a hard disc and stores programs executed by the CPU 901, and various data. The functions of the storage unit 150 of the UAV 100, the storage unit 230 of the server 200, or the storage unit 340 of the mobile robot 300 are implemented by the storage device 910.
The drive 911 is a reader/writer for a storage medium, and is built in the information processing device 900 or externally attached thereto. The drive 911 reads out information recorded in the attached removable storage medium 913 such as a magnetic disc, an optical disk, a magnetooptical disk, or a semiconductor memory, and outputs the information to the RAM 903. Furthermore, the drive 911 can write information into the removable storage medium 913.
The communication device 912 is a communication interface including a communication device or the like for connecting to a communication network 914, for example. The functions of the communication unit 120 of the UAV 100, the communication unit 220 of the server 200, or the communication unit 320 of the mobile robot 300 are implemented by the communication device 912.
As described above, the UAV 100 according to the present disclosure can search for a specific person from above and track the specific person, by analyzing captured image data acquired by the imaging unit 130. Then, because the UAV 100 can identify the position of a person appearing in captured image data, and a direction that the person faces, the server 200 can move the mobile robot 300 toward the person on the basis of the position and the direction identified by the UAV 100, and cause the mobile robot 300 to provide a predetermined service.
Furthermore, in a case where a feature region of a face of a person does not clearly appear in captured image data by image capturing being performed against light, an image of a region with a shadow being captured, or image capturing being performed at a location with many obstacles, or the like, the UAV 100 can control the flight of the own device in such a manner as to eliminate a cause for disturbing the identification of a direction that a person faces.
Heretofore, a preferred embodiment of the present disclosure has been described in detail with reference to the attached drawings, but the technical scope of the present disclosure is not limited to this example. It should be appreciated that a person who has general knowledge in the technical field of the present disclosure can conceive various change examples and modified examples within the scope of the technical idea described in the appended claims, and these change examples and modified examples are construed as naturally falling within the technical scope of the present disclosure.
Furthermore, the effects described in this specification are merely provided as explanatory or exemplary effects, and the effects are not limited. That is, the technology according to the present disclosure can bring about another effect obvious for the one skilled in the art, from the description in this specification, in addition to the above-described effects or in place of the above-described effects.
Note that the following configurations also fall within the technical scope of the present disclosure.
(1) An information processing device including:
an acquisition unit configured to acquire captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing; and
a direction identification unit configured to identify a direction that a target appearing in the captured image data faces, by analyzing the captured image data.
(2) The information processing device according to (1) described above, in which
the direction identification unit identifies a direction that a person being the target faces.
(3) The information processing device according to (2) described above, in which
the direction identification unit identifies the direction on the basis of a feature of a face or a head of the person that appears in the captured image data.
(4) The information processing device according to (3) described above, in which
the direction identification unit identifies the direction on the basis of a feature of at least any one of both eyes, one eye, an eyebrow, a nose, a mouth, a tooth, an ear, or hair.
(5) The information processing device according to (2) described above, in which
the direction identification unit identifies the direction on the basis of a feature of a body of the person that appears in the captured image data.
(6) The information processing device according to (2) described above, in which
the direction identification unit identifies the direction on the basis of a feature of an object worn by the person that appears in the captured image data.
(7) The information processing device according to any one of (1) to (6) described above, further including:
a flight control unit configured to recognize that identification of the direction has been disturbed, and control flight of the UAV toward front or back, left or right, up or down, or by combining these.
(8) The information processing device according to (7) described above, further including
a target identification unit configured to identify a target appearing in the captured image data,
in which the flight control unit controls the flight in such a manner as to track at least one of the identified targets.
(9) The information processing device according to (8) described above, in which
the flight control unit controls the flight on the basis of a change in a relative positional relationship between the UAV and the target.
(10) The information processing device according to (8) or (9) described above, in which
in a case where the target identified by the target identification unit is identified by appearing also in captured image data obtained by a different UAV by performing aerial image capturing,
the flight control unit controls the flight in such a manner that at least one of the UAV or the different UAV tracks the target.
(11) The information processing device according to any one of (1) to (10) described above, further including:
a position identification unit configured to identify a position of a target appearing in the captured image data, by analyzing the captured image data; and
a robot control unit configured to control a movement of a predetermined robot on the basis of the position and the direction.
(12) The information processing device according to (11) described above, in which
the robot control unit moves the robot closer to the target from the direction or a different direction determined on the basis of the direction.
(13) The information processing device according to (12) described above, in which
the robot control unit stops the robot at a position distant from the target by a predetermined distance.
(14) The information processing device according to any one of (11) to (13) described above, in which
in a case where the target is moving,
the robot control unit controls a movement of the robot on the basis of a prediction result of a position of the target or the direction at a time point at which a certain time elapses.
(15) The information processing device according to according to any one of (11) to (14) described above, in which
the robot control unit controls a movement of the robot on the basis of a prediction result of a position of a tangible object existing around the target, at a time point at which a certain time elapses.
(16) The information processing device according to according to any one of (11) to (15) described above, further including:
an operation analysis unit configured to analyze an operation of the target appearing in the captured image data,
in which the robot control unit controls a movement of the robot on the basis of the operation.
(17) The information processing device according to any one of (1) to (16) described above, in which
the information processing device is any of the UAV, a predetermined device mounted on the UAV, or a cloud server.
(18) An information processing method executed by a computer, the information processing method including:
acquiring captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing; and
identifying a direction that a target appearing in the captured image data faces, by analyzing the captured image data.
(19) A program for causing a computer to execute:
acquiring captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing; and
identifying a direction that a target appearing in the captured image data faces, by analyzing the captured image data.
(20) An information processing system including: an information processing device; and a robot,
in which the information processing device includes
an acquisition unit configured to acquire captured image data obtained by an unmanned aerial vehicle (UAV) by performing aerial image capturing,
a direction identification unit configured to identify a direction that a target appearing in the captured image data faces, by analyzing the captured image data,
a position identification unit configured to identify a position of a target appearing in the captured image data, by analyzing the captured image data, and
a robot control unit configured to control a movement of the robot on the basis of the position and the direction.
(21) The information processing device according to any one of (1) to (6) described above, further including:
a flight control unit configured to control flight of the UAV in such a manner as to eliminate a cause for disturbing identification of the direction.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2018-114695 | Jun 2018 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/015241 | 4/8/2019 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2019/239694 | 12/19/2019 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20080269973 | Kuroda | Oct 2008 | A1 |
| 20110019003 | Asa | Jan 2011 | A1 |
| 20110311112 | Matsuyama | Dec 2011 | A1 |
| 20120210853 | Abershitz | Aug 2012 | A1 |
| 20150134143 | Willenborg | May 2015 | A1 |
| 20160232650 | Christ | Aug 2016 | A1 |
| 20160280397 | Christ | Sep 2016 | A1 |
| 20170134631 | Zhao | May 2017 | A1 |
| 20170166299 | Fujimura | Jun 2017 | A1 |
| 20170374277 | Dobashi | Dec 2017 | A1 |
| 20180129212 | Lee | May 2018 | A1 |
| 20180247421 | DeAngelis | Aug 2018 | A1 |
| 20180268202 | Yu | Sep 2018 | A1 |
| 20180314251 | Kamalakantha | Nov 2018 | A1 |
| 20190002104 | Kageyama | Jan 2019 | A1 |
| 20190011921 | Wang | Jan 2019 | A1 |
| 20190130594 | Seyfi | May 2019 | A1 |
| 20190146512 | Sindlinger | May 2019 | A1 |
| 20190163966 | Moriya | May 2019 | A1 |
| 20190176967 | Ohata | Jun 2019 | A1 |
| 20190212742 | Qian | Jul 2019 | A1 |
| 20190227489 | Tokuchi | Jul 2019 | A1 |
| 20190227491 | Tokuchi | Jul 2019 | A1 |
| 20190318634 | Ma | Oct 2019 | A1 |
| 20200018291 | Rosen | Jan 2020 | A1 |
| 20200150662 | Zhou | May 2020 | A1 |
| 20200349704 | Pang | Nov 2020 | A1 |
| 20210034078 | Gomez Gutierrez | Feb 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 2552092 | Jan 2018 | GB |
| 2003-263641 | Sep 2003 | JP |
| 2003263641 | Sep 2003 | JP |
| 2005-329515 | Dec 2005 | JP |
| 2009086471 | Apr 2009 | JP |
| 2016-1140 | Oct 2016 | JP |
| 2016177740 | Oct 2016 | JP |
| 2017-56904 | Mar 2017 | JP |
| 2017-58831 | Mar 2017 | JP |
| 2017056904 | Mar 2017 | JP |
| 2017-68298 | Apr 2017 | JP |
| 2017140899 | Aug 2017 | JP |
| 2017-212528 | Nov 2017 | JP |
| 2017212528 | Nov 2017 | JP |
| WO-2017049816 | Mar 2017 | WO |
| 2017115448 | Jul 2017 | WO |
| 2017130407 | Aug 2017 | WO |
| WO-2017130407 | Aug 2017 | WO |
| WO-2017170651 | Oct 2017 | WO |
| Entry |
|---|
| P. Kakvand, M. Jaberzadeh, M. M. Inallou and Y. Alborz, “Smart on-board UAV system: Using computer vision system to find a movable and stationary target,” 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI), 2015, pp. 694-699, doi: 10.1109/KBEI.2015.7436128. (Year: 2015). |
| M. M. Kurdi, A. K. Dadykin and I. Elzein, “Navigation of mobile robot with cooperation of quadcopter,” 2017 Ninth International Conference on Advanced Computational Intelligence (ICACI), 2017, pp. 30-36, doi: 10.1109/ICACI.2017.7974481. (Year: 2017). |
| B. Grocholsky, J. Keller, V. Kumar and G. Pappas, “Cooperative air and ground surveillance,” in IEEE Robotics & Automation Magazine, vol. 13, No. 3, pp. 16-25, Sep. 2006, doi: 10.1109/MRA.2006.1678135. (Year: 2006). |
| Article title “Curve”, Encyclopaedia Britannica, Publisher: Encyclopaedia Britannica, Inc., Date Published: Mar. 21, 2012, URL: https://www.britannica.com/science/curve Access Date: Jan. 22, 2022 (Year: 2012). |
| International Search Report and Written Opinion dated Jun. 25, 2019 for PCT/JP2019/015241 filed on Apr. 8, 2019, 11 pages including English Translation of the International Search Report. |
| Number | Date | Country | |
|---|---|---|---|
| 20210209332 A1 | Jul 2021 | US |
