Patent Application
20250190891
This application claims priority to Japanese Patent Application No. 2023-209379 filed on Dec. 12, 2023, incorporated herein by reference in its entirety.
The present disclosure relates to a remote operation system, a remote operation management method, and a non-transitory storage medium.
US Unexamined Patent Application Publication No. 2020/0062267 discloses a system for remote operation of a vehicle. The system assigns a suitable remote operator to remote operation of the vehicle, based on the skills and experiences of remote operators.
It is herein assumed that a remote operator remotely operates a mobility. There may be cases where there is a plurality of types of mobilities for remote operation. Remote operators do not necessarily have licenses for all types of mobilities. If a remote operator who does not have a license to operate a target mobility is assigned to remote operation of the target mobility by mistake, the assignment process needs to be started over, which is inefficient.
The present disclosure provides a technique that allows to efficiently perform an assignment process for assigning a remote operator to remote operation of a mobility.
A remote operation system of remote operation of a plurality of types of mobility according to a first aspect of the present disclosure includes one or more processors. The one or more processors are configured to: acquire operator license information indicating a license each of a plurality of remote operators has for operation of the plurality of types of mobilities; acquire required capability information indicating at least a required license required for the remote operation of a target mobility; and assign a first remote operator who has the required license out of the remote operators to the remote operation of the target mobility, based on the operator license information and the required license.
A remote operation management method of managing remote operation of a plurality of types of mobilities by a computer according to a second aspect of the present disclosure includes: acquiring operator license information indicating a license each of a plurality of remote operators has for operation of the plurality of types of mobilities; acquiring required capability information indicating at least a required license required for the remote operation of a target mobility; and assigning a first remote operator who has the required license out of the remote operators to the remote operation of the target mobility, based on the operator license information and the required license.
A non-transitory storage medium according a third aspect of the present disclosure storing instructions of managing remote operation of a plurality of types of mobilities that are executable by one or more processors and that cause the one or more processors to perform functions includes: acquiring operator license information indicating a license each of a plurality of remote operators has for operation of the plurality of types of mobilities; acquiring required capability information indicating at least a required license required for the remote operation of a target mobility; and assigning a first remote operator who has the required license out of the remote operators to the remote operation of the target mobility, based on the operator license information and the required license.
According to the present disclosure, the licenses the individual operators have and the required license required for the remote operation of the target mobility are taken into consideration. The first remote operator having the required license is assigned to the remote operation of the target mobility. This avoids a situation in which a remote operator who does not have the required license is assigned to the target mobility by mistake. This eliminates the need to start over the assignment process, and implements an efficient assignment process.
Features, advantages, and technical and industrial significance of exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings, in which like signs denote like elements, and wherein:
An embodiment of the present disclosure will be described with reference to the accompanying drawings.
The mobility 100 is a moving object that can move. The mobility 100 may be manually operated by an operator in the mobility 100. The mobility 100 may have an autonomous moving function. In any case, the mobility 100 is configured to be operated even remotely as needed. That is, the mobility 100 is an object to be remotely operated by the remote operation system 1.
The number of types of mobility 100 to be remotely operated is not limited to one, and may be two or more. For example, the mobility 100 is a vehicle that travels on public roads (e.g., a passenger car, a truck, a bus, a Mobility as a Service (MaaS) vehicle, an automated driving vehicle, etc.). As another example, the mobility 100 may be a vehicle that is used in a factory (e.g., a forklift, an industrial dolly, etc.). As still another example, the mobility 100 may be a special small vehicle (e.g., a golf cart, a personal mobility, an electric wheelchair, etc.). As yet another example, the mobility 100 may be a construction machine (e.g., a power shovel, a bulldozer, etc.). As a further example, the mobility 100 may be a robot (e.g., a logistics robot, a work robot, etc.). As a still further example, the mobility 100 may be a flying object (e.g., a drone, etc.). As a yet further example, the mobility 100 may be a watercraft (e.g., a small boat, a large cruise ship, etc.). As a yet further example, the mobility 100 may be a ride in an amusement park (e.g., a cart, an attraction, etc.).
The remote operator terminal 200 is a terminal device that a remote operator O uses when remotely operating the mobility 100. That is, the remote operator terminal 200 is configured to be used by the remote operator O for remote operation of the mobility 100. Examples of the remote operator terminal 200 include a cockpit-type terminal, a personal computer (PC), a tablet, and a smartphone. A single remote operator terminal 200 may be configured to support remote operation of various types of mobilities 100. Alternatively, a single remote operator terminal 200 may be dedicated to remote operation of a specific mobility 100. The combination of the remote operator O and the remote operator terminal 200 may be determined in advance, or may be changeable as desired. That is, a single remote operator terminal 200 may be used only by a specific remote operator O, or may be used by various remote operators O in turns.
The management system (manager) 300 manages the remote operation system 1. The management system 300 may be composed of a plurality of servers that performs distributed processing. For example, the management system 300 manages a plurality of remote operators O and a plurality of remote operator terminals 200. The management system 300 assigns a remote operator O and a remote operator terminal 200 to remote operation of a mobility 100, in response to a remote operation request. The management system 300 may also manage the state of the mobility 100 during remote operation. The management system 300 will be described in detail later.
The mobility 100, the remote operator terminal 200, and the management system 300 can communicate with each other via a communication network. For example, the mobility 100 can wirelessly communicate with the remote operator terminal 200 and the management system 300 via a wireless communication network. The remote operator terminal 200 and the management system 300 can communicate with each other via a wired communication network or a wireless communication network. The mobility 100 and the remote operator terminal 200 may communicate with each other via the management system 300, or may communicate directly with each other without via the management system 300.
The flow of information during remote operation of the mobility 100 is roughly as follows.
The mobility 100 is equipped with various sensors including a camera. The camera captures a video of the surroundings of the mobility 100. The video of the surroundings of the mobility 100 is thus obtained by the camera. Mobility information MOV is information obtained by various sensors, and includes at least the video captured by the camera. The mobility information MOV may include the location and state (e.g., speed, steering angle, etc.) of the mobility 100. The mobility 100 sends the mobility information MOV to the remote operator terminal 200.
The remote operator terminal 200 receives the mobility information MOV sent from the mobility 100. The remote operator terminal 200 presents the mobility information MOV to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device, and displays the video etc. on the display device. The remote operator O grasps the surroundings of the mobility 100 by looking at the displayed information, and remotely operates the mobility 100. Remote operation information OPE is information about remote operation by the remote operator O (steering operation, acceleration operation, deceleration operation, forward and reverse movement operation, lateral movement operation, etc.). For example, the remote operation information OPE includes an amount of operation input by the remote operator O. The remote operation information OPE can be said to be information reflecting the degree of remote operation by the remote operator O. The remote operator terminal 200 sends the remote operation information OPE to the mobility 100.
The mobility 100 receives the remote operation information OPE sent from the remote operator terminal 200. The mobility 100 performs mobility control according to the received remote operation information OPE. Remote operation of the mobility 100 is implemented in this manner.
The communication device 110 wirelessly communicates with the outside of the mobility 100. For example, the communication device 110 wirelessly communicates with the remote operator terminal 200 and the management system 300.
The sensor group 120 includes a perception sensor, a mobility state sensor, and a location sensor. The perception sensor perceives (detects) the surroundings of the mobility 100. Examples of the perception sensor include a camera C, a Laser Imaging Detection and Ranging (LIDAR) sensor, and a radar. The mobility state sensor detects the state of the mobility 100. The mobility state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. The location sensor detects the location and orientation of the mobility 100. For example, the location sensor includes a Global Navigation Satellite System (GNSS).
The actuator 130 moves the mobility 100. For example, the actuator 130 includes a forward and reverse movement actuator for moving (accelerating and decelerating) the mobility 100 forward and backward. As another example, the actuator 130 may include a lateral movement actuator for moving the mobility 100 laterally. As still another example, when the mobility 100 is equipped with an arm, the actuator 130 may include an arm actuator for moving the arm.
For example, when the mobility 100 is a common vehicle, the actuator 130 includes a steering device, a drive device, and a braking device. The steering device steers wheels. For example, the steering device includes an electric power steering (EPS) system. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.
The control device 150 is a computer that controls the mobility 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as “processor 160”) and one or more storage devices 170 (hereinafter simply referred to as “storage device 170”). The processor 160 performs various processes. Examples of the processor 160 include a general purpose processor, a special purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, a conventional circuit, and/or combinations thereof. The processor 160 can also be referred to as “circuitry” or “processing circuitry.” Circuitry is hardware programmed to implement the described functions, or hardware that performs the functions. The storage device 170 stores various kinds of information. Examples of the storage device 170 include a volatile memory, a nonvolatile memory, a hard disk drive (HDD), and a solid state drive (SSD).
A control program PROG1 is a computer program that is executed by the processor 160. The functions of the control device 150 may be implemented through cooperation between the processor 160 that executes the control program PROG1 and the storage device 170. The control program PROG1 is stored in the storage device 170. Alternatively, the control program PROG1 may be recorded on a computer-readable recording medium.
The control device 150 acquires driving environment information ENV indicating the driving environment of the mobility 100. The driving environment information ENV is stored in the storage device 170.
The driving environment information ENV includes surroundings information indicating the perception results from the perception sensor. For example, the surroundings information includes a video captured by the camera C. The surroundings information may include object information on objects around the mobility 100. Examples of the objects around the mobility 100 include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, stop lines, traffic lights, signs, and roadside structures. The object information indicates the relative positions and relative speeds of the objects with respect to the mobility 100. For example, the objects can be identified and the relative positions of the objects can be calculated by analyzing the video obtained by the camera. Alternatively, the objects can be identified and the relative positions and relative speeds of the objects can be acquired based on point cloud information obtained by a LIDAR sensor.
The driving environment information ENV may include mobility state information indicating the detection results from the mobility state sensor. The mobility state information indicates the speed, acceleration (longitudinal acceleration, lateral acceleration), yaw rate, steering angle, etc. of the mobility 100.
The driving environment information ENV may further include location information indicating the location and moving direction (orientation) of the mobility 100. The location information is obtained by the location sensor. Accurate location information may be acquired by localization using map information and the surroundings information (object information).
The control device 150 performs movement control to control movement of the mobility 100. The movement control includes forward and reverse movement control and lateral movement control. The control device 150 performs the movement control by controlling the actuator 130.
The control device 150 may perform autonomous movement control based on the driving environment information ENV. More specifically, the control device 150 generates a movement plan for the mobility 100 based on the driving environment information ENV. The control device 150 also generates a target trajectory that is necessary for the mobility 100 to travel according to the movement plan, based on the driving environment information ENV. The target trajectory includes a target location and a target speed. The control device 150 then performs the movement control so that the mobility 100 follows the target trajectory.
When remote operation of the mobility 100 is performed, the control device 150 communicates with the remote operator terminal 200 via the communication device 110.
The control device 150 sends the mobility information MOV to the remote operator terminal 200. The mobility information MOV is information necessary for the remote operator O to remotely operate the mobility 100, and includes at least part of the driving environment information ENV described above. In particular, the mobility information MOV includes the video captured by the camera C. The mobility information MOV may include other surroundings information. The mobility information MOV may include the mobility state information. The mobility information MOV may include the location information.
The control device 150 receives the remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information about remote operation by the remote operator O (steering operation, acceleration operation, deceleration operation, forward and reverse movement operation, lateral movement operation, etc.). For example, the remote operation information OPE includes an amount of operation input by the remote operator O. The control device 150 performs the movement control according to the received remote operation information OPE.
The communication device 210 communicates with the mobility 100 and the management system 300.
The display device 220 displays various kinds of information to the remote operator O who performs remote operation. In other words, the display device 220 presents various kinds of information to the remote operator O by displaying them. The display device 220 is typically a display (monitor) such as a liquid crystal display or an organic electroluminescence (EL) display. The display device 220 may be a touch panel.
The input device 230 receives inputs from the remote operator O. For example, the input device 230 includes a remote operation member that the remote operator O operates when remotely operating the mobility 100. Examples of the remote operation member include a steering wheel, an accelerator pedal, a brake pedal, a turn signal lever, a joystick, a directional pad, and a switch. The remote operation member may be a touch panel. The input device 230 may include a member other than the remote operation member, such as a keyboard, a mouse, and a touch panel.
The operator sensor 240 is a sensor for monitoring the state of the remote operator O. For example, the operator sensor 240 includes a biosensor that detects biological information of the remote operator O. Examples of the biological information include body temperature, heart rate, blood pressure, and amount of perspiration.
The control device 250 controls the remote operator terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as “processor 260”) and one or more storage devices 270 (hereinafter simply referred to as “storage device 270”). The processor 260 performs various processes. Examples of the processor 260 include a CPU, a GPU, an ASIC, and an FPGA. The storage device 270 stores various kinds of information. Examples of the storage device 270 include a volatile memory, a nonvolatile memory, an HDD, and an SSD. The processor 260 performs various processes. Examples of the processor 260 include a general purpose processor, a special purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, a conventional circuit, and/or combinations thereof. The processor 260 can also be referred to as “circuitry” or “processing circuitry.” Circuitry is hardware programmed to implement the described functions, or hardware that performs the functions. The storage device 270 stores various kinds of information. Examples of the storage device 270 include a volatile memory, a nonvolatile memory, an HDD, and an SSD.
A control program PROG2 is a computer program that is executed by the processor 260. The functions of the control device 250 may be implemented through cooperation between the processor 260 that executes the control program PROG2 and the storage device 270. The control program PROG2 is stored in the storage device 270. Alternatively, the control program PROG2 may be recorded on a computer-readable recording medium. The control program PROG2 may be provided via a network.
The control device 250 communicates with the mobility 100 via the communication device 210. The control device 250 receives the mobility information MOV sent from the mobility 100. The control device 250 presents the mobility information MOV including the video to the remote operator O by displaying the mobility information MOV on the display device 220. The remote operator O can grasp the state and surroundings of the mobility 100 based on the mobility information MOV displayed on the display device 220.
The remote operator O operates the remote operation member of the input device 230. The amount of operation of the remote operation member is detected by a sensor installed on the remote operation member. The control device 250 generates the remote operation information OPE reflecting the amount of operation of the remote operation member by the remote operator O. The remote operation information OPE can be said to be information reflecting the degree of driving operation by the remote operator O. The control device 250 sends the remote operation information OPE to the mobility 100 via the communication device 210.
The communication device 310 communicates with the mobility 100 and the remote operator terminal 200.
The control device 350 controls the management system 300. The control device 350 includes one or more processors 360 (hereinafter simply referred to as “processor 360”) and one or more storage devices 370 (hereinafter simply referred to as “storage device 370”). The processor 360 performs various processes. Examples of the processor 360 include a general purpose processor, a special purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, a conventional circuit, and/or combinations thereof. The processor 360 can also be referred to as “circuitry” or “processing circuitry.” Circuitry is hardware programmed to implement the described functions, or hardware that performs the functions. The storage device 370 stores various kinds of information. Examples of the storage device 370 include a volatile memory, a nonvolatile memory, an HDD, and an SSD.
A management program PROG3 is a computer program that is executed by the processor 360. The functions of the control device 350 may be implemented through cooperation between the processor 360 that executes the management program PROG3 and the storage device 370. The management program PROG3 is stored in the storage device 370. Alternatively, the management program PROG3 may be recorded on a computer-readable recording medium. The management program PROG3 may be provided via a network.
The storage device 370 further stores operator management information MGT-O and terminal management information MGT-T. The operator management information MGT-O is information for managing the remote operators O. For example, the operator management information MGT-O indicates information on the individual remote operators O, such as an operator identification (ID), license(s) each remote operator O has, availability, assignment status, and work history. The terminal management information MGT-T is information for managing the remote operator terminals 200. For example, the terminal management information MGT-T indicates information on the individual remote operator terminals 200, such as a terminal ID, specifications, availability, assignment status, and operation history.
The control device 350 communicates with the mobility 100 and the remote operator terminal 200 via the communication device 310. The control device 350 may relay communication between the mobility 100 and the remote operator terminal 200. That is, the control device 350 may relay the mobility information MOV and the remote operation information OPE between the mobility 100 and the remote operator terminal 200.
Before the start of remote operation of the mobility 100, the control device 350 performs an “assignment process” of assigning a suitable remote operator O and a suitable remote operator terminal 200 to remote operation of the mobility 100. The assignment process that is performed by the management system 300 (control device 350) will be described in more detail below.
The management system 300 (control device 350) includes an assignment processing unit 400 that performs the assignment process. First, the assignment processing unit 400 receives a remote operation request REQ regarding the target mobility 100-X. For example, the remote operation request REQ is made by the target mobility 100-X itself. In this case, the target mobility 100-X sends the remote operation request REQ to the management system 300 via wireless communication. As another example, the remote operation request REQ may be sent from the service provider affiliated with the remote operation system 1 to the management system 300.
The remote operation request REQ indicates the type of the target mobility 100-X and the content of desired remote operation. In response to the received remote operation request REQ, the assignment processing unit 400 assigns a first remote operator O-X and a first remote operator terminal 200-X to remote operation of the target mobility 100-X. More specifically, the assignment processing unit 400 assigns a suitable first remote operator O-X out of the remote operators O to remote operation of the target mobility 100-X, based on the remote operation request REQ and the operator management information MGT-O. The assignment processing unit 400 also assigns a suitable first remote operator terminal 200-X out of the remote operator terminals 200 to remote operation of the target mobility 100-X, based on the remote operation request REQ and the terminal management information MGT-T.
The following three aspects will be considered in order to implement an appropriate assignment process: first aspect: the state of each remote operator O and the state of each remote operator terminal 200, second aspect: the license(s) each remote operator O has and a required license required for remote operation of the target mobility 100-X, and third aspect: terminal specifications of each remote operator terminal 200 and required terminal specifications required for remote operation of the target mobility 100-X. The assignment processing unit 400 performs the assignment process by taking at least one of the first to third aspects into consideration. The assignment processing unit 400 may perform the assignment process by taking two or more of the first to third aspects into consideration. The first, second, and third aspects will be described in detail below in Sections 4, 5, and 6, respectively.
Regarding the remote operator O, there may be states that are suitable for remote operation and states that are not. If a first remote operator O-X who is in a state not suitable for remote operation is selected, the accuracy of remote operation will decrease. Similarly, regarding the remote operator terminal 200 as well, there may be states that are suitable for remote operation and states that are not. If a first remote operator terminal 200-X that is in a state not suitable for remote operation is used, the accuracy of remote operation will decrease.
Therefore, in the first aspect, the assignment process is performed by taking the state of each remote operator O and the state of each remote operator terminal 200 into consideration. For this purpose, the state of each remote operator O and the state of each remote operator terminal 200 are quantified in the form of a “score.” The score is a quantitative parameter indicating suitability for remote operation. For example, the score is calculated in the range of 0 to 100. The higher the score, the higher the suitability for remote operation. Based on the scores, a suitable first remote operator O and a suitable first remote operator terminal 200-X are selected and assigned to remote operation of the target mobility 100-X. The first aspect will be described in more detail below.
An operator state acquisition unit 510 acquires operator state information STA-O indicating the state of the remote operator O. An operator score calculation unit 520 calculates an operator score SCR-O based on the operator state information STA-O. The operator score SCR-O indicates suitability of the remote operator O for remote operation.
In a first example, the operator state acquisition unit 510 acquires the health state of the remote operator O. That is, the state of the remote operator O includes the health state of the remote operator O. For example, the remote operator O reports his/her own health state using the remote operator terminal 200 or his/her own terminal (e.g., smartphone). As another example, the operator sensor 240 may be used to detect the health state of the remote operator O. The operator sensor 240 may be included in the remote operator terminal 200 (see
In a second example, the operator state acquisition unit 510 acquires the amount of working time the remote operator O was engaged in remote operation over a certain period in the past. That is, the state of the remote operator O includes the amount of working time the remote operator O was engaged in remote operation over a certain period in the past. For example, the amount of working time can be obtained from operation logs recorded in the remote operator terminals 200 that the remote operator O operated in the past. Alternatively, the amount of working time is obtained from the working history of each remote operator O included in the operator management information MGT-O. The operator score calculation unit 520 calculates a second operator score based on the amount of working time of the remote operator O. As shown in
In a third example, the operator state acquisition unit 510 acquires the hourly wage of the remote operator O. That is, the state of the remote operator O includes the hourly wage of the remote operator O. For example, the remote operator O registers his/her own hourly wage using the remote operator terminal 200 or his/her own terminal (e.g., smartphone). The hourly wage of each remote operator O may be registered in advance in the operator management information MGT-O and may be acquired from the operator management information MGT-O. The operator score calculation unit 520 calculates a third operator score based on the hourly wage of the remote operator O. As shown in
Two or more of the above first to third examples may be combined. In that case, the scores in the examples are added up.
Typically, the operator state acquisition unit 510 acquires in real time the operator state information STA-O indicating the state of the remote operator O. The operator score calculation unit 520 calculates the operator score SCR-O in real time based on the operator state information STA-O.
The operator state acquisition unit 510 and the operator score calculation unit 520 may be included in the remote operator terminal 200, may be included in the management system 300, or may be distributed between the remote operator terminal 200 and the management system 300. When the operator state acquisition unit 510 is included in the management system 300, the remote operator terminal 200 sends the biological information, information on the amount of working time, etc. of the remote operator O to the management system 300. When the operator state acquisition unit 510 and the operator score calculation unit 520 are included in the remote operator terminal 200, the remote operator terminal 200 need only send information on the operator score SCR-O to the management system 300. In this case, since there is no need to send the biological information, information on the amount of working time, etc. of the remote operator O to the management system 300, the amount of communication between the remote operator terminal 200 and the management system 300 is reduced. This contributes to reducing communication resources that are used.
A terminal state acquisition unit 530 acquires terminal state information STA-T indicating the state of the remote operator terminal 200. A terminal score calculation unit 540 calculates a terminal score SCR-T based on the terminal state information STA-T. The terminal score SCR-T indicates suitability of the remote operator terminal 200 for remote operation.
In a first example, the terminal state acquisition unit 530 acquires the communication state of the remote operator terminal 200. That is, the state of the remote operator terminal 200 includes the communication state of the remote operator terminal 200. Examples of the communication state include a transmission speed (throughput) and a transmission delay. The remote operator terminal 200 can measure the transmission speed, transmission delay, etc. based on the reception state of data received from a communication partner. As another example, the remote operator terminal 200 measures the transmission speed, transmission delay, etc. based on data the remote operator terminal 200 sent to the communication partner and feedback received from the communication partner. The terminal state acquisition unit 530 acquires the communication state measured by the remote operator terminal 200. The terminal score calculation unit 540 calculates a first terminal score based on the communication state of the remote operator terminal 200. As shown in
In a second example, the terminal state acquisition unit 530 acquires the degree of abnormality of the control device 250 (see
The above first and second examples may be combined. In that case, the scores in the first and second examples are added up.
Typically, the terminal state acquisition unit 530 acquires in real time the terminal state information STA-T indicating the state of the remote operator terminal 200. The terminal score calculation unit 540 calculates the terminal score SCR-T in real time based on the terminal state information STA-T.
The terminal state acquisition unit 530 and the terminal score calculation unit 540 may be included in the remote operator terminal 200, may be included in the management system 300, or may be distributed between the remote operator terminal 200 and the management system 300. When the terminal state acquisition unit 530 is included in the management system 300, the remote operator terminal 200 sends information on the communication state of the remote operator terminal 200 or the degree of abnormality of the control device 250 to the management system 300. When the terminal state acquisition unit 530 and the terminal score calculation unit 540 are included in the remote operator terminal 200, the remote operator terminal 200 need only send information on the terminal score SCR-T to the management system 300. In this case, since there is no need to send the communication state of the remote operator terminal 200 or the degree of abnormality of the control device 250 to the management system 300, the amount of communication between the remote operator terminal 200 and the management system 300 is reduced. This contributes to reducing communication resources that are used.
For example, the score integration unit 550 calculates the average value of the operator score SCR-O and the terminal score SCR-T. When the average value is less than 100, the average value is used as the integrated score SCR. When the average value is 100 or more, the integrated score SCR is always set to 100.
The score integration unit 550 may be included in the remote operator terminal 200, or may be included in the management system 300. The operator state acquisition unit 510, the operator score calculation unit 520, the terminal state acquisition unit 530, the terminal score calculation unit 540, and the score integration unit 550 may all be included in the remote operator terminal 200. In this case, the remote operator terminal 200 need only send information on the integrated score SCR to the management system 300. Therefore, the amount of communication between the remote operator terminal 200 and the management system 300 is reduced. This contributes to reducing communication resources that are used.
The score acquisition unit 410 acquires the operator scores SCR-O from the operator score calculation unit 520. More specifically, the score acquisition unit 410 acquires the operator scores SCR-O of the individual remote operators O indicated by the operator management information MGT-O. The score acquisition unit 410 may register the operator scores SCR-O of the individual remote operators O in the operator management information MGT-O. Typically, the score acquisition unit 410 acquires the operator scores SCR-O of the individual remote operators O in real time.
The selection unit 450 selects a suitable first remote operator O-X from the remote operators O, based on the operator scores SCR-O and the operator management information MGT-O. More specifically, the selection unit 450 extracts any available (free) remote operators O from the remote operators O, based on the operator management information MGT-O. The selection unit 450 then selects a first remote operator O-X from the available remote operators O, based on the operator scores SCR-O. For example, the selection unit 450 preferentially selects a remote operator O with a high operator score SCR-O as a first remote operator O-X. That is, the operator scores SCR-O are used to determine operator priority. When there is a plurality of candidates with the same operator score SCR-O, the selection unit 450 may preferentially select a remote operator O with a longer waiting time as a first remote operator O-X. The selected first remote operator O-X is assigned to remote operation of the target mobility 100-X.
The score acquisition unit 410 acquires the terminal scores SCR-T from the terminal score calculation unit 540. More specifically, the score acquisition unit 410 acquires the terminal scores SCR-T of the individual remote operator terminals 200 indicated by the terminal management information MGT-T. The score acquisition unit 410 may register the terminal scores SCR-T of the individual remote operator terminals 200 in the terminal management information MGT-T. Typically, the score acquisition unit 410 acquires the terminal scores SCR-T of the individual remote operator terminals 200 in real time.
The selection unit 450 selects a suitable first remote operator terminal 200-X from the remote operator terminals 200, based on the terminal scores SCR-T and the terminal management information MGT-T. More specifically, the selection unit 450 extracts any available (ready for use) remote operator terminals 200 from the remote operator terminals 200, based on the terminal management information MGT-T. The selection unit 450 then selects a first remote operator terminal 200-X from the available remote operator terminals 200, based on the terminal scores SCR-T. For example, the selection unit 450 preferentially selects a remote operator terminal 200 with a high terminal score SCR-T as a first remote operator terminal 200-X. That is, the terminal scores SCR-T are used to determine terminal priority. When there is a plurality of candidates with the same terminal score SCR-T, the selection unit 450 may preferentially select a remote operator terminal 200 with a longer waiting time as a first remote operator terminal 200-X. The selected first remote operator terminal 200-X is assigned to remote operation of the target mobility 100-X.
Alternatively, in the example shown in
The selection unit 450 selects a suitable combination of a first remote operator O-X and a first remote operator terminal 200-X from the combinations, based on the integrated scores SCR, the operator management information MGT-O, and the terminal management information MGT-T. More specifically, the selection unit 450 extracts any available (free) remote operators O from the remote operators O, based on the operator management information MGT-O. The selection unit 450 extracts any available (ready for use) remote operator terminals 200 from the remote operator terminals 200, based on the terminal management information MGT-T. The selection unit 450 then selects a first remote operator O-X from the available remote operators O and selects a first remote operator terminal 200-X from the available remote operator terminals 200, based on the integrated scores SCR. For example, the selection unit 450 preferentially selects a combination of a remote operator O and a remote operator terminal 200 with a high integrated score SCR as a combination of a first remote operator O-X and a first remote operator terminal 200-X. That is, the integrated scores SCR are used to determine priority. When there is a plurality of candidates with the same integrated score SCR, the selection unit 450 may preferentially select a combination including a remote operator O with a longer waiting time. The selected combination of the first remote operator O-X and the first remote operator terminal 200-X is assigned to remote operation of the target mobility 100-X.
As described above, according to the first aspect, the operator scores SCR-O indicating suitability of the individual remote operators O for remote operation are calculated based on the states of the individual remote operators O. A first remote operator O-X to be assigned to remote operation of the target mobility 100-X is selected based on the operator scores SCR-O. That is, a suitable first remote operator O-X is assigned to remote operation of the target mobility 100-X by taking the suitability into consideration. This ensures the accuracy of remote operation of the target mobility 100-X.
The terminal scores SCR-T indicating suitability of the individual remote operator terminals 200 for remote operation are also calculated based on the states of the individual remote operator terminals 200. A first remote operator terminal 200-X to be assigned to remote operation of the target mobility 100-X is selected based on the terminal scores SCR-T. That is, a suitable first remote operator terminal 200-X is assigned to remote operation of the target mobility 100-X by taking the suitability into consideration. This ensures the accuracy of remote operation of the target mobility 100-X.
The score calculation may be performed by each remote operator terminal 200. In this case, the remote operator terminal 200 need only send score information to the management system 300. Therefore, the amount of communication between the remote operator terminal 200 and the management system 300 is reduced. This contributes to reducing communication resources that are used.
It is herein assumed that there is a plurality of types of mobilities 100 for remote operation (see
Therefore, in the second aspect, the assignment process is performed by taking the licenses the individual remote operators O have into consideration. More specifically, a first remote operator O-X having the license required for remote operation of the target mobility 100-X is assigned to remote operation of the target mobility 100-X. The second aspect will be described in more detail below.
The operator license information LIC is included in the operator management information MGT-O. For example, each remote operator O reports the license(s) he/she has using the remote operator terminal 200 or his/her own terminal. The management system 300 collects information on the licenses the individual remote operators O have to generate the operator license information LIC in advance and update the operator license information LIC as needed.
The required license acquisition unit 420 receives a remote operation request REQ. The remote operation request REQ indicates the type of the target mobility 100-X. The required license acquisition unit 420 can recognize a “required license REQ-LIC” required for remote operation of the target mobility 100-X, based on the type of the target mobility 100-X.
Required capability information REQ-ABL indicates the capabilities required for remote operation of the target mobility 100-X. The required capability information REQ-ABL includes at least information of the required license REQ-LIC.
The selection unit 450 includes an operator selection unit 450-0. The operator selection unit 450-O selects a suitable first remote operator O-X from the remote operators O, based on the operator management information MGT-O and the required capability information REQ-ABL. More specifically, the operator selection unit 450-0 extracts any available (free) remote operators O from the remote operators O, based on the operator management information MGT-O. The operator selection unit 450-O also acquires the operator license information LIC included in the operator management information MGT-O, and the information on the required license REQ-LIC included in the required capability information REQ-ABL. The operator selection unit 450-O then selects a first remote operator O-X having the required license REQ-LIC from the available remote operators O, based on the operator license information LIC and the required license REQ-LIC. The selected first remote operator O-X is assigned to remote operation of the target mobility 100-X.
When there is a plurality of remote operators O who has the required license REQ-LIC, the operator selection unit 450-O may select a first remote operator O-X by taking the operator scores SCR-O described above in Section 4 into consideration. More specifically, the operator selection unit 450-O may preferentially select a remote operator O with a high operator score SCR-O as a first remote operator O-X. When there is a plurality of candidates with the same operator score SCR-O, the operator selection unit 450-O may preferentially select a remote operator O with a longer waiting time as a first remote operator O-X.
As another example, when there is a plurality of remote operators O who has the required license REQ-LIC, the operator selection unit 450-O may select a remote operator O having the fewest types of licenses as a first remote operator O-X for this time. In this case, the remote operators O having licenses for more types of mobilities 100 are saved for later. This increases the probability that a suitable remote operator O can be assigned to the subsequent target mobility 100-X for which remote operation is requested.
As described above, according to the second aspect, the licenses the individual remote operators O have and the required license REQ-LIC required for remote operation of the target mobility 100-X are considered. A first remote operator O-X having the required license REQ-LIC is assigned to remote operation of the target mobility 100-X. This ensures the accuracy of remote operation of the target mobility 100-X. Moreover, this avoids a situation in which a remote operator O who does not have the required license REQ-LIC is assigned to the target mobility 100-X by mistake. This eliminates the need to start over the assignment process, and implements an efficient assignment process.
The specifications required for a remote operator terminal 200 may vary depending on the type of the target mobility 100-X. The specifications required for a remote operator terminal 200 may also vary depending on the content of the requested remote operation. If a remote operator terminal 200 that does not meet the specifications required for remote operation of the target mobility 100-X is assigned to remote operation of the target mobility 100-X, the accuracy of remote operation will decrease. Moreover, it is inefficient to start over the assignment process.
Therefore, in the third aspect, the assignment process is performed by taking the specifications of the individual remote operator terminals 200 into consideration. More specifically, a first remote operator terminal 200-X that meets the required terminal specifications required for remote operation of the target mobility 100-X is assigned to remote operation of the target mobility 100-X. The third aspect will be described in more detail below.
Examples of the types of remote operator terminals 200 include a cockpit-type terminal, a personal computer (PC), a tablet, and a smartphone. If the type of remote operator terminal 200 changes, the type(s) of mobility (ies) 100 supported by the remote operator terminal 200 and the content of remote operation(s) supported by the remote operator terminal 200 also change. For example, vehicle cockpits are suitable for remote operation of various types of vehicles, and are also suitable for driving on public roads and long distance driving. Tablets are not necessarily suitable for driving on public roads or long distance driving. Tablets can be used for driving a vehicle a short distance in a limited area such as a parking lot, for automated driving assist such as safely pulling over to a road shoulder in case of emergency, for remote operation of a small vehicle in a factory, etc.
The terminal specification information SPC is included in the terminal management information MGT-T. For example, the management system 300 collects information on the type of remote operator terminal from each of the remote operator terminals 200. The management system 300 generates in advance the terminal specification information SPC illustrated in
The required terminal specification acquisition unit 430 receives a remote operation request REQ. The remote operation request REQ indicates either or both of the “type of the target mobility 100-X” and the “content of desired remote operation of the target mobility 100-X.” The required terminal specification acquisition unit 430 can recognize “required terminal specifications REQ-SPC” required for remote operation of the target mobility 100-X, based on the remote operation request REQ. In this example, the remote operation request REQ is used as it is as the required terminal specifications REQ-SPC.
Required capability information REQ-ABL indicates the capabilities required for remote operation of the target mobility 100-X. The required capability information REQ-ABL includes at least information on the required terminal specifications REQ-SPC.
The selection unit 450 includes a terminal selection unit 450-T. The terminal selection unit 450-T selects a suitable first remote operator terminal 200-X from the remote operator terminals 200, based on the terminal management information MGT-T and the required capability information REQ-ABL. More specifically, the terminal selection unit 450-T extracts any available (ready for use) remote operator terminals 200 from the remote operator terminals 200, based on the terminal management information MGT-T. The terminal selection unit 450-T acquires the terminal specification information SPC contained in the terminal management information MGT-T, and the information on the required terminal specifications REQ-SPC included in the required capability information REQ-ABL. The terminal selection unit 450-T then selects a first remote operator terminal 200-X that meets the required terminal specifications REQ-SPC from the available remote operator terminals 200, based on the terminal specification information SPC and the required terminal specifications REQ-SPC. The selected first remote operator terminal 200-X is assigned to remote operation of the target mobility 100-X.
When there is a plurality of remote operator terminals 200 that meets the required terminal specifications REQ-SPC, the terminal selection unit 450-T may select a first remote operator terminal 200-X by taking the terminal scores SCR-T described above in Section 4 into consideration. More specifically, the terminal selection unit 450-T may preferentially select a remote operator terminal 200 with a high terminal score SCR-T as a first remote operator terminal 200-X. When there is a plurality of candidates with the same terminal score SCR-T, the terminal selection unit 450-T may preferentially select a remote operator terminal 200 with a longer waiting time as a first remote operator terminal 200-X.
Dedicated remote operator terminals 200 (remote cockpits) may be prepared for the individual types of mobilities 100. However, it is preferable from the standpoints of cost reduction, reduction in installation area, efficiency, convenience, etc. if various types of mobilities 100 can be remotely operated by a single remote operator terminal 200. A second example considers the case where various types of mobilities 100 are remotely operated by a single remote operator terminal 200. For this purpose, the specifications (equipment) of the remote operator terminals 200 are redefined in a more abstract way. More specifically, as will be described below, the specifications (equipment) of the remote operator terminals 200 are defined by “operation system capabilities” and “display system capabilities.” The same description as in the first example will be omitted as appropriate.
The input device 230 (see
For example, forward, reverse, and lateral operations of a joystick are associated with an accelerator pedal operation, a brake pedal operation, and a steering wheel operation, respectively. As another example, the operation systems of various types of mobilities 100 can be replicated by a user interface displayed on a touch panel. In any case, it is possible to convert the amount of operation of one remote operation member to the amount of operation of another remote operation member. Therefore, another remote operation member can be substituted for one remote operation member.
The operation system capabilities of the remote operator terminals 200 are defined in view of the above. Specifically, the operation system capabilities of the remote operator terminals 200 include the “number of inputs for a forward and reverse operation of the mobility 100” and the “number of inputs for a lateral operation of the mobility 100.” The operation system capabilities of the remote operator terminals 200 may further include “presence or absence of operation feedback.” The presence or absence of operation feedback means whether the remote operator terminal 200 includes an operation reaction force mechanism that can generate an operation reaction force to the remote operator O.
The required terminal specifications REQ-SPC include the operation system capabilities required for remote operation of the target mobility 100-X. As shown in
The required terminal specification acquisition unit 430 receives a remote operation request REQ. The remote operation request REQ indicates either or both of the “type of the target mobility 100-X” and the “content of desired remote operation of the target mobility 100-X.” The required terminal specification acquisition unit 430 can recognize required terminal specifications REQ-SPC including the operation system capabilities required for remote operation of the target mobility 100-X, based on the remote operation request REQ.
For example, a conversion table 435 showing correspondence between remote operation requests REQ and required terminal specifications REQ-SPC is prepared in advance (see
The terminal selection unit 450-T acquires the terminal specification information SPC and the information on the required terminal specifications REQ-SPC. The terminal selection unit 450-T then selects a first remote operator terminal 200-X that meets the required terminal specifications REQ-SPC from the available remote operator terminals 200, based on the terminal specification information SPC and the required terminal specifications REQ-SPC. More specifically, the terminal selection unit 450-T selects a first remote operator terminal 200-X that has at least the required operation system capabilities, based on the terminal specification information SPC (operation system information) and the required terminal specifications REQ-SPC. The selected first remote operator terminal 200-X is assigned to remote operation of the target mobility 100-X.
When there is a plurality of remote operator terminals 200 that has the required operation system capabilities, the terminal selection unit 450-T may select a first remote operator terminal 200-X by taking the terminal scores SCR-T described above in Section 4 into consideration. More specifically, the terminal selection unit 450-T may preferentially select a remote operator terminal 200 with a high terminal score SCR-T as a first remote operator terminal 200-X. When there is a plurality of candidates with the same terminal score SCR-T, the terminal selection unit 450-T may preferentially select a remote operator terminal 200 with a longer waiting time as a first remote operator terminal 200-X.
As another example, when there is a plurality of remote operator terminals 200 that meets the required operation system capabilities, the terminal selection unit 450-T may select a remote operator terminal 200 having minimum operation system capabilities as a first remote operator terminal 200-X for this time. In this case, the remote operator terminals 200 having higher operation system capabilities are saved for later. This increases the probability that a suitable remote operator terminal 200 can be assigned even when high operation system capabilities are required for remote operation of the subsequent target mobility 100-X.
As a modification, under a specific situation, the terminal selection unit 450-T may select a remote operator terminal 200 that falls slightly short of the required operation system capabilities as a first remote operator terminal 200-X. The specific situation may be when there is no other option in case of emergency. For example, even when a remote operator terminal 200 with operation feedback is preferred, a remote operator terminal 200 with no operation feedback may be selected as a first remote operator terminal 200-X in case of emergency. In this case, however, the terminal selection unit 450-T instructs the target mobility 100-X and the first remote operator terminal 200-X to restrict functions. Examples of the restriction on the functions include lowering the upper speed limit to a limit value lower than a default value, and narrowing the steering range to a limit range narrower than a default range. Instructions to the target mobility 100-X and the first remote operator terminal 200-X are given through communication.
There are various types of display devices 220 (see
The required terminal specifications REQ-SPC include the display system capabilities required for remote operation of the target mobility 100-X. As shown in
The required terminal specification acquisition unit 430 receives a remote operation request REQ. The remote operation request REQ indicates either or both of the “type of the target mobility 100-X” and the “content of desired remote operation of the target mobility 100-X.” The required terminal specification acquisition unit 430 can recognize required terminal specifications REQ-SPC including the display system capabilities required for remote operation of the target mobility 100-X, based on the remote operation request REQ.
For example, the conversion table 435 showing correspondence between remote operation requests REQ and required terminal specifications REQ-SPC is prepared in advance (see
The terminal selection unit 450-T acquires the terminal specification information SPC and the information on the required terminal specifications REQ-SPC. The terminal selection unit 450-T then selects a first remote operator terminal 200-X that meets the required terminal specifications REQ-SPC from the available remote operator terminals 200, based on the terminal specification information SPC and the required terminal specifications REQ-SPC. More specifically, the terminal selection unit 450-T selects a first remote operator terminal 200-X that has at least the required display system capabilities, based on the terminal specification information SPC (display system information) and the required terminal specifications REQ-SPC. The selected first remote operator terminal 200-X is assigned to remote operation of the target mobility 100-X.
When there is a plurality of remote operator terminals 200 that has the required display system capabilities, the terminal selection unit 450-T may select a first remote operator terminal 200-X by taking the terminal scores SCR-T described above in Section 4 into consideration. More specifically, the terminal selection unit 450-T may preferentially select a remote operator terminal 200 with a high terminal score SCR-T as a first remote operator terminal 200-X. When there is a plurality of candidates with the same terminal score SCR-T, the terminal selection unit 450-T may preferentially select a remote operator terminal 200 with a longer waiting time as a first remote operator terminal 200-X.
As another example, when there is a plurality of remote operator terminals 200 that meets the required display system capabilities, the terminal selection unit 450-T may select a remote operator terminal 200 having minimum display system capabilities as a first remote operator terminal 200-X for this time. In this case, the remote operator terminals 200 having higher display system capabilities are saved for later. This increases the probability that a suitable remote operator terminal 200 can be assigned even when high display system capabilities are required for remote operation of the subsequent target mobility 100-X.
As a modification, under a specific situation, the terminal selection unit 450-T may select a remote operator terminal 200 that falls slightly short of the required display system capabilities as a first remote operator terminal 200-X. The specific situation may be when there is no other option in case of emergency. For example, even when two types of auxiliary videos are preferred to be displayed in addition to three types of necessary videos, a remote operator terminal 200 that can display only three types of videos may be selected as a first remote operator terminal 200-X in case of emergency. In this case, however, the terminal selection unit 450-T instructs the target mobility 100-X and the first remote operator terminal 200-X to restrict functions. Examples of the restriction on the functions include lowering the upper speed limit to a limit value lower than a default value, and narrowing the steering range to a limit range narrower than a default range. Instructions to the target mobility 100-X and the first remote operator terminal 200-X are given through communication.
The remote operator O can customize the video display mode as desired according to the content of the remote operation request REQ. The video display mode includes layout, a video display position, a video size, and whether to display a video(s). The remote operator O can set whether to display an auxiliary image(s) as desired. The auxiliary images include lane markings, a trajectory line, a line representing a maximum turning radius, and a safe stopping position.
As described above, according to the third aspect, the specifications of the individual remote operator terminals 200 and the required terminal specifications REQ-SPC required for remote operation of the target mobility 100-X are considered. A first remote operator terminal 200-X that meets the required terminal specifications REQ-SPC is assigned to remote operation of the target mobility 100-X. This ensures the accuracy of remote operation of the target mobility 100-X. This also avoids a situation in which a remote operator terminal 200 that does not have the required terminal specifications REQ-SPC is assigned to the target mobility 100-X by mistake. This eliminates the need to start over the assignment process, and implements an efficient assignment process.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2023-209379 | Dec 2023 | JP | national |
