Patent Grant
12108996
The disclosure relates to, for example, a method of remotely supporting a surgery assistant robot.
In recent years, surgery assistant robots have been used in surgery such as laparoscopic surgery. The necessity to provide detailed supports for the surgery assistant robots has been increasing. The specification of United States Patent Application Publication No. 2012/0330613 (“Patent Literature 1”) discloses a method of monitoring a medical robot for preventive maintenance.
In remote support for surgery assistant robots by conventional methods including the method disclosed in Patent Literature 1, log data (text data) concerning an operation state of a surgery assistant robot is collected and analyzed by a remote server. When a trouble occurs in the surgery assistant robot, a person in charge in a call center receives telephone communication from a medical facility and grasps details of the trouble while viewing an analysis result of the log data of the surgery assistant robot. However, the details of the trouble cannot be grasped in some case. In that case, a serviceperson S sometimes has to visit a site (for example, an operation room where the surgery assistant robot is installed). In this case, there is a problem in terms of quickness due to a time required from the occurrence of the trouble until solution of the trouble.
One or more aspects have been devised in view of the problem described above, and aim to provide a remote support method and a remote support system that, in a site where surgery or the like using a surgery assistant robot is performed, can grasp details of a trouble in a short time and quickly perform appropriate remote support.
A method according to one or more embodiments may be for remotely supporting a surgery assistant robot. The method may include: receiving at least one piece of operation information concerning an operation of the surgery assistant robot, by a server device that performs a remote support for the surgery assistant robot; and transmitting, in response to a predetermined event, at least one of a sound, an image, or a text from the server device to at least one of the surgery assistant robot or a terminal device.
A method according to one or more embodiments may be for remotely supporting a surgery assistant robot. The method may include: adjusting a frame rate of an image photographed by at least one of a surgery assistant robot or a camera that photographs an inside of a room in which the surgery assistant robot is installed; and transmitting the image, the frame rate of which is adjusted, to a server device that performs a remote support for the surgery assistant robot.
A method according to one or more embodiments may be for remotely supporting a surgery assistant robot. The method may include: processing an image photographed by at least one of a surgery assistant robot or a camera that photographs an inside of a room where the surgery assistant robot is installed, by image processing of visually obscuring identifiable information that enables a person to be identified; and transmitting an image obtained by the image processing to a server device that performs a remote support for the surgery assistant robot.
A remote support system according to one or more embodiments may include: a server device that performs a remote support for a surgery assistant robot; and a communication control device that communicates with the server device to transmit operation information concerning an operation of the surgery assistant robot to the server device and to receive information from the server device. In response to a predetermined event, the server device may perform processing of establishing bidirectional communication between at least one of the surgery assistant robot or a first terminal device used in a facility in which the surgery assistant robot is installed and a second terminal device that performs a remote support for the surgery assistant robot.
A method of remotely supporting a surgery assistant robot according to one or more aspects includes: receiving operation information concerning the operation of a surgery assistant robot (20) by a server device (40) that performs a remote support for the surgery assistant robot (20); and, when a predetermined event is detected, transmitting at least one of a sound, an image, or a text from the server device (40) to at least one of the surgery assistant robot (20) or a terminal device (25).
A method of remotely supporting a surgery assistant robot according to one or more aspects includes: adjusting a frame rate of an image photographed by at least one of a surgery assistant robot (20) or a camera (24) that photographs the inside of a room where the surgery assistant robot (20) is installed; and transmitting the image, the frame rate of which is adjusted, to a server device (40) that performs a remote support for the surgery assistant robot (20).
A method of remotely supporting a surgery assistant robot according to one or more aspects includes: processing an image photographed by at least one of a surgery assistant robot (20) or a camera (24) that photographs the inside of a room where the surgery assistant robot (20) is installed, by image processing of visually obscuring identifying information which enables a person to be identified; and transmitting the image processed by the image processing to a server device (40) that performs a remote support for the surgery assistant robot (20).
A remote support system according to one or more aspects includes: a server device (40) that performs a remote support for a surgery assistant robot (20); and a communication control device (261) that communicates with the server device (40) in order to transmit operation information concerning the operation of the surgery assistant robot (20) to the server device (40) and receive information from the server device (40). When detecting a predetermined event, the server device (40) performs processing of establishing bidirectional communication between at least one of the surgery assistant robot (20) or a first terminal device (25) used in a facility where the surgery assistant robot (20) is installed and a second terminal device (42) that performs a remote support for the surgery assistant robot (20).
According to one or more aspects, it is possible to, in a site where surgery or the like using a surgery assistant robot is performed, grasp details of a trouble in a short time and quickly perform appropriate remote support.
An embodiment is explained below with reference to
An embodiment is explained with reference to
It is assumed that the call center 4 performs a remote support job for medical-related facilities 2. In
A major characteristic of an embodiment is that, whereas, at normal time, only communication in one direction in which at least one of operation information (explained below) of the surgery assistant robot 20 of the medical-related facility 2 and an image photographed by a network camera 24 is transmitted to a server device 40 of the call center 4 is performed, when a predetermined event (explained below) occurs, bidirectional communication becomes possible between a terminal device 25 of the call center 4 and a device (at least one of the surgery assistant robot 20, the terminal device 25, and the network camera 24) of the medical-related facility 2. In other words, a major characteristic of an embodiment is that, whereas, at normal time, only communication in one direction in which at least one of operation information (explained below) of the surgery assistant robot 20 of the medical-related facility 2 or an image photographed by a network camera 24 is transmitted to a server device 40 of the call center 4 is performed, when a predetermined event (explained below) occurs, bi-directional communication becomes possible between a terminal device 25 of the call center 4 and a device (at least one of the surgery assistant robot 20, the terminal device 25, or the network camera 24) of the medical-related facility 2.
A flow of processing until the bidirectional communication becomes possible is explained with reference to
Subsequently, as an example of an event, when the surgery assistant robot 20 receives operation for requesting the bidirectional communication from the medical stall M (S17), a request for establishment of the bidirectional communication is transmitted from the surgery assistant robot 20 to the server device 40 via the communication control device 261 (S18 to S19). When the request is received by the server device 40 (S20), an inquiry about possibility of the bidirectional communication is transmitted from the server device 40 to a terminal device 42 of the call center 4 (S21). The terminal device 42 outputs the received inquiry to a monitor (S22). When a response to the inquiry is input from the serviceperson S who uses the terminal device 42, the terminal device 42 transmits the response to the server device 40 (S23 to S24). When the response is a response permitting the bidirectional communication (YES in S25), the server device 40 transmits a notification to the effect that the bi-directional communication is permitted (hereinafter referred to as “permission notification”) to the surgery assistant robot 20 that transmits the request (S26 to S28). In this way, the request for establishment of the bidirectional communication is transmitted (S18) and, at a stage when the permission notification is received as the response to the request (S28), the bidirectional communication is established.
After the bidirectional communication is established, data (for example, at least one of a sound, an image, and a text) input to the terminal device 42 and transmitted to the server device 40 is transmitted to the surgery assistant robot 20 via the server device 40 and the communication control device 261 (S29 to S32). In other words, data (for example, at least one of a sound, an image, or a text) input to the terminal device 42 and transmitted to the server device 40 is transmitted to the surgery assistant robot 20 via the server device 40 and the communication control device 261 (S29 to S32). Data received by the server device 40 in a period when the bidirectional communication is established is stored in the storage 41 as a log during the bidirectional communication (S33).
Note that, in
Each of components for realizing a remote support method according to an embodiment is explained with reference to
Medical-related Facility 2
A LAN (local area network) (hereinafter referred to as “intra-facility LAN”) is provided in the medical-related facility 2. The intra-facility LAN is communicably connected to an external network 3. The surgery assistant robot 20, the terminal device 25, the network camera 24, and the communication control device 261 are communicably connected to the intra-facility LAN. Note that the communication control device 261 is a main device configuring a network service system 26.
Surgery Assistant Robot 20
The surgery assistant robot 20 is installed in an operation room (a room) of the medical-related facility 2. The surgery assistant robot 20 includes a patient-side device 21 to which medical instruments are attached, a remote operation device 22 that remotely operates the medical instruments attached to the patient-side device 21, and an image processing device 23 that processes an image photographed by an endoscope 201b, which is one of the medical instruments. The surgery assistant robot 20 includes a system linkage device 27 that mediates communication between the patient-side device 21, the remote operation device 22, and the image processing device 23 and the communication control device 261.
When an operation command for operation that should be executed by the patient-side device 21 is input to the remote operation device 22 by the medical staff M, the remote operation device 22 transmits the operation command to a controller 206 of the patient-side device 21. The controller 206 is a computer that performs operation control of the patient-side device 21. The patient-side device 21 operates, in response to the operation command transmitted from the remote operation device 22, medical instruments such as a surgical instrument and an endoscope respectively gripped by surgical manipulators 201. Consequently, minimally invasive surgery is performed.
The patient-side device 21 is arranged besides an operating table 300 on which a patient P lies. One (a first manipulator) of the surgical manipulators 201 included in the patient-side device 21 grips the endoscope 201b. The other surgical manipulators 201 (second manipulators) grip instruments 201a, which are surgical instruments. The surgical manipulators 201 that grip the instruments 201a function as instrument arms 201A. The surgical manipulator 201 that grips the endoscope 201b functions as a camera arm 201B. The instrument arms 201A and the camera arm 201B are supported by a platform 203. The surgical manipulators 201 include joints. Driving units including servomotors and position detectors such as encoders are provided in the respective joints. The surgical manipulators 201 are configured to be controlled such that medical instruments attached to the surgical manipulators 201 perform desired operations according to driving signals given via the controller 206. The platform 203 is supported by a positioner 202 placed on a floor of a room (typically an operation room) in which the surgery assistant robot 20 is installed. The positioner 202 is configured to, after being positioned with respect to the operating table 300 on which the patient P lies, move the platform 203 and the surgical manipulators 201 to positions where surgery preparation is performed. Movement of the positioner 202, the platform 203, and the surgical manipulators 201 is performed according to an input via a touch panel 207 provided in the patient-side device 21.
The instruments 201a are detachably attached to the distal end portions of the instrument arms 201A. The instruments 201a include housings attached to the instrument arms 201A and end effectors provided at the distal end portions of shafts having an elongated shape. Examples of the end effectors include gripping forceps, scissors, hooks, high-frequency knives, snare wires, clamps, and staplers. However, the end effectors are not limited to these. Various treatment instruments can be applied. The instrument arms 201A are introduced into the body of the patient P via a trocar placed on the body surface of the patient P. The end effectors of the instruments 201a are positioned near a surgical site.
The endoscope 201b is detachably attached to the distal end portion of the camera arm 201B. The camera arm 201B is introduced into the body of the patient P via the trocar placed on the body surface of the patient P. Consequently, the endoscope 201b is positioned near a surgical site and photographs the inside of the body cavity of the patient P. An image photographed by the endoscope 201b is processed by image processing by the image processing device 23 and then displayed on a monitor 221 of the remote operation device 22. The medical staff M inputs an operation command to the remote operation device 22 while viewing the image displayed on the monitor 221.
The image photographed by the endoscope 201b is processed by the image processing by the image processing device 23 and then transmitted from the system linkage device 27 to the server device 40 via the communication control device 261. As explained below, in transmitting the image to the server device 40, from the viewpoint of personal information protection, it is desirable that mask processing of masking a part of an image region can be applied by the communication control device 261 and transmission can be limited according to necessity.
The image photographed by the endoscope 201b may be processed by the image processing by the image processing device 23 and then displayed on a monitor 231 connected to the image processing device 23. Consequently, the medical staff M not operating the remote operation device 22 can also view the image photographed by the endoscope 201b.
The remote operation device 22 is a device for the medical staff M to operate the medical instruments attached to the surgical manipulators 201. An operation handle 233, an operation pedal unit 234, and the monitor 221 are provided in the remote operation device 22. The operation handle 233 is provided for the medical staff M to remotely operate the medical instruments supported by the surgical manipulators 201. The operation handle 233 is configured from a handle for right hand operated by the right hand of the medical staff M and a handle for left hand operated by the left hand of the medical staff M. For example, the medical staff M can operate the instrument 201a of the instrument arm 201A with the handle for right handle and operate the instrument 201a of another instrument arm 201A with the handle for left hand. Operation pedals are provided in the operation pedal unit 234. An operation pedal for executing a function of the instrument 201a operated by the handle for right hand, an operation pedal for executing a function of the instrument 201a operated by the handle for left hand, an operation pedal for operating the camera arm 201B with the operation handle, and the like are provided. That is, the remote operation device 22 is configured to be capable of transmitting, to the controller 206 of the patient-side device 21, an operation command for operation, which should be executed by the instrument 201a and the endoscope 201b, input by the medical staff M using the operation handle 233 and the operation pedal unit 234.
The system linkage device 27 is a computer and has functions that the computer usually has. The system linkage device 27 is communicably connected to the intra-facility LAN. The system linkage device 27 mediates communication between the patient-side device 21, the remote operation device 22, and the image processing device 23 and the communication control device 261. The system linkage device 27 is mounted with an API (application program interface). According to a request from the communication control device 261, the system linkage device 27 acquires information concerning the operation of the surgery assistant robot 20 (hereinafter referred to as “operation information”) from the patient-side device 21, the remote operation device 22, and the image processing device 23 by using the API and transmits the acquired operation information to the communication control device 261. Timing when the system linkage device 27 transmits the operation information to the communication control device 261 may be immediately after the acquisition or may be after a certain degree of data amount is accumulated. Typical examples of the operation information are described in (1) to (13) below.
(1) to (13) described above are illustrations. The operation information is not limited to these.
Note that, at a stage when the predetermined event is not detected by the server device 40, in transmitting the operation information from the surgery assistant robot 20 to the server device 40, unidirectional communication is established between the surgery assistant robot 20 and the server device 40.
Network Service System 26
The network service system 26 provides various network services to the devices in the medical-related facility 2. Therefore, various APIs are prepared in the network service system 26.
The network service system 26 includes the communication control device 261. The communication control device 261 is a computer and has functions that the computer usually has. As illustrated in
The data acquiring unit 263 acquires operation information of the surgery assistant robot 20. This function is realized by invoking an API mounted on the system linkage device 27. The data acquiring unit 263 acquires an image photographed by the network camera 24.
An overview of processing performed by the data acquiring unit 263 is explained with reference to
The data processing unit 264 has functions described below.
(Function 1) Mask of Personal Information
When an image photographed by the network camera 24 and an image photographed by the endoscope 201b are transmitted via the communication control device 261, if information that can specify a person (the face of the person or the like) (hereinafter referred to as “identifiable information”) is photographed, it is desirable to apply image processing of masking the identifiable information (processing of visually obscuring the identifiable information, for example, applying mosaic) from the viewpoint of personal information protection. Therefore, the data processing unit 264 inspects, sees, checks, or determines whether the identifiable information is photographed in the image to be transmitted, and processes the image by the image processing of masking the identifiable information when the identifiable information is photographed.
(Function 2) Adjustment of a Frame Rate
As a data size of a video transmitted to the server device 40 via the communication control device 261 is larger with respect to line speed of the intra-facility LAN, a more time is required for the video transmission, causing a delay. Therefore, the data processing unit 264 adjusts the number of frames per second (a frame rate) of the video to be transmitted to be small (that is, thins out frames). Specifically, the data processing unit 264 adjusts the number of frames per second to six. Note that the data processing unit 264 may monitor communication loads of the intra-facility LAN and the external network 3 and adjust the number of frames per second of the video to be small according to the communication loads.
(Function 3) Analysis of a Video and Transmission of an Analysis Result
Under the condition where a video photographed by the network camera 24 is transmitted to the server device 40 via the communication control device 261, the operation of various medical systems connected to the intra-facility LAN may be affected if a communication load is applied to the intra-facility LAN. Therefore, the data processing unit 264 analyzes a movement of a person photographed in the video to be transmitted. A transmission target to the server device 40 by the data transmitting unit 265 is only text data obtained as an analysis result. Consequently, it is possible to reduce a communication load of the intra-facility LAN compared with when the video is transmitted. “Analysis of a movement of a person” indicates that, for example, an AI (artificial intelligence) mounted on the communication control device 261 replaces a movement of a person photographed in an analysis target video with a skeleton (a bar-like model) and converts a movement of a joint of the skeleton into a numerical value.
(Function 4) Processing (Summarization) of Text Data Such as Output Values of Various Sensors
The data processing unit 264 may summarize, for example, text data such as output values of various sensors acquired at an interval of one second from the surgery assistant robot 20 into an average per hour, an average per day, or the like. Consequently, in transmission to the server device 40 by the data transmitting unit 265, a transmission amount and a transmission frequency per one time are reduced. Therefore, it is possible to reduce a communication load of the external network 3.
An overview of processing performed by the data processing unit 264 is explained with reference to
The data transmitting unit 265 transmits the data acquired by the data acquiring unit 263 and the data processed by the data processing unit 264 to the server device 40. Further, the data transmitting unit 265 may be configured to be able to limit the transmission of the data. Specifically, the data transmitting unit 265 does not transmit only a sound included in a video or does not transmit both of the video and the sound. Consequently, it is possible to avoid unnecessary data transmission.
An overview of processing performed by the data transmitting unit 265 is explained with reference to
[Terminal Device 25]
The terminal device 25 is a computer terminal used by the medical staff M in the room where the surgery assistant robot 20 is installed. Typical examples of the terminal device 25 are a personal computer, a tablet terminal, a smartphone, and the like. The terminal device 25 has functions that the computer terminal usually has such as a communication function with other devices, a data input function of a keyboard and the like, a data display function of a monitor and the like, a sound input function of a microphone and the like, and a sound output function of a speaker and the like.
The terminal device 25 is communicably connected to the intra-facility LAN and accesses the network service system 26 via a Web application. For example, the terminal device 25 acquires, via the communication control device 261, data (for example, operation information of the surgery assistant robot 20) stored in the storage 41 of the call center 4 and displays the acquired data on the monitor.
The terminal device 25 is capable of receiving, from the medical staff M, operation for requesting the bidirectional communication with the terminal device 42. The terminal device 25 transmits a request corresponding to the operation to the server device 40. Examples of the operation are click of a button displayed on the monitor and a sound input to the microphone. However, the operation is not limited to these.
The terminal device 25 is capable of executing various functions for performing bidirectional communication (for example, voice call or audio call, video call, and chat) with the terminal device 42 with which the bidirectional communication with the server device 40 is established. That is, the terminal device 25 transmits data input to perform the bidirectional communication (hereinafter referred to as “communication data”) to the server device 40 via the communication control device 261 in order to share the data with the terminal device 42. On the other hand, the terminal device 25 receives communication data, which is input to the terminal device 42, from the server device 40 via the communication control device 261 and outputs the communication data. Examples of the communication data are, for example, when the voice call is performed, sound, when the video call is performed, video with sound, when the chat is performed, text data, and, when data sharing using a whiteboard function is performed, graphics drawn by a paint tool. However, the communication data is not limited to these.
[Network Camera 24]
The network camera 24 is a photographing device that is installed in the room where the surgery assistant robot 20 is installed and photographs a state of the room as a video or a still image. The video is desirably a video with a sound. A photographing target may be the entire inside of the room, may be the patient-side device 21, or may be a specific medical staff M. The network camera 24 is desirably capable of optionally adjusting tilt and zoom according to an instruction from the outside.
An image photographed by the network camera 24 is transmitted to the server device 40 via the communication control device 261. As explained above, in transmitting the image to the server device 40, from the viewpoint of personal information protection, it is desirable that image processing of, for example, masking a part (for example, the face of the patient P) of an image region can be applied by the communication control device 261 and transmission can be limited according to necessity. In transmitting a video photographed by the network camera 24 to the server device 40, a frame rate of the video may be adjusted in the communication control device 261.
[Call Center 4]
The call center 4 is a facility different from the medical-related facility 2 and is a facility for performing a remote support job for the medical-related facility 2. A LAN (hereinafter referred to as “intra-call center LAN”) is provided in the call center 4. The intra-call center LAN is communicably connected to the external network 3. The server device 40, the storage 41, and the terminal device 42 are communicably connected to the intra-call center LAN.
[Server Device 40]
The server device 40 is a computer and has functions that the computer usually has. The server device 40 is communicably connected to the intra-call center LAN. As illustrated in
[Data Storing Unit 401]
The data storing unit 401 stores, in the storage 41, data transmitted from the surgery assistant robot 20, the network camera 24, the terminal device 25, and the terminal device 42 to the server device 40. Typical examples of the data stored in the storage 41 by the data storing unit 401 are (1) to (5) described below.
In a period with which the bidirectional communication is established, the data storing unit 401 stores, in the storage 41, as a log for the bidirectional communication, data transmitted from a device with which the bidirectional communication is established to the server device 40.
An overview of processing performed by the data storing unit 401 is explained with reference to
[Communication Control Unit 402]
(Communication Establishment Processing)
The communication control unit 402 controls communication between the server device 40 and a device communicably connected to the server device 40. The communication control unit 402 performs processing of establishing the unidirectional communication and the bidirectional communication.
An overview of establishment processing of the bidirectional communication by the communication control unit 402 is explained with reference to
Note that the communication control unit 402 is capable of establishing bidirectional communications in parallel. Specifically, the communication control unit 402 is capable of establishing the bidirectional communication between surgery assistant robots 20 set in the same medical-related facility 2 and the terminal device 42. The communication control unit 402 is also capable of establishing the bi-directional communication between the surgery assistant robots 20 set in different medical-related facilities 2 and the terminal device 42.
A trigger for the communication control unit 402 to perform the processing of establishing the bidirectional communication is detection of the predetermined event. Typical examples of the predetermined event are described in (event example 1) to (event example 5) below.
(Event example 1) A request transmitted from the surgery assistant robot 20 or the terminal device 25 according to operation for requesting the bi-directional communication is received by the server device 40.
The request is transmitted from the surgery assistant robot 20 or the terminal device 25 in which operation for requesting the bidirectional communication is performed by the medical staff M to the server device 40. When some abnormality such as a failure or an error occurs in the surgery assistant robot 20, it is highly likely that the medical staff M needs support of the serviceperson S. Therefore, it is useful to establish the bidirectional communication between the server device 40 and at least one of the surgery assistant robot 20 and the terminal device 25 and between the server device 40 and the terminal device 42. In other words, it is useful to establish the bidirectional communication between the server device 40 and at least one of the surgery assistant robot 20 or the terminal device 25 and between the server device 40 and the terminal device 42.
An example of the operation performed by the medical staff M is operation on an operation unit communicably connected to the surgery assistant robot 20 or an operation unit provided in the surgery assistant robot 20. For example, the operation performed by the medical staff M is pressing of a switch 222 provided near the remote operation device 22 (for example, at hand of the medical staff M operating the remote operation device 22) or touch on a button displayed on the touch panel 207 provided in the patient-side device 21. The former is mainly performed by the medical staff M operating the remote operation device 22. The latter is mainly performed by the medical staff M present near the patient-side device 21. The controller 206 of the surgery assistant robot 20 transmits a request for the bidirectional communication to the server device 40 in response to this operation. The communication control unit 402, which detects the request, performs processing of establishing the bi-directional communication (1) between the server device 40 and at least one of the surgery assistant robot 20 that transmits the request and the terminal device 25 and (2) between the server device 40 and the terminal device 42. In other words, the communication control unit 402, which detects the request, performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the surgery assistant robot 20 that transmits the request or the terminal device 25 and (2) between the server device 40 and the terminal device 42.
Another example of the operation performed by the medical staff M is click of a button displayed on the monitor of the terminal device 25. A request in this case is transmitted from the terminal device 25 to the server device 40. The communication control unit 402, which detects the request, performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 that transmits the request and the surgery assistant robot 20 and (2) between the server device 40 and the terminal device 42. In other words, the communication control unit 402, which detects the request, performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 that transmits the request or the surgery assistant robot 20 and (2) between the server device 40 and the terminal device 42.
Still another example of the operation performed by the medical staff M is input of a sound or a voice requesting establishment of the bidirectional communication to the surgery assistant robot 20 or the terminal device 25.
(Event example 2) A request transmitted from the terminal device 42 according to operation for requesting the bidirectional communication is received by the server device 40.
This request is transmitted from the terminal device 42 in which operation for requesting the bidirectional communication is performed by the serviceperson S to the server device 40. This operation is performed when the serviceperson S, who detects an abnormality or the like of the surgery assistant robot 20, determines that the serviceperson S should support the medical staff M. For example, when an abnormal value indicating an error, a failure, or the like is included in the operation information of the surgery assistant robot 20, the medical staff M is likely to need support of the serviceperson S. Therefore, it is useful to establish the bidirectional communication between the server device 40 and the terminal device 42 and between the server device 40 and at least one of the surgery assistant robot 20 and the terminal device 25. In other words, it is useful to establish the bidirectional communication between the server device 40 and the terminal device 42 and between the server device 40 and at least one of the surgery assistant robot 20 or the terminal device 25.
Typical examples of the abnormal value included in the operation information are an axis value and a current value from which it is surmised that the instrument arm 201A and the camera arm 201B are in a state in which the instrument arm 201A and the camera arm 201B interfere with each other and do not move and an error code indicating that restart of the surgery assistant robot 20 is necessary.
An example of the operation performed by the serviceperson S is designation of the surgery assistant robot 20 that should be remotely operated in the terminal device 42. The request is transmitted from the terminal device 42 to the server device 40. The communication control unit 402, which detects the request, performs processing of establishing the bidirectional communication (1) between the server device 40 and the terminal device 42 that transmits the request and (2) between the server device 40 and at least one of the designated surgery assistant robot 20 and the terminal device 25. In other words, the communication control unit 402, which detects the request, performs processing of establishing the bidirectional communication (1) between the server device 40 and the terminal device 42 that transmits the request and (2) between the server device 40 and at least one of the designated surgery assistant robot 20 or the terminal device 25.
Another example of the operation performed by the serviceperson S is designation of the terminal device 25 of the medical staff M who should be supported in the terminal device 42. In this case as well, the request is transmitted from the terminal device 42 to the server device 40. The communication control unit 402, which detects the request, performs processing of establishing the bidirectional communication (1) between the server device 40 and the terminal device 42 that transmits the request and (2) between the server device 40 and at least one of the designated terminal device 25 and the surgery assistant robot 20. In other words, the communication control unit 402, which detects the request, performs processing of establishing the bi-directional communication (1) between the server device 40 and the terminal device 42 that transmits the request and (2) between the server device 40 and at least one of the designated terminal device 25 or the surgery assistant robot 20.
(Event example 3) Detection of the abnormal value included in the operation information.
As explained above, typical examples of the abnormal value included in the operation information are an axis value and a current value from which it is surmised that the instrument arm 201A and the camera arm 201B are in a state in which the instrument arm 201A and the camera arm 201B interfere with each other and do not move and an error code indicating that restart of the surgery assistant robot 20 is necessary. In this case, the medical staff M is likely to need support of the serviceperson S. Accordingly, the serviceperson S is desirably able to quickly perform data sharing with the medical staff M or remote operation of the surgery assistant robot 20.
In this case, the communication control unit 402 performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 and the surgery assistant robot 20 provided in the same medical-related facility 2 as the medical-related facility 2 in which the network camera 24 is provided and (2) between the server device 40 and the terminal device 42. In other words, the communication control unit 402 performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 or the surgery assistant robot 20 provided in the same medical-related facility 2 as the medical-related facility 2 in which the network camera 24 is provided and (2) between the server device 40 and the terminal device 42.
(Event example 4) Detection of an abnormal situation (specifically, detection of unusual behavior of the medial staff M) based on an image analysis of a video photographed by the network camera 24.
The unusual behavior is, for example, behavior different from normal time (for example, suddenly busily moving around) and behavior of operating an unusual device (for example, behavior of operating a specific telephone). In this case, the serviceperson S is desirably able to quickly perform data sharing with the medical staff M or remote operation of the surgery assistant robot 20.
In this case, the communication control unit 402 performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 and the surgery assistant robot 20 provided in the same medical-related facility 2 as the medical-related facility 2 in which the network camera 24 is provided and (2) between the server device 40 and the terminal device 42. In other words, the communication control unit 402 performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 or the surgery assistant robot 20 provided in the same medical-related facility 2 as the medical-related facility 2 in which the network camera 24 is provided and (2) between the server device 40 and the terminal device 42.
(Event 5) Detection of an abnormal situation (specifically, detection of a specific phrase or volume or frequency exceeding a threshold) based on a sound analysis of a video photographed by the network camera 24.
When a phrase seeking help of the call center 4 is detected or volume or frequency exceeding a threshold is detected, it is highly likely that utterance is performed because of occurrence of some problem and it is highly likely that the medical staff M needs support of the serviceperson S. Accordingly, the serviceperson S is desirably able to quickly perform data sharing with the medical staff M and remote operation of the surgery assistant robot 20.
In this case, the communication control unit 402 performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 and the surgery assistant robot 20 provided in the same medical-related facility 2 as the medical-related facility 2 in which the network camera 24 is provided and (2) between the server device 40 and the terminal device 42. In other words, the communication control unit 402 performs processing of establishing the bidirectional communication (1) between the server device 40 and at least one of the terminal device 25 or the surgery assistant robot 20 provided in the same medical-related facility 2 as the medical-related facility 2 in which the network camera 24 is provided and (2) between the server device 40 and the terminal device 42.
(Data Transmission Processing)
The communication control unit 402 transmits, to a partner device with which the bidirectional communication is established, various data transmitted from the device with which the bidirectional communication is established to the server device 40. Transmission examples are described in (transmission example 1) to (transmission example 7) below.
(Transmission example 1) In a condition in which the bidirectional communication is established, the communication control unit 402 transmits data (at least one of a sound, an image, and a text) input to one of at least one of the surgery assistant robot 20 and the terminal device 25 and the terminal device 42 and transmitted to the server device 40 to the other of the at least one of the surgery assistant robot 20 and the terminal device 25 and the terminal device 42. In other words, in a condition in which the bidirectional communication is established, the communication control unit 402 transmits data (at least one of a sound, an image, or a text) input to one of at least one of the surgery assistant robot 20 or the terminal device 25 and the terminal device 42 and transmitted to the server device 40 to the other of the at least one of the surgery assistant robot 20 or the terminal device 25 and the terminal device 42. Consequently, it is possible to quickly share advices and instructions between the medical staff M and the serviceperson S. Therefore, effects equivalent to effects obtained when the serviceperson S is present and supports the medical staff M in the site can be obtained. Note that, if the data providing unit 403 is always transmitting operation information of the surgery assistant robot 20 to the terminal device 42, the serviceperson S can estimate, based on operation information provided before the bi-directional communication is established, a cause of a trouble and a method of coping with the trouble. Therefore, the serviceperson S can quickly and appropriately support the medical staff M immediately after the bidirectional communication is established.
As an example, an example of a flow of processing with which the bi-directional communication is established by the event example 3 or the event example 4 explained above and the transmission example 1 is performed is explained with reference to
(Transmission example 2) When the bidirectional communication is established, the communication control unit 402 desirably transmits, to at least the terminal device 42, an image photographed by the network camera 24 and received by the server device 40. The terminal device 42 displays the transmitted image on the monitor. Consequently, the medical staff M and the serviceperson S can visually share a present state of the operating room equivalently to or better than when the serviceperson S is present in the site. Therefore, the serviceperson S can appropriately support the medical staff M. Note that the terminal device 25 may be added as a transmission destination of the image.
(Transmission example 3) When the bidirectional communication is established, the communication control unit 402 desirably transmits, to at least the terminal device 42, an image photographed by the endoscope 201b and received by the server device 40. The terminal device 42 displays the transmitted image on the monitor. Consequently, the medical staff M and the serviceperson S can visually share a state in the body cavity of the patient P equivalently to or better than when the serviceperson S is present in the site. Therefore, the serviceperson S can appropriately support the medical staff M. Note that the terminal device 25 may be added as a transmission destination of the image.
(Transmission example 4) The transmission explained in the transmission example 2 and the transmission explained in the transmission example 3 are desirably simultaneously performed. That is, it is desirable to transmit, to at least the terminal device 42, an image photographed by the network camera 24 and received by the server device 40 and an image photographed by the endoscope 201b and received by the server device 40. The terminal device 42 simultaneously displays both the transmitted images on the monitor. Consequently, the serviceperson S can confirm a present state of the operating room and a state in the body cavity of the patient P. Therefore, the serviceperson S can appropriately support the medical staff M. Note that the terminal device 25 may be added as a transmission destination of the image.
(Transmission example 5) It is assumed that the communication control unit 402 transmits an image (at least one of an image photographed by the network camera 24 and an image photographed by the endoscope 201b) to the terminal device 42 with which the bidirectional communication is established and the image is displayed on a monitor of the terminal device 42. In other words, it is assumed that the communication control unit 402 transmits an image (at least one of an image photographed by the network camera 24 or an image photographed by the endoscope 201b) to the terminal device 42 with which the bidirectional communication is established and the image is displayed on a monitor of the terminal device 42. Further, it is assumed that the communication control unit 402 also transmits the image to at least one of the surgery assistant robot 20 and the terminal device 25 with which the bi-directional communication is established and the image is displayed on a monitor included in the surgery assistant robot 20 (the touch panel 207 provided in the patient-side device 21 and the monitor 231 connected to the image processing device 23) or the monitor of the terminal device 42. In other words, it is assumed that the communication control unit 402 also transmits the image to at least one of the surgery assistant robot 20 or the terminal device 25 with which the bidirectional communication is established and the image is displayed on a monitor included in the surgery assistant robot 20 (the touch panel 207 provided in the patient-side device 21 and the monitor 231 connected to the image processing device 23) or the monitor of the terminal device 42. In this case, data (for example, graphics drawn by a paint tool in a display region of the image) to be superimposed and displayed on the image is input to the terminal device 42, the data is transmitted to the server device 40. The communication control unit 402 transmits the data to at least one of the surgery assistant robot 20 and the terminal device 25 with which the bidirectional communication is established. In other words, the communication control unit 402 transmits the data to at least one of the surgery assistant robot 20 or the terminal device 25 with which the bidirectional communication is established. The surgery assistant robot 20, which receives the data, superimposes and displays the data on the image transmitted from the communication control unit 402 and displayed on the monitor. Similarly, the terminal device 25, which receives the data, superimposes and displays the data on the image transmitted from the communication control unit 402 and displayed on the monitor. Consequently, both of the medical staff M and the serviceperson S can share a processed image, the image on which the data is superimposed and displayed. Therefore, for example, by drawing marking in a noticeable part in the image in the terminal device 42, both of the medical staff M and the serviceperson S can easily share the noticeable part.
As an example, an example of a flow of processing in which the transmission examples 2 to 5 are performed in a state with which the bidirectional communication is established between the server device 40 and the surgery assistant robot 20 and the terminal device 25 and between the server device 40 and the terminal device 42 is explained with reference to
(Transmission example 6) The communication control unit 402 transmits, to the surgery assistant robot 20, an operation command and a warning output instruction for the surgery assistant robot 20 input to the terminal device 42 and transmitted to the server device 40. Consequently, the serviceperson S can quickly remotely operate the surgery assistant robot 20.
(Transmission example 7) When data such as a sound, an image, and a text can be input to the surgery assistant robot 20, the communication control unit 402 may transmit, to the terminal device 42, the data input to the surgery assistant robot 20 and transmitted to the server device 40. Conversely, when data such as a sound, an image, and a text can be output in the surgery assistant robot 20, the communication control unit 402 may transmit, to the surgery assistant robot 20, the data input to the terminal device 42 and transmitted to the server device 40. Consequently, communication between the medical staff M and the serviceperson S is possible even if the terminal device 25 is not used.
[Data Providing Unit 403]
The data providing unit 403 transmits data corresponding to a request from the terminal device 25 to the terminal device 25 at the request source. Similarly, the data providing unit 403 provides data corresponding to a request from the terminal device 42 to the terminal device 42 at the request source. Therefore, various APIs are prepared in the data providing unit 403. Typical examples of the data provided to the terminal device 25 and the terminal device 42 are data stored in the storage 41, in particular, operation information of the surgery assistant robot 20, an image photographed by the network camera 24, and an image photographed by the endoscope 201b.
Note that the data providing unit 403 desirably always transmits the operation information received from the surgery assistant robot 20 to the terminal device 42. Consequently, the serviceperson S can always monitor a state of the surgery assistant robot 20 in the terminal device 42. Therefore, the serviceperson S can confirm, beforehand, a cause of a trouble and a method of coping with the trouble based on the operation information of the surgery assistant robot 20 before the bi-directional communication is established. Therefore, the serviceperson S can quickly and appropriately support the medical staff M immediately after bidirectional communication with the medical staff M is started.
When an error is included in the data to be provided, the data providing unit 403 transmits a notification concerning the error to the terminal device 25 or the terminal device 42 at the request source. A method of coping with the error may be included in the notification concerning the error. The method of coping with the error is, for example, a hyper link with a link destination set in a relevant part in an electronic manual and data obtained by dynamically summarizing only necessary information among accumulated information concerning the method of coping with the error. However, the method of coping with the error is not limited to these.
Note that the data providing unit 403 may statistically analyze, with the data analyzing unit 404, the data to be provided and then transmit the data or may organize a display form of the data to be provided and then transmit the data.
An overview of processing performed by the data providing unit 403 is explained with reference to
[Data Analyzing Unit 404]
The data analyzing unit 404 performs an analysis of various data according to a request from the terminal device 25 or the terminal device 42 and transmits an analysis result to the request source. Therefore, various APIs for analysis are prepared in the data providing unit 403. A typical example of analysis target data is data (typically, operation information of the surgery assistant robot 20) stored in the storage 41.
An overview of processing performed by the data analyzing unit 404 is explained with reference to
[Terminal Device 42]
The terminal device 42 is a computer terminal used in the call center 4 by the serviceperson S of the call center 4. Typical examples of the terminal device 42 are a personal computer, a tablet terminal, a smartphone, and the like. The terminal device 42 is communicably connected to the server device 40 via a LAN in the call center 4.
Note that the terminal device 42 has functions that the computer terminal usually has such as a communication function with other devices, a data input function of a keyboard and the like, a data display function of a monitor and the like, a sound input function of a microphone and the like, and a sound output function of a speaker and the like.
The terminal device 42 acquires data (typically, the operation information of the surgery assistant robot 20) stored in the storage 41 via the server device 40 and displays the acquired data on the monitor.
The terminal device 42 is capable of receiving, from the serviceperson S, operation for requesting the bidirectional communication with the surgery assistant robot 20 or the terminal device 25. The terminal device 42 transmits a request corresponding to the operation to the server device 40. Examples of the operation are click of a button displayed on the monitor and a sound input to the microphone. However, the operation is not limited to these.
The terminal device 42 is capable of executing various functions for performing bidirectional communication and data sharing with the terminal device 25 with which the bidirectional communication is established. That is, the terminal device 42 transmits communication data to the server device 40 in order to share the communication data with the terminal device 25. On the other hand, the terminal device 42 receives, from the server device 40, communication data input to the terminal device 25
The terminal device 42 is capable of receiving a command to the surgery assistant robot 20 with which the bidirectional communication is established. Examples of the command are a warning output command from the surgery assistant robot 20 and an operation command for remotely operating the surgery assistant robot 20.
The terminal device 42 can acquire a screen displayed on the monitor 221 of the remote operation device 22 of the surgery assistant robot 20 with which the bi-directional communication is established and display the screen on the monitor of the terminal device 42.
When the surgery assistant robot 20 includes an output unit that outputs data such as a sound, an image, and a text, the terminal device 42 may transmit, through the server device 40, the data to the surgery assistant robot 20 with which the bi-directional communication is established.
[Storage 41] The storage 41 is a device that stores various data and is, for example, a magnetic disk such as a hard disk.
Examples of display screens of the monitors of the terminal device 42 and the terminal device 25 are explained with reference to
It is assumed that the bidirectional communication is established between the server device 40 and the terminal device 42 and the terminal device 25. It is assumed that an image of the operating room photographed by the network camera 24 and an image of the inside of the body cavity of the patient P photographed by the endoscope 201b are transmitted from the communication control unit 402 to the terminal device 42 and the terminal device 25.
The transmitted image of the operating room is displayed in a display region A1 (see
The transmitted image of the inside of the body cavity of the patient P is displayed in a display region A2 (see
In this way, the same image is simultaneously displayed on both of the monitor of the terminal device 42 and the monitor of the terminal device 25. Consequently, both of the medical staff M and the serviceperson S can share a present state of the operating room and a present state of the inside of the body cavity of the patient P equivalently to or better than when the serviceperson S is present in the site. Therefore, the service person S can appropriately support the medical staff M.
In a region A3 illustrated in
A sound input and output device and an image output device may be provided in the patient-side device 21 besides the touch panel 207. With the sound input and output device and the image output device, it is possible to perform operation for requesting establishment of the bidirectional communication in the patient-side device 21 and it is possible to perform, after the establishment of the bi-directional communication, bidirectional communication and data sharing between the medical staff M who uses the patient-side device 21 and the serviceperson S who uses the terminal device 42.
Similarly, an operation device that receives input operation, a sound input and output device, and an image output device may be provided in the remote operation device 22 besides the monitor 221. With the operation device, the sound input and output device, and the image output device, it is possible to perform operation for requesting establishment of the bidirectional communication in the remote operation device 22 and it is possible to perform, after the establishment of the bi-directional communication, bidirectional communication and data sharing between the medical staff M who uses the remote operation device 22 and the serviceperson S who uses the terminal device 42.
Similarly, an operation device that receives input operation, a sound input and output device, and an image output device may be connected to the image processing device 23 besides the monitor 231. With the operation device, the sound input and output device, and the image output device, it is possible to perform operation for requesting establishment of the bidirectional communication in the image processing device 23 and it is possible to perform, after the establishment of the bi-directional communication, bidirectional communication and data sharing between the medical staff M who uses the image processing device 23 and the serviceperson S who uses the terminal device 42.
The terminal device 25 is usually communicably connected to the intra-facility LAN. However, the terminal device 25 may be communicably connected to the external network 3 not via the intra-facility LAN.
The configuration example in which the server device 40 is installed in the call center 4 is explained above. However, a setting place of the server device 40 is not limited to this. As illustrated in
In an embodiment, transmission and reception of data between devices in the medical-related facility 2b and the server device 40 is performed via a LAN. Transmission and reception of data between the storage 41 and the terminal device 42 of the call center 4b and the server device 40 is performed via the external network 3.
The configuration example in which the surgery assistant robot 20 includes one patient-side device 21 including the surgical manipulators 201 is explained above. However, the configuration of the surgery assistant robot 20 is not limited to this. For example, the surgery assistant robot 20 may have a configuration including patient-side devices instead of the patient-side device 21 and each of the patient-side devices includes a manipulator to which a surgical instrument such as an endoscope or forceps is attached.
A surgery assistant robot 20b according to an embodiment is illustrated in
In an embodiment, the communication control unit 402 performs processing of establishing the bidirectional communication between the server device 40 and at least one of the surgery assistant robot 20b (the first patient-side device 211, the second patient-side device 212, the third patient-side device 213, the remote operation device 22, and the image processing device 23) and the terminal device 25. In other words, the communication control unit 402 performs processing of establishing the bi-directional communication between the server device 40 and at least one of the surgery assistant robot 20b (the first patient-side device 211, the second patient-side device 212, the third patient-side device 213, the remote operation device 22, or the image processing device 23) or the terminal device 25.
The configuration example in which the surgery assistant robots 20 and 20b include the patient-side devices 21, 211, 212, and 213, the remote operation device 22, and the image processing device 23 is explained above. However, the configuration of the surgery assistant robot 20 is not limited to this.
A surgery assistant robot 20c according to an embodiment is illustrated in
In the operation device 226, a button for communication 228 for requesting the bidirectional communication is provided. The medical staff M presses the button for communication 228, whereby a request for the bidirectional communication is transmitted from the operation device 226 to the server device 40. In an embodiment, the communication control unit 402 performs processing of establishing the bilateral communication between the server device 40 and at least one of the surgery assistant robot 20c (the robot operating table 225 and the operation device 226) and the terminal device 25. In other words, the communication control unit 402 performs processing of establishing the bilateral communication between the server device 40 and at least one of the surgery assistant robot 20c (the robot operating table 225 or the operation device 226) or the terminal device 25.
Note that the operation device 226 is capable of inputting and outputting data such as an image, a sound, and a text. After the establishment of the bidirectional communication, it is possible to perform bidirectional communication and the data sharing between the medical staff M who uses the operation device 226 and the serviceperson S who uses the terminal device 42. The image and the text are output to a monitor 229.
[Additional Note 1]
Timing when the medical staff M needs support by the serviceperson S could occur not only in a period in which the surgery assistant robot 20 is operating (that is, during surgery or training) but also in a period in which the surgery assistant robot 20 is not operating. For example, when the patient-side device 21, the remote operation device 22, and the image processing device 23 are connected by wire, in preparation before surgery and clean-up after the surgery, the medical staff M sometimes desires to seek support of the serviceperson S concerning wiring of these devices. Therefore, the bidirectional communication may be established in the period in which the surgery assistant robot 20 is not operating and bidirectional communication may be performed between the medical staff M and the serviceperson S.
[Additional Note 2]
As the bidirectional communication, there a form with which the bidirectional communication is performed between two devices and a form with which the bidirectional communication is performed among devices. A typical example of the former is suitable in a case in which the serviceperson S remotely supports one medical-related facility 2 (that is, a one-to-one relation). In this case, the bidirectional communication is performed between one terminal device 42 and the surgery assistant robot 20 or the terminal device 42. A typical example of the latter is suitable in a case in which an instructor remotely trains training target persons in training of the surgery assistant robot 20 (that is, a one-to-many relation). In this case, the bidirectional communication is performed between one terminal device 42 and the surgery assistant robots 20.
[Additional Note 3]
The server device 40 does not always need to be set in the facility of the call center 4 as in a first embodiment. The server device 40 may be set on the outside of the call center 4 and realized to be communicatively connected to the call center 4 via the external network 3.
[Additional Note 4]
In an embodiment, all the functions of the communication control device 261 do not always need to be included in the communication control device 261. For example, the surgery assistant robot 20 or the server device 40 may have a function of performing, on an image photographed by at least one of the endoscope 201b and the network camera 24, image processing of visually obscuring identifying information which enables a person to be identified and a function of adjusting a frame rate of an image photographed by at least one of the endoscope 201b and the network camera 24. In other words, the surgery assistant robot 20 or the server device 40 may have a function of performing, on an image photographed by at least one of the endoscope 201b or the network camera 24, image processing of visually obscuring identifying information which enables a person to be identified and a function of adjusting a frame rate of an image photographed by at least one of the endoscope 201b or the network camera 24.
[Additional Note 5]
In an embodiment, the request for the bidirectional communication is received by the server device 40, whereby the communication from the call center 4 to the medical-related facility 2 becomes possible. It goes without saying that, at emergency time, communication is possible from the call center 4 to the medical-related facility 2 even if the request for the bidirectional communication is not transmitted.
[Realization By Software]
Control blocks (in particular, the system linkage device 27) of the surgery assistant robots 20, 20b, and 20c, control blocks (in particular, the control unit 262) of the communication control device 261, and control blocks (in particular, the control unit 400) of the server device 40 may be realized by a logic circuit (hardware) formed on an integrated circuit (an IC chip) or may be realized by software.
In the latter case, the surgery assistant robots 20, 20b, and 20c, the communication control device 261, and the server device 40 include a computer that executes commands of programs, which are software, for realizing the functions. The computer includes, for example, one or more processors and includes a computer-readable recording medium storing the programs. In the computer, the processor reads the programs from the recording medium and executes the programs, whereby the object of one or more aspects is achieved. As the processor, for example, a CPU and a GPU (Graphics Processing Unit) can be used. As the recording medium, a “non-transitory tangible medium”, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, and the like can be used besides, for example, a ROM (Read Only Memory). The computer may further include a RAM (Random Access Memory) on which the programs are developed. The programs may be supplied to the computer via any transmission medium (a communication network, a broadcasting wave, or the like) capable of transmitting the programs. Note that an embodiment can also be realized in a form of a data signal embedded in a carrier wave in which the programs are embodied by electronic transmission.
The present invention is not limited to the embodiments explained above. Various changes are possible in a range described in claims. An embodiment obtained by combining, as appropriate, the technical means respectively disclosed in the different embodiments is also included in the technical scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2018-190527 | Oct 2018 | JP | national |
This application is a continuation of application Ser. No. 16/592,819, filed on Oct. 4, 2019, which is based upon and claims the benefit of priority from the prior Japanese Patent Application No. 2018-190527, filed with the Japan Patent Office on Oct. 6, 2018 the entire contents of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10360531 | Stallman | Jul 2019 | B1 |
| 20020018587 | Ueda | Feb 2002 | A1 |
| 20030135203 | Wang et al. | Jul 2003 | A1 |
| 20050057689 | Sakagami | Mar 2005 | A1 |
| 20060087746 | Lipow | Apr 2006 | A1 |
| 20070122783 | Habashi | May 2007 | A1 |
| 20070167702 | Hasser et al. | Jul 2007 | A1 |
| 20100217991 | Choi | Aug 2010 | A1 |
| 20100241693 | Ando | Sep 2010 | A1 |
| 20120330613 | Sillman et al. | Dec 2012 | A1 |
| 20140267658 | Speier et al. | Sep 2014 | A1 |
| 20150157411 | Choi | Jun 2015 | A1 |
| 20160041727 | Choi | Feb 2016 | A1 |
| 20160210419 | Kuji et al. | Jul 2016 | A1 |
| 20160314716 | Grubbs | Oct 2016 | A1 |
| 20170165837 | Asano et al. | Jun 2017 | A1 |
| 20180338806 | Grubbs | Nov 2018 | A1 |
| 20190110856 | Barral | Apr 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 1426564 | Jun 2003 | CN |
| 104298499 | Jan 2015 | CN |
| 104688347 | Jun 2015 | CN |
| 105138815 | Dec 2015 | CN |
| 106096301 | Nov 2016 | CN |
| 107106246 | Aug 2017 | CN |
| 108472085 | Aug 2018 | CN |
| 2002-058017 | Feb 2002 | JP |
| 2002-092183 | Mar 2002 | JP |
| 2003-22326 | Jan 2003 | JP |
| 2005-111080 | Apr 2005 | JP |
| 2005-118232 | May 2005 | JP |
| 2005-135344 | May 2005 | JP |
| 2006-071359 | Mar 2006 | JP |
| 2007-007040 | Jan 2007 | JP |
| 2010-029511 | Feb 2010 | JP |
| 2010-278596 | Dec 2010 | JP |
| 2015-532040 | Nov 2015 | JP |
| 2017-104456 | Jun 2017 | JP |
| 2017-192043 | Oct 2017 | JP |
| 2016149794 | Sep 2016 | WO |
| 2017083768 | May 2017 | WO |
| 2018150489 | Aug 2018 | WO |
| Entry |
|---|
| Japanese Office Action mailed on Aug. 1, 2023 in a counterpart Japanese patent application No. 2022-140835. |
| Japanese Office Action mailed on Aug. 1, 2023 in a counterpart Japanese patent application No. 2022-140836. |
| The Office Action issued on Mar. 28, 2023 in a counterpart Japanese patent application No. 2022-140835, with English translation. |
| The Office Action issued on Mar. 28, 2023 in a counterpart Japanese patent application No. 2022-140836, with English translation. |
| (Submitted in Parent U.S. Appl. No. 16/592,819), Nikkei Linux, “The Linux Kernel: Q&A 48”, Nov. 2015 issue, p. 34, vol. 17th, No. 11, Nikkei BP, Japan; Cited in the JPOA issued on Jan. 24, 2023. |
| (Submitted in Parent U.S. Appl. No. 16/592,819), the Office Action issued on Jan. 24, 2023 in a counterpart Japanese patent application with English translation. |
| (Submitted in Parent U.S. Appl. No. 16/592,819), the Office Action issued on Jul. 26, 2022 in a counterpart Japanese patent application with English translation. |
| (Submitted in Parent U.S. Appl. No. 16/592,819), the extended European search report (EESR) issued on Mar. 12, 2020 in a counterpart European patent application. |
| The Office Action (JPOA) issued on Oct. 17, 2023 in a counterpart Japanese patent application No. 2022-140835, with English translation. |
| The Office Action (JPOA) issued on Oct. 17, 2023 in a counterpart Japanese patent application No. 2022-140836, with English translation. |
| The Office Action (CNOA) issued on Nov. 4, 2023 in a counterpart Chinese patent application, with English translation. |
| The Office Action (CNOA) issued on Mar. 8, 2024 in a counterpart Chinese patent application No. 201910916739.X, with English translation. |
| Annex 3, Code of Practice for the Management of Telemedicine Services (Trial), Jul. 17, 2018, 1-7 pages, The National Health Council and State Administration of Traditional Chinese Medicine, Cited in the Office Action (CNOA) issued on Mar. 8, 2024 in a counterpart Chinese patent application No. 201910916739.X, with English machine translation. |
| A Communication pursuant to Article 94 (3) EPC issued on May 7, 2024 in a counterpart European patent application No. 19201434.8. |
| Number | Date | Country | |
|---|---|---|---|
| 20230181271 A1 | Jun 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16592819 | Oct 2019 | US |
| Child | 18166066 | US |
