The present disclosure relates to monitoring systems, monitoring methods, and monitoring devices.
In recent years, vehicle systems become complicated in order to provide users with advanced functions such as autonomous driving. In order to solve the problem of increase in development period and cost caused by the complication, there is movement to integrate functions which have been conventionally mounted separately on a plurality of electronic control units (ECUs) into a single ECU. In such integration of ECUs, virtual technology is used to cause a single ECU to operate a plurality of virtual computers. Such virtual computers are referred to as virtual machines. Software that is a virtualization infrastructure for operating the plurality of virtual machines is referred to as hypervisor. When vulnerability in virtual software or a device driver in a hypervisor or a defect in specifications becomes noticeable, there is a problem that one or more memory regions that are assigned to the hypervisor or virtual machines may be manipulated.
As a countermeasure against such a problem, Patent Literature 1 discloses a device that periodically monitors manipulation in an operating system.
Security in vehicles is desired to be increased.
In view of this, the present disclosure provides monitoring systems, monitoring methods, and monitoring devices with increased security in vehicles.
A monitoring system according to an aspect of the present disclosure is monitoring system for monitoring a vehicle or a first monitoring target that operates inside the vehicle, the monitoring system including: a reliability manager that manages first reliability indicating a security protection state of the first monitoring target, according to a vehicle event of the vehicle; and a function restrictor that places a restriction on at least a part of functions of the first monitoring target, according to the first reliability.
A monitoring method according to an aspect of the present disclosure is a monitoring method for monitoring a vehicle or a monitoring target that operates inside the vehicle, the monitoring system including: managing reliability indicating a security protection state of the monitoring target, according to a vehicle event of the monitoring target; and placing a restriction on at least a part of functions of the monitoring target, according to the reliability.
A monitoring device according to an aspect of the present disclosure is a monitoring device for monitoring a vehicle or a monitoring target that operates inside the vehicle, the monitoring device being included in a monitoring system that includes the monitoring device and a function restricting device for placing a restriction on at least a part of functions of the monitoring target, the monitoring device further including: a reliability manager that manages reliability indicating a security protection state of the monitoring target, according to a vehicle event of the vehicle.
A function restricting device according to an aspect of the present disclosure is a function restricting device for placing a restriction on at least a part of functions of the monitoring target, the function restricting device being included in a monitoring system that includes a monitoring device for monitoring a vehicle or a monitoring target that operates inside the vehicle and the function restricting device, the function restricting device further including: a function restrictor that places a restriction on at least a part of functions of the monitoring target, according to reliability indicating a security protection state of the monitoring target, according to a vehicle event of the vehicle.
The monitoring system, etc., according to aspects of the present disclosure increase security in vehicles.
These and other advantages and features will become apparent from the following description thereof taken in conjunction with the accompanying Drawings, by way of non-limiting examples of embodiments disclosed herein.
Circumstances leading to the present disclosure are described before the present disclosure is described.
When the device of PTL 1 described above is applied to a single ECU configured to operate a plurality of virtual computers, check-target regions become huge. Thus, the time required for completion of the check of each of the check targets increases, which inevitably makes large differences in the points of end time of check between the check targets. In addition, since check is performed periodically, it is difficult to detect an attack when the attack is made in the time between check and check. For this reason, it is desired that security of a vehicle in which ECUs are mounted be increased.
In view of this, the inventors of the present application have keenly studied monitoring systems, etc., capable of increasing security of vehicles, and have arrived at monitoring systems, etc., that are described below. The inventors of the present application have arrived at monitoring systems, etc., capable of implementing more safety ECUs by setting numerical values based on which each of check targets is determined to be normal and restricting a part of functions of the check target according to the numerical value in an example case where large differences in the points of end time of check are made between the check targets. This allows to appropriately determine and restrict the usable functions of the monitoring target according to the state of the monitoring target, and to maintain a safe state as the entirety of the system.
A monitoring system according to an aspect of the present disclosure is a monitoring system for monitoring a vehicle or a first monitoring target that operates inside the vehicle, the monitoring system including: a reliability manager that manages first reliability indicating a security protection state of the first monitoring target, according to a vehicle event of the vehicle; and a function restrictor that places a restriction on at least a part of functions of the first monitoring target, according to the first reliability.
This allows to appropriately determine and restrict the usable functions of the monitoring target according to the state of the monitoring target, and to maintain a safe state as the entirety of the system. In short, it is possible to increase security in the vehicle.
In addition, for example, the first reliability may be a variable capable of taking at least two levels each of which indicates a degree of the security protection state of the first monitoring target, and the reliability manager may change a current level among the at least two levels each time a vehicle event occurs.
This allows to manage the state of the monitoring target in more detail.
In addition, for example, the vehicle event that causes change of the current level among the at least two levels may include integrity check of software that is the first monitoring target.
This allows to detect manipulation of software that is the monitoring target, and reduce influence onto another monitoring target, etc.
In addition, for example, the reliability manager may change the current level among the at least two levels by performing at least one of: adding a value to the first reliability when the integrity check has been successfully completed; or subtracting a value from the first reliability when the integrity check has not been successfully completed.
This allows to update the reliability according to the result of the integrity check.
In addition, for example, the vehicle event that causes change of the current level among the at least two levels may include an elapse of a predetermined time.
This allows to appropriately manage the state of the monitoring target for which risks such as manipulation increase over time, and reduce influence onto another monitoring target, etc., by restricting the part of the functions of the monitoring target according to the state of the monitoring target.
In addition, for example, the reliability manager may subtract a value from the first reliability when the predetermined time has elapsed.
This enables to update the reliability when the predetermined time elapsed.
In addition, for example, the vehicle event that causes change of the current level among the at least two levels may include reboot of the first monitoring target.
This allows to appropriately manage the state of reboot that may or may not be successfully performed depending on manipulation, etc., and reduce influence onto another monitoring target, etc., by restricting the part of the functions of the monitoring target according to the state of the reboot.
In addition, for example, the reliability manager may change the current level among the at least two levels by either adding a value to the first reliability or setting the first reliability to an initial value when the reboot has been successfully completed.
This allows to update the reliability according to the success or failure of the reboot.
In addition, for example, the reliability manager may determine whether the first reliability of the first monitoring target is less than a threshold value, and cause the first monitoring target to execute the reboot when the first reliability is less than the threshold value.
This allows to automatically perform reboot when the reliability decreased, and update the reliability.
In addition, for example, the restriction on the at least the part of the functions of the first monitoring target according to the first reliability of the first monitoring target may include stop of an access right that allows the first monitoring target to access a particular resource.
This allows to appropriately manage access to the resource according to the state of the monitoring target.
In addition, for example, the restriction on the at least the part of the functions according to the first reliability of the first monitoring target may include stop of a communication function of the first monitoring target.
This allows to restrict an action by the monitoring target whose reliability is low, specifically the action of trying to expanding influence onto another monitoring target, etc.
In addition, for example, in a case where the first monitoring target tries to communicate with a communication target, the function restrictor may forbid the first monitoring target from communicating with the communication target when the first reliability is less than the threshold value.
This allows to restrict an action by the monitoring target whose reliability is low, specifically the action of starting communication and expanding influence onto another monitoring target, etc.
In addition, for example, the monitoring system may further monitor the vehicle or a second monitoring target that operates inside the vehicle, the reliability manager may further manage second reliability indicating a security protection state of the second monitoring target, according to a vehicle event of the vehicle, and in a case where the first monitoring target and the second monitoring target try to communicate with each other, the function restrictor may forbid the first monitoring target and the second monitoring target from communicating with each other when at least one of the first reliability or the second reliability is less than the threshold value.
This allows to restrict an action by the monitoring target whose reliability is low, specifically the action of starting communication and expanding influence onto another monitoring target, etc.
In addition, for example, the restriction on the at least the part of the functions according to the first reliability of the first monitoring target may include stop of an operation of the first monitoring target.
This allows to prevent the monitoring target whose reliability is low from keeping operating and placing bad influence onto the entirety of the system.
In addition, for example, a unit of the first monitoring target for which the first reliability is calculated may include at least one of a software process, an operating system, a hyperviser, a virtual machine that operates on the hyperviser, an electronic control unit that operates in the vehicle, or the vehicle.
This allows to restrict functions for each unit, and perform wide monitoring that covers the entirety of the system.
In addition, for example, the monitoring system may further include: a display unit that displays the first monitoring target and the first reliability together.
This allows to check the current state of the monitoring target from outside, which allows to maintain a safe state as the entirety of the system.
In addition, for example, the reliability manager and the function restrictor may be mounted in the vehicle.
This allows to increase security in the vehicle without communicating with something outside the vehicle.
In addition, for example, the monitoring system may include a server that is communicatively connected to the vehicle, and at least one of the reliability manager or the function restrictor may be implemented in the server.
This allows to accurately make a determination as to at least one of reliability or function restriction even when the software, etc., that is the monitoring target has been manipulated.
A monitoring method according to an aspect of the present disclosure is a monitoring method for monitoring a vehicle or a monitoring target that operates inside the vehicle, the monitoring method including: managing reliability indicating a security protection state of the monitoring target, according to a vehicle event of the monitoring target; and placing a restriction on at least a part of functions of the monitoring target, according to the reliability.
With this, advantageous effects similar to those provided by the monitoring system described above can be provided. Specifically, this allows to appropriately determine the usable functions of the monitoring target according to the state of the monitoring target, and to maintain a safe state as the entirety of the system.
A monitoring device according to an aspect of the present disclosure is a monitoring device for monitoring a vehicle or a monitoring target that operates inside the vehicle, the monitoring device being included in a monitoring system that includes the monitoring device and a function restricting device for placing a restriction on at least a part of functions of the monitoring target, the monitoring device further including: a reliability manager that manages reliability indicating a security protection state of the monitoring target, according to a vehicle event of the vehicle.
This allows to manage the state of the monitoring target appropriately.
A function restricting device according to an aspect of the present disclosure is a function restricting device for placing a restriction on at least a part of functions of a monitoring target, the function restricting device being included in a monitoring system that includes a monitoring device for monitoring a vehicle or the monitoring target that operates inside the vehicle and the function restricting device, the function restricting device further including: a function restrictor that places a restriction on at least a part of functions of the monitoring target, according to reliability indicating a security protection state of the monitoring target, according to a vehicle event of the vehicle.
This allows to appropriately determine the usable functions of the monitoring target according to the state of the monitoring target.
Hereinafter, a fraud countermeasure method relating to the embodiments of the present disclosure is described with reference to the drawings. The embodiments described below each indicate a preferred specific example of the present disclosure. In other words, the numerical values, shapes, materials, elements, the arrangement and connection of the elements, steps, the order of the steps, etc., in the following embodiments are mere examples of the present disclosure, and therefore are not intended to limit the present disclosure. The present disclosure is identified based on the recitation of the claims. Accordingly, among the elements in the embodiments below, the elements that are not recited in any of the independent claims that define the most generic concept of the present disclosure are not always required to solve the problem of the present disclosure, but are explained as elements included in one or more preferred embodiments.
In addition, it is to be noted that the respective diagrams are schematic diagrams, and are not necessarily precisely illustrated. Accordingly, for example, the scales in the respective diagrams do not always match. In addition, in each of the diagrams, elements that are substantially the same as those in one or more of the diagrams are assigned with the same reference signs, and overlapping descriptions are omitted or simplified.
In addition, in the present specification, terms indicating relationships between elements such as “match”, terms indicating numerical values and numerical ranges refer not only to their strict meanings, but each encompass a range of substantial equivalents, such as a range of deviations of a few percent (for example, approximately 10%).
These general and specific aspects may be implemented using a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of systems, methods, integrated circuits, computer programs, or computer-readable recording media.
Hereinafter, certain exemplary embodiments are described in greater detail with reference to the accompanying Drawings.
Each of the exemplary embodiments described below shows a general or specific example.
The numerical values, shapes, materials, elements, the arrangement and connection of the elements, steps, the processing order of the steps etc. shown in the following exemplary embodiments are mere examples, and therefore do not limit the scope of the appended Claims and their equivalents. Therefore, among the elements in the following exemplary embodiments, those not recited in any one of the independent claims are described as optional elements.
Here, a vehicle network system according to an embodiment of the present disclosure is described with reference to the drawings.
As illustrated in
A processing instruction according to a driver operation is transmitted from integrated ECU 1100 to each of brake 1011 and engine 1012, and the results are fed back.
A processing instruction according to a driver operation is transmitted from integrated ECU 1100 to display 1013, and content is displayed thereon. In addition, upon reception of the driver operation onto display 1013, the content is fed back to integrated ECU 1100.
It is to be noted that display 1013 may, for example, display a monitoring target to be described later and the reliability of the monitoring target together. In addition, display 1013 may display determination results such as the results of integrity check and maintenance processing to be described later.
As illustrated in
In the present embodiment, monitoring-target module 1160 and monitoring-target module 1170 which are monitoring targets that operate on OS 1140 and OS 1150, respectively. In addition, monitoring module 1180 which monitors monitoring-target module 1160 and monitoring-target module 1170 operates on TEE 1130.
TEE 1130 operates independently of OSs 1140 and 1150. Although an example where the monitoring targets of monitoring module 1180 are monitoring-target modules 1160 and 1170 is described in the present embodiment, the number of monitoring targets is not particularly limited, and may be any as long as the number is one or more.
As illustrated in
Communication unit 1161 takes a role for communicating with hypervisor 1120, the module, the process, etc., that operate in the OS which operates in another VM. Communication unit 1161 asks monitoring module 1180 about the reliability determination result, and receives the reliability determination result. Communication unit 1161 is configured to include a communication module (communication circuit).
It is to be noted that the reliability is an indicator indicating whether a monitoring target (here, monitoring-target module 1160) is safe in terms of security. For example, the reliability is the degree of the security protection state of the monitoring target. The reliability indicates, for example, a likelihood that no manipulation has been made onto the monitoring target ((here, monitoring-target module 1160), or the monitoring target has not been suffered from hacking.
Function processor 1162 is a processor for implementing original functions of monitoring-target module 1160. Examples of processing for implementing the original functions include rendering processing for display 1013 and instructions for running control for brake 1011 or engine 1012.
Function restrictor 1163 restricts at least a part of functions for each monitoring target according to the reliability of the monitoring target. For example, function restrictor 1163 receives the reliability determination result from monitoring module 1180, and restricts the at least part of the functions of function processor 1162 according to the reliability of monitoring module 1180.
It is to be noted that, hereinafter, monitoring-target module 1160 is also referred to as monitoring-target module A, and monitoring-target module 1170 is also referred to as monitoring-target module B. In addition, monitoring-target modules 1160 and 1170 are mounted in vehicle 10.
As illustrated in
Although an example in which the monitoring targets of monitoring module 1180 are monitoring-target modules is described in the present embodiment, it is to be noted that the unit of monitoring targets may be any one of a software process, an operating system, a hypervisor, virtual machines which operate on a hypervisor, electronic control units (ECUs) which operate in vehicle 10, or vehicle 10.
Communication unit 1181 takes a role for communicating with hypervisor 1120, the OS which operates on another VM, the module which operates in the OS, a process, or the like. Communication unit 1181 mainly receives an inquiry from another module, and returns the determination result. Communication unit 1181 is configured to include a communication module (communication circuit).
Vehicle state obtainer 1182 obtains, as vehicle state information, the running state of vehicle 10, a particular event which occurs inside vehicle 10, etc. Vehicle state obtainer 1182 notifies reliability manager 1186 of the vehicle state information received.
Monitor 1183 checks the integrity of a module, a VM file, or a memory region which is a monitoring target, using a hash value required for integrity check stored in monitoring information storage 1184. Monitor 1183 stores the check result into monitoring log storage 1185, and notifies reliability manager 1186 of the check result.
Reliability manager 1186 manages, the reliability of each of monitoring-target modules (monitoring targets) according to a vehicle event, using the vehicle state information received from vehicle state obtainer 1182, the check result received from monitor 1183, and a reliability table (to be described later with reference to
As indicated in
With reference to
As indicated in
Each action is one example of a vehicle event that is a target according to which the reliability is changed, that is, whose level of reliability is changed. Examples of actions include “(i)ntegrity check completed”, “(p)redetermined time elapsed”, “(r)unning of predetermined distance finished”, “(p)articular vehicle event occurred (positive)”, “(p)articular vehicle event occurred (negative)”, “(v)ehicle state changed (positive)”, “(v)ehicle state changed (negative)” and “(r)eset”. Examples are not limited thereto, and it is only necessary that at least one of these is included.
Here, “(i)ntegrity check completed” indicates that integrity check of monitoring-target software has been successfully completed; “(p)redetermined time elapsed” indicates that predetermined time has elapsed from a reference time or the time at which the previous integrity check was performed; and “(r)unning of predetermined distance finished” indicates that running of a predetermined distance from a reference position or the position at which the previous integrity check was performed has been finished. In addition, “(p)articular vehicle event occurred (positive)” indicates that a vehicle event which increases the security of a monitoring target has occurred, and “(p)articular vehicle event occurred (negative)” indicates that a vehicle event which decreases the security of a monitoring target has occurred. A particular vehicle event is, for example, an event based on an action onto vehicle 10 by an operator. Non-limiting examples of particular vehicle events include maintenance, repairment, etc., of a vehicle.
In addition, “(v)ehicle state changed (positive)” indicates that a vehicle state change which increases the security of a monitoring target has occurred, and, “(v)ehicle state changed (negative)” indicates that a vehicle state change which decreases the security of a monitoring target has occurred. In addition, “(r)eset indicates that a monitoring target has been rebooted (reboot has been successfully performed).
For example, 5 is added to the reliability of the monitoring target whose integrity check has been completed regardless of which one of monitoring targets the monitoring target is. In addition, when running of a predetermined distance has been finished, 1 is subtracted from the reliability of only monitoring-target module A among the monitoring targets.
As described above, vehicle events which change the levels of reliability may include integrity check of monitoring-target software, an elapse of predetermined time, and/or reset (reboot) of a monitoring target.
It is to be noted that the table in
With reference to
As indicated in
Each restriction detail indicates the detail of the function restriction that is placed onto a corresponding one of the monitoring targets whose reliability has fallen below the threshold value. The restriction detail is one example of restriction on at least a part of the functions of the corresponding one of the monitoring targets according to the reliability of the monitoring target. The threshold value indicates the reliability value based on which restriction is executed. In the example of
As described above, restriction on at least a part of functions according to the reliability of each monitoring target may include restriction on a communication function (for example, stop of a communication function) of the monitoring target or stop of operation by the monitoring target.
It is to be noted that the restriction on the communication function of the monitoring target is one example of stop of an access right that allows the monitoring target to access a particular resource. The access to the particular resource includes access to a communication function (communication module), access to a (particular) file, access to a part of functions of an OS, etc.
It is to be noted that in the present embodiment, each of the elements of monitoring module 1180 is mounted in vehicle 10. In other words, in the present embodiment, reliability manager 1186 and function restrictor 1163 are mounted in vehicle 10. In addition, vehicle network system 1000 may include at least reliability manager 1186 and function restrictor 1163.
(S1101) Monitoring module 1180 performs integrity check processing on monitoring-target module 1160 periodically. The integrity check processing here is intended to read a memory region of a module which is the target for the integrity check in order to detect manipulation of the memory region, calculate a hash value, compares the hash value calculated with the hash value that is held as correct data in advance to determine whether the hash value calculated matches the hash value as the correct data.
(S1102) Monitor 1183 determines whether the integrity check processing in Step S1101 has been successfully finished. In the case where monitor 1183 has determined that the integrity check processing in step S1101 has been successfully finished, for example, that the hash value calculated and the correct data match (“Y” indicated in S1102), reliability manager 1186 determines that no manipulation has been detected, and adding a value to the reliability of monitoring-target module A (monitoring-target module 1160) which is the check target. It is to be noted that, in the case where monitor 1183 has determined that the hash value calculated and the correct data match, reliability manager 1186 may change the reliability so that the possible maximum value is set (for example, reset the reliability).
In addition, in the case where monitor 1183 has determined that the integrity check processing in step S1101 has not been successfully finished, for example, that the hash value calculated and the correct data do not match (“N” indicated in S1102), reliability manager 1186 determines that an error has occurred, and skips the processing for changing the reliability of monitoring-target module A (monitoring-target module 1160).
It is to be noted that in the case where monitor 1183 has determined that the hash value calculated and the correct data do not match, reliability manager 1186 may subtract a value from the reliability of monitoring-target module A which is the check target. It is only necessary for reliability manager 1186 to change (update) the reliability by performing at least one of: adding the value to the reliability in the case where the integrity check has been successfully completed in step S1102; or subtracting the value from the reliability in the case where the integrity check has not been successfully completed in step S1102.
(S1103) Monitoring-target module 1170 receives maintenance processing from an outside tool (not illustrated), and notifies monitoring module 1180 that the maintenance processing has been successfully completed.
(S1104) Monitor 1183 of monitoring module 1180 determines whether or not the vehicle event notified corresponds to a particular vehicle event. For example, monitor 1183 checks (determines) whether or not the vehicle event notified is indicated in the reliability adjustment table. In the case where monitor 1183 has determined that the vehicle event notified is indicated in the reliability adjustment table (“Y” indicated in S1104), reliability manager 1186 changes the reliability of monitoring-target module B (monitoring-target module 1170) which is the target. The successful completion of the maintenance processing corresponds to “(p)articular vehicle event occurred (positive)”, reliability manager 1186 adds “1” to the reliability of monitoring-target module 1170. In the opposite case where monitor 1183 has determined that the vehicle event notified is not indicated in the reliability adjustment table (“N” indicated in S1104), reliability manager 1186 skips the processing for changing the reliability of monitoring-target module B (monitoring-target module 1170).
It is to be noted that reliability manager 1186 may: subtract a value from the reliability of monitoring-target module B which is the check target in the case where “(p)articular vehicle event occurred (negative)” has been obtained as the result of determining whether or not the vehicle event is the particular vehicle event (negative) in Step S1104; and skip the processing for changing the reliability of monitoring-target module B in the case where “(p)articular vehicle event occurred (negative)” has not been obtained as the result of the same.
(S1105) Reliability manager 1186: subtracts a value from the reliability of each of all the monitoring-target modules in the case where predetermined time has elapsed (“Y” indicated in S1105) as the result of determining whether the predetermined time (one example of predetermined time) has elapsed; and skips the processing for changing the reliability of each of all the monitoring-target modules in the case where predetermined time has not elapsed (“N” indicated in S1105) as the result of determining whether the predetermined time has elapsed. Reliability manager 1186 may subtract a preset value from the reliability of each of all the monitoring-target modules every time preset time has elapsed. It is to be noted that the subtraction of the preset value from the reliability of each of all the monitoring-target modules performed in the case where predetermined time has elapsed is a non-limiting example, and a value may be subtracted from the reliability of only one or more particular monitoring-target modules among all the monitoring-target modules.
(S1201) Monitoring-target module 1160 requests a connection to monitoring-target module 1170.
(S1202) Monitoring-target module 1170 asks monitoring module 1180 whether or not connection to monitoring-target module 1160 is allowed.
(S1203) Monitoring module 1180 compares the current reliability of monitoring-target module 1160 with a threshold value, makes a determination as to whether or not monitoring-target module 1160 is allowed to communicate with another module, and transmits the determination result to monitoring-target module 1170.
(S1204) Monitoring-target module 1170 responds to the connection request from monitoring-target module 1160, based on the determination result received from monitoring module 1180. Monitoring-target module 1170: transmits a connection response indicating that connection is allowed to monitoring-target module 1160 in the case where monitoring-target module 1160 is allowed to communicate with another module (“Y” indicated in S1204); and transmits an error notification to monitoring-target module 1160 in the case where monitoring-target module 1160 is not allowed to communicate with another module (“N” indicated in S1204), and ends the processing for connection to monitoring-target module 1160. It is to be noted that display according to the error notification may be provided by display 1013.
Function restrictor 1163 of monitoring-target module 1160: does not restrict communication with monitoring-target module 1170 in the case where it has obtained the connection response; and restricts communication (forbids communication) with monitoring-target module 1170 in the case where it has obtained the error notification.
In this way, function restrictor 1163 forbids monitoring-target module 1160 from communicating with monitoring-target module 1170 when the reliability of monitoring-target module 1160 is less than the threshold value in the case where monitoring-target module 1160 tries to perform communication with monitoring-target module 1170. In this way, it is possible to prevent monitoring-target module 1160 whose reliability is low from communicating with monitoring-target module 1170. In an example case where monitoring-target module 1160 is suffering from hacking, it is possible to prevent monitoring-target module 1170 from being affected by the hacking by function restrictor 1163 restricting communication therebetween.
The processing in steps S1201 to S1204 indicated in
(S1301) Reliability manager 1186 determines whether or not the current reliability of monitoring-target module A (monitoring-target module 1160) is less than a threshold value. The threshold value here is a threshold value for determining whether or not to execute reboot. Reliability manager 1186 checks the current reliability of monitoring-target module 1160, and ends the processing in the case where the current reliability is larger than or equal to the threshold value (“N” indicated in S1301). In the opposite case where the current reliability of monitoring-target module 1160 is less than the threshold value (“Y” indicated in S1301), reliability manager 1186 instructs monitoring-target module 1160 to reboot itself.
(S1302) Monitoring-target module 1160 reboots itself, and notifies monitoring module 1180 of completion of the reboot. The completion of the reboot here means that the reboot has been successfully completed.
(S1303) Reliability manager 1186 adds a value to the reliability of monitoring-target module A (monitoring-target module 1160). It is only necessary for reliability manager 1186 to perform at least one of: adding the value to the reliability of monitoring-target module A or resetting the reliability to an initial value when the reboot has been successfully completed, or subtracting a value from the reliability when the reboot has not been successfully completed.
Vehicle network system 1000 (monitoring system) indicated in Embodiment 1 is capable of securing safety as the entirety of integrated ECU 1100 by restricting the part of functions according to the time elapsed after the date and time of the last check of the monitoring-target module that is the monitoring target to reduce the possibility that the monitoring-target module whose reliability has decreased is manipulated and becomes a threat against other modules.
It is to be noted that the present disclosure may be implemented as monitoring module 1180 (one example of a monitoring device) for monitoring vehicle 10 or each of monitoring targets that operate inside the vehicle. The present disclosure may be implemented as monitoring module 1180 included in vehicle network system 1000, together with function restrictor 1163 (one example of a function restricting device) for placing a function restriction on monitoring vehicle 10 or monitoring targets that operate inside the vehicle. Monitoring module 1180 may be configured to include reliability manager 1186 that manages the reliability of each monitoring target according to a vehicle event of the monitoring target. In addition, the present disclosure may be implemented as a function restricting device that is include in vehicle network system 1000 and is configured to include function restrictor 1163 which restricts at least a part of functions of each monitoring target based on the reliability according to a vehicle event. In other words, each of reliability manager 1186 and function restrictor 1163 may be implemented as an individual device.
Although a method performed by monitoring module 1180 in order to manage the reliability of each monitoring target has been described in Embodiment 1, in this variation, an example of a method in which a monitoring-target module manages the own reliability is described with reference to the drawings. It is to be noted that descriptions of elements similar to those in Embodiment 1 are omitted. Furthermore, elements and processing steps similar to those in Embodiment 1 are assigned with the same numerical references, and descriptions thereof are omitted.
As illustrated in
Vehicle state obtainer 11164 obtains, as vehicle state information, the running state of vehicle 10, a particular event which occurs inside vehicle 10, etc. For example, vehicle state obtainer 11164 may obtain vehicle state information from each of sensors, etc., mounted in vehicle 10. Vehicle state obtainer 11164 notifies reliability manager 11165 of the vehicle state information obtained.
Reliability manager 11165 manages own reliability (here, monitoring-target module 11160) using a reliability management table. Reliability manager 11165 stores the reliability management table in which the value of the own reliability is stored into table storage 11166.
As illustrated in
With reference to
In addition, reliability manager 11165 restricts a part of functions of the monitoring-target module (that is the monitoring target itself) using a function restriction table. The function restriction table is, for example, stored in table storage 11166 in advance. One example of the function restriction table is the same as in Embodiment 1 (see
As illustrated in
Communication unit 11181 takes a role for communicating with hypervisor 1120, the OS which operates in another VM, the module and a process which operate in the OS, etc. Communication unit 11181 transmits mainly the result of check by monitor 11183 to another module. Communication unit 11181 is configured to include communication circuitry.
Monitor 11183 checks the integrity of a module, a VM file, or a memory region which is a monitoring target, using a hash value (correct data), etc., required for integrity check stored in monitoring information storage 1184. Monitor 11183 stores the check result into monitoring log storage 1185, and transmits the check result to another module via communication unit 11181.
(S11001) Monitor 11183 of monitoring module 11180 determines whether the integrity check processing in Step S1101 has been successfully finished. Monitor 11183 determines that no manipulation, etc., have been detected in the case where the integrity check processing in Step S1101 has been successfully finished (“Y” indicated in S11001), and notifies monitoring-target module 11160 of the check result notification indicating the fact. In the opposite case where the integrity check processing in Step S1101 has not been successfully finished (“N” indicated in S11001), monitor 11183 determines that an error has occurred, and skips issuing a notification to monitoring-target module 11160.
Monitor 11183 performs, for example, Steps S1101 and S11001 for each of a plurality of monitoring targets. It is to be noted that, when the integrity check processing in Step S1101 has not been successfully finished, monitor 11183 may notify monitoring-target module 11160 of the check result notification indicating the fact.
(S11002) Reliability manager 11165 of monitoring-target module 11160 that has received the notification updates (here, adds a value to) the own reliability that monitoring-target module 11160 itself manages. It is to be noted that reliability manager 11165 may change the own reliability (of monitoring-target module 11160) upon obtaining the check result notification indicating that the integrity check processing has been successfully finished.
(S11003) Monitoring-target nodule 11170 executes maintenance processing upon receiving maintenance processing from an outside tool (not illustrated).
(S11004) A reliability manager of monitoring-target module 11170 determines whether or not a vehicle event that has occurred corresponds to a particular vehicle event. For example, the reliability manager checks (determines) whether or not the vehicle event occurred is indicated in the reliability adjustment table. When determining that the vehicle event is indicated therein (“Y” indicated in S11004), the reliability manager changes the own reliability (monitoring-target module B). Successful completion of the maintenance processing corresponds to “(p)articular vehicle event occurred” indicated in
(S11005) Each of reliability manager 11165 of monitoring-target module 11160 and the reliability manager of monitoring-target module 11170 determines whether or not predetermined time has elapsed. Reliability manager 11165 updates (here, subtracts a value from) the own reliability (the reliability of monitoring-target module A) when predetermined time has elapsed (“Y” indicated in S11005), and skips the processing for changing the own reliability when predetermined time has not elapsed (“N” indicated in S11005). Likewise, the reliability manager of monitoring-target module 11170 updates (here, subtracts a value from) the own reliability (the reliability of monitoring-target module B) when predetermined time has elapsed (“Y” indicated in S11005), and skips the processing for changing the own reliability when predetermined time has not elapsed (“N” indicated in S11005).
In this way, in Step S11005, each of monitoring-target module 11160 and monitoring-target module 11170 subtracts a preset value from the own reliability that the module itself manages.
(S11201) Monitoring-target module 11160 determines whether or not the own reliability is larger than or equal to a threshold value. It can be said that monitoring-target module 11160 compares the current own reliability with the threshold value, and makes a determination as to whether or not monitoring-target module 11160 itself can communicate with the another module. Monitoring-target module 11160 ends the processing as an error when the determination result indicates that communication is impossible (“N” indicated in S11201), that is, when the reliability has been determined to be less than the threshold value. When the determination result indicates that communication is possible (“Y” indicated in S11201), that is, when it has been determined that the reliability is larger than or equal to the threshold value, monitoring-target module 11160 makes a request for connection to monitoring-target module 11170.
It is to be noted that, when the determination result indicates that communication is impossible, monitoring-target module 11160 may notify monitoring module 11180 of the determination result. In addition, the threshold value may be preset and, for example, may be stored in table storage 11166.
(S11202) Upon receiving the connection request from monitoring-target module 11160, monitoring-target module 11170 determines whether or not the current own reliability is larger than or equal to the threshold value. It can be said that monitoring-target module 11170 compares the current own reliability with the threshold value, and makes a determination as to whether or not monitoring-target module 11170 itself can communicate with the another module. Based on the determination result, monitoring-target module 11170 responds to the connection request from monitoring-target module 11160. In an example case where the determination result indicates that communication is impossible (“N” indicated in S11202), that is, when the reliability has been determined to be less than the threshold value, monitoring-target module 11170 transmits an error notification to monitoring-target module 11160, and ends the connection processing to monitoring-target module 11160. In the opposite case where the determination result indicates that communication is possible (“Y” indicated in S11202), that is, when the reliability has been determined to be larger than or equal to the threshold value, monitoring-target module 11170 transmits a connection response that allows connection to monitoring-target module 11160.
In this way, in the present variation, each of monitoring-target module 11160 and monitoring-target module 11170 determines the own reliability. Only when both the reliability of monitoring-target module 11160 and the reliability of monitoring-target module 11170 are greater than or equal to the threshold value, monitoring-target module 11160 and monitoring-target module 11170 are to communicate with each other. In other words, in the present variation, monitoring module 11180 does not get involved in the determination as to whether communication is possible or impossible. It is to be noted that the threshold value used in the determination in Step S11201 and the threshold value used in the determination in Step S11202 may be a common value, or may be mutually different values.
It can also be said that, in the case where monitoring-target module 11160 and monitoring-target module 11170 try to communicate with each other, the function restrictors (here, function restrictor 1163 and the function restrictor included in monitoring-target module 11170) included in the vehicle network system according to the present variation forbid communication between monitoring-target module 11160 and monitoring-target module 11170 when at least one of the reliability of monitoring-target module 11160 or the reliability of monitoring-target module 11170 is less than the threshold value.
(S11301) Monitoring-target module 11160 determines whether or not current own reliability is less than a threshold value. Monitoring-target module 11160 ends the processing when the reliability is greater than or equal to the threshold value (“N” indicated in S11301). In addition, monitoring-target module 11160 instructs reboot when the own reliability of the module is less than the threshold value (“Y” indicated in S11301).
(S11302) When the result in Step S11301 indicates that the own reliability is less than the threshold value, monitoring-target module 11160 reboots.
(S11303) When the reboot has been successfully completed, monitoring-target module 11160 adds a value to the own reliability. Monitoring-target module 11160 adds the value to the own reliability because there is a possibility that the reboot is not successfully completed when the memory region of the module, or the like has been manipulated. It is to be noted that, when the reboot has been successfully completed, monitoring-target module 11160 sets the own reliability as an initial value.
It is to be noted that the processing indicated in
In the vehicle network system indicated in the variation of Embodiment 1, it is possible to reduce the size of codes required for monitoring module 11180 by the self-management of the reliability of each module, and facilitate operation on the TrustedOS. In this way, a cost reduction effect can be expected, and it becomes possible to secure safety of the entirety of integrated ECU 1100 by reducing the possibility that the monitoring-target module whose reliability has decreased in a wider system is manipulated and becomes a threat against the other modules.
Monitoring module 1180 (one example of a monitoring device) described in Embodiment 1 (1) is present on the ECU, (2) handles the module unit that operates on the particular ECU in vehicle 10 as a monitoring target whose reliability is to be monitored, and (3) represents the reliability as a numerical value. However, monitoring module 1180 may (1′) be present outside vehicle 10, (2′) handle, for example, a unit of a VM, an ECU, or vehicle 10 as a monitoring target whose reliability is to be monitored, without being limited to the module unit, and (3′) represent the reliability as a state of OK, NG, or the like instead of the numerical value. Embodiment 2 describes one example as indicated above.
Here, network system 2000 according to an embodiment of the present disclosure is described with reference to the drawings. It is to be noted that descriptions of drawings similar to those in Embodiment 1 are omitted. Furthermore, elements and processing steps similar to those in Embodiment 1 are assigned with the same numerical references, and descriptions thereof are omitted.
As illustrated in
As illustrated in
In the present embodiment, monitoring-target OS 2140 and monitoring-target OS 2150 are monitoring targets. Monitoring module 2180 which monitors monitoring-target OS 2140 and monitoring-target OS 2150 operate on TEE 1130. Furthermore, function restricting module 2160 operates on monitoring-target OS 2140, and function restricting module 2170 operates on monitoring-target OS 2150. Function restricting module 2160 and function restricting module 2170 each have functions similar to those of function restrictor 1163 illustrated in
It is to be noted that, hereinafter, monitoring-target OS 2140 is also referred to as monitoring-target OS-A, and monitoring-target OS 2150 is also referred to as monitoring-target OS-B.
As illustrated in
Communication unit 2181 performs communication with hypervisor 1120, the OS that operates on each of other VMs, the module, the process, etc., that operate in the OS which operates in another VM, and further takes a role for communicating with server 20 outside vehicle 11. Communication unit 2181 notifies server 20 of mainly the result of check by monitor 2183, a vehicle state received by vehicle state obtainer 2182, etc., and notifies the another module of the result of a determination by server 20. Communication unit 2181 includes communication circuitry (a communication module).
Vehicle state obtainer 2182 obtains, as vehicle state information, the running state of vehicle 11, a particular event which occurs inside vehicle 11, etc. Vehicle state obtainer 2182 notifies server 20 of the vehicle state information received via communication unit 2181. It is to be noted that the vehicle state information can be obtained from various kinds of sensors mounted in vehicle 11.
Monitor 2183 checks the integrity of a VM file, or a memory region which is a monitoring target, using for example a hash value (correct data), etc., required for integrity check stored in monitoring information storage 1184. Monitor 2183 stores the check result into monitoring log storage 1185, and notifies server 20 of the check result via communication unit 2181.
As illustrated in
Communication unit 21 takes a role for communicating with vehicle 11.
Display unit 22 is a display device which displays information indicating a monitoring target (for example, monitoring-target module) and the reliability of the monitoring target together. Display unit 22 is implemented as, for example, a liquid crystal display, or the like.
Reliability manager 23 manages the reliability of the monitoring-target module, using vehicle state information, a check result, and a reliability management table which have been received from vehicle 11. The reliability management table indicates, for each of monitoring-target modules, the reliability of the monitoring target, and is stored in table storage 24.
As indicated in
With reference to
As indicated in
With reference to
As indicated in
Although each of
(S2103) Monitoring module 2180 determines whether an error has occurred, for example, whether the integrity check processing in Step S1101 has been successfully finished.
(S2104) When determining that an error has occurred (“Y” indicated in S2103), that is, when the integrity check processing in Step S1101 has not been successfully finished, monitoring module 2180 changes the reliability of monitoring-target OS-A (monitoring-target OS 2140) with reference to a reliability adjustment table. For example, monitoring module 2180 sets the reliability to an OK state when the integrity check has been successfully finished, or the number of times of success of integrity check has exceeded a predetermined number of times, and sets the reliability to an NG state when the integrity check has been failed, or the number of times of error of integrity check has exceeded a predetermined number of times.
(S2105) Monitoring module 2180 stores, as a log, the result of the reliability changed in Step S2104, and uploads the same result to server 20.
(S2201) Monitoring-target OS 2140 requests for connection to monitoring module 2180.
(S2202) Monitoring module 2180 asks server 20 whether connection with monitoring-target OS 2140 is allowed.
(S2203) Server 20 determines whether connection to outside is allowed based on the reliability of monitoring-target OS-A (monitoring-target OS 2140). For example, server 20 checks the reliability of monitoring-target OS 2140 and a function restriction table, makes a determination as to whether monitoring-target OS 2140 is allowed to communicate with outside, and transmits the determination result to monitoring module 2180.
(S2204) Monitoring module 2180 responds to the connection request from monitoring-target OS 2140 based on the determination result received from server 20. Monitoring module 2180 transmits, to monitoring-target OS 2140, a connection allowance response that allows connection when monitoring-target OS 2140 is allowed to communicate with outside (“Y” indicated in S2204). When monitoring-target OS 2140 is not allowed to communicate with outside (“N” indicated in S2204), monitoring module 2180 transmits an error notification to monitoring-target OS 2140, and ends the processing for external connection in monitoring-target OS 2140.
Upon obtaining the connection allowance response, function restricting module 2160 (one example of a function restrictor) of monitoring-target OS 2140 does not restrict communication with a device (one example of a communication target) present outside vehicle 11. Upon obtaining the error notification, function restricting module 2160 restricts (forbids) communication with the device present outside vehicle 11.
In this way, in the case where monitoring-target OS 2140 tries to communicate with the outside device of vehicle 11, function restricting module 2160 forbids monitoring-target OS 2140 from communicating with the outside device when the reliability of monitoring-target OS 2140 is less than a threshold value. In this way, it is possible to prevent monitoring-target OS 2140 whose reliability is low from communicating with the device present outside vehicle 11.
The processing in Steps S2201 to S2204 indicated in
(S2301) Server 20 determines whether the current reliability of monitoring-target OS-A (monitoring-target OS 2140) is less than a threshold value. Server 20 ends the processing when the reliability of monitoring-target OS 2140 is not less than the threshold value (the reliability is larger than or equal to the threshold value) (“N” indicated in S2301), that is, when the reliability indicates an OK state. When the reliability of monitoring-target OS 2140 is less than the threshold value (“Y” indicated in S2301), that is, when the reliability indicates an NG state, server 20 instructs reboot of monitoring-target OS 2140.
(S2302) Upon obtaining the instruction indicating reboot of monitoring-target OS 2140 from server 20, monitoring module 2180 instructs monitoring-target OS 2140 to reboot (transmits a reboot instruction).
(S2303) In the case where the reboot has been successfully completed, monitoring-target OS 2140 notifies monitoring module 2180 of the completion of the reboot (transmits a reboot completion notification).
(S2304) Upon obtaining the notification indicating the completion of the reboot, monitoring module 2180 notifies server 20 that the reboot of monitoring-target OS 2140 has been completed (transmits a reboot completion notification).
(S2305) Server 20 changes the reliability of monitoring-target OS 2140. Here, server 20 adds a value to the reliability of monitoring-target OS 2140.
Monitoring device indicated in Embodiment 2 places a function restriction according to the check result by the monitoring-target OS, thereby reducing the possibility that the monitoring-target module whose reliability has decreased is manipulated and becomes a threat against the other modules, which makes it possible to secure safety of the entirety of integrated ECU 2100.
Although the present disclosure has been described based on each of the embodiments, the present disclosure is not limited to the embodiments as a matter of course. Cases as described below are also included in the present disclosure. It is to be noted that each of
(1) Although, in each of the embodiments, integrity check of software is indicated as one example where a value is added to reliability, and a vehicle event that is an elapse of predetermined time is indicated as one example where a value is subtracted from reliability, vehicle events are not limited thereto. For example, as indicated in
(2) Although the variation of Embodiment 1 describes the example in which the monitoring module performs integrity check of software of the monitoring-target module, and directly notifying the result, all the modules inside a vehicle may be notified of such a result as a vehicle event. The monitoring-target module can receive the vehicle event, and reflect the result to the own reliability.
(3) Although each of the above embodiments describes the monitoring device which handles, as its target, the integrated ECU that is an introduction environment for the hypervisor. For example, as illustrated in
(4) Although each of the above embodiments describes the example in which the unit of a monitoring target to which reliability is to be assigned is a module or an OS that operates on the OS, the units of a monitoring target are not limited thereto. As illustrated in
(5) The monitoring module that operates on the TEE monitors all the monitoring targets in each of the above embodiments, which is a non-limiting example. For example, a monitoring module may operate on a hypervisor as illustrated in each of
(6) Although each of the above embodiments describes the example in which the module or OS reboots itself as one example of making reliability closer to an initial value, an exclusive module for making reliability closer to an initial value may be disposed in a region outside a target. For example, as illustrated in
(7) Although each of the above embodiments describes the example in which the module or OS places a restriction as one example of function restriction, an exclusive module for function restriction may be disposed in a region outside a target. For example, as illustrated in
(8) Although each of the above embodiments uses reliability for determination in internal processing, a display screen that displays the reliability of each of monitoring targets may be provided so that the reliability is managed inside a server or vehicle.
(9) Each of the devices according to the above embodiments is, specifically, a computer system configured with a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, and so on. A computer program is stored in the RAM or the hard disk unit. The respective devices achieve their functions through the microprocessor's operations according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve predetermined functions.
(10) A part or all of the elements of the respective devices may be configured with a single system-LSI (Large-Scale Integration). The system-LSI is a super-multi-function LSI manufactured by integrating structural units on a single chip, and is specifically a computer system configured to include a microprocessor, a ROM, a RAM, and so on. A computer program is stored in the RAM. The system-LSI achieves its function through the microprocessor's operations according to the computer program.
(11) Furthermore, each of units of the respective devices may be made as separate individual chips, or as a single chip to include a part or all thereof.
Furthermore, although system-LSI is mentioned here, but there are instances where, due to a difference in the degree of integration, the designations IC, LSI, super LSI, and ultra LSI may be used. Furthermore, the means for circuit integration is not limited to an LSI, and implementation with a dedicated circuit or a general-purpose processor is also available. It is also possible to use a Field Programmable Gate Array (FPGA) that is programmable after the LSI is manufactured, and a reconfigurable processor in which connections and settings of circuit cells within the LSI are reconfigurable.
Furthermore, if integrated circuit technology that replaces LSI appear through progress in semiconductor technology or other derived technology, that technology can naturally be used to carry out integration of the functional blocks. Biotechnology is anticipated to apply.
(12) A part or all of the elements constituting the respective devices may be configured as an IC card which can be attached to and detached from the respective devices or as a stand-alone module. The IC card or the module is a computer system configured from a microprocessor, a ROM, a RAM, and so on. The IC card or the module may also be included in the aforementioned super-multi-function LSI. The IC card or the module achieves its functions through the microprocessor's operations according to the computer program. The IC card or the module may also be implemented to be tamper-resistant.
(13) The present disclosure may be implemented as the above method. In addition, the present disclosure may be implemented as computer programs for executing the above method, using a computer, and may also be implemented as digital signals including the computer program. The computer program may be a computer program for causing a computer to execute each of unique steps included in the monitoring method indicated in any one of
Furthermore, the present disclosure may also be implemented as a computer program or digital signals stored on computer-readable recording media such as a flexible disc, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), and a semiconductor memory. Furthermore, the present disclosure may also be implemented as the digital signals stored on these recording media.
Furthermore, the present disclosure may also be implemented as the above computer program or digital signals transmitted via a telecommunication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, and so on.
The present disclosure may also be implemented as a computer system including a microprocessor and a memory, in which the memory stores the above computer program and the microprocessor operates according to the computer program.
Furthermore, it is also possible to execute another independent computer system by transmitting the program or the digital signals stored on the aforementioned recording media, or by transmitting the program or digital signals via the above network and the like.
(14) In addition, the vehicle network system according to each of the embodiment, etc., may be implemented as a single device, or a plurality of devices. When the vehicle network system is implemented by a plurality of devices, the elements of the vehicle network system may be divided into the plurality of devices in any way. When the vehicle network system is implemented by a plurality of devices, communication methods used between the plurality of devices are not particularly limited, and may be wireless communication or wired communication. In addition, wireless communication and wired communication may be combined between the plurality of devices.
(15) Furthermore, the order of steps which are executed in each of sequence diagrams are provided as an example for specifically explaining the present disclosure, and thus an order other than the above order is also possible. In addition, a part of the steps may be executed at the same time (in parallel) with another one or more steps, or a part of the steps may not be executed.
(16) In addition, at least one of the reliability management table, the reliability adjustment table, and the function restriction table may be different for each monitoring target or may be common to each monitoring target.
(17) In addition, although each of the above embodiments describes the example in which functions of a monitoring target are restricted using the reliability to which a value is added when the safety of the monitoring target has been confirmed by, for example, completion of integrity check, it is also possible to restrict functions of a monitoring target using a risk to which a value is added when the safety of the monitoring target has not been confirmed by, for example, uncompletion of integrity check. The reliability and risk are examples of indicators indicating security protection states of a vehicle.
(18) The embodiments and the variation may be combined in any way. In addition, the techniques according to the present disclosure are not limited thereto, and are applicable to other embodiments obtainable by optionally performing change, replacement, addition, omission, etc. For example, embodiments that may be conceived by those skilled in the art, as well as embodiments resulting from optional combinations of elements and functions from different embodiments may be included in the present disclosure as long as such embodiments do not depart from the scope of the present disclosure.
The present disclosure makes it possible to clarify the reliability of each of elements of a system, which allows to reduce damage that may be generated if abnormality occurs, thereby maintaining a safety state of the system.
Number | Date | Country | Kind |
---|---|---|---|
PCT/JP2021/020681 | May 2021 | WO | international |
This is a continuation application of PCT International Application No. PCT/JP2022/018819 filed on Apr. 26, 2022, designating the United States of America, and which is based on and claims priority of PCT International Application No. PCT/JP2021/020681 filed on May 31, 2021. The entire disclosures of the above-identified applications, including the specifications, drawings and claims are incorporated herein by reference in their entirety.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2022/018819 | Apr 2022 | US |
Child | 18517128 | US |