Patent Application
20240353201
The present invention relates to a personalized combat simulation equipment according to the preamble of claim 1. Furthermore, it relates to a use of the personalized combat simulation equipment and a combat simulation method using the personalized combat simulation equipment.
State of the art in live simulation is that weapon effects are simulated using laser or radio signals. The effect of the ammunition is transmitted to a target equipped with laser detectors by means of a coded laser signal or radio telegrams. The resulting simulated effect of the weapon is decoded at the target, evaluated, and registered and/or transmitted to a higher-level control center via a radio network. In addition, each exercise participant must be located in the open terrain as well as inside buildings using additional infrastructure so that the respective positions of the targets and their condition are monitored and recorded at the control center. Thus, the use of weapons, their time of impact on the targets and the position of all participants are recorded at the control center.
However, this state of the art in live simulation still maintains disadvantages compared to real missions. The high level of infrastructure required to perform live simulations means that realistic training is only possible in a fixed instrumented environment on the one hand, and only with prior complex and mobile instrumentation on the other hand. This increases the costs for such exercises and the training sessions require correspondingly long preparation times. A lot of important information are not collected and evaluated with the current state of live simulation. E.g. the handling of weapons is not evaluated, only the effect on the targets.
Especially in the area of training of special forces, police and military, it is today only possible to a limited extent to conduct such exercises in a real environment (department stores, airplanes, public buildings) because its prior instrumentation would be too costly.
Therefore, an object of the present invention is a weapon simulating device which imposes fewer demands regarding the training environment.
Such a device is defined in claim 1. The further claims define preferred embodiments.
Another, preferred aim of the invention is, on the one hand, to expand the training with weapon systems by means of the use of artificial intelligence and additional sensor technology with further collected information and, on the other hand, to reduce the expenditure on instrumentation of weapons, of participants and of infrastructure as far as possible. Training in the use of weapon systems is to be simplified as far as possible. This should make it possible to increase the efficiency of the exercises. In addition, the quality of the evaluations with regard to the behaviour of the training participants should be improved.
The advantage of the invention is that by using it, on the one hand, exercises are very quickly and efficiently carried out in previously unknown terrain or buildings with a minimum of preparation time. On the other hand, the invention allows a much higher degree of additional evaluation of the correct behaviour of the exercise participants to increase the quality of the trainings. It is possible to record wrong behaviour at the wrong time. The goal is to optimally prepare the participants for their real mission and to optimize their behaviour during the real mission, which ultimately leads to their increased safety and success.
The invention will be further explained by way of preferred exemplary embodiment with reference to the Figures:
Artificial intelligence (AI) is a broader term existing since the mid-1950s in computer sciences. Nevertheless, even today no unique and clear definition of the term exists among scholars in the field, but rather a broader understanding of a task that a machine aims to perform that is to mimic human behaviour. Scholars in the field of computational sciences face the same problem regarding the task to find one unique and clear definition of further developments in the field. In the mid-1980s the term machine learning came up from the definition issues of AI to help describe how a task is performed by the computer. Nevertheless, still an abundance of definitions exists about the term machine learning. Although more progress is made in the field of computational sciences, as for example the development of deep learning from the field of machine learning that, broadly described, uses neural network structures to solve complex problems, in this patent the term artificial intelligence (AI) is used to describe highly complex algorithms in the field of machine learning and deep learning.
A Deep Neural Network is a neural work with at least one, generally more layers of nodes between the input and the output layer of nodes.
Human Recognition and Pose Recognition defines the problem of localization of human joints (key points as elbows, wrists, etc.) in images or videos. It also defines the search for specific human poses in space of all articulated poses. There exist different approaches to 2D and 3D Human Pose Estimation. The main component of human pose estimation builds the modelling of the human body. The three most used types of body models are: skeleton-based model, contour-based and volume-based model. With the help of Deep Neural Networks, such models could be recognized and tracked in real time. Within this invention no specific method is mentioned because most of them could be applied for this application. The only big differences between different Human Pose Estimation methods are the accuracy of recognition and the performance in such frameworks. To reach a higher accuracy of the human body recognition such a Deep Neural Network can be deeper trained with additional training data of available recorded poses. Training data may be images (e.g. still images taken by a camera) of a person in different poses with and without weapon and each time a thread label assigned which indicates the thread level. The images may also be simplified. The same is done for distinguishing if objects are dangerous: Images of objects to which thread levels are assigned are shown to the KI. Generally, the training is performed in a separate process ahead of the actual simulation, and during simulation, no further training of the KI occurs.
The Hit zone recognition allows detecting the exact hit zone on the human body during the simulated shot firing. It is based on the Human Body and Pose Recognition and captures during the shot triggering the aimed zone. This captured zone will be treated and recorded afterwards.
The Threat Recognition is also based on the Human Body and Pose Recognition methods. Specific dangerous appearing poses could be trained with the help of a Deep Neural Network. Such dangerous poses are then recognized in a real time video stream. This is combined with an object detection also based on Deep Neural Networks to detect dangerous objects carried by such a recognized human body.
According to one aspect, the Identification is based on Face Detection and Recognition with the help of Deep Learning, i.e. an AI technique. Also Face recognition is based on different existing methods. Most of the methods have in common that first a face must be recognized in a real time video or in a picture. It is often done by a method invented in 2005 called Histogram of Oriented Gradients (HOG, cf. U.S. Pat. No. 4,567,610, herein incorporated by reference). Then, the face is often identified with a method called face landmark estimation. The faces to be identified must be provided in a training set and trained with such a facial recognition Deep Neural Network.
The Georeferencing is based on the detection of beacons or markers called April Tags or Fiducial markers. April tags have been developed in the APRIL Robotics Laboratory at the University of Michigan led by Edwin Olson, cf. https://april.eecs.umich.edu/software/apriltag. An April tag resembles a QR code, yet with a far simpler structure and using bigger blocks. Its data payload is lower, e.g. 4 to 12 bits, and it is designed that the 3D position in relation to a sensor or a camera can be precisely determined. These tags or markers will be mounted on georeferenced points in the simulated area. The markers could be detected based on image treatment and decoded to read out the marked position.
Object Detection/Object Recognition describes a collection of related computer vision tasks that involve object identification in images or real time video streams. Object recognition allows identifying the location of one or more objects in an image or video stream. The objects are recognized and localized. This is performed with the help of AI, in particular Deep Learning algorithms used in many other applications. There exist a lot of pretrained Object Detection networks which could be applied in this invention. Additional objects which are not known in those pretrained networks could be trained additionally with additional prepared trainings sets.
Simultaneous Localization and Mapping (SLAM) allows the tracking and mapping within an unknown environment. Multiple SLAM algorithms exist and can be applied to such an application. Those systems are often based on the use of a camera and they allow to create a map in real time as well as to recognize the position based on the treatment of the video stream. The map is created online and georeferencing to a real map is possible at any time.
Speech Recognition/Speech to Text algorithms enable the recognition and translation of spoken language into text with the help of Neural Networks. There exist a lot of different possible speech recognition algorithms applicable in this invention. The trained networks could be extended with additional training sets used in tactical operations. After the speech to text translation, the text will be analysed for used keywords important for the tactical operations.
The task of the preferred embodiment is to provide weapon systems and exercise participants with an intelligent weapon simulator. This intelligent weapon simulator shall provide the following capabilities to better evaluate the correct behaviour of exercise participants:
As shown in
The AI Unit 6 and C2 Unit 8 communicate with each other via a radio link 51. Accordingly, the AI Unit 6 and the C2 Unit 8 each have an antenna 53. The AI Unit 6 and the Sensor Unit Arrangement 9 are centrally supplied with power by a power supply 7, e.g. a modular battery pack. The Sensor Unit Arrangement 9 controls the various sensors, including the distance sensor 1, camera 2, camera-based position sensor 3, and the shot trigger unit 5 via a Sensor Control subunit 4. All sensor information is collected by the Sensor Control 4 and then forwarded to the AI Unit 6 for further processing. The AI Unit 6 calculates the various results using Artificial Intelligence. The AI Unit 6 is arranged to send the resulting data to the C2 Unit 8 for real-time display and central history recording. Therefore, all data is sent to the C2 unit 8 in real time. The AI Unit 6 provides to the Unit C2 8, among other things, the 3D position of the Sensor Unit Arrangement 9, the Human Recognition Information, the Pose Recognition Information, the Shot Trigger Information, the Identified Objects Information, the Attack Threat Detection Information, the Distance to the Targeted Objects, and the Geo-Reference Information.
A gyroscope unit may be present in order to determine angular movement, angular speed and orientation in space.
It is not required that the personalized combat simulation equipment described in its entirety is attached to a weapon system. Merely, “attached to a weapon system” includes the situation, where parts of the equipment required for observing the use of the weapon system (e.g. camera, stereo-camera, LIDAR, gyro) are actually attached to the weapon, and the remaining parts are worn by the participant, f. i. integrated in a harness or as a belt accessory, or are remotely positioned. Thereby, the handling properties (weight, dimensions) of the weapon system are less influenced. Any intermediate distribution of the parts of the personalized combat simulation equipment between actually attached to the weapon system and a component of the section worn by the participant himself, is conceivable.
As the training system is informed of the position of the participants 12, 47, it is able to derive from the position of the targeting participant 47, the virtual trajectory of the shot and the position of the possibly targeted other persons and objects 12 if a participant in simulation or an object has been hit. Face recognition may support distinguishing between persons 47 the positions of which are too close to each other to determine which one has been virtually hit.
According to a specific, preferred aspect, the position of a weapon, or more exactly the location of the round before firing, and its target vector (corresponding to the simulated trajectory of the fired round) are published to the targets 12. Their equipment determines on the basis of these data and its own position if the person is hit. In the affirmative, the respective information (who hast been hit, where has the hit occurred) is published to the other devices and in particular to the C2 unit 8.
The Human Recognition unit 20 recognizes people on the Video Data Stream and tracks them in real time. The AI unit 6 thus distinguishes humans from all other objects in the video stream. The Pose Recognition unit 21 detects the limbs and joints of the detected humans in the video stream. This detection is used for the Threat Recognition unit 24 to detect and process threatening and attacking people in the video stream. The Hit Zone Recognition unit 22 compares the aimed and targeted areas on the recognized human body and forwards the results (head, neck, upper arms, lower arms, various torso areas, as well as upper and lower legs (femoral/shank)). The Identification Unit 23 is arranged to recognize people by means of facial recognition or tag recognition. Recognition via the position of the target is also possible. Thus, the AI Unit 6 is capable of assigning hits on targets to the corresponding recognized individuals. The Object Recognition function 19 is capable of recognizing, evaluating, and to pass on further object information within the video stream. The Georeferencing function 25 is used for geolocation in 3D space. By detecting a previously georeferenced marker, the Sensor Unit Arrangement 9 and, therefore, the weapon 10 is georeferenced as well. The Distance Measurement function 26 measures the distance to the respective targets in real time by means of the Distance Measurement unit 15. Thus, in addition to the distance and tracking unit (26 & 27), the respective target line and orientation of the weapon is determined. The Positioning Sensor unit 16, together with the Indoor/Outdoor Tracking unit 27, enables absolute positioning in space in real time using a SLAM (Simultaneous Localization and Mapping) software. This absolute positioning method is called an infrastructureless localisation, i.e. a localisation without relying on data furnished by the infrastructure of a known simulation environment thoroughly equipped with sensors, cameras and the like.
The Shot Trigger unit 28 detects fired shots of the weapon (e.g. by detecting a typical vibration and/or flash of the weapon) or is arranged to be triggered manually by the user via a manual Shot Trigger Button 5. All these data are collected by the Data Out Pipeline & Transmission unit 29 and transmitted as a data stream via a radio network 51 to the C2 unit 8 for display and recording for further processing. At the AI unit 6, a connected audio source (e.g. headset splitter) 55 is used to evaluate the radio transmissions during the mission. This is treated over the Audio Stream Treatment 17 and forwarded to the Speech Recognition 18. The Speech Recognition 18 analyses the radio transmission and detects mission commands using predefined keywords.
The main function of the C2 unit 8 is the management of the different sensor units and the AI unit 6 as well as their data processing and recording of the actions with the simulator. The Weapon Communication Management takes care of the management and communication with the different Sensor Unit Arrangements 9 and AI units 6 in the network. Thus, the system is arranged to track multiple participants 47, 12 and to record, manage and store their actions simultaneously. The 2D/3D mapping and georeferencing unit georeferences the weapon 10 in the 3D system of the C2 unit 8 as soon as a tag (e.g. an APRIL tag) is detected by the corresponding Sensor Unit Arrangement 9. The georeferencing is done automatically. The direction of the weapon, as well as its orientation in 3D space, is detected by the Weapon Tracking System and displayed in the MMI (Display). Selecting the Live View Control (sub-display) in the MMI, the system is arranged to show the live video image of the individual Sensor Unit Arrangements 9.
The Shot Action View unit displays and stores the current live image of the corresponding unit with all associated information (position, hit information or hit zone, stress level based on attached vitality sensors, direction of the weapon, distance, threat detection and identification of the target). Vitality sensors retrieve values of physiological parameters like speed of movement, electrical resistance of skin, temperature of skin, temperature of blood, pulse etc.
The Pose Recognition Unit 21 together with the Threat Recognition Unit 24 recognizes the detected poses and their threat potential in the live image of the corresponding Sensor Unit Arrangements 9. This information is displayed in the MMI. The Object Recognition unit 19 allows recognition of further important objects which have been previously defined. This information is displayed only via the MMI or logged. Via the Identification unit 23 the hit target is identified and thus assigned. The corresponding hit information is processed by the Impact Transmission unit and displayed in the MMI. By means of the Shot Performance History Recording all these actions of the different simulators are recorded for the After-Action Review with the actual hit image. The system is arranged that the paths of the respective exercise participants 47 are traced on a map and all Sensor Unit Arrangements 9 are tracked and recorded (stored/saved). By means of the unit After Action Review History the whole history (tracking and action history) is stored in order to be evaluated and displayed on the MMI on a 3D view map. Via the Speech Recognition Module 18, recognized keywords (pre-defined) are logged during the operation and then evaluated for the correct or incorrect commands given during the corresponding firing actions.
The Shot Performance is derived from the direction of the aiming of the weapon 10 in space, the distance of the target aimed at supplied by the distance sensor 15, and the geolocation of the weapon 10 equivalent of that of the participant 47 which altogether allow to determine a vector or trajectory in space of a simulated shot. Thereby, it is as a minimal requirement feasible to determine if a shot has hit a target and which target. According to a preferred aspect, in view of an enhanced shot performance determination, it is additionally determined where the shot has hit, i.e. the hit accuracy. Other aspects of the shot performance which may be determined by the equipment are the correct handling and bearing of the weapon. As well, the correct reaction and time involved for reacting to a threat (e.g. detected by face or pose recognition) may be determined and used for shot performance. The location of a hit (e.g. head, left/right shoulder, torso, left/right leg) may be displayed on the MMI. The location of a hit is determined by Hit Zone Recognition 22.
By means of the deployed solution, the performance during a mission as well as during an exercise can be ascertained, recorded and analysed. The AI Unit 6 described in this patent is arranged to efficiently monitor, record, and evaluate the performance of armed forces by means of artificial intelligence and additional sensor technology and is thus apt to increase the quality of training, to check the participants' correctness, and to improve the quality of the training. Furthermore, it reduces the amount of instrumentation of participants, weapons and infrastructure required for training to a minimum.
Thereby, it is feasible to determine the activity, or the performance, of a participant in a training unit exclusively on the basis of predetermined data, like a map of the training environment, positions of passive markers like, e.g. APRIL tags, and data furnished by the personalized combat simulation equipment worn by the participant herself and optionally other participants, in particular participants acting as opponents. As a consequence, it is not required to prepare a training environment by installing sensors, cameras and other means for determining the activity of the participants.
The one skilled in the art is able to conceive various modifications and variants on the basis of the preceding description without leaving the scope of protection of the invention which is defined by the appended claims. Conceivable is, f. i.:
SLAM Simultaneous Localization and Mapping, cf. p. 4.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/070037 | 7/16/2022 | WO |
