Patent Application
20250225787
The disclosure relates to sports timing based on a camera system and, in particular, though not exclusively, to methods and systems for sports timing based on a camera system, and a computer program product enabling a computer system to perform such methods.
Sports events such as car- or motor racing, cycling, athletics, drones matches and ice-skating, typically require accurate and fast time registration for tracking objects (persons or vehicles) during the event. Such timing system is usually based on an RFID system, wherein each participant in the event is provided with an RFID transponder, e.g. an UHF back scattering tag or an LF tag based on magnetic induction, which can be read out by RFID readers that are positioned along the track. Such readers may be implemented in the form of an antenna mat, side antenna's and/or antenna's mounted on a frame above a track. Each transponder is configured to transmit packets at a certain frequency and to insert a unique identifier into the packet such that a detector is able to associate a packet with a certain transmitter.
Drawbacks related to timing systems based on RFID technology include the necessity to provide every participant with an UHF tag, the sensitivity of UHF signals to environmental influences, e.g. moisture and rain, the detuning of UHF tags when it is positioned close to the human body, reflection of UHF signals by ‘hard’ objects, e.g. street surface, walls, etc. and collisions of UHF signals when a large number of participants simultaneously pass an RFID detector, e.g. an RFID antenna mat, that is provided across the track.
WO2021/048446 describes an example of a vision-based timing system for measuring a passing time for participants in a mass sports event, wherein many participants may simultaneously or almost simultaneously pass a virtual timing line. The system includes one or more cameras that capture images of participants passing the virtual timing line. The system may further include a processor that is configured to analyze the images, i.e. detect objects such as participants in a sports event, in the images, and to determine a passing time for detected objects that pass the virtual timing line. Further, based on the captured images, the processor is configured to detect identification information carrier, e.g. identification number and/or symbols (e.g. a QR code or the like) printed on a so-called BIB that is attached to the clothing of the participant, and to associate the detected identification information to the passing time of a participant. The advantage of such vision-based timing system is that it does not exhibit the drawbacks of RFID-based timing systems as mentioned above.
While the vision-based timing system allows accurate determination of passing times of participants passing the virtual timing line, some challenges still exist to accurately time and identify real time all participants that pass the virtual timing line. In particular, cameras positioned to capture the area around the timing line, such as the finish line, typically capture a scene that may substantially extend beyond the timing line, including the areas around, and in particular on both sides of the timing line which will including a large number of people, not only athletes but also other people, which do not participate in the sports event such spectators and officials.
This makes accurate, reliable and fast determination and identification of participants passing a timing line, e.g. a start line or a finish line, particular challenging, especially when dealing with mass sports events (e.g. a marathon or the like) wherein high detection accuracies are required for example detection accuracies of more than 99.9%. Additionally, different privacy regulations add further technical challenges in that data, especially biometric data, may only be used under stricture conditions
Hence, from the above, it follows that there is a need in the art for improved vision-based timing of sports events, that allows highly accurate and reliable determination of passing times and identification of participants in a sports event, in particular mass sports events.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system”. Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including a functional or an object oriented programming language such as Java™, Scala, C++, Python or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer, server or virtualized server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the embodiments are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU), or graphics processing unit (GPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
It is an objective of the embodiments in this disclosure to reduce or eliminate at least one of the drawbacks known in the prior art. In an aspect, the embodiments may relate to a method for determining a passing time of objects passing a virtual timing line.
In an embodiment, the method may comprise: receiving video frames captured by a calibrated camera system, each video frame representing a picture of scene including one or more objects moving along a track, each video frame being associated with a time instance; determining or receiving calibration data, the calibration data defining a virtual timing zone positioned at a predetermined distance from the calibrated camera system, the virtual timing zone extending across the track and extending along the sides of the track and including a virtual timing line or virtual timing plane across the track; detecting objects in the video frames using an object detection algorithm, the objects detected by the detection algorithm defining detected objects; determining depth information for at least part of the video frames, the depth information comprising information regarding a distance between detected objects and the calibrated camera system; determining detected objects that are positioned within the virtual timing zone based on the calibration data and the depth information; and, determining a passing time for detected objects in the virtual timing zone that passed the virtual timing line based on one or more time instances of one or more video frames comprising the at least one detected object passing the timing line.
In another embodiment, the method may comprise: receiving video frames captured by a calibrated camera system, each video frame representing a picture of scene including one or more objects moving on a track and objects moving outside of the track, each video frame being associated with a time instance, the calibrated camera system comprising calibration data defining a virtual timing zone in the video frames positioned at a predetermined distance from the calibrated camera system, the virtual timing zone extending across the track and extending along the sides of the track; detecting objects in the video frames using an object detection algorithm, the objects detected by the detection algorithm defining detected objects; determining depth information for at least part of the video frames, the depth information comprising information regarding the position of the detected objects relative to the calibrated camera system; determining whether detected objects are positioned inside the virtual timing zone or outside the virtual timing zone based on the calibration data and the depth information; applying a feature analysis algorithm to at least part of the detected objects in the video frames that are determined to be inside the virtual timing zone, the feature analysis algorithm determining identifying features for the one or more detected objects inside the virtual timing zone; and, determining a passing time for detected objects that determined to be positioned inside the virtual timing zone and pass a virtual timing line across the track based on the calibration data, the depth information and one or more time instances of one or more video frames of the detected objects that pass the virtual timing line.
Thus, the timing system detect objects in the images and use the depth information to determine whether the detected objects are inside or outside the virtual timing zone, i.e. zone that is positioned over the track, wherein the sides of the timing zone substantially coincide with the sides of the track. The computer may track detected objects inside the virtual timing zone and execute an identification algorithm to identify these objects (the athletes on the track) and/or to determine a passing time for these objects, while it will not process objects that are outside the virtual timing zone. This way, the computer is able to only process and identify athletes within the virtual timing zone that will pass the virtual timing line. Further, the computer is able to disregard objects, e.g. spectators and/or officials, outside the virtual timing zone. This way, the privacy of the spectators is respected, while athletes that participate in the sports event (which gave their consent for being timed by the vision-based timing system) are being timed and identified without wasting computer resources. The virtual timing zone effectively enables spatial filtering based on the depth information to filter out detected objects that are positioned outside the timing zone, before determining a passing time for detected objects and/or before executing an identification algorithm for identifying the timed objects.
The timing system is calibrated such that the camera system not only includes information, e.g. coordinates, of a virtual timing line across the track positioned at a predetermined distance from the camera system, but also the information, e.g. coordinates, of an zone, e.g. a 2D area or 3D volume, that is positioned over the track and having sides that extend along the track. Preferably, the sides of the timing and identification zone substantially coincide with the edges or borders of the track. The virtual timing line may be positioned within the virtual timing zone. In case the coordinates of the virtual timing zone define a 3D volume, the virtual timing and identification zone may define a volume of interest (VOI) positioned over the track, having sides extending along the track. Preferably, the sides of the timing and identification zone substantially coincide with the edges or borders of the track.
In an embodiment, the width of the virtual timing zone may be equal or larger than the width of the track and wherein the length of the virtual timing zone is equal or larger than the width of the track.
In an embodiment, the virtual timing zone may be substantially rectangularly shaped wherein long side of the rectangularly shaped virtual timing zone is aligned with the sides of the track.
The virtual timing zone thus defines part on the track where only athletes are allowed to come, so that the timing system can efficiently determine if an object is an athlete or not. Objects within the virtual timing zone extending are monitored, identified and timed when passing a virtual timing line (or plane). The virtual timing zone may have many different shapes depending on the situation as long as the virtual timing zone identifies an area in which (only) athletes move and which includes a virtual timing line.
In an embodiment, the method may further comprise: applying a feature analysis algorithm to detected objects in the video frames that are within the virtual timing zone, the feature analysis algorithm determining identifying features for the one or more detected objects in the video frames; and, determining the identity of the detected object for which the passing time is determined based on the identifying features of the detected object that has passed the timing line.
In an embodiment, the identifying features of a detected object may include one or more an optically readable identification markers such as a race bib or a printed mark; and/or, one or more characteristics about the shape and/or colour of the detected object; and/or, in case the detected object is an animal or a human, one or more biometric identifiers of the detected object.
In an embodiment, the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, for example a deep learning algorithm such as a convolutional deep neural network system, that is trained to detected one or more objects in a video frame and to determine identifying features associated with detected objects.
In an embodiment, objects may be detected in the images using an instance object segmentation algorithm based on deep neural network. In an embodiment, the instance object segmentation algorithm may be based on a R-CNN, fast R-CNN, faster R-CNN, mask R-CON algorithm or related algorithms. Such algorithm is capable of detecting and classifying objects in an image in (almost) real-time.
In an embodiment, identification of an object may be based on an optical character recognition (OCR) algorithm, or an optical code recognition algorithm may be used to recognize (printed) identification markers in a ROI. In an embodiment, the identification markers may include printed characters. In an embodiment, the printed characters may be only visible using an infrared camera.
In a further embodiment, identification of an object may be based on an algorithm that is configured to identify an object based on biometric features, e.g. a face recognition algorithm that is configured to identify persons based on facial features.
In an embodiment, the calibration data may include coordinates defining the virtual timing zone, preferably the coordinates defining a 2D area over the track, wherein the sides of the 2D area are substantially parallel to the sides of the track; or, the coordinates defining a 3D area over the track, wherein the side surfaces of the 3D area are substantially parallel to the sides of the track.
In an embodiment, the determining detected objects that are positioned within the virtual timing zone includes: using the depth information to determine position of the objects; determining on the basis of the positions and the coordinates of the virtual timing zone if an object is positioned in or outside the virtual timing zone.
In an embodiment, the detecting one or more objects in the video frames may include: determining one or more regions of interest ROls in a video frame, each ROI comprising pixels representing an object; determine identifying features in one of the one or more ROIs; and, determine an object in the ROI based on the determined identifying features.
In an embodiment, the camera system may comprise a plurality of camera modules, preferably two camera modules forming a stereo camera, the stereo camera being configured to generate at each time instance at least a first video frame and a second video frame of the scene and wherein the depth map is determined based on a disparity mapping algorithm configured to determine a disparity between pixels of the first and second video frame.
In an embodiment, the passing time may be determined based on at least one video frame of the scene wherein a predetermined part of the detected object that has passed the virtual plane.
In a further aspect, the embodiments may relate to a method for calibrating a timing system configured to determine a passing time of an object passing a timing line across a sports track, wherein the method comprise: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of a timing system, each video frame representing a picture of scene including the track and one or more calibration markers, preferably at least two calibration markers defining positions at opposite sides of the track; determining depth information, e.g. one or more depth maps, based on the video frames, the depth information comprising information regarding the distance between one or more objects in the picture of a video frame; using the depth information to determine a relative position between of the one or more calibration markers and the camera system; determining calibration data, the calibration data including position information of a virtual timing line oriented across the track at a predetermined position relative to the position of the one or more calibration markers; and, position information associated with a virtual timing zone positioned over the track at a predetermined position relative to the position of the one or more calibration markers, the timing zone extending across the track and along the track; and, storing the coordinates of the virtual plane in a memory of the timing system.
In an embodiment, the calibration data may include coordinates defining the virtual timing zone, preferably the coordinates defining a 2D area, e.g. a rectangular shaped area, over the track, wherein the sides of the 2D area are substantially parallel to and/or substantially coincide with the sides of the track; or, the coordinates defining a 3D volume, e.g. a rectangular shaped volume, over the track, wherein the sides surfaces of the 3D volume are substantially parallel to and/or substantially coincide with the sides of the track. The sides of the virtual timing zone may extend over a substantial length along the sides of the track so the system is able to distinguish athletes from spectators.
In another aspect, the embodiments may relate to a system for determining a passing time of an object passing a timing line across a sports track comprising: at least one camera system connected to a computer; the computer comprising a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: receiving video frames captured by a calibrated camera system, each video frame representing a picture of scene including one or more objects moving along a track, each video frame being associated with a time instance; determining or receiving calibration data, the calibration data defining a virtual timing zone positioned at a predetermined distance from the calibrated camera system, the virtual timing zone extending across the track and along the track and including a virtual timing line across the track; detecting objects in the video frames using an object detection algorithm, the objects detected by the detection algorithm defining detected objects; determining depth information for at least part of the video frames, the depth information comprising information regarding a distance between detected objects and the calibrated camera system; determining detected objects that are positioned within the virtual timing zone based on the calibration data and the depth information; and, determining a passing time for detected objects that pass the virtual timing zone within the virtual timing zone, based on the calibration data, the depth information and one or more time instances of one or more video frames of the detected objects.
In a further embodiment, the executable operations may further comprise any of the method steps as described above.
In yet a further aspect, the embodiments may relate to a calibration module for a timing system configured to determine a passing time of an object passing a timing line across a sports track, wherein the module may comprise: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of a timing system, each video frame representing a picture of scene including the track and one or more calibration markers, preferably at least two calibration markers defining positions at opposite sides of the track; determining depth information, e.g. one or more depth maps, based on the video frames, the depth information comprising information regarding the distance between one or more objects in the picture of a video frame; using the depth information to determine a relative position between of the one or more calibration markers and the camera system; determining calibration data, the calibration data including position information of a virtual timing line oriented across the track at a predetermined position relative to the position of the one or more calibration markers; and, position information associated with a virtual timing zone over the track at a predetermined position relative to the position of the one or more calibration markers, the timing zone extending across the track and along the track; and, storing the coordinates of the virtual plane in a memory of the timing system.
The embodiments may also relate to a computer program or suite of computer programs comprising at least one software code portion or a computer program product storing at least one software code portion, the software code portion, when run on a computer system, being configured for executing any of the method steps described above.
The embodiments may further relate to a non-transitory computer-readable storage medium storing at least one software code portion, the software code portion, when executed or processed by a computer, is configured to perform any of the method steps as described above.
The embodiments will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the invention is not in any way restricted to these specific embodiments.
For example, a group of pixels in a video frame may be part of an object in the scene that is imaged by the camera system. In that case, the depth map may indicate the relative distance between the camera (the viewpoint) and the surface of the object in the scene. Hence, during capturing of a sequence of time-stamped video frames of an object, e.g. an athlete or a vehicle that is moving along the sports track, the associated depth maps may provide information about the distance between the moving object in the video frames and the (static) camera system as a function of time.
Camera systems that are capable of generating depth-maps are known. For example, in an embodiment, a camera may be implemented as a 3D camera system e.g. stereo camera comprising two or more camera modules, wherein each camera module has its own lens system. An example of a top-view of such 3D imaging system is depicted in
The 3D camera system (or the computer system controlling the camera system) may include a module for computing a depth map based on video frames captured by the two (or more) camera modules. In an embodiment, the module may use a disparity mapping technique compute a depth map-based images generated by the two image sensors.
To compute a depth map based on the video frames, a matching algorithm may be executed to match corresponding pixels of the left and right video frame. Hence, an object 300 imaged by two synchronized camera modules is positioned in the same position 3041,2 but separated by a baseline distance 308. In that case, the object will appear on similar positions in both images. The distance between the objects in the left and right image is known as the disparity 306. An algorithm for constructing the disparity map based on the two images is known as a stereo matching algorithm. Various stereo matching algorithm exist, which needs to be both accurate and fast for real-time applications.
The camera systems used in the embodiments of this application are not limited to stereo based-imaging techniques and other imaging techniques may be used as well. For example, a depth map may be generated based on an RGB/infrared (IR) technique (as used by the Kinect) or a 3D time-of-flight (TOF) technique, LIDAR or combinations thereof, wherein infrared radiation, e.g. an IR camera or IR laser, may be used to determine depth information of a scene that is captured by a camera. In some embodiments, the camera system may be a color-infrared camera configured to capture image of the visible spectrum (e.g. RGB) and the infrared (IR) spectrum to capture images of objects. For example, the use of an infrared images may be advantageous when light conditions are not optimal, e.g. due to weather conditions (e.g. fog, rain, etc.) or during twilight.
To increase the angel of view of the camera system, in some embodiments, one or more wide angle camera modules may be used, e.g. a 180-degree camera or a 360-degree camera. Also, for such type of video formats, such as 360-video or immersive video, which is generated using special 360 camera systems, wherein the video is projected onto a 2D video frame using e.g. an equirectangular projection, depth maps can be generated. To enable accurate image processing, the cameras used in the timing system may have a frame-rate of at least 30 frames per second or more. Depending on the application, the frame rate may be at least 60 frames per seconds or more. Additionally, the camera may be a high-resolution camera having an image resolution of 4K or more.
As shown in
Based on the implementation, the virtual timing reference may define a virtual timing line or a virtual timing plane arranged at a distance z from the camera system. A virtual timing line may be used when determining a passing based on two coordinates, e.g. an object passing the time line at position x or based on three coordinates, e.g. part of an object passing the time line at position x and height y. A virtual timing plane may for example be used to determine a passage time when a specific part of an object has passed the virtual timing plane. For example, for certain use cases it is desired to determine a passing time when a specific part of an object, e.g. the chest of an athlete comprising a BIB, passes the virtual timing plane.
Below, whenever reference is made to virtual timing line, other types of virtual timing references such as a virtual timing plane should also be included. The timing line may be arranged across the track at the position of (or relative to) the one or more calibration markers. For example, when using two calibration markers, the virtual timing line may be positioned between the two calibration markers so that it is arranged across the track between (or relative to) the two calibration markers.
The calibration process requires a 3D camera system to accurately detect the position and orientation of the calibration markers under all outdoor circumstances. Therefore, the calibration markers may be designed to have predetermined distinct shape and/or color combination so that during calibration an object detection program may easily and accurately determine the position of the (edges of) markers in video frames so that the coordinates of the virtual plane can be accurately determined. When the (calibrated) timing system is in operation, the 3D camera system may capture video frames that include athletes passing the through the virtual plane. While the figure illustrates a camera system along the side of the racing track, in other embodiments, one or more of the camera systems may be mounted above the sports track using a suitable mounting structure and/or integrated in a mat that at is arranged across the sports track.
As will be described hereunder in more detail, the timing system depicted in
The computer for controlling the one or more 3D camera systems and executing the calibration and timing methods may be implemented as a stand-alone computer or a set of (wirelessly) connected computers. For example, the 3D camera systems that are used for determining the passing time based on virtual plane located across the track may be controlled by a computer that includes a wireless interface for wireless communication with the computers that control the other 3D camera systems.
A plurality of timing systems as depicted in
The data processing methods that are used by the timing system to calculate the depth maps and analyze the video frames may require real-time imaging processing so in some embodiments the computer system 1041,2 may include one or more special purpose processors, such as a GPU, to execute the computation intensive parts of calibration and timing process. In other embodiments, the one or more 3D camera systems may be connected to cloud resources which may run the computation intensive parts of the processes. A CPU clock or a GPS clock may be used to link the video frames with time information. For example, in an embodiment, each or at least part of the video frames may be linked to a time instance by time stamping the video frames.
The timing system in
The computer system of the timing system may include one or more image processing units configured to execute various image processing algorithms. For example, in an embodiment, the computer system may be configured to execute an object detection algorithm for detecting objects in images captured by the camera system. In particular, an instance object segmentation algorithm may be used to detect and delimit an object in an image from other objects in the image. As shown in
This way, objects in an image, such as athletes, cars, bikes, etc. may be accurately and efficiently detected and classified and represented as labelled ROIs in an image. In an embodiment, information about the ROls and the IDs may be added or linked as metadata to an image. Efficient instance object segmentation algorithms based on deep learning may be used, including but not limited to algorithms such as R-CNN, fast R-CNN or faster R-CNN or related algorithms, which are capable of detecting and classifying objects in an image in (almost) real-time. An overview of state-of-the-art instance segmentation algorithms is provided in the article by A. Hafiz et al, A survey on Instance Segmentation: State of The Art, International Journal of Multimedia Information Retrieval volume 9, pages 171-189 (2020), which hereby is incorporated by reference into this application. These algorithms provide very efficient run-time performance so that they can be executed locally, e.g. using the computer of the timing system.
To speed up the object detection and identification process, the CPU-based local computer of the timing system that runs the image processing algorithms may include one or more special purpose processors, e.g. one or more graphical processing units GPUs, tensor processing units TPUs or field programmable gate arrays FPGAs, which may be especially adapted to accelerate computations, in particular computations associated with neural networks, performed by the various image processing algorithms.
Further, in an embodiment, the computer system may be configured to execute an object identification algorithm. Here, object identification refers to the process of assigning an identity, typically personal data (e.g. name of a person, an animal, a vehicle or a team that is stored prior to the event in a database), to a detected object based on these object features. An example of an object feature may be (part of) a number or a code (e.g. a QR code) on the BIB of a participant. When a participant registers himself with a sports event, he will receive identification information, e.g. identification number such as a BIB number or a code, which may be used by the timing system to link data, e.g. information about passing times, to personal data, e.g. a name of a registered participant, that is stored in a central database. Different object features may be used to link an object to an identity. For example, in case the objects represent persons, object features may include (but is not limited to) (a part of) a race BIB, colors, biometric information of a person, visual identification codes, e.g. a QR code, object items, such as shoes, shirts, branding marks, etc. In case the objects represent vehicles, object features may include characteristics of the vehicle, e.g. number, color, shape, branding marks, license or registration plate, etc. The computer may include or be connected to a wireless radio module for wirelessly connecting the timing system to a further computer system (not shown), e.g. a remote computer system e.g. a server system, which is configured to centrally process determined passing times of detected and identified objects.
As shown in
The ROIs may be analyzed by one or more algorithms that are configured to detect object features in a ROI and extracting identification information from the ROI. For example, an optical character recognition (OCR) algorithm may be used to recognize (printed) identification markers in a ROI. Typically, an OCR algorithm may include a text detection algorithm and a text recognition algorithm. These printed identification markers may be based on characters and/or symbols, such as example numbers and/or characters on a BIB or a vehicle. Alternatively, and/or in addition, a code detection algorithm may be used to detect and encode a visual code, typically a standardized code, in a ROI. For example, BIB or clothes of an athlete may include a QR code or a variant thereof, in which identification information is encoded as (for example) a geometrical structure. Further, a face detection and recognition algorithm may be used for identifying a person in a ROI based on facial features.
In some embodiments, a pose estimation algorithm which is configured to identify a pose of an object. For example, in an embodiment, a pose estimation algorithm may determine key joints (shoulders/hips/knees/head) of an object, which may be used to identify possible locations in a ROI that may represent object identification features, i.e. features of an object that can be used for identification. For example, based on the location of the key joints, specific parts of a body, e.g. chest, arm, head, legs or feet, may be determined. Based on the locations of the key joints, locations within a ROI may be determined that have a high chance of comprising one or more object features, e.g. a face, a chest or feet.
When using a vision-based timing system for timing a mass-event such as a marathon or the like, typically the timing area will be very crowded with both athletes and other persons e.g. spectators or officials, that do not participate to the sports event.
To address this problem, a spatial filtering technique based on depth information in the is used to filter out detected objects. This is schematically shown in
As will be described below in more detail, the calibrated timing system is configured to capture images and depth information associated with the images. Further, it is configured to detect objects in the images and use the depth information to determine whether the detected objects are inside the virtual timing zone. If that is the case, the computer will track detected objects within the virtual timing zone and execute an identification algorithm to identify objects detected within the virtual timing zone (the athletes on the track), while it will not process detected objects that are located outside the virtual timing zone. This way, the computer will only process, e.g. time and identify, athletes within the virtual timing zone that will pass the virtual timing line. The computer will disregard objects, e.g. spectators and/or officials, outside the virtual timing zone. This way, the privacy of the spectators is respected while athletes that participate in the sports event (which gave their consent for being timed by the vision-based timing system) are being timed without wasting computer resources.
Any type of marker may be used for calibration, e.g. an existing object, e.g. a tree, a traffic sign or a pole, on the side of the track. In a further embodiment, geo-coordinates may be used as calibration markers. In such case, the camera may be associated with a geo-coordinate using e.g. GPS or the like and a line or plane across a track that is visible in the FOV of the camera may define may be defined by one or more geocoordinates relative to the geo-coordinates of the camera.
Alternatively and/or in addition, specifically designed calibration marks may be used. For example, in an embodiment, calibration markers may be designed to have features that allow accurate calibration under different outdoor conditions. For example, the shape, edges and/or colors of the marker may be designed to allow accurate detection in the pictures. The depth map associated with the video frames may be used to determine the distance between the camera and the detected calibration markers. Alternatively, if a sufficiently accurate depth map can be constructed, an object detection program may also determine the position the calibration markers directly based on the depth map.
Once the position of the one or more markers has been detected by the computer, it may determine a virtual timing line across the track 700. The virtual timing line may be a virtual line 709 or virtual plane 711 across the track located at a predetermined position relative to the position of the one or more markers. In an embodiment, the virtual timing line may be a line that extends from a marker positioned at a first side of the track in a direction across the track to the other side of the track. In another embodiment, the virtual timing line may extend from a first marker at one side of the track to a second marker at the other side of the track. The virtual timing line or virtual timing plane may be used as the location at which the timing system determines a passing time. In an embodiment, the virtual line or plane may be positioned within a rectangular 3D volume 712 in space, wherein the width of the volume may be determined by the calibration markers and the height and the depth of the volume may be determined by the computer. The 3D volume may define a 3D detection zone in which the timing system will acquire the video data (e.g. a plurality of video frames) for accurately determining a passing time and for identifying the object associated with the passing time.
In addition to the timing line, the calibration process may also be used to determine a virtual timing zone 706 which is positioned across the track and along the track at a predetermined position relative to the one or more markers. In an embodiment, the virtual timing zone may define a 3D volume of interest VOI in which the virtual timing line or plane is positioned. In another embodiment, the virtual timing zone may be 2D virtual timing area in which the virtual timing line or plane is positioned. In an embodiment, the one or more markers may define the position and/or (part of) the dimensions of the virtual timing zone. For example, after positioning and detection of the markers, the calibration program may generate a virtual timing zone of a certain width and length and, optionally, height, based the position of the one or more markers. As the markers are positioned along the edges of the track, the virtual timing zone may have sides that run substantially parallel to the edge of the track. This way, position information, e.g. coordinates, of a virtual timing zone that is positioned over the track can be determined during calibration using the one or more markers and stored in the memory of the timing system. This way, a calibrated timing system 702,704 as shown in
Thereafter, a depth map may be determined based on the video frames, the depth map may comprise information regarding the distance between one or more objects in the picture and the 3D camera system (step 806). For example, a depth map may be generated based on two video frames generated by two camera modules in the stereo camera and disparity mapping may be used to generate the depth map in the same way as described with reference to
Then, coordinates of a virtual timing line across the track may be computed that is positioned relative to the one or more calibration markers, e.g. between calibration markers at opposite side of the track. The virtual timing line may be used by the timing system to determine a passing time of objects passing the virtual plane (step 810). Further, on the basis of the positions of the calibration markers position information, e.g. coordinates or information to compute coordinates, of a virtual timing zone, e.g. a 2D area or a 3D volume, that is oriented over the sports track may be computed (step 812). In an embodiment, the virtual timing line may be positioned within the virtual timing zone, which defines a zone where objects are tracked, identified, and timed when passing the virtual timing line. The coordinates of the virtual timing line and the virtual timing zone may be stored in a memory of the timing system (step 814).
In operation, a calibrated timing system no longer needs the calibration markers. The calibrated camera system will use the calibration data defining the virtual timing line and the virtual timing zone to efficiently determine a passing time and one or more identification features of an object that is passing the virtual timing line.
It is submitted that the calibration process depicted in
A calibration process may be used to install and calibrate one or more further camera systems along the track so that each of these camera systems may capture video frames of objects passing a timing line. The camera system may (wirelessly) communicate with each other that the video capturing process can be time-synchronized. This way, at one time instance, each of the camera systems will procedure one or more time-stamped video frames of the sports track. The time-stamped video frames (and associated depth maps) of the different viewing angles may be used for determining passing times of objects passing the virtual plane and identification of objects for which a passing time has been determined.
Thus, when an object moves along the track, the 3D camera system will capture images (pairs of images in case of a stereo camera) of a scene that includes the 3D detection zone. For each image (video frame) the 3D image system may compute a depth map. An object detection and tracking algorithm may be used to detect and track a predetermined object, e.g. a human object or an object representing an object, in subsequent video frames.
Based on the depth maps, the computer may determine that a detected object enters the first part of the 3D detection zone. In that case, the computer may start storing video frames and associated depth maps in a buffer until the object leaves the 3D detection zone via the second part. In another embodiment, only the pairs of video frames are stored, and the depth maps are determined later. These video frames and depth maps may be used by the computer to determine a passing time and to identify the object associated with the passing time.
A passing time module in the computer of the timing system may analyse the sequence of time-stamped video frames to determine at what time instance the athlete has passed the virtual plane. To that end, an object detection and classification algorithm may be applied to each video frame. The algorithm may determine for example in the video frame region of interests 9081-3 (ROIs) that belong to an object. Further, for each of these ROIs, the algorithm may classify pixels as belonging to the athlete or not (the background).
Further, a depth map associated with each of the video frames may be used to determine distance values belonging to pixels that are classified as belonging to the object. These distance values may be compared with the distance between the camera and the virtual plane. This way, when the 3D camera system captures an object crossing the virtual plane, for each video frame the part of the pixels of the object that have crossed the virtual plane can be determined. This is visible in the video frames of
For the video frame at time instance T1 only pixels 9041 representing part of a hand and pixels 9061 representing a shoe of the athlete are associated with distance values smaller than the distance between the virtual plane and the 3D camera system. Similarly, for the video frame at time instance T2, pixels 9041 representing part of the upper body and pixels 9062 representing part of a leg are associated with distance values smaller than the distance between the virtual plane and the 3D camera system. Finally, for the video frame at T3 all pixels 908 representing the athlete are associated with distance values smaller than the distance between the virtual plane and the 3D camera system. Based on this analysis, the computer may determine that at T2, a substantial part of the body of the athlete has crossed the virtual plane. For example, the computer may determine that if a part of object that has crossed the virtual plane is larger than a certain threshold value that in that case, it is determined that the athlete has crossed the plane. Hence, the time-stamp T2 may in that case define the passing time 510, in this example 2:34. Different rules may be defined in order to determine if an object has crossed the virtual plane.
In a further step 1104, the method may include determining or receiving calibration data, e.g. pre-defined calibration data, which may define a virtual timing zone positioned over the track at a predetermined distance from the calibrated camera system, wherein the virtual timing zone includes a virtual timing line. In an embodiment, the calibration data may include position information, e.g. coordinates, of the virtual timing zone, e.g. a 2D area over and along the track or a 3D volume positioned over and along the track. The calibration data may be stored in a memory of the timing system and retrieved when the timing process is executed.
Objects in the video frames may be detected using an object detection algorithm, the objects detected by the detection algorithm defining one or more detected objects (step 1106). Thus, within each video frame a plurality of objects may be identified using a known object detection algorithm and a relative position between the identified object and the camera system may be determined. In an embodiment, the object detection step may include determining regions of interest ROIs comprising the object and for each ROI subjecting the pixels in the ROI to a classification algorithm for classifying whether a pixel represents part of the object or part of the background.
Further, depth information, such as one or more depth maps, may be determined for the sequence of video frames, wherein the depth information may comprise information regarding the position of detected objects in the video frames relative to the camera system (step 1108). Typically, the 3D camera system may be a stereo camera so that it produces pairs of video frames which can be used to determine a depth map. Alternatively, any other camera system that can produce depth information associated with captured video frames may be used.
The calibration data and the depth information may then be used to determine if the detected objects that are positioned within the virtual timing zone based (step 1110). Finally, a passing time for detected objects that pass the virtual timing zone within the virtual timing zone may be determined based on the calibration data (in particular the position information of the timing line), the depth information and one or more time instances of one or more video frames of the detected objects (step 1112).
The method provides an efficient way of determining passing times using a vision-based timing system for crowded timing locations, e.g. the finish of a mass sports event such as a marathon. The virtual timing zone is used to spatially filter out a substantial part of detected objects that are not participates of the sports event that is timed so that computational resources are only assigned to detection, timing and identification of athletes captured by the 3D camera of the timing system. The method further provides a vision-based timing scheme that is privacy friendly in the sense that it only applies identification algorithms to detected objects in the video frames that represent athletes.
The passing of an object over the virtual timing line may be determined by comparing a distance between the 3D camera system and the virtual timing line with the distance between the 3D camera system and detected objects within the virtual timing zone. A passing time may then be determined based on time information, for example time instances, e.g. time stamps, associated with video frames comprising a detected object passing the virtual timing line. For example, to that end, one or more video frames may be determined wherein a part of the object that has passed the virtual plane has certain dimensions.
Hence, certain rules may be used to determine if the object has passed the virtual plane. The time instance, e.g. time stamp, associated with the video frame that depicts that situation defines the passing time. Thereafter, a feature analysis algorithm may be applied to the one or more detected objects in the video frames, the feature analysis algorithm determining identifying features for the one or more detected objects in the video frames and the identity of the detected object for which the passing time is may be determined based on the identifying features of the detected object that has passed the timing line.
In an embodiment, the the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, preferably a deep learning algorithm such as a convolutional deep neural network system, that is trained to detected one or more objects in a video frame and to determine identifying features associated with detected objects.
Thus, different pictures from the sequence of video frames may be used by the identification of the object that has crossed the virtual plane at the passing time. Hence, the video frame that is used for determining the passing time of an object may be different from the one or more video frames that are used for determining the identity of the object.
Memory elements 1304 may include one or more physical memory devices such as, for example, local memory 1308 and one or more bulk storage devices 1310. Local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code to reduce the number of times program code must be retrieved from bulk storage device 1310 during execution.
Input/output (I/O) devices depicted as input device 1312 and output device 1314 optionally can be coupled to the data processing system. Examples of input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like. Examples of output device may include, but are not limited to, for example, a monitor or display, speakers, or the like. Input device and/or output device may be coupled to data processing system either directly or through intervening I/O controllers. A network adapter 1316 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and/or networks to said data and a data transmitter for transmitting data to said systems, devices and/or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 1100.
As pictured in
In one aspect, for example, data processing system 1300 may represent a client data processing system. In that case, application 1318 may represent a client application that, when executed, configures data processing system 1300 to perform the various functions described herein with reference to a “client”. Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like.
In another aspect, data processing system may represent a server. For example, data processing system may represent an (HTTP) server in which case application 1318, when executed, may configure data processing system to perform (HTTP) server operations. In another aspect, data processing system may represent a module, unit or function as referred to in this specification.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2031336 | Mar 2022 | NL | national |
This Application is a 371 National Stage Application of International Application No. PCT/NL2023/050145, filed Mar. 20, 2023 and published as WO 2023/177302 A1 on Sep. 21, 2023, and further claims priority to Netherlands Application Ser. No. 2031336, filed Mar. 18, 2022.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/NL2023/050145 | 3/20/2023 | WO |
