This disclosure relates generally to processors, and, more particularly, to methods and apparatus to improve performance data collection of a high performance computing application.
High performance computing (HPC) is utilized in various types of technologies to perform complex tasks. In HPC systems, individual computers (e.g., nodes) may be configured in clusters. Each computer may have multiple cores capable of running multiple processes. HPC utilizes multiple nodes of a cluster together to solve a problem larger than a single computer can easily solve. HPC systems run based on instructions from HPC applications. An HPC application includes instructions to be executed by the nodes of a HPC system. Most HPC applications include computation and communication phases that execute at alternate times. Instructions corresponding to initialization of variables, preprocessing data, parsing data, semantic analysis, lexical analysis, etc. are executed during computational phases. Instructions corresponding to communication with other nodes in a HPC system that are executed during communication phases. Performance analysis tools may be used by HPC software developers to collect performance data corresponding to communication operations of an HPC application to improve the performance of the application, identify errors, identify issues, etc.
The figures are not to scale. In general, the same reference numbers will be used throughout the drawing(s) and accompanying written description to refer to the same or like parts.
Descriptors “first,” “second,” “third,” etc. are used herein when identifying multiple elements or components which may be referred to separately. Unless otherwise specified or understood based on their context of use, such descriptors are not intended to impute any meaning of priority or ordering in time but merely as labels for referring to multiple elements or components separately for ease of understanding the disclosed examples. In some examples, the descriptor “first” may be used to refer to an element in the detailed description, while the same element may be referred to in a claim with a different descriptor such as “second” or “third.” In such instances, it should be understood that such descriptors are used merely for ease of referencing multiple elements or components
High performance computing (HPC) systems include multiple processing nodes working together to perform one or more tasks based on instructions of an HPC application. As used herein, a “node” is defined to be an individual computer (e.g., a service, a personal computer, a virtual machine, etc.) that is part of an HPC cluster. A node may include one or more CPUs. Each CPU may include one or more processor cores. Each node of a HPC system may exhibit a computation phase (e.g., for performing computations locally) and a communication phase (e.g., for transmitting data to one or more other nodes in the HPC system). To implement communication operations between nodes, HPC nodes include one or more hardware-based host fabric interfaces (HFIs) (e.g., network interface cards (NICs)) designed to transmit (e.g., broadcast) data to one or more of the other nodes in the HPC system to write data (e.g., using an remote direct memory access (RDMA) operation) from the first node into the memory of one or more of the other nodes. In known systems, the first node transmits instructions to the HFI to cause transmission of data to the other node(s) immediately or after some event(s) occur(s). The HFI includes hardware event counters to track when certain events occur. Accordingly, when the instruction from the CPU of the first node corresponds to a triggered operation (e.g., an instruction to transmit data after some event occurs), the HFI can monitor the count of a corresponding event counter to identify when to transmit the data identified by instructions from the CPU of the first node to one or more of the other nodes in the HPC system.
Some CPUs in one or more nodes of known HPC systems utilize a software-based collector or collector thread monitor performance of the application running on one or more of the main executor threads of a CPU in the node. In this manner, the collector thread can provide useful information to a user and/or developer to improve (e.g., optimize) the application. The collector may collect performance data (e.g., pull data from hardware performance counters) to measure and/or improve (e.g., optimize) the progress of one or more communication operations. The collector, or another component, can process the performance data to identify any potential issue(s) corresponding to communication operations. The collector may continuously measure the performance of the communication operations by polling hardware performance counters at execution time. However, such polling consumes resources of a CPU in the node, which is a valuable commodity for HPC systems. Accordingly, such polling may degrade overall HPC application performance. Although the polling performed by the collector is important for measuring progress of communication operations (perhaps justifying a degree of degradation of overall performance), it may degrade the overall performance if polling is enabled during the computation phase of an application.
Some known techniques reduce performance data collection by sampling or polling in response to an application's runtime behavior. The polling interval can be increased if no changes are observed for a threshold period of time and/or the polling interval can be decreased when an event of interest occurs. Such techniques adapt the sampling frequency online to increase (e.g., maximize) the information content of the samples and reduce (e.g., minimize) the collection of low-information samples, to reduce the overhead associated with performance monitoring. However, such techniques may miss critical events that occur spontaneously at the beginning and/or end of a program phase. Additionally, tuning polling parameters is difficult because the optimal values depend on various complex characteristics (e.g., system configurations, available resources, dynamic behavior of an application, etc.). Additionally, such techniques restrict the actions that the collector or other tool of the CPU can take during collection (e.g., a tool may not be able to allocate memory, perform input/output (I/O) operations needed to capture samples, etc.).
Examples disclosed herein improve performance data collection for HPC applications by leveraging the host fabric interface (HFI). For example, although HFIs are typically structured to forward data to other nodes in a HPC system (e.g., by writing data into memory of the other nodes for collective communication operations), examples disclosed herein instruct the HFI to perform a triggered put operation (e.g., a write data operation) to write data back to the memory of the node forwarding the data (i.e., the node that includes the collector) as opposed to another node in the HFI. The triggered put operation occurs in response to one or more conditions corresponding to communication phase events that will trigger a wakeup of the collector. In this manner, the collector can enter a sleep mode during computation phases to reduce or cease polling (e.g., to conserve CPU resources) while the host fabric interface tracks one or more events using hardware event counters. The triggered put operation causes a write operation back to the memory (e.g., a user memory space) of the node originating the triggered put operation at a memory address location specified by the collector. In this manner, when one or more events specified for monitoring occur, the host fabric interface identifies the condition and writes to the memory address location of the node specified by the collector. In sleep mode, the collector monitors the memory address location to identify when the host fabric interface writes to the memory address location, thereby indicating the condition has been satisfied (e.g., the one or more triggering events have occurred). In response to the collector identifying that the data in the memory address location has been updated, the collector wakes up and increases the polling frequency and/or restarts the polling process. Because monitoring one or more memory addresses uses less CPU resources than polling the event counters directly, examples disclosed herein significantly reduce the amount of CPU resources needed to execute performance data collection of HPC applications. Examples disclosed herein use a direct memory access (DMA) operation to write data into the memory of the source node. As used herein, a DMA operation corresponds to an HFI of source node writing data to memory of the source node and a RDMA operation corresponds to an HFI of a source node writing data to memory of a destination node different than the source node.
The example node 100 of
The example node 100 of
The example CPU 103 of
The example application 104 of
The example main executor threads 106 of the application of
The example collector 108 of
Monitoring a change in a specific memory address location utilizes less processor resources of the example CPU 103 than polling performance data from the example event counters 118. Accordingly, the sleep mode of the collector saves power by allowing CPU resources (e.g., cores) to be powered down. In this manner, the CPU may improve performance by allowing other cores to run at a higher frequency. Additionally, the HFI 102 does not utilize processor resources of the example CPU 103 of the source node. Accordingly, the collector 108 (which executes on the CPU 103 of the source node) can enter sleep mode and wake up based on a trigger from the HFI 102 (e.g., data being written to the memory address location) to thereby utilize less processor resources of the example CPU 103 of the source node, while maintaining overall application performance monitoring by polling when polling is necessary to maintain application performance and preventing polling when polling is not necessary to maintain application performance data. The write back instructions generated by the example collector 108 may include threshold count(s) corresponding to count(s) of the event counters 118 that trigger execution of a put operation. However, because the event counters 118 may be continuously operational, the collector 108 may need to identify the starting (e.g., current) event count of the event counters 118 at the time of receiving the write back instruction(s)to be able to determine when the number of events identified in the write back instructions has occurred. Accordingly, the collector 108 adds the wake up count to the current event count to generate a threshold count (e.g., whose satisfaction triggers the put operation to be executed). For example, if the put operation corresponds to writing data into a memory address location in response to a particular event occurring 5 times, the collector 108 reads the event count of the counter that corresponds to the particular event (e.g., 100). In such an example, the collector 108 adds 5 (e.g., the wake up count specified in the write back instructions) and 100 (e.g., the current event count of the event counter) to generate a threshold count of 105. An example implementation of the example collector 108 is further described below in conjunction with
The example memory 109 of
The HFI 102 of the example of
The example HFI 102 includes triggered operations circuitry 112 to receive write back instruction(s) from the example collector 108 and to track one or more of the example event counters 118 based on the write back instruction(s). Based on the write back instructions, the example triggered operations circuitry 112 performs an action (e.g., transmit a queued put operation) in response to the event count of one of more of the event counters 118 reaching a threshold count. For example, the collector 108 may transmit write back instruction(s) including an operation (e.g., a triggered put operation, a triggered atomic operation, and/or an instruction to perform one or more such operations) to the triggered operations circuitry 112. The write back instruction(s) further indicate that the operation (e.g., read, write, etc.) is to occur in response to one or more events. For example, the write back instruction(s) may identify a triggered put operation that instructs and/or causes the communication engine 116 of the HFI 102 to write data to a particular memory address location in response to a triggered event (e.g., more than a threshold number of event(s) occurring as measured by the event counters 118). A triggered atomic operation instructs and/or causes the communication engine 116 of the HFI 102 to write to and/or update a particular memory address location without allowing other intervening instructions. The triggered operations circuitry 112 queues (e.g., stores in a register) the put operation (e.g., a memory write operation corresponding to a memory address location) and monitors the example event counters 118 until the threshold number of events occur. For example, when the triggered operations circuitry 112 determines that more than a threshold number of the particular event has occurred, the queued put operation is released, thereby causing the triggered operation to be executed (e.g., by transferring the operation from the queue of the triggered operations circuitry 112 to the core of the command processor 114 to be executed).
As described above, the write back instruction(s) may identify a number of events (e.g., a wake up count) that should occur before causing the triggered operation to be executed based on the threshold wake up count(s) of the write back instructions. The triggered operation circuitry 112 monitors the event counter until the event count satisfies (e.g., equals, reaches, exceeds etc.) the threshold count (e.g., 105). In response to the satisfying of the threshold count, the triggered operations circuitry 112 launches (e.g., transmits) the queued put operation to the example command processor 114 to be executed to cause the communication engine to write data to memory at the source node. In some examples, rather than adding the wake-up count (e.g., 5) to the current event count (e.g., 100), the triggered operation circuitry sets the threshold value directly.
The example command processor 114 of the example of
The example communication engine 116 of
The example event counters 118 of
The example on-chip interface 200 of
The example performance data comparator 201 of
In response to determining that sleep mode should be initiated, the example instructions generator 202 of
The example adder 204 of
To enter sleep mode, the example frequency selector 205 of
Once in sleep mode, the example memory monitor 206 monitors the selected memory address location included in the write back instructions that the HFI will write to when the threshold number of event(s) has been satisfied to trigger a wake-up of the collector 108. The example memory monitor 206 monitors the value stored in the selected memory address location until the value changes. For example, the memory monitor 206 performs a read operation to access (e.g., using the example memory interface 208), the data stored in the selected memory address of the example user memory space 110. In response to the value changing (e.g., the read value of the data stored in the selected memory address being different from the initial stored value at the selected memory address and/or being equal to a predetermined values (e.g., logic 1) as determined by a comparator or the like in the memory monitor 206), the collector 108 wakes up (e.g., the frequency selector 205 resumes the polling protocol of the example event counters 118 of
The example memory interface 208 of
The example communication interface 300 of
The example instructions queue 302 of
The example comparator 310 of
While an example manner of implementing the example collector 108 of
While an example manner of implementing the example triggered operations circuitry 112 of
When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example event counters 118, the example on-chip interface 200, the example performance data comparator 201, the example instructions generator 202, the example frequency selector 205, the example memory monitor 206, the example memory interface 208, the example collector 108 of
Flowcharts representative of example hardware logic, machine readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the example collector 108 and/or the example HFI 102 of
The machine readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a packaged format, etc. Machine readable instructions as described herein may be stored as data (e.g., portions of instructions, code, representations of code, etc.) that may be utilized to create, manufacture, and/or produce machine executable instructions. For example, the machine readable instructions may be fragmented and stored on one or more storage devices and/or computing devices (e.g., servers). The machine readable instructions may require one or more of installation, modification, adaptation, updating, combining, supplementing, configuring, decryption, decompression, unpacking, distribution, reassignment, etc. in order to make them directly readable and/or executable by a computing device and/or other machine. For example, the machine readable instructions may be stored in multiple parts, which are individually compressed, encrypted, and stored on separate computing devices, wherein the parts when decrypted, decompressed, and combined form a set of executable instructions that implement a program such as that described herein. In another example, the machine readable instructions may be stored in a state in which they may be read by a computer, but require addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc. in order to execute the instructions on a particular computing device or other device. In another example, the machine readable instructions may need to be configured (e.g., settings stored, data input, network addresses recorded, etc.) before the machine readable instructions and/or the corresponding program(s) can be executed in whole or in part. Thus, the disclosed machine readable instructions and/or corresponding program(s) are intended to encompass such machine readable instructions and/or program(s) regardless of the particular format or state of the machine readable instructions and/or program(s) when stored or otherwise at rest or in transit.
As mentioned above, the example process of
“Including” and “comprising” (and all forms and tenses thereof) are used herein to be open ended terms. Thus, whenever a claim employs any form of “include” or “comprise” (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within a claim recitation of any kind, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase “at least” is used as the transition term in, for example, a preamble of a claim, it is open-ended in the same manner as the term “comprising” and “including” are open ended. The term “and/or” when used, for example, in a form such as A, B, and/or C refers to any combination or subset of A, B, C such as (1) A alone, (2) B alone, (3) C alone, (4) A with B, (5) A with C, (6) B with C, and (7) A with B and with C. As used herein in the context of describing structures, components, items, objects and/or things, the phrase “at least one of A and B” is intended to refer to implementations including any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects and/or things, the phrase “at least one of A or B” is intended to refer to implementations including any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the performance or execution of processes, instructions, actions, activities and/or steps, the phrase “at least one of A and B” is intended to refer to implementations including any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing the performance or execution of processes, instructions, actions, activities and/or steps, the phrase “at least one of A or B” is intended to refer to implementations including any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.
At block 402, the example performance data comparator 201 collects (e.g., via the example on-chip interface 200) performance data of the example application 104. For example, the on-chip interface 200 polls counter values from the example event counters 118 of the example HFI 102 corresponding to communication events occurring at the example HFI 102. Because the application 104 corresponds to the instructions that cause the communication events, tracking the event counts corresponds to the performance of the example application 104. At block 404, the example performance data comparator 201 processes the collected performance data. The example performance data comparator 201 processes the collected performance data to determine if there is a period of low activity (e.g., low communication activity). Periods of low activity periodically occur in bulk-synchronous HPC applications, for example. The example performance data comparator 201 may determine that there is a period of low activity if less than a threshold number of communication operations occurred within a duration of time.
At block 406, the example performance data comparator 201 determines if the example collector 108 should enter sleep mode. For example, if the performance data comparator 201 determines that there is a period of low activity based on current and/or previous polled data, the performance data comparator 201 determines that sleep mode should be entered. Additionally or alternatively, the example performance data comparator 201 may determine that sleep mode should be entered based on a triggered signal from the example application 104 and/or another component.
If the example performance data comparator 201 determines that the collector 108 should not enter sleep mode (block 406: NO), the process returns to block 402 and the example collector 108 continues to poll performance data at a frequency corresponding to wake-up mode. If the example performance data comparator 201 determines that the collector 108 should enter sleep mode (block 406: YES), the example instructions generator 202 determines which and/or how many event(s) to correspond to a wake-up trigger (block 407). For example, the instructions generator 202 may determine that the collector 108 should be awaken in response to three message arrivals at the HFI 102, five messages have been transmitted by the HFI 102, and/or 100 bytes have been received by the HFI 102. The wake up parameters may be based on user and/or manufacturer preferences.
At block 408, the example instructions generator 202 obtains event count(s) of the event counter(s) 118 (e.g., via the example on-chip interface 200) corresponding to the events to be tracked. For example, if an event to be tracked corresponds to a number of received messages, and the corresponding event counter is currently at a count of one hundred, the example instructions generator 202 identifies the count as one hundred. At block 409, the example adder 204 determines the threshold count(s) by adding the wake-up count to the identified count of the corresponding event counters 118. For example, if the wake up count is five and the current count of the corresponding event counter 118 is one hundred, the example adder 204 determines the threshold count for the corresponding counter to be one-hundred five.
At block 410, the example instructions generator 202 allocates address(es) in the example user memory space 110 to correspond to a triggered operation(s). As described above, the triggered operation will instruct the example HFI 102 to write to a selected address in the user memory space 110 in response to the number of selected events occurring. Accordingly, the example instructions generator 202 allocates the memory space to be able to determine when the HFI 102 has written to the memory, thereby triggering a wake-up of the collector 108. At block 412, the example memory monitor 206 reads the initial data stored at the allocated address(es). In some examples, the memory monitor 206 may write (e.g., using the example memory interface 208) a preset initial value to the allocated address(es) to ensure that the HFI does not write the same data as the initial data.
At block 414, the example on-chip interface 200 transmits write back instructions (e.g., one or more data packet(s) including the triggered operation(s), the allocated memory address location(s), and the wake-up parameters (e.g., the type of events and/or event counters to trigger wake-up, the threshold count(s), etc.)) to the example HFI 102. At block 416, the frequency selector 205 enters sleep mode by reducing the polling frequency from a first frequency (e.g., an awake polling frequency) to a second frequency (e.g., a sleep polling frequency). As described above, reducing or otherwise halting performance polling conserves CPU resources.
At block 418, the example memory monitor 206 reads the current data at the allocated address(es) by instructing the memory interface 208 to read the value stored at the allocated address. At block 420, the example memory monitor 206 determines if the current data (e.g., the data read from the allocated memory address(es) at block 418) is the same as the initial data (e.g., the data read from the allocated memory address(es) at block 412). As described above, if the event counter(s) associated with the triggered operations reach the threshold value, the example HFI 102 writes data to the allocated memory address of the user memory space 110. Accordingly, the current data being the different from the initial data corresponds to a wake-up trigger for the collector 108.
If the example memory monitor 206 determines that the current data is the same as the initial data (block 420: YES), the process returns to block 418 to continue to monitor the data in the allocated memory address(es) and the collector 108 remains in sleep mode. If the example memory monitor 206 determines that the current data is not the same as (e.g., is different than) the initial data (block 420: NO), the example frequency selector 205 wakes the collector 108 up by increasing the polling frequency from the second frequency to the first frequency and/or any other frequency faster than the second frequency (block 422) and the process returns to block 402 to collect performance data, thereby waking up the collector 108.
At block 502, the communication interface 300 of the example triggered operations circuitry 112 obtains write back instructions from the collector 108 corresponding to a triggered put operation. As described above, the example collector 108 may transmit the write back instructions corresponding to a triggered put operation when the collector 108 enters into a sleep-mode. At block 504, the example triggered operations circuitry 112 determines the event(s) to be tracked, the threshold count(s) (e.g., the count of one or more event count(s) that must occur before the triggered operation is executed to wake up the collector 108), and/or the corresponding memory address location(s) for writing once the wake-up count(s) is/are satisfied based on the obtained write back instructions.
At block 506, the example instructions queue 302 of the example triggered operation circuitry 112 stores the triggered operation(s) specified in the obtained data packet(s). As described above, the instructions queue 302 stores the triggered operation(s) (e.g., triggered put operation(s) or triggered atomic operation(s)) until the count(s) of the event counter(s) 118 corresponding to the determined event(s) satisfies the wake-up count(s). At block 510, the example threshold register 308 stores the threshold count(s) specified in the write back instructions.
At block 512, the example communication engine 116 determines if an event corresponding to one of the example event counters 118 has occurred. If the example communication engine 116 determines that an event corresponding to one of the example event counters 118 has not occurred (block 512: NO), the example communication engine 116 continues to monitor events. If the example communication engine 116 determines that an event corresponding to one of the example event counters 118 occurred (block 512: YES), the example communication engine 116 increments the corresponding event counter 118 (block 514).
At block 516, the example comparator 310 of the triggered operations circuitry 112 determines if the current count of the corresponding event counter(s) 118 (e.g., the event counter(s) corresponding to the events identified in the obtained data packed) reached the trigger threshold. If the comparator 310 determines that the current count of the corresponding event counter(s) 118 does not satisfy the threshold count(s) (block 516: NO), the process returns to block 512 until one or more of the corresponding event counters 118 satisfies the threshold count(s). If the comparator 310 determines that the current count of the corresponding event counter(s) 118 satisfies the threshold count(s) (block 516: YES), the example triggered operations circuitry 112 launches the example queued operation(s) (block 518) by popping the queued put operation(s) and transmitting the put operation(s) to the example command processor 114. At block 520, the example command processor 114 executes the triggered operation by instructing the example communication engine 116 to write data (e.g., using a DMA/RDMA operation) to the allocated memory address(es) (e.g., specified in the put operation of the obtained write back instructions) of the example user memory space 110.
The processor platform 600 of the illustrated example includes a processor 612. The processor 612 of the illustrated example is hardware. For example, the processor 612 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor implements the example on-chip interface 200, the example performance data comparator 201, the example instructions generator 202, the example frequency selector 205, the example memory monitor 206, and the example memory interface 208.
The processor 612 of the illustrated example includes a local memory 613 (e.g., a cache). In some examples, the local memory 613 implements the example cache 120 of
The processor platform 600 of the illustrated example also includes an interface circuit 620. The interface circuit 620 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), a Bluetooth® interface, a near field communication (NFC) interface, and/or a PCI express interface.
In the illustrated example, one or more input devices 622 are connected to the interface circuit 620. The input device(s) 622 permit(s) a user to enter data and/or commands into the processor 612. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system.
One or more output devices 624 are also connected to the interface circuit 620 of the illustrated example. The output devices 624 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer and/or speaker. The interface circuit 620 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip and/or a graphics driver processor.
The interface circuit 620 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and/or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 626. The communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc. In the example of
The processor platform 600 of the illustrated example also includes one or more mass storage devices 628 for storing software and/or data. Examples of such mass storage devices 628 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.
The machine executable instructions 632 of
The processor platform 700 of the illustrated example includes a processor 712. The processor 712 of the illustrated example is hardware. For example, the processor 712 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor implements the example triggered operations circuitry 112, the example command processor 114, the example communication engine 116, and the example event counters 118.
The processor 712 of the illustrated example includes a local memory 713 (e.g., a cache). The processor 712 of the illustrated example is in communication with a main memory including a volatile memory 714 and a non-volatile memory 716 via a bus 718. The volatile memory 714 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®) and/or any other type of random access memory device. The non-volatile memory 716 may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory 714, 716 is controlled by a memory controller.
The processor platform 700 of the illustrated example also includes an interface circuit 720. The interface circuit 720 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), a Bluetooth® interface, a near field communication (NFC) interface, and/or a PCI express interface.
In the illustrated example, one or more input devices 722 are connected to the interface circuit 720. The input device(s) 722 permit(s) a user to enter data and/or commands into the processor 712. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system.
One or more output devices 724 are also connected to the interface circuit 720 of the illustrated example. The output devices 724 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer and/or speaker. The interface circuit 720 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip and/or a graphics driver processor.
The interface circuit 720 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and/or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 726. The communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.
The processor platform 700 of the illustrated example also includes one or more mass storage devices 728 for storing software and/or data. Examples of such mass storage devices 728 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.
The machine executable instructions 732 of
Example methods, apparatus, systems, and articles of manufacture to collect performance data collection in cooperation with a host fabric interface are disclosed herein. Further examples and combinations thereof include the following: Example 1 includes an apparatus to collect performance data collection in cooperation with a host fabric interface, the apparatus comprising a performance data comparator of a source node to collect the performance data of an application of the source node from the host fabric interface at a polling frequency, an interface to transmit a write back instruction to the host fabric interface, the write back instruction to cause data to be written to a memory address location of memory of the source node to trigger a wake up mode, and a frequency selector to start the polling frequency to a first polling frequency for a sleep mode, and increase the polling frequency to a second polling frequency in response to the data in the memory address location identifying the wake mode.
Example 2 includes the apparatus of example 1, further including an instructions generator to generate the write back instruction corresponding to a threshold number of events.
Example 3 includes the apparatus of example 2, wherein the write back instruction is to cause host fabric interface to write the data to the memory address in response to the threshold number of events.
Example 4 includes the apparatus of example 1, wherein the memory is accessible to the application.
Example 5 includes the apparatus of example 1, wherein the first polling frequency is zero.
Example 6 includes the apparatus of example 1, further including a memory monitor to monitor the data at the memory address location changes.
Example 7 includes the apparatus of example 6, wherein the memory monitor is to monitor the data at the memory address location by reading an initial value of the memory address location, reading a current value of the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 8 includes the apparatus of example 6, wherein the memory monitor is to monitor the memory address location by writing an initial value to the memory address location of the memory, reading a current value stored at the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 9 includes a non-transitory computer readable storage medium comprising instructions which, when executed, cause a processor to at least collect performance data of an application of a source node at a polling frequency, transmit a write back instruction to host fabric interface, the write back instruction to cause data to be written to a memory address location of memory of the source node to trigger a wake mode, start the polling frequency to a first polling frequency for a sleep mode, and increase the polling frequency to a second polling frequency in response to the data in the memory address location identifying the wake mode.
Example 10 includes the non-transitory computer readable storage medium of example 9, wherein the instructions cause the processor to generate the write back instructions corresponding to a threshold number of events.
Example 11 includes the non-transitory computer readable storage medium of example 10, wherein the write back instructions is to cause a host fabric interface to write the data to the memory address location in response to the threshold number of events.
Example 12 includes the non-transitory computer readable storage medium of example 9, wherein the memory is accessible to the application.
Example 13 includes the non-transitory computer readable storage medium of example 9, wherein the first polling frequency is zero.
Example 14 includes the non-transitory computer readable storage medium of example 9, wherein the instructions cause the processor to monitor data in the memory address location.
Example 15 includes the non-transitory computer readable storage medium of example 14, wherein the instructions cause the processor to monitor the data at the memory address location by reading an initial value of the memory address location, reading a current value of the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 16 includes the non-transitory computer readable storage medium of example 14, wherein the instructions cause the processor to monitor the memory address location by writing an initial value to the memory address location of the memory, reading a current value stored at the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 17 includes a source node comprising a processor, memory, and a collector to collect performance data corresponding to a high performance computing application to be executed by the processor, transmit a write back instruction to a host fabric interface, the write back instruction to cause the host fabric interface to initiate an update of a memory address location of the memory of the source node, enter into a sleep mode, and wake up from the sleep mode in response to the update to the memory address location.
Example 18 includes the source node of example 17, wherein the write back instruction is to cause a write operation to the memory address location of the memory in response to a threshold number of events.
Example 19 includes the source node of example 17, wherein the collector is to monitor the data at the memory address location for changes.
Example 20 includes the source node of example 19, wherein the collector is to monitor the data at the memory address location by reading an initial value of the memory address location, reading a current value of the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 21 includes the source node of example 19, wherein the collector is to monitor the memory address location by writing an initial value to the memory address location of the memory, reading a current value stored at the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 22 includes an apparatus to collect performance data collection in cooperation with a host fabric interface, the apparatus comprising means for collecting the performance data of an application of the source node from the host fabric interface at a polling frequency, means for transmitting a write back instruction to the host fabric interface, the write back instruction to cause data to be written to a memory address location of memory of the source node to trigger a wake up mode, and means for starting the polling frequency to a first polling frequency for a sleep mode, and increasing the polling frequency to a second polling frequency in response to the data in the memory address location identifying the wake mode.
Example 23 includes the apparatus of example 22, further including means for generating the write back instruction corresponding to a threshold number of events.
Example 24 includes the apparatus of example 23, wherein the write back instruction is to cause host fabric interface to write the data to the memory address in response to the threshold number of events.
Example 25 includes the apparatus of example 22, wherein the memory is accessible to the application.
Example 26 includes the apparatus of example 22, wherein the first polling frequency is zero.
Example 27 includes the apparatus of example 22, further means for monitoring the data at the memory address location changes.
Example 28 includes the apparatus of example 27, wherein the means for monitoring is to monitor the data at the memory address location by reading an initial value of the memory address location, reading a current value of the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
Example 29 includes the apparatus of example 27, wherein the means for monitoring is to monitor the memory address location by writing an initial value to the memory address location of the memory, reading a current value stored at the memory address location, and identifying that the data in the memory address location has changed when the initial value is different than the current value.
From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed herein to improve performance data collection in high performance computing applications. Disclosed methods, apparatus and articles of manufacture improve performance data collection for HPC applications by leveraging the possible capability of an HFI to wake up a collector from sleep mode. For example, although HFIs are typically structured and/or programmed to forward data to other nodes in a HPC system by writing data into memory of the other nodes for collective communication operations, examples disclosed herein utilize a collector of a node to instruct the HFI to initiate a trigger put operation (e.g., a write data operation) in the memory of the node that includes the sleeping collector and requested the write back (as opposed to another node in the HFI). In sleep mode, the collector monitors a specified memory address location to identify when the memory address location is written to a trigger value, thereby corresponding to the condition being satisfied (e.g., the one or more events occurring). In response to the collector identifying that the data in the memory address location has been updated, the collector then wakes up and increases the polling frequency or restarts the polling process. Because monitoring one or more memory addresses uses less CPU resources than polling the event counters directly, examples disclosed herein significantly reduce the amount of CPU resources needed to perform performance data collection of HPC applications. Disclosed methods, apparatus and articles of manufacture are accordingly directed to one or more improvement(s) in the functioning of a computer.
Although certain example methods, apparatus and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.
This patent arises from a continuation of U.S. patent application Ser. No. 16/286,095, which was filed on Feb. 26, 2019. U.S. patent application Ser. No. 16/286,095 is hereby incorporated herein by reference in its entirety. Priority to U.S. patent application Ser. No. 16/286,095 is hereby claimed.
Number | Date | Country | |
---|---|---|---|
Parent | 16286095 | Feb 2019 | US |
Child | 17856148 | US |