Patent Grant
7249270
1. Field of the Invention
The present invention relates to a data processing apparatus and method for controlling access to a shared resource in a data processing apparatus, and in particular controlling such access in a data processing apparatus having a plurality of processors operable to perform respective data processing operations requiring access to the shared resource.
2. Description of the Prior Art
It is known to provide multi-processing systems in which two or more processor cores share access to one or more shared resources, for example one or more data structures provided within a shared memory region. Such systems are typically used to gain higher performance by arranging the different processor cores to execute respective data processing operations in parallel. Known data processing systems which provide such multi-processing capabilities include IBM370 systems and SPARC multi-processing systems. These particular multi-processing systems are high performance systems where power efficiency and power consumption is of little concern and the main objective is maximum processing speed.
An important consideration in multi-processing systems where the multiple processors have access to a shared resource is the provision of a mechanism to control access to that shared resource to ensure that the data of that resource does not become inconsistent or corrupted due to multiple processors accessing the resource simultaneously. To deal with this issue, a typical multi-processor operating system may provide an access control mechanism such as a locking mechanism to protect the shared resource from being accessed by multiple processors simultaneously to prevent the data of the shared resource becoming inconsistent or corrupted if conflicting changes are made. If a lock is busy, i.e. has been set by one of the processors, it is often not practical for another processor that requires access to the shared resource associated with that lock to do anything except wait for the lock to be released, also referred to herein as the lock being cleared. For example, if a processor is handling an interrupt from a device it may need to add data received from the device to a shared queue. If another processor is removing data from the queue, the first processor must wait until the queue is in a consistent state and the lock has been released before it can then set the lock and add the new data to the queue. It cannot return from the interrupt handler until the data has been added to the queue, and hence it must wait.
In circumstances such as these, it is known to provide a “spin-lock” mechanism. In accordance with this mechanism, a processor requiring access to the protected data will attempt to obtain the lock, this involving reading the lock and then setting the lock if the read operation indicates that the lock is currently cleared. If the lock cannot be obtained, the processor repeatedly reads the lock value in a tight loop until the lock becomes available. At this point, the processor can then attempt once again to obtain the lock.
In a multi-processing system where low power consumption is desirable, this tight read loop is undesirable because energy is being consumed to no useful effect, and additionally in systems implementing multi-threading, the execution of spin-locks by waiting threads may be highly detrimental to performance.
Accordingly, it would be desirable to provide an improved technique for controlling access to a shared resource.
Viewed from a first aspect, the present invention provides a data processing apparatus comprising: a plurality of processors operable to perform respective data processing operations requiring access to a shared resource; an access control mechanism operable to control access to the shared resource by the plurality of processors; a path interconnecting the plurality of processors; each processor being operable to enter a power saving mode if access to said shared resource is required but the access control mechanism is preventing access to the shared resource by that processor; each processor further being operable, when that processor has access to the shared resource, to issue a notification on said path when access to the shared resource is no longer required by that processor; a processor in said power saving mode being operable, upon receipt of said notification, to exit said power saving mode and to seek access to the shared resource.
In accordance with the present invention, if a processor seeks to access a shared resource but is prevented from doing so by the access control mechanism, it is arranged to enter a power saving mode. Whilst in this power saving mode, that processor does not continue to seek access to the shared resource, and accordingly consumes significantly less power than would otherwise be the case. Furthermore, a processor that does have access to the shared resource is arranged to issue a notification when it no longer requires access to the shared resource, this notification being issued on a path interconnecting the plurality of processors within the data processing apparatus. Upon receipt of the notification by a processor that has entered the power saving mode, that processor is then operable to exit the power saving mode and to seek access to the shared resource. Hence, this notification is used as a mechanism to cause a processor in the power saving mode to “wake up” from that mode, and re-seek access to the shared resource. It has been found that this approach enables significant power savings to be achieved within the data processing apparatus.
It will be appreciated that the path may take a variety of forms. However, in one embodiment, the path comprises a dedicated path for the transfer of said notification between said plurality of processors, each of said processors having a port connected to said path. This ensures that there is no delay in the propagation of the notification.
It will be appreciated that in one embodiment a processor entering the power saving mode may be arranged to register an interest in a particular shared resource, if more than one such shared resource exists, with that information then being used by the processor that currently has access to that shared resource to ensure that the notification that it subsequently sends when it no longer requires access to that shared resource is directed to the appropriate processor(s). However, in one embodiment this approach is not used, and instead each processor is operable, when access to the shared resource is no longer required, to broadcast the notification to the remaining processors of said plurality. This enables a simplified implementation, in which no attempt is made to indicate which shared resource has become available when the notification is broadcast. If the notification relates to a different shared resource to that required by a processor in the power saving mode (or if another processor obtains the resource more quickly following that notification), then that processor on exiting the power saving mode and then failing to acquire access to the shared resource will merely re-enter the power saving mode and await receipt of the next notification.
In one embodiment, each processor has control storage which is set when said notification is received by that processor from the path. This enables a notification to be buffered at the processor, such that the processor can then determine from the value in the control storage whether a notification has been received by it. This is particularly useful in the situation where another processor determines that it no longer requires access to the shared resource (and hence issues the notification) between the time the processor seeking access to that resource determines that access is prevented and the time that processor enters the power saving mode. The use of the control storage avoids the possibility of the notification in such a situation being overlooked.
It will be appreciated that there are a number of ways in which a processor seeking access to a shared resource, but being prevented from such access, may enter the power saving mode. However, in one embodiment each processor is operable, if access to said shared resource is required but the access control mechanism is preventing access to the shared resource by that processor, to execute a wait for notification instruction to cause said power saving mode to be entered. Hence, in accordance with this embodiment, a processor that is seeking access to a shared resource but is prevented from access to that shared resource by the access control mechanism is arranged to execute a specific instruction which causes the power saving mode to be entered, until such time as a notification is received.
In one particular embodiment where the control storage is provided within each processor, the processor is operable upon execution of the wait for notification instruction to check whether the control storage has been set, and if so to prevent the power saving mode being entered. Instead, if it is determined that the control storage has been set, the processor can be arranged to merely clear the control storage and immediately seek access to the shared resource. This hence avoids the processor unnecessarily entering the power saving mode.
It will be appreciated that there are a number of ways in which a processor that has access to the shared resource can be arranged to issue the notification when it no longer requires access to that resource. However, in one embodiment, each processor is operable, when access to the shared resource is no longer required by that processor, to execute a send notification instruction to cause said notification to be issued on said path. Hence, in this embodiment each processor is arranged, when it no longer requires access to a shared resource that it currently does have access to, to execute a particular instruction in order to cause the notification to be issued.
It will be appreciated that the access control mechanism can take a variety of forms. However, in one embodiment, the access control mechanism comprises a lock associated with the shared resource, when that shared resource is accessed by one of said processors the lock being set, and when access to the shared resource is no longer required by that processor the lock being cleared prior to that processor issuing said notification on said path.
Such a lock will typically be maintained within a shared memory region, so that the value of the lock can be read by the various processors as and when required, and can be set by a processor when it gains access to the shared resource.
In one embodiment, when one of said processors wishes to access said shared resource, that processor is operable to perform an atomic operation to check whether the associated lock is cleared and if so to set the associated lock. The atomic operation ensures that the value of the lock cannot be changed by another processor between the time the lock is read to determine whether it is clear, and the time it is set by the processor. This is important to ensure that a processor does not read a lock that is cleared, but which is then set by a different processor prior to the lock being set by the processor that has read the “clear” value of the lock.
The atomic operation can be implemented in a variety of ways. However, in one embodiment, the processor is operable to perform said atomic operation by executing a swap instruction to cause the processor to read the value of the lock, to write a set value to the lock and to determine from the read value whether the associated lock was cleared. The nature of the swap instruction itself ensures that an atomic operation takes place, since the swap instruction reads the value of the lock and writes a set value to the lock as part of a single operation. If it transpires that the value of the lock that has been read is cleared, then the fact that the lock has been set by this processor ensures that no other processor subsequently reading the lock will see a value other than the set value. Similarly, if the processor executing the swap instruction determines from the read value that the associated lock was already set, then the fact that it has written a set value to the lock will be of no consequence. In this instance, the processor will merely determine that it cannot access the shared resource, and accordingly will enter the power saving mode to await notification from the processor that does have access to that shared resource when that processor has subsequently cleared the lock.
In an alternative embodiment, the data processing apparatus further comprises: monitor logic operable to monitor accesses to the lock by the plurality of processors; wherein the processor is operable to perform said atomic operation by: executing a load instruction to read the value of the lock and to initiate monitoring of the lock by the monitor logic; and executing a store instruction to store a set value to the lock if the monitor logic indicates that none of the other processors have modified the value of the lock since the value of the lock was read using the load instruction. Such load and store instructions will be referred to herein as exclusive load and store instructions, which when executed in combination ensure that an atomic operation takes place, thus ensuring that if a processor determines from the read value of the lock that it is clear, that processor will only be able to store a set value to the lock if it is apparent that no other processor has in the meantime modified the value of the lock.
The shared resource may take a variety of forms. However, in one embodiment the shared resource is a data structure, which may for example comprise a single data value, or multiple data values.
In one embodiment the data structure is provided within a slave device. In one particular embodiment, the data structure is provided within a shared memory region of a memory.
It will be appreciated that the techniques of the present invention may be employed in a data processing apparatus that only has a single shared resource. However, in one embodiment, the data processing apparatus comprises a plurality of shared resources, the access control mechanism being operable to control access to each of the shared resources. The access control mechanism may be a global access control mechanism provided for all of the shared resources, or alternatively a separate access control mechanism may be provided for each shared resource. In one embodiment the access control mechanism comprises a lock for each shared resource.
It will be appreciated that in certain embodiments other actions (other than receipt of the notification) could also be used to cause the processor to exit the power saving mode. For example, in one embodiment, each processor, when in said power saving mode, is operable to exit said power saving mode if an exception is received by that processor. Such an exception may for example be an enabled (i.e. non-masked) interrupt, an imprecise data abort, etc. By allowing the processor to exit the power saving mode in this way, this enables the processor to handle interrupts or other exceptions without impacting the latency.
Viewed from a second aspect, the present invention provides a method of controlling access to a shared resource in a data processing apparatus having a plurality of processors operable to perform respective data processing operations requiring access to the shared resource, a path being provided interconnecting the plurality of processors, the method comprising the steps of: (i) providing an access control mechanism to control access to the shared resource by the plurality of processors; (ii) if access to said shared resource is required by one of said processors but the access control mechanism is preventing access to the shared resource by that processor, causing that processor to enter a power saving mode; (iii) when one of said processors has access to the shared resource, causing that processor to issue a notification on said path when access to the shared resource is no longer required by that processor; (iv) causing a processor in said power saving mode, upon receipt of said notification, to exit said power saving mode and to seek access to the shared resource.
The present invention will be described further, by way of example only, with reference to preferred embodiments thereof as illustrated in the accompanying drawings, in which:
When one of the processors 20, 30, 40 wishes to access one of the data structures within the shared memory 70, it needs to obtain the associated lock. In order to obtain the lock, the processor needs to read the current value of the lock in order to assess whether it can proceed with accessing the data structure. If it can, it will then update the lock and proceed with the access. The operation of reading the lock to assess its current contents, followed by any appropriate update to the lock, needs to occur as an atomic operation, since it is important that no other processor has the opportunity to modify the value of the lock between the time it is read and the time it is modified dependent upon that read value.
In a typical system, only one processor at a time may be allowed to have access to the data structure. When a processor wishes to have access to the shared data structure, then it can only consider the lock to be cleared if the lock indicates that no other processor has access. If this is the case, then that processor can set the lock. Whilst the lock is set in such a manner, no other processor can obtain access to the associated data structure. When the processor having access no longer requires access to the data structure, it will then clear the lock.
Each processor 20, 30, 40 incorporates an execution unit 22, 32, 42, coupled to a set of working registers 24, 34, 44. Each execution unit can execute data processing operations on data values held within the working registers.
As will be discussed in more detail later, when a processor has obtained the lock for a particular data structure, and accordingly has the required access rights to that data structure, it will be arranged when it no longer requires access to that shared data structure to execute a Send Event instruction in order to issue an event on to a dedicated path established between the processors 20, 30, 40. Each of the processors 20, 30, 40 has a port 26, 36, 46 via which the processor is connected to this dedicated path, each port having an event output pin EVOUT and an event input pin EVIN. The event is issued from a processor via its EVOUT pin, and this signal is then routed to the EVIN pin of the other processors. As can be seen from
In addition, each of the processors 20, 30, 40 is arranged such that if it seeks access to a data structure, but the current value of the associated lock indicates that it cannot have access at that time, that processor is then arranged to enter a power saving mode of operation and to remain in that power saving mode of operation until it receives an event at its EVIN pin.
As can be seen from
Further, as will be discussed in more detail with reference to the embodiment of
Thereafter, the process proceeds to step 110, where it is determined whether the shared resource is currently locked, this being indicated by the value of the lock read into the working register of the processor during the swap operation. If it is determined at step 110 that the resource is currently locked, then the process proceeds to step 115 where a wait operation is performed. This process will be discussed in more detail later with reference to
If at step 110 it is determined from the value read from the lock that the shared resource is not locked, then the swap operation will have already resulted in the lock being set to indicate that the shared resource is now locked. Hence, the process can now proceed to step 120, where the processor performs one or more operations using the data of the shared resource.
Each of the processors 20, 30, 40 may typically have a write buffer associated therewith into which write accesses to data within the shared memory 70 are placed. When the processor has finished performing any operations that require access to the shared resource, it will need to perform a write to the associated lock in order to clear the lock in order to release the associated data structure for use by another processor. However, some systems provide mechanisms whereby write accesses pending in a write buffer may complete out of order, and it is important to ensure that any write accesses to the data structure occur prior to the writing to the associated lock in order to release that lock.
Accordingly, prior to writing to the lock, the process will proceed from step 120 to step 125, where the processor will cause its write buffer to be drained. Then, at step 130, it will issue a write operation to the relevant lock in order to clear that lock, also referred to herein as releasing the lock. At this point, the processor is then ready to issue the event (also referred to herein as the notification) from its EVOUT pin onto the path interconnecting the various processors, but it must be ensured that the update to the lock has actually taken place before this occurs. Accordingly, prior to issuing the event, the processor proceeds to step 135, where again the processor causes its write buffer to be drained. Thereafter, at step 140 the event is sent over the path to the other processors, after which the process exits at step 145.
If however it was determined as a result of executing the compare operation that the current content of register r0 is a logic one value, then this will indicate that the lock had already been set by a different processor, and accordingly the processor cannot proceed with its required access to the shared resource. Instead, a wait for event instruction, also referred to herein as a wait for notification instruction, is executed. The operations performed as a result of executing this instruction will be discussed later with reference to
Once a processor has access to the shared resource, and has performed the necessary operations that it requires using the shared resource, it needs to implement the unlock routine. As shown in
It at step 205 it is determined that an event is not registered in the event register, then the process proceeds to step 210, where the processor enters a power saving mode. In one embodiment, this power saving mode places the processor in as low a power state as possible consistent with a fast wake up without the need for software state restoration. Thereafter, the process proceeds to step 215, where it is determined whether an event has been received at the EVIN port of the processor. If not, the process proceeds to step 217 where it is determined whether there is a pending exception. Such an exception may for example be an enabled (i.e. non-masked) interrupt, an imprecise data abort, etc. The provision of such a step is beneficial since it enables the processor to be able to handle interrupts or other exceptions without impacting the latency. In the event that no such exception is detected at step 217, the process returns to step 215.
Once either receipt at the EVIN port of an event is detected at step 215, or a pending exception is detected at step 217, the process proceeds to step 220, to cause the power saving mode to be exited. Thereafter, the wait for event operation exits at step 230. With reference to
At step 215, the processor is arranged to directly receive the signal received at the EVIN port, rather than it first being latched within the event register 28, 38, 48. However, if instead all signals received at the EVIN port are first latched in the event register, then on exiting the power saving mode at 220 due to detection of an event at step 215, the processor would also be arranged to clear the event register prior to exiting at step 230.
It will be appreciated that such an approach results in significant power savings when compared with a traditional prior art approach, during which the processor would repeatedly read the lock value in a tight loop until the lock became available. Instead, the processor enters a power saving mode and waits to be woken up by receipt of an event from a processor indicating that a lock has been released.
It will be appreciated that in certain embodiments other actions could also be used to cause the processor to exit the power saving mode. For example, the processor may be arranged to exit the power saving mode if a debug entry request is received by the processor.
At step 310, it is determined whether the shared resource is locked by reviewing the value loaded from the lock address. If the resource is locked, then the process proceeds to step 315 where a wait for event operation is performed. Step 315 of
If however it is determined that the shared resource is not locked at step 310, then the process proceeds to step 320 where an exclusive store operation is performed. As will be discussed later with reference to
At step 325, it is determined whether the store operation passed its conditions for execution, i.e. there were no intervening modifications to the lock. If the store did not pass its conditions, and hence the write did not take place, the process returns to step 305.
Assuming that the store operation did pass its conditions, and accordingly was executed, the process proceeds from step 325 to step 330, where any required operations using the shared resource are performed. Steps 330, 335, 340, 345, 350 and 355 are analogous to steps 120, 125, 130, 135, 140 and 145 discussed earlier with reference to
Thereafter a compare instruction is executed to determine whether the content of register r0 (i.e. the lock value as read from the lock address) is a logic zero value. If the content of register r0 is equal to zero, this indicates that the lock was cleared at the time it was read, and accordingly a store exclusive operation is then executed to store to the lock address the contents of register r1, i.e. to store a logic one value at the lock address. However, execution of this store exclusive instruction is conditional upon the monitor logic 60 indicating that no intervening modifications to the lock value have been made since the lock value was read by the load exclusive instruction. The store exclusive instruction also causes register r0 to be updated with a value indicating whether the store took place or not. In particular, r0 is set to a logic zero value if no intervening modifications to the lock were made, and accordingly the store took place, or is set to a logic one value if intervening modifications to the lock did take place, and accordingly the store did not proceed.
If the result of the previous compare instruction was that the contents of register r0 were not equal to zero (i.e. indicating that the lock had already been set by a different processor), then instead of executing the store exclusive instruction the processor will execute a wait for event instruction, this causing the processing steps discussed earlier with reference to
However, this second compare instruction is executed following execution of the store exclusive instruction, since the store exclusive instruction is only executed if the first comparison instruction concluded that the contents of register r0 were equal to zero. As discussed earlier, execution of the store exclusive instruction also results in register r0 being updated with a value indicating whether the store actually took place or not. This second compare instruction then determines whether the content of r0 is equal to zero, this indicating that the store did take place, and that accordingly the processor has locked the shared resource and can proceed to have access to the resource. However, if as a result of execution of this second compare instruction it is determined that the contents of r0 are not equal to zero, this will indicate that the store did not take place, and accordingly the processor then executes the branch instruction to return to the load exclusive instruction.
As will be appreciated from the above description, the technique of embodiments of the present invention provides a mechanism to allow a processor waiting a spin-lock to enter a low power state. The processor that currently holds the desired spin-lock sends an event when releasing the lock, which causes the waiting processor to wake up and attempt to gain the lock again. A complex operating system may contain thousands of distinct locks, and in the above described embodiments no attempt is made to indicate which lock has been released when the event is sent. If the event relates to a different lock (or if another processor obtains the lock more quickly), the processor that has woken up will fail to acquire the lock, and will re-enter the low power state waiting for the next event.
The technique of the above described embodiments of the present invention relies on hardware and software to achieve the energy saving. The hardware is adapted to provide a path to enable an event issued by one processor to be routed to all of the other processors having access to the shared resource, and each processor is further provided with a register which can be set to indicate whether an event has been received by that processor. Two new instructions, namely a send event instruction and a wait for event instruction are added. The hardware hence provides the mechanism to enter the wait for event low power state, and the operating system software is responsible for issuing wait for event instructions and send event instructions when waiting for a contended lock and releasing a lock respectively.
It will be appreciated by those skilled in the art that the same wait for event and send event instructions could be used with more complex locks than those described in the above embodiments, for example read/write locks, to achieve a similar energy saving.
It will further be appreciated by those skilled in the art that the techniques of the present invention may be utilised in embodiments using atomic operations other than the two types of atomic operation explicitly described in the above embodiment description.
In one embodiment of the present invention, the process of acquiring locks may be nested, and hence when the processor acquires a lock at step 110 of
Although a particular embodiment has been described herein, it will be appreciated that the invention is not limited thereto and that many modifications and additions thereto may be made within the scope of the invention. For example, various combinations of the features of the following dependent claims could be made with the features of the independent claims without departing from the scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 0411792.5 | May 2004 | GB | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5502840 | Barton | Mar 1996 | A |
| 5875320 | Gu | Feb 1999 | A |
| 6910212 | Brenner et al. | Jun 2005 | B2 |
| 7100021 | Marshall et al. | Aug 2006 | B1 |
| 7162666 | Bono | Jan 2007 | B2 |
| 20020004810 | Reneris | Jan 2002 | A1 |
| 20050166206 | Parson | Jul 2005 | A1 |
| 20050289365 | Bhandarkar | Dec 2005 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 2004006097 | Jan 2004 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20050268106 A1 | Dec 2005 | US |
