Modern servers contain both “system processors” and “service processors.” Management of server hardware is commonly performed by a management application running on either a server or servers (e.g., in a blade or partitioned server system) being managed or on a remote management station (another computer). The management application obtains its data from the service processor(s) that is part of the server(s) being managed. The communication channel used to obtain this data typically has high latency and low bandwidth. Current protocols and formats used for this data transfer are, however, very inefficient over such a channel, leading to long application response times. Further, current methods require significant processing on the service processor for each data request, which also increases the latency due to the service processor's limited processing power.
When the amount of information to be transferred is small or the data rate is low, current methods can give acceptable performance. But the trend for server management applications is to request increasingly more information at higher rates; the point has been reached where current data transfer methods create a performance bottleneck that leads to unacceptable application performance. This is particularly a problem in large, partitionable servers and blade systems.
One current method for data access to a service processor is IPMI (Intelligent Platform Management Interface). IPMI uses a relatively efficient format for data transfer, but requires the application to perform schema accesses over the slow channel and requires the application to perform many accesses to obtain its required data. The processing of IPMI request on the service processor is relatively simple and efficient, but IPMI supports only a simple data model and a limited number of objects.
Another current method is WBEM (Web Based Enterprise Management). Originally designed to run on powerful computers and communicate over a fast data channel, it is being used on low performance service processors and slow communication channels. Besides having the schema access and multiple access limitations of IPMI, WBEM also uses a particularly inefficient data format and is complex, requiring significant memory and processing on the service processor to parse a request and form a response. WS-manage (Web Services for Management) is an emerging Microsoft method for obtaining management data with all of the disadvantages of WBEM.
An advantage of the embodiments described herein is that they overcome the disadvantages of the prior art. These advantages and others are achieved by a system and method for efficient data access. A method for efficient remote data access for use by server management applications, includes transferring server schema from a service processor on a server to an application machine, receiving a request for server data on application machine, processing the request for server data on application machine, and forming, in a self-describing format, a single, consolidated query to obtain information not obtained from the server schema on the application machine. Processing the request obtains server data from the server schema on the application machine and the consolidated query is formatted in self-describing format. The method further includes transferring the consolidated query to the service processor, the service processor: processing the consolidated query, obtaining the requested data, and forming a response with the requested data in the self-describing format. The method transmits the formatted, retrieved data to application machine.
These advantages and others are also provided by a computer readable medium containing instructions for performing the above method. Likewise, these advantages and others are also provided by a system for efficient remote data access. The system includes an application machine including a memory that includes a remote server management application and instructions for executing the above method, a processor for executing the remote server management application and the instructions, a server including a service processor for receiving and processing queries from the application machine and a network connection between the application machine and the service processor, over which a copy of schema, consolidated queries, and retrieved data are transferred between application machine and server.
The detailed description will refer to the following drawings, wherein like numerals refer to like elements, and wherein:
Described herein are a system and method for efficient remote data access. The embodiments described herein provide efficient data access for remote server management applications. The performance of remote server management applications is significantly improved by these embodiments. Embodiments provide the efficient data access using a combination of software techniques.
As noted above, modern servers contain both “system processors” and “service processors.” The system processors are powerful processors, e.g., Itanium and X86 processors, which have gigabytes of memory, are connected to large external data stores (disks) with terabytes (TBs) of storage, and have high-bandwidth network connections. Except for the smallest servers, there are multiple system processors in one server. The system processors run an operating system of an administrator's choice, e.g., Linux, Unix or Windows, and various applications such as a relational database (e.g., Oracle, SQL server) or enterprise ERP applications (e.g., SAP) or a high performance web or email server.
The service processor is an embedded microprocessor that controls the hardware the server comprises. Servers have more than just processors and memory; servers have bulk power supplies, dc-dc converters, fans and other more arcane components. The service processor is responsible for controlling these various hardware components. A couple of simple examples: servers have a network of temperature sensors that sense the air temperature at strategic locations in the server cabinet. The service processor monitors these sensors and controls the speeds of the fans to keep the temperatures within acceptable limits while running the fans as slowly as possible to keep noise and power consumption as low as possible. During power up and power down of the server, the multiple power supplies and power rails must be turned on and off in a strict sequence with strict timing—the service processor controls this activation of the power supplies and power rails and the sequence and timing of the activation.
The service processor typically has limited memory compared to the system processors (MB vs GB), is clocked at low frequencies (MHz vs GHz), has no external storage (versus the system processor's TB of external disks). The service processor runs a specialized embedded operating system. The service processor's programming is fixed in ROM at the server factory; unlike the system processors, the server administrator cannot alter the service processor's programming.
Although in small servers there is indeed typically just one service processor, in large partitioned servers and blade systems, the service processor may be a network of cooperating microprocessors. For example, in Hewlett Packards's (“HP's”) current blade product, there is a service processor on each blade as well as a redundant pair in the server cabinet. In HP's current partitioned server family there is a service processor on each cell board and a redundant pair in the cabinet.
Service processor is a generic term; manufacturers have their own brand names for the service processors. HP cellular servers call them “the MP (Management Processor)” for the ones in the cabinet and “cell controller” for the ones on the cells. HP x86 servers call then “iLo (Integrated Lights Out).” HP blade servers call them “iLo” for the ones on the blades and “OA (Onboard Administrator)” for the ones in the cabinet. The IPMI specification uses the term “BMC (Baseboard Management Controller).”
Unlike the system processors, service processors are always on (assuming the server is plugged in), running on standby power. Service processors communicate with remote management applications via a network connection, but this network connection is typically a relatively low bandwidth connection compared to what the system processors have.
Server management applications both display information about the state of the server hardware and control the server hardware. To do this they communicate with the service processor, not with the system processors. After all, if a server administrator wants to remotely power up a server, the server management application cannot “talk” to a system processor since the system processor will be off. Likewise, if an administrator wants to know the temperatures in the server cabinet, the system management application must talk to the service processor because the system processor is not connected to the temp sensors.
Server management applications typically run on a server administrator's workstation in his office, well away from the computer room where the server(s) being managed resides. Modern workstations are far more powerful than the little embedded microprocessor that is the server's service processor.
Large modern servers are modular systems. Using an HP bladed system as an example, the blade chassis has slots for multiple bulk power supplies, slots for multiple fans, slots for multiple system processors and storage blades of various types in front and slots for multiple I/O blades of various types in back. The cabinet may be populated with an almost infinite combination of these components; not every slot needs to be filled. HP's partitioned servers are similar.
A typical function for a server management application is to display the inventory of the chassis (what's plugged in where). It is the service processor that knows this information, as well as detailed information about the components. Some examples of the detailed information about server hardware components the service processor can supply to a server management application are: the amount of memory a processor blade has installed, the number of system processors on a blade, the version of each system processor, the type of I/O card in each I/O slot, and so, on.
A server management application also typically displays hardware health and status information that it obtains from the service processor. Some examples: the power state (on or off) for each blade and bulk power supply, fan speeds, cabinet temperatures, power consumption, power rail voltage levels, and so on. If a component fails (e.g., a fan stops turning), the service processor can signal that to a server management application, too. The server management application can then be configured to alert an administrator (e.g., via e-mail or pager).
In bladed and partitioned servers, server management applications also configure the server. In bladed systems, a server management application remotely configures a blade in some slot to connect to one of the multiple networks connected to the blade chassis via an I/O switch blade. In a partitioned server, a management application configures processors and I/O cards into partitions (independent servers). All of this control is affected through the service processor.
The service processor needs to present an information rich interface to the server management application to accomplish all these tasks. This interface typically resembles a simple database interface. The server management application is making requests to the service processor such as: “What is the version number of processor 2 on blade 4?” or “Is there a fan in slot 5 and if so, what is its speed?” or “How much power is blade 6 consuming?” These sorts of requests to the service processor and the service processor's responses fit naturally into a database model. However, in this case the data is not written first by an application and then later retrieved but is real time information about the hardware state of the server.
Control over the hardware can also be achieved via this sort of database model using writes to object properties. So for example, one may have a “blade” entity that has a binary “power” property. If the server management application reads the power property for a blade it learns whether the blade is on or off. If the server management application writes the power property for a blade it turns the blade's power on or off.
For server management applications not to have to have embedded, a priori knowledge of the servers they manage, the service processor may also supply basic configuration information about the server. For example, HP has two different blade chassis, each of which has a different number of blade slots (among other things). Consequently, a server management application could be written with embedded knowledge of how many blade slots each chassis model has, and then the server management application could request of the service processor which model of chassis is its server. Alternatively, the server management application could be written to handle an arbitrary number of slots and ask the service processor how many slots the service processor's chassis has. For various reasons the later is more desirable.
The description of all the data the service processor can supply to a server management application, how the data is organized, what data may be read only and what data may be written and the format of each data item is the “schema.” The schema is the information a server management application must have in order to know how to compose a request to the service processor. For example, a schema may say a server chassis has blade slots, blades have memory DIMMS, memory DIMMs have serial numbers, etc.
In current interfaces the schema resides on the service processor and the interface supports only simple requests. This means the server management application ends up making a long sequential string of requests to get information. For example, suppose a server management application wants to list the serial number of all the DIMMs in a server. A very much simplified version of the “conversation” with the service processor may be something like the following:
Step 1: First the server management application needs to see if the service processor can provide DIMM serial numbers and the path through the hardware decomposition to get them. The server management application sends a series of requests, somewhat akin to the following requests, receiving responses from the service processor for each (the actual conversation is much longer and more complex—the following is simplified to illustrate the concept of multiple requests and responses presently required to obtain the necessary information):
“What sort of server are you?”
“How many chassis are in this server?”
“Do the chassis have blades?”
“Do blades have DIMMS?”
“Do the DIMMs have serial numbers?”
“What's the format of a DIMM serial number?”
Step 2: Second, after it has received responses to the above, the server management application has to actually obtain the serial numbers. Consequently, the server management application sends a series of requests, somewhat akin to the following requests, receiving responses from the service processor for each (again, the actual conversation is much longer and more complex):
For each chassis, “How many blade slots does this chassis have?”
As can be seen by the above illustration, there is considerable message traffic between the server management application and the service processor is required to implement even the simplest management function. The service processor in a big server can literally have tens of thousands of pieces of information about the server hardware, so it is easy to see that the communication path between the server management application and the service processor can be a performance bottleneck, especially if the latency is high (e.g., the administrator is operating a server management application running on a application machine in India to remotely managing a server in a US data center).
Embodiments of the system and method described herein overcome this performance bottleneck by:
A) Transferring the schema in total from the service processor to the server management application. This is typically done when the server management application first contacts the service processor on a server being managed. With the schema transferred, the first set of questions above can be answered without going to the service processor at all. Indeed, all schema queries by the server management application can be handled by the server management application process on the application machine. This eliminates the need to send a class of requests and responses over the slow channel between the server management application and the service processor and moves the bulk of the processing to the higher performance application machine.
B) Populating the transferred schema with not just the format and organization of the data but also hardware fixed data (e.g., this server has 8 blade slots) and relatively static population information (e.g., slot 5 has a blade installed, the blade in slot 5 has DIMMs in memory slots 0, 1, 2) that may be transferred to the server management application from the service processor. This eliminates the “How many” and “Is there one in this slot” questions in the second set of questions above from requiring traffic over the slow channel.
C) Provide a server management application interface that can handle a request for and supply multiple arbitrary data items at once. So, for example, instead of asking for each DIMM serial number in a single, separate request, the management application can form one request to get all the serial numbers at once in a single response: “Give me the DIMM serial numbers for blade 2 DIMMs 1,2,3, blade 3 DIMMs 1,2, blade 5 DIMMS 1,2,3,4” or “Give me the temperature sensor readings for the temperature sensors at blade 2, blade 3 and blade 5.” In an embodiment, the server management application interface includes query code running on the application machine that allows the server management application to form a single query that retrieves all the information the server management application needs that it cannot obtain from the transferred schema and the data in the transferred schema. This further reduces the number of round trips over the slow channel and reduces the processing performed by the service processor.
D) Use an efficient, self-describing data format to transfer both the requests to the service processor and the responses back to the server management application. The efficient, self-describing data format reduces the number of bytes in a message over the slow channel and further reduces the processing performed by the service processor.
The four parts are independent of each other. Consequently, implementations may be created containing any combination of the four. However, if all four are implemented, then the server management application can, e.g., get all the DIMM serial numbers for the server with one request to the service processor. With the proper interface the processing required on the little weak microprocessor that is the service processor is minimal even for a request for a large amount of disparate information.
With reference now to
Servers can be changed or modified; for example, additional blades can be added to a bladed server. Accordingly, the server schema, which is the information about the server that the service processor can provide to the server management application, can change as well. Accordingly, a schema version identifier may be transferred with the schema. In this manner, the schema copy at the application machine can be checked versus the schema at the service processor and updated as needed.
With continuing reference to
As noted above, the server management application will need to access a lot of server information to manage the server. Consequently, the server management application will generate numerous requests for data. In the prior art, these requests would each be transmitted to the service processor for processing as a series or class of requests and responses to and from the service processor. Each of theses requests and responses would need to be transmitted over the slow channel between the client machine and the server. Likewise, each request would have to be processed and each response generated by the service processor. In method 10, however, the server management application instead seeks to obtain as much data as possible from the transferred schema before contacting the service processor. Accordingly, the server management application process receives a request for server information, block 16, and processes the request locally on the application machine, block 18. The server management application process processes 18 the request locally by obtaining all of the relevant, responsive data available from the schema on the application machine. This, in affect answers the first set of questions illustrated in step 1 above and some of the second set of questions illustrated in step 2 above (e.g., “how many blade slots” and “is a blade in this slot”).
With continuing reference to
The consolidated query is formatted in an easily parsable format. The format is preferably an efficient, self-describing data format. Looking at how the consolidated query may be encoded to send to the service processor gives a notion of the self-describing format. Self-describing is a term of art in computer science that generally has to do with tagging data so that one does not have to have a priori knowledge of context or format to know what a value means. So a sequence of bytes ‘123456789’ might mean anything; it is only when it is known that this is a string, not an integer or floating point number (its format or syntax) and is a serial number (its semantic) and is the serial number of DIMM 2 on Blade 1 (its instance) that this sequence of bytes has meaning. The idea behind a self-describing format is to pack the most information into the smallest number of bytes possible.
Continuing this example, a request for the serial number DIMM 1 on blade 2 may be encoded as the tuple: <B, 2, D, 1, N>; where B means blade, 2 is the blade number (i.e., the slot number the blade is in), D means DIMM, 1 is the DIMM number and N means serial number. The English decoding of this request tuple is then “Give me the serial number for the DIMM in memory slot 1 of the blade in blade slot 2.” By sending a simple sequence of tuples such as these as a request, method 10 can provide a consolidated request for a list of any set of information the service processor can supply. For example, a sequence to get the serial numbers for blade 2 DIMMs 1,2,3, blade 3 DIMMs 1,2, blade 5 DIMMS 1,2,3,4 from the above example would be:
<B, 2, D, 1, N><B, 2, D, 2, N><B, 2, D, 3, N><B, 3, D, 1, N><B, 3, D, 2, N><B, 5, D, 1, N><B, 5, D, 2, N><B, 5, D, 3, N><B, 5 D, 4, N>
The response from the service processor might be the tuple <B, 2, D, 1, N, S10, 1234567890>, i.e., blade 2, DIMM 1, serial number, a string of 10 characters whose value is “1234567890.” The data is self-describing, in that it contains its description as well as its value.
There are many of different schemes for encoding self-describing data. The one outlined above is both simple and flexible. Other embodiments could make it more complex and more compact. But since there is a shared schema between the server management application and the service processor, an embodiment can go so far as to assign each individual data item (e.g., each individual DIMM serial number) a unique identifier (aka, GUID, Globally Unique IDentifier). Then all the server management application and the service processor would have to do is transfer the GUIDs in a request and the GUIDs and values in a response. Note: the above is talking implicitly about read requests to the service processor; write requests would be similar but also contain the data values to be written, of course.
Efficiency can be increased by eliminating all descriptive information in a read response if sequence is preserved. Since the server management application has the schema, it knows a DIMM serial number is a fixed length string of 10 characters (that sort of information is in the schema), and it can know the sequence of the serial numbers it requested (e.g., first is blade 2, DIMM 1, next is blade 2 DIMM 2, and so on), then the service processor could just return the serial number strings in the sequence they were requested; the server management application then remembers the sequence of the request and matches up the returned values to their meanings.
With continued reference to
For example, HP servers have many different of serial numbers for their various components and they do not all have the same format (for many very practical reasons). So, although a DIMM serial number may be 10 characters on an Itanium partitioned server, it might be 12 characters on a blade server. An I/O card serial number might be 8 characters. An IBM server might have 24 character DIMM serial numbers. To keep the server management applications flexible so they do not have to be rewritten every time a new server is introduced, they are written to handle arbitrary data and look up the formats of that data at the time it is being used.
With continued reference to
With continuing reference to
Upon receipt of the single, consolidated query, the service processor processes the consolidated query, block 24. As noted above, using the self-describing format reduces the processing that is performed by the slow service processor. In processing 24 the consolidated query, the service processor extracts the location and information necessary to retrieve the requested data. Accordingly, the service processor retrieves the requested data, block 26.
With continued reference to
Additionally, in an embodiment, a schema version identifier is embedded in the data request (the formatted, consolidated query), block 32. The embedded schema version identifier may be the identifier described above that is transferred with the schema in 12. Consequently, the service processor may receive and compare the embedded schema version identifier the current version of the server schema, block 34. If the server schema has been changed and updated, the service processor may transfer a copy of the updated schema, along with a new schema version identifier as the response, block 36. In this manner, server schema changes may be detected, the schema on the application machine updated, and data requests reformed using the updated schema and re-sent to the service processor (e.g., steps 18-24 repeated).
The method described herein allows an application to obtain all the data it needs from the service processor in a single request/response pair from the service processor plus the overhead of one schema transfer over the slow channel, as opposed to the overhead of multiple request/response pairs and significant processing by the service processor previously required. Indeed, existing solutions require many requests and responses (typically hundreds of request/response pairs in a mid-range partitioned server). Since the slow channel is the performance bottleneck, this solution can provide up to orders of magnitude increases in performance.
The efficient data transfer format of this method further increases performance. WBEM, for example, can quite literally require the transfer of megabytes over the slow channel to obtain kilobytes of information; its data transfer format is that inefficient. The method described herein reduces the processing required on the service processor to a minimum, consequently simplifying embedded firmware and reducing the processing required on the low capability service processor, further improving performance. The method described herein supports complex data and complex data organization. The method described herein can be implemented to preserve the data models and APIs of existing management methods such as WBEM and WS-manage, consequently providing a large performance boost to management applications simply by re-linking the application to a new access library. The method described herein may also be implemented as a new API and access standard, providing an architectural control point in the industry. Moreover, the method described herein is applicable not just for server management data, but for a general class of data access over a slow channels.
With reference now to
Application machine 52 illustrates typical components of a computer on which server management application 56 is run. Application machine 52 typically includes a memory 60, a secondary storage device 62, a processor 64, an input device 66, a display device 68, and an output device 70. Memory 60 may include random access memory (RAM) or similar types of memory, and it may store one or more applications during execution, including remote server management application 56 and data access instructions 58, for execution by processor 64. Memory 60 may also store a copy of the server database schema while application 56 is running. Secondary storage device 62 may include a hard disk drive, floppy disk drive, CD-ROM drive, or other types of non-volatile data storage, and it may store application 56, data access instructions 58 and copy of the server database schema. Processor 64 may execute applications or programs stored in memory 60 or secondary storage 62, or received from the network 72 (e.g., Internet, LAN, etc.), and the processing may be implemented in software, such as software modules, for execution by computers or other machines. As noted above, remote server management application 56 and data access instructions 58 preferably include instructions executable to perform the methods described above with reference to
Application machine 52 typically connects to service processor on server 54 through network 72. As discussed above, network 72 is typically a low latency connection, at least relative to the size and number of data requests and responses typically required by server management applications in the prior art. System 50 and method 10 overcome this limitation as described above.
A server, such as server 54, of course has many more components then those illustrated in
Server 54 shown only illustrates some of the components associated with a service processor. These components typically include a memory 74, the service processor 76, input devices 78 and output devices 80. Memory 74 is typically ROM and RAM or similar types of memory and it may store specialized embedded operating system and other programming fixed for execution by service processor 78. Input devices 78 may typically include sensors, such as temperature sensors, power usage sensors, etc. Output devices 80 may include various controls, switches, etc., whereby the service processor 76 controls aspects of the server 54 (e.g., turns on fans, system processors, etc.). Some service processors also have human interaction input devices 78 and output devices 80. For example, the OA (Onboard Administrator) that is the cabinet level service processor in HP's blade chassis has a small display (about the size of a palm device) and a little keypad with navigation keys (up, down, left, right, enter, etc).
Service processor 76 stores the server schema as described above. Service processor 76 retrieves requested data and performs actions as instructed by server management applications 56.
Multiple servers 54′ are shown to illustrate a blade or partitioned server system. In addition, although application machine 52 and server 54 are depicted with various components, one skilled in the art will appreciate that these machines and the server can contain additional or different components. As noted above, server 54 only illustrates components associated with the service processor. In addition, although aspects of an implementation consistent with the above are described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices, including hard disks, floppy disks, or CD-ROM; a carrier wave from the Internet or other network; or other forms of RAM or ROM. The computer-readable media may include instructions for controlling a computer system, such as application machine 52, to perform a particular method, such as method 10.
It is also noted that server management applications may also run on the system processor of the server being managed. This used to be common for some server management applications on HP's partitioned servers. The management application would present a remote GUI to the administrator sitting at his workstation (via a web browser, for example). The server management application running on the system processor communicates with the service processor via either an internal, special communication path in the hardware, or out the system processor's network interface to the service processors network interface. The embodiments described herein may be implemented in such a configuration.
It is further noted that a server management application may obtain information not only from a service processor on a server, but also from the server's system processors. For example, if the management application wishes to display the amount of free external storage, the amount of main memory allocated by the operating system to each application or the percentage of system CPU time each application is consuming it obtains this information from a system processor, not the service processor. The embodiments described herein may also be used to advantage for these data accesses.
The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention as defined in the following claims, and their equivalents, in which all terms are to be understood in their broadest possible sense unless otherwise indicated.
This application claims the benefit of U.S. Provisional Application Ser. No. 61/064060, filed Feb. 14, 2008, titled “System And Method For Efficient Remote Data Access For Server Management” which is hereby incorporated by reference herein as if reproduced in full bellow.
Number | Name | Date | Kind |
---|---|---|---|
4493083 | Kinoshita | Jan 1985 | A |
5084816 | Boese et al. | Jan 1992 | A |
5261055 | Moran et al. | Nov 1993 | A |
5442771 | Filepp et al. | Aug 1995 | A |
5479637 | Lisimaque et al. | Dec 1995 | A |
5535357 | Moran et al. | Jul 1996 | A |
5579522 | Christeson et al. | Nov 1996 | A |
5596738 | Pope | Jan 1997 | A |
5598534 | Haas | Jan 1997 | A |
5608910 | Shimakura | Mar 1997 | A |
5622954 | Henrie, II et al. | Apr 1997 | A |
5628016 | Kukol | May 1997 | A |
5666293 | Metz et al. | Sep 1997 | A |
5752039 | Tanimura | May 1998 | A |
5778440 | Yiu et al. | Jul 1998 | A |
5790974 | Tognazzini | Aug 1998 | A |
5832000 | Lin et al. | Nov 1998 | A |
5832520 | Miller | Nov 1998 | A |
5878256 | Bealkowski et al. | Mar 1999 | A |
5896566 | Averbuch et al. | Apr 1999 | A |
5960445 | Tamori et al. | Sep 1999 | A |
5968182 | Chen et al. | Oct 1999 | A |
5974250 | Angelo et al. | Oct 1999 | A |
6009497 | Wells et al. | Dec 1999 | A |
6018747 | Burns et al. | Jan 2000 | A |
6038636 | Brown, III et al. | Mar 2000 | A |
6064814 | Capriles et al. | May 2000 | A |
6073206 | Piwonka et al. | Jun 2000 | A |
6073214 | Fawcett | Jun 2000 | A |
6088759 | Hasbun et al. | Jul 2000 | A |
6105063 | Hayes, Jr. | Aug 2000 | A |
6112024 | Almond et al. | Aug 2000 | A |
6112197 | Chatterjee et al. | Aug 2000 | A |
6126327 | Bi et al. | Oct 2000 | A |
6128695 | Estakhri et al. | Oct 2000 | A |
6138249 | Nolet | Oct 2000 | A |
6157559 | Yoo | Dec 2000 | A |
6163274 | Lindgren | Dec 2000 | A |
6198946 | Shin et al. | Mar 2001 | B1 |
6212659 | Zehavi | Apr 2001 | B1 |
6279153 | Bi et al. | Aug 2001 | B1 |
6311322 | Ikeda et al. | Oct 2001 | B1 |
6301710 | Ozawa et al. | Jan 2002 | B1 |
6381740 | Miller et al. | Apr 2002 | B1 |
6408434 | Fujiwara | Jun 2002 | B1 |
6209111 | Morie et al. | Aug 2002 | B1 |
6438585 | Capriles et al. | Aug 2002 | B2 |
6526574 | Jones | Feb 2003 | B1 |
6546552 | Peleg | Apr 2003 | B1 |
6591272 | Williams | Jul 2003 | B1 |
6615038 | Moles et al. | Sep 2003 | B1 |
6615404 | Garfunkel et al. | Sep 2003 | B1 |
6659345 | Sukeda et al. | Dec 2003 | B2 |
6668336 | Lasser | Dec 2003 | B2 |
6675382 | Foster | Jan 2004 | B1 |
6725056 | Moles et al. | Apr 2004 | B1 |
6725392 | Frey et al. | Apr 2004 | B1 |
6728950 | Davis et al. | Apr 2004 | B2 |
6832373 | O'Neill | Dec 2004 | B2 |
6836657 | Ji et al. | Dec 2004 | B2 |
6925467 | Gu et al. | Aug 2005 | B2 |
6928108 | Nelson et al. | Aug 2005 | B2 |
6968543 | Takahara et al. | Nov 2005 | B2 |
7031972 | Ren et al. | Apr 2006 | B2 |
7069545 | Wang et al. | Jun 2006 | B2 |
7073017 | Yamamoto | Jul 2006 | B2 |
7143115 | Jones et al. | Nov 2006 | B2 |
7242929 | Draluk et al. | Jul 2007 | B2 |
7275243 | Gibbons et al. | Sep 2007 | B2 |
7292846 | Mittal | Nov 2007 | B2 |
7313791 | Chen et al. | Dec 2007 | B1 |
7367027 | Chen et al. | Apr 2008 | B1 |
7369851 | Okkonen et al. | May 2008 | B2 |
7433936 | Zhu et al. | Oct 2008 | B2 |
7555750 | Lilley | Jun 2009 | B1 |
7818556 | Lima et al. | Oct 2010 | B2 |
20010029178 | Criss et al. | Oct 2001 | A1 |
20010047363 | Peng | Nov 2001 | A1 |
20010048728 | Peng | Dec 2001 | A1 |
20020065919 | Taylor et al. | May 2002 | A1 |
20020072359 | Moles et al. | Jun 2002 | A1 |
20020078209 | Peng | Jun 2002 | A1 |
20020116261 | Moskowitz et al. | Aug 2002 | A1 |
20020131404 | Mehta et al. | Sep 2002 | A1 |
20020152005 | Bagnordi | Oct 2002 | A1 |
20020156863 | Peng | Oct 2002 | A1 |
20030023964 | Rajaram et al. | Jan 2003 | A1 |
20030027563 | Herle et al. | Feb 2003 | A1 |
20030033590 | Leherbauer | Feb 2003 | A1 |
20030033599 | Rajaram et al. | Feb 2003 | A1 |
20030037075 | Hannigan et al. | Feb 2003 | A1 |
20030055919 | Fong et al. | Mar 2003 | A1 |
20030061384 | Nakatani | Mar 2003 | A1 |
20030074658 | Kim | Apr 2003 | A1 |
20030081786 | Nakano et al. | May 2003 | A1 |
20030084435 | Messer et al. | May 2003 | A1 |
20030093433 | Seaman et al. | May 2003 | A1 |
20030103484 | Oommen et al. | Jun 2003 | A1 |
20030121032 | Cho et al. | Jun 2003 | A1 |
20030154471 | Teachman et al. | Aug 2003 | A1 |
20030162533 | Moles et al. | Aug 2003 | A1 |
20030186689 | Herle et al. | Oct 2003 | A1 |
20040015952 | Lajoie et al. | Jan 2004 | A1 |
20040017831 | Shen et al. | Jan 2004 | A1 |
20040031031 | Rudelic | Feb 2004 | A1 |
20040034853 | Gibbons et al. | Feb 2004 | A1 |
20040054995 | Lee | Mar 2004 | A1 |
20040068724 | Gardner et al. | Apr 2004 | A1 |
20040095457 | Pokorny et al. | May 2004 | A1 |
20040111723 | Moles et al. | Jun 2004 | A1 |
20040117785 | Kincaid | Jun 2004 | A1 |
20040123282 | Rao | Jun 2004 | A1 |
20040133887 | Herle et al. | Jul 2004 | A1 |
20040152455 | Herle | Aug 2004 | A1 |
20040166839 | Okkonen et al. | Aug 2004 | A1 |
20040194081 | Qumei et al. | Sep 2004 | A1 |
20040215755 | O'Neill | Oct 2004 | A1 |
20040243992 | Gustofson et al. | Dec 2004 | A1 |
20040243993 | Okkonen et al. | Dec 2004 | A1 |
20040261072 | Herle | Dec 2004 | A1 |
20040261073 | Herle et al. | Dec 2004 | A1 |
20040267833 | Meller et al. | Dec 2004 | A1 |
20050005268 | Zilvay et al. | Jan 2005 | A1 |
20050039178 | Marolia et al. | Feb 2005 | A1 |
20050060699 | Kim et al. | Mar 2005 | A1 |
20050071385 | Rao | Mar 2005 | A1 |
20050096025 | Qumei et al. | May 2005 | A1 |
20050097544 | Kim | May 2005 | A1 |
20050102615 | Roman et al. | May 2005 | A1 |
20050114504 | Marolia et al. | May 2005 | A1 |
20050144609 | Rothman et al. | Jun 2005 | A1 |
20050144612 | Wang et al. | Jun 2005 | A1 |
20050160195 | Bruner et al. | Jul 2005 | A1 |
20050204068 | Zhu et al. | Sep 2005 | A1 |
20050216902 | Schafer | Sep 2005 | A1 |
20050216903 | Schafer | Sep 2005 | A1 |
20050227683 | Draluk et al. | Oct 2005 | A1 |
20060039561 | Ypya et al. | Feb 2006 | A1 |
20060101040 | Ren et al. | May 2006 | A1 |
20060174242 | Zhu et al. | Aug 2006 | A1 |
20080016122 | Liu et al. | Jan 2008 | A1 |
20090327343 | McCormack et al. | Dec 2009 | A1 |
Number | Date | Country | |
---|---|---|---|
20090210401 A1 | Aug 2009 | US |
Number | Date | Country | |
---|---|---|---|
61064060 | Feb 2008 | US |