A microprocessor is an electronic device capable of performing the processing and control functions for computing devices such as desktop computers, laptop computers, server computers, cell phones, laser printers, and so on. Conventionally, a microprocessor comprises a small plastic or ceramic package that contains and protects a small piece of semiconductor material that includes a complex integrated circuit. Leads connected to the integrated circuit are attached to pins that protrude from the package allowing the integrated circuit to be connected to other electronic devices and circuits. Microprocessors are usually plugged into or otherwise attached to a circuit board containing other electronic devices.
While a microprocessor integrated circuit may include only one computing unit, i.e., one processor, it is possible to include multiple processors in a microprocessor integrated circuit. The multiple processors, which are often referred to as “cores,” are included in the same piece of semiconductor material and connected to the microprocessor package pins. Having multiple cores increases the computing capability of the microprocessor. For example, a microprocessor with four cores can provide almost the same amount of computing capability as four single-core microprocessors.
There has been an increase in the use of multiple microprocessors and multiple-core microprocessors in traditional computing devices. Traditional computing devices are capable of running only one instance of an operating system. Even traditional computing devices that contain multiple-core microprocessors, multiple microprocessors, or multiple multiple-core microprocessors are only capable of running one instance of an operating system. Still, harnessing the increased computing capability that multiple-core microprocessors provide allows computing functions, which were previously executed by multiple computing devices, to be executed with fewer computing devices.
For example, a server is a computing device connected to a network that provides a service or set of services to other entities connected to the network. A server comprising 32 traditional computing devices, i.e., a 32 way server, may be comprised of eight microprocessors, each having four cores. Taking the concept one step further, if each individual core is eight times more capable than one of the 32 computing devices, the 32-way server's capabilities can be provided by the four core microprocessor. A clear advantage of such a four core server is that computing resource redundancy is more affordable than that provided by traditional servers. In addition, reducing the number of microprocessors reduces the cost of the server, the amount of energy used to power the server, and the amount of maintenance the server requires.
It is possible to use “partitions” to take greater advantage of the computing capabilities of multiple-core microprocessors A partition is an electrically isolatable set of electronic devices, e.g., processors, memory, etc., within a computing device that can run an independent instance of an operating system, i.e., a local operating system. A partitionable computing device is a computing device that can be divided into partitions and thus is able to run multiple local operating systems. A partitionable server is a server that is a partitionable computing device and thus able to run multiple local operating systems. A partition of a partitionable server may also be referred to as a “logical server.” That is, to other entities on a network a logical server appears to be a stand-alone server, even though it is not. It also possible to assemble a plurality of servers, logical or otherwise, into a “server cluster.” A server cluster is a plurality of servers that behave as a unit to provide a service or set of services.
The advantages of using multiple-core microprocessors is driving a trend toward “server consolidation.” Server consolidation is the process of replacing multiple servers, for example in a server cluster, with fewer servers, e.g., one server. A server that replaces multiple servers may contain computing capability that equals or exceeds the capabilities of the multiple servers. While reducing costs, energy, and maintenance, server consolidation has the effect of putting all of one's eggs into one basket. Server consolidation may increase the impact of a server failure. For example, if multiple applications, which used to run on multiple servers, are all run on the same server, and that server fails, the impact is likely to affect all of the applications. In the worst case, this means application downtime. To guard against such an impact, many high end servers, i.e., servers with a large amount of computing capability, apply a portion of their capabilities to reliability features.
One such reliability feature is “failover” capability. Failover is the ability of a first entity to pass the information the first entity contains onto a second similar entity preferably before the first entity completely fails. Techniques have been developed for traditional servers, i.e., servers based on traditional computing devices, to perform failover in a controlled and orderly fashion to ensure that no data is lost and no ongoing processes are interrupted during the transition from the failing server to the replacement server.
In order to create multiple-core microprocessor servers that are as robust and reliable as traditional servers, similar techniques that operate at the processor level are useful.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Methods and apparatus for correlating the identities of hardware devices, such as the processors and memory controllers of multiple-core microprocessors, between a local operating system and global management entity is disclosed.
When an operating system detects a faulting device, the operating system generates a fault message and transmits the fault message to a management entity. The management entity determines the identity of the faulting device based on the fault message, selects an appropriate replacement device, and changes a routing table to map the identity of the replacement device to the identity of the faulting device. The management entity then transmits the global identity of the replacement device to the operating system and the operating system correlates the local identity of the replacement device with the global identity of the replacement device.
A server is a computing device connected to a network that provides a service or set of services to other entities, e.g., computing devices, connected to the network. For example, a web page server provides a service that returns web pages in response to web page requests. Other exemplary servers are an email server that returns email messages for particular users, a video server that returns video clips from a video archive, etc. An exemplary server contains a microprocessor, a memory controller, and memory blocks controlled by the memory controller. The memory controller and the memory blocks controlled by the memory controller are often referred to as a unit, i.e., a memory unit. Servers may also contain additional microprocessors, memory controllers, memory blocks, and other electronic devices such as interrupt processors. Hence, servers containing only a microprocessor and memory unit should be construed as exemplary and not limiting.
As with many types of computing devices, the operation of a server is controlled by a software program called an operating system. Traditional computing devices are capable of running only one instance of an operating system. Hence a traditional server, i.e., a server based on a traditional computing device or traditional computing devices, executes the instructions contained in a copy of the operating system, i.e., an instance of the operating system. For example, a server comprising 32 traditional computing devices, i.e., a 32 way server, may be comprised of eight microprocessors, each having four cores and yet run one operating system. Reducing the number of microprocessors reduces the cost of the server, the amount of energy u to power the server, and the amount of maintenance the server requires.
Partitions make it possible to take even greater advantage of the computing capabilities of multiple-core microprocessors A partition is an electrically isolatable set of electronic devices, e.g., processors, memory, etc., within a computing device that can run an independent instance of an operating system, i.e., a local operating system. A partitionable computing device is a computing device that can be divided into partitions and thus is able to run multiple local operating systems. A partitionable server is a server that is a partitionable computing device and thus able to run multiple local operating systems. A partition of a partitionable server may also be referred to as a “logical server.” Hence, one partitionable server may contain multiple logical servers. A plurality of servers, logical or otherwise, may be assembled into a “server cluster” that behaves as a unit to provide a service or set of services.
Preferably, partitioning is dynamic. That is, partition units are assigned to, or removed from, partitions with little or no impact on the services the server provides. A server that is capable of being partitioned is a partitionable server. A server system, i.e., system, comprising partitionable servers is a partitionable system. A partitionable system provides flexibility in the number and configuration of partition units and electronic devices assigned to a partition and makes it easier and more cost-effective to support “server consolidation.”
Server consolidation is the process of replacing multiple servers with fewer servers or perhaps even only one server. An exemplary server that is the result of a server consolidation may contain computing capability that equals or exceeds the capabilities of the multiple servers that the server replaces. Server consolidation may increase the impact of a server failure. For example, imagine multiple applications that used to run on the multiple servers are all run on the one server. If the server fails, the impact is likely to affect all of the applications and even cause application downtime.
Traditional servers guard against such an impact by applying a portion of the servers' computing capability to reliability features such as “failover” capability. Techniques have been developed for traditional servers to perform failover in a controlled and orderly fashion to ensure that no data is lost and no ongoing processes are interrupted during the transition from the failing server to the replacement server. Since traditional servers connect to each other through a network and are thus not tightly tied together, work is broken into small pieces and shared across the servers, i.e., packetized. This makes it easy to replace a failing server since the failing server's work packets can be re-routed during failover. Notice that in order to implement failover, there must be more than one traditional server available. That is, a failing traditional server needs another similar traditional server able to accept data from the failing traditional server.
Since a partitionable server may contain multiple logical servers, which can communicate more easily than traditional servers tied together by a network, a partitionable server has the potential to provide reliability more easily and cost-effectively than a group of traditional servers. Processes for controlled and orderly failover that operate using the partitions in a partitionable server help realize the reliability partitionable servers can provide.
It is impractical to make partitionable servers more reliable by notifying each of the high-level software applications when a failover is implemented. To enable high-level software applications to respond to such a notification would require that the computer code for each application be modified to adapt to the failover. Even notifying applications would probably not be enough to provide failover without a mechanism to replace a portion of a running server. Instead, it is more practical and advantageous to involve only the lowest level software in the failover and allow the upper level software, e.g., applications, to behave as though no hardware change has happened.
An implementation of an orderly, low-level, partitionable server failover involves a global management entity and one or more local operating systems. Examples of a global management entity are a service processor (SP) and a baseboard management controller (BMC). An SP is a specialized microprocessor or microcontroller that manages electronic devices attached to a circuit board or motherboard, such as memory controllers and microprocessors. A BMC is also a specialized microcontroller embedded on a motherboard. In addition to managing electronic devices, a BMC monitors the input from sensors built into a computing system to report on and/or respond to parameters such as temperature, cooling fan speeds, power mode, operating system status, etc. Other electronic devices may fulfill the role of a global management entity. Hence, the use of an SP or BMC as a global management entity should be construed as exemplary and not limiting.
A local operating system is an instance of an operating system that runs on one partition. Partition units are assigned to a specific partition to ensure that the devices in the partition unit cannot be shared with devices in other partitions, ensuring that a failure will be isolated to a single partition. Such a partition unit may indicate which physical addresses are serviced by a given memory controller and, thereby, map the physical memory addresses to the memory controller and to the physical partition unit containing the memory controller. More than one partition unit may be used to boot and operate a partition. Unused or failing partition units may be electrically isolated. Electrically isolating partition units is similar to removing a server from a group of traditional servers with the advantage that partition units may be dynamically reassigned to different partitions.
In the foregoing discussion, unless otherwise noted, a partition unit comprises a single core and a single memory unit. However, partition units may comprise more than one core, memory unit, interrupt processor, and/or other devices that provide computing services and/or support. Hence, the use of partition units comprising a core and a memory controller should be construed as exemplary and not limiting. Managing, e.g., adding or replacing, the partition units in a partitionable server allows a failover to be performed in a controlled and orderly fashion to ensure that the partitionable server is as robust and reliable as traditional servers.
An exemplary computing device 100 for implementing a partitionable server capable of supporting partitions and partition unit addition and/or replacement is illustrated in block diagram form in
In
The replacement of partition units may be understood by comparing the block diagram shown in
The partition 200a illustrated in
Replacing a partition unit involves identifying the hardware devices that require replacement and the replacement hardware devices. It is common for a processor, such as processor A 202, to have an Advanced Programmable Interrupt Controller ID (APIC ID) identifying the processor; and for a memory unit, such as memory unit 204, to have a physical address identifying the memory unit. Within a partition's local operating system, such as partition 200a's local operating system, a processor's APIC ID is uniquely identifies the processor. Similarly, within a partition's local operating system, a memory unit's physical address uniquely identifies the memory unit.
A computing device, such as computing device 100, shown in
During a partition unit replacement, such as the partition unit replacement shown in
When a partition unit is replaced, each of the hardware devices in the partition unit is replaced. For example, as shown in
Preferably, replacing a partition unit is an “atomic” process. An atomic process is a process that is executed in a way that insures that an entity performing an operation that requires accessing the partition unit accesses the old partition unit, i.e., the partition unit to be replaced, or the “new” partition unit, i.e., the replacement partition unit, but not both during the same operation.
It is also possible for the SP to receive information from the OS and make the decision to replace a processor. In such an implementation, preferably, the OS provides a self-contained error record that allows the SP to analyze one record, instead of having to collect the information from the faulting processor and possibly other sources. The OS can generate such a self-contained error message that is understood by the SP in the global namespace.
Continuing in
Continuing in
Mapping a processor's global representation to the processor's local representation and checking the validity of the mapping by mapping the local presentation back to the global representation is particularly useful when more than one processor is failing and so more than one processor must be replaced. The validity of the local representation is then checked to make sure the local representation maps back into the global representation. Checking the validity of replacement processor with a faulting processor enables the SP to make a more suitable selection of a replacement processor to replace a faulting processor. For example, an “idle” processor, i.e., a processor that is active but performing no useful work, can be described in a global namespace such that an SP can access the idle processor without the OS's accessing the idle processor. The SP then indicates to an OS that the idle processor has been brought into the OS's partition for the purpose of replacing a faulting processor in the OS's partition. The replacement processor takes over the local identity of the processor being replaced. In the global context, the replacement processor is uniquely identified to the SP. That is, the SP can distinguish between all of the processors in all of the partitions. For example, the SP 102 can distinguish between processor A 108, processor B 112, processor C 116, and processor D 120 in computing device 100 shown in
A process similar to the process illustrated in
At block 504 the operating system transmits the fault message to the SP, such as the SP 102 shown in
Continuing in
In the processes illustrated in
While various embodiments have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the invention. For example, in the processes described above, information about the faulting hardware devices primarily originates in the local operating systems and the decision to replace a faulting hardware device originates with the local operating systems. It is possible for the faulting information to be transmitted to the global management entity and have the global management entity make the decision to replace the faulting hardware device. Also, while the various embodiments described above deal with physical processors, it is also possible to apply similar processes to virtual processors.
This application is a continuation of, and claims priority to, U.S. patent application Ser. No. 11/675,261, filed Feb. 15, 2007, which is incorporated herein by reference. This application is also related to U.S. patent application Ser. No. 11/675,272, filed on Feb. 15, 2007, U.S. patent application Ser. No. 11/675,290, filed on Feb. 15, 2007, and U.S. patent application Ser. No. 11/675,243, filed on Feb. 15, 2007, which are incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5345590 | Ault et al. | Sep 1994 | A |
5491788 | Cepulis et al. | Feb 1996 | A |
5689701 | Ault et al. | Nov 1997 | A |
5815731 | Doyle et al. | Sep 1998 | A |
5913058 | Bonola | Jun 1999 | A |
6199179 | Kauffman et al. | Mar 2001 | B1 |
6226734 | Kleinsorge et al. | May 2001 | B1 |
6247109 | Kleinsorge et al. | Jun 2001 | B1 |
6421679 | Chang et al. | Jul 2002 | B1 |
6502162 | Blumenau et al. | Dec 2002 | B2 |
6546415 | Park et al. | Apr 2003 | B1 |
6574748 | Andress et al. | Jun 2003 | B1 |
6598174 | Parks et al. | Jul 2003 | B1 |
6640278 | Nolan et al. | Oct 2003 | B1 |
6684230 | Momoh et al. | Jan 2004 | B1 |
6792564 | Ahrens et al. | Sep 2004 | B2 |
6895586 | Brasher et al. | May 2005 | B1 |
6907474 | Oshins et al. | Jun 2005 | B2 |
6986014 | Qureshi et al. | Jan 2006 | B2 |
7007192 | Yamazaki | Feb 2006 | B2 |
7039827 | Meyer et al. | May 2006 | B2 |
7051243 | Helgren et al. | May 2006 | B2 |
7065761 | Foster et al. | Jun 2006 | B2 |
7114064 | Ramesh et al. | Sep 2006 | B2 |
7171590 | Kadoi | Jan 2007 | B2 |
7275180 | Armstrong et al. | Sep 2007 | B2 |
7343515 | Gilbertson et al. | Mar 2008 | B1 |
7363394 | Shen et al. | Apr 2008 | B2 |
7404105 | Arai | Jul 2008 | B2 |
7426657 | Zorek et al. | Sep 2008 | B2 |
7480911 | Lee et al. | Jan 2009 | B2 |
7493515 | Armstrong et al. | Feb 2009 | B2 |
7509375 | Christian et al. | Mar 2009 | B2 |
7543182 | Branda et al. | Jun 2009 | B2 |
7562262 | Kondajeri et al. | Jul 2009 | B2 |
7934121 | Ritz et al. | Apr 2011 | B2 |
8086906 | Ritz | Dec 2011 | B2 |
20020016892 | Zalewski et al. | Feb 2002 | A1 |
20020053009 | Selkirk et al. | May 2002 | A1 |
20020144193 | Hicks et al. | Oct 2002 | A1 |
20020170039 | Kovacevic | Nov 2002 | A1 |
20030037278 | Olarig | Feb 2003 | A1 |
20030163744 | Yamazaki | Aug 2003 | A1 |
20030212884 | Lee et al. | Nov 2003 | A1 |
20040107383 | Bouchier et al. | Jun 2004 | A1 |
20040193950 | Gagne et al. | Sep 2004 | A1 |
20040221193 | Armstrong et al. | Nov 2004 | A1 |
20050283641 | Clark et al. | Dec 2005 | A1 |
20060010344 | Zorek, Sr. et al. | Jan 2006 | A1 |
20060095624 | Raj et al. | May 2006 | A1 |
20070061634 | Marisetty et al. | Mar 2007 | A1 |
20070067673 | Avizienis | Mar 2007 | A1 |
20080010527 | Lu | Jan 2008 | A1 |
20080028413 | Conklin et al. | Jan 2008 | A1 |
20080120486 | Ritz et al. | May 2008 | A1 |
20080120515 | Ritz et al. | May 2008 | A1 |
20080120518 | Ritz et al. | May 2008 | A1 |
20080201603 | Ritz | Aug 2008 | A1 |
20100262590 | Holt | Oct 2010 | A1 |
20110161729 | Ritz et al. | Jun 2011 | A1 |
Number | Date | Country |
---|---|---|
101542433 | Sep 2009 | CN |
101553791 | Oct 2009 | CN |
2010510592 | Apr 2010 | JP |
20090081405 | Jul 2009 | KR |
20090082242 | Jul 2009 | KR |
WO-2008064198 | May 2008 | WO |
WO-2008064213 | May 2008 | WO |
WO-2008073683 | Jun 2008 | WO |
Entry |
---|
“Final Office Action”, U.S. Appl. No. 11/675,261, (Oct. 5, 2010), 10 pages. |
“Final Office Action”, U.S. Appl. No. 11/675,261, (Aug. 4, 2011), 4 pages. |
“International Search Report”, PCT/US2007/085180, (Apr. 21, 2008), 2 pages. |
“International Search Report”, PCT/US2007/085184, (Mar. 28, 2008), 2 pages. |
“International Search Report”, PCT/US2007/085216, (Apr. 22, 2008), 2 pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,261, (Mar. 29, 2010), 9 pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,261, (Mar. 4, 2011), 10 pages. |
“Notice of Allowance”, U.S. Appl. No. 11/675,261, (Aug. 24, 2011), 8 pages. |
“Written Opinion of the International Searching Authority”, PCT/US2007/085180, (Apr. 21, 2008), 3 pages. |
“Written Opinion of the International Searching Authority”, PCT/US2007/085216, (Apr. 22, 2008), 4 pages. |
“Written Opinion of the International Searching Authority”, PCT/US2007/085184, (Mar. 28, 2008), 3 pages. |
Fletcher, M et al., “Realtime Reconfiguration Using an IEC 61499 Operating System”, Proceedings of the 15th International Parallel & Distributed Processing Symposium, San Francisco, Calif, Apr. 23-27, 2001., (Apr. 2001), 7 pages. |
Milojicic, D et al., “Global Memory Management for a Multi Computer System”, Proceedings of the 4th USENIX Windows Symposium, Seattle, Wash., Aug. 3-4, 2000., (Aug. 2000), 12 pages. |
Moore, R.W. et al., “Knowledge-Based Grids”, Proceedings of the 18th IEEE Symposium on Mass Storage Systems, Apr. 17-20, 2001, San Diego, Calif., 2000, (2000), pp. 29-39. |
“Advisory Action”, U.S. Appl. No. 11/675,290, (Dec. 7, 2010), 3 Pages. |
“Chinese First Office Action”, Application No. 200780042949.6, (Aug. 4, 2010), 8 Pages. |
“Chinese First Office Action”, Application No. 200780042958.5, (Sep. 13, 2010), 3 Pages. |
“Chinese First Office Action”, Application No. 200780042960.2, (Nov. 4, 2010), 10 Pages. |
“Chinese Notice of Grant”, Application No. 2000780042960.2, (Jul. 26, 2012), 4 pages. |
“Chinese Notice of Grant”, Application No. 200780042958.5, (Nov. 6, 2012), 4 pages. |
“Chinese Office Action”, Application No. CN200780042958.5, Decision of Rejection, (Jun. 24, 2011), 5 pages. |
“Chinese Second Office Action”, Application No. 200780042958.5, (Apr. 17, 2012), 3 pages. |
“Chinese Second Office Action”, Application No. 200780042960.2, (Nov. 30, 2011), 6 pages. |
“Chinese Third Office Action”, Application No. 200780042960.2, (Mar. 22, 2201), 6 pages. |
“European Search Report”, Application No. 07864653.6, (Nov. 17, 2011), 7 Pages. |
“Final Office Action”, U.S. Appl. No. 11/657,243, (Feb. 19, 2010), 12 Pages. |
“Final Office Action”, U.S. Appl. No. 11/675,243, (Jun. 1, 2012),11 pages. |
“Final Office Action”, U.S. Appl. No. 11/675,272, (Mar. 18, 2009), 9 Pages. |
“Final Office Action”, U.S. Appl. No. 11/675,290, (Jan. 27, 2010), 9 Pages. |
“Final Office Action”, U.S. Appl. No. 11/675,290, (Oct. 4, 2010), 11 pages. |
“Intel Hyper-Threading Technology Technical Users Guide”, Intel Corporation,(Jan. 2003), 44 pages. |
“Issue Notification”, U.S. Appl. No. 11/675,272, (Jan. 5, 2011), 1 Page. |
“Issue Notification”, U.S. Appl. No. 11/675,290, (Apr. 6, 2011), 1 Page. |
“Non Final Office Action”, U.S. Appl. No. 13/044,391, (Nov. 6, 2012), 11 Pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,243, (Aug. 7, 2009),12 Pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,243, (Dec. 8, 2011),15 Pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,272, (Aug. 7, 2009), 10 Pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,290, (May 24, 2010), 10 Pages. |
“Non-Final Office Action”, U.S. Appl. No. 11/675,290, (Jun. 24, 2009), 9 Pages. |
“Notice of Allowance”, U.S. Appl. No. 11/675,243, (Feb. 26, 2013), 8 Pages. |
“Notice of Allowance”, U.S. Appl. No. 11/675,272, (Sep. 21, 2010), 19 Pages. |
“Notice of Allowance”, U.S. Appl. No. 11/675,290, (Jan. 19, 2011), 7 Pages. |
“Notice of Allowance”, U.S. Appl. No. 11/675,243, (Sep. 27, 2012), 8 Pages. |
“Notice of Allowance”, Application No. 70080042960.2, (Jul. 25, 2012), 6 Pages. |
“Notice of Allowance”, Application No. CN200780042958.5, (Nov. 6, 2012), 6 Pages. |
“Reply to Chinese Office Examination along with Amended Action”, Application No. 200780042958.5, Request for Re-Examination along with Amended Claims in the response, (Sep. 23, 2011), 14 pages. |
“Reply to EP Communication”, Application No. EP0786465.6, (Jun. 15, 2012), 14 pages. |
“Reply to First Chinese Office Action”, Application No. 200780042958.5, (Nov. 18, 2010), 13 Pages. |
“Reply to First Chinese Office Action”, Application No. 200780042960.2, (Dec. 10, 2010), 7 Pages. |
“Reply to Second Chinese Office Action”, Application No. 200780042958.5, (Jun. 15, 2012), 12 Pages. |
“Reply to Second Chinese Office Action”, Application No. 200780042960.2, (Dec. 16, 2011), 2 pages. |
“Reply to Third Chinese Office Action”, Application No. 200780042960.2, (May 21, 2012), 8 Pages. |
Nguyen, Khang et al., “Detecting Multi-Core Processor Topology in an IA-32 Platform”, Intel Corporation,(2006), 17 pages. |
Shanley, Tom et al., “The Unabridged Pentium 4 IA32 Processor Genealogy”, Addison Wesley, Retrieved from Safari Books Online,(Jul. 26, 2004), 8 pages. |
“Final Office Action”, U.S. Appl. No. 13/044,391, (Apr. 11, 2013), 17 Pages. |
Number | Date | Country | |
---|---|---|---|
20120054538 A1 | Mar 2012 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 11675261 | Feb 2007 | US |
Child | 13289776 | US |