Method and apparatus for improved graphics/image processing using a processor and a memory

Abstract

A smart video memory (10) is provided that includes data storage (12 and 18), a serial access memory (19), and a processing core (14 and 16) for executing instructions stored in the data storage area (12 and 18). Externally, smart memory (10) is directly accessible as a standard video memory device.

Description

TECHNICAL FIELD OF THE INVENTION

This invention relates generally to processing, and more particularly to a method and apparatus for improved graphics/image processing.

BACKGROUND OF THE INVENTION

Advances in processor technology have allowed for significant increases in processing speed. However, in applications that are intensive in off-processor chip memory accesses, such as speech, signal, and image processing applications, the gain in raw processing speed is often lost because of relatively slow access times to the off-chip memories. This problem is further aggravated since memory technology has focused on increased device density. With increased device density, the maximum bandwidth of a system decreases because multiple bus architectures are defeated. For example, a graphics application requiring storage of a 480.times.240 sixteen-bit image has four times the bandwidth if eight 256K memory chips are used, rather than two of the more dense 1 megabyte chips.

Several strategies have been proposed to overcome these difficulties. One such solution involves using an application specific integrated circuit ("ASIC") to offload time-intensive tasks from the host CPU to increase overall system throughput. This alternative, however, requires one ASIC for each function to be offloaded, and requires dedicated memory for each ASIC. Consequently, a higher overall system cost is involved, and the system throughput is increased only for those tasks for which the ASIC was designed to handle, and not for tasks in general.

Another alternative involves the use of a co-processor. Such a solution allows for tasks to be offloaded from a host CPU and allows system memory to be shared by both the host CPU and the co-processor. With this system, however, total system bandwidth is decreased because of arbitration between the host processor and the co-processor. Furthermore, well-developed software is required to make full use and provide for "seamless integration" of the co-processor.

Another alternative involves the use of an application specific processor for offloading tasks from a host CPU. This alternative may require an expensive dedicated static RAM ("SRAM") for use by the application specific processor. Thus, this alternative involves increased system cost. Furthermore, the SRAM is not available even when the attached application specific processor is idle, and well-developed software is needed for "seamless integration".

As another solution to these difficulties, significant research and effort has been directed towards multiprocessing systems for increasing throughput as the limits of decreasing processor cycle times are approached. However, difficulties in designing multiprocessing systems, developing communication protocols for such systems, and designing software support routines have deterred proliferation of multiprocessing systems. Nonetheless, many applications in signal, speech and image processing are structured and lend themselves to partitioning and parallel processing.

These problems present themselves in many environments, and a particular area in which incrased processor to memory bandwidth is critical is graphics and imaging processing, since significant amounts of memory and associated data processing are required.

Thus, a need has arisen for a device and method allowing for execution of several self-contained graphics and imaging tasks in parallel within existing architectural frameworks. Furthermore, a need has arisen for improving processor to memory bandwidth in graphics and imaging applications without significant cost increases and without requiring customized, specific solutions for increasing system throughput.

SUMMARY OF THE INVENTION

In accordance with the present invention, an improved method and apparatus for graphics and imaging processing is provided. In particular, data is stored in a data storage of a smart video memory. Within the smart video memory, a processing core is operable to execute instructions stored in the storage area and to read and write data stored in that storage. External connections to the smart video memory are arranged such that the smart video memory appears as a standard video memory device to external devices.

An important technical advantage of the present invention is the fact that system throughput can be increased through use of the present invention, since it allows for parallel processing.

Another important technical advantage of the present invention is the fact that existing systems can be easily upgraded through use of the present invention because it appears externally as a standard video memory device. Because the present invention appears externally as a standard video memory device, parallel processing can be more easily implemented.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and the advantages thereof, reference is now made to the following description taken in conjunction with the accompanying drawings in which like reference numbers indicate like features and wherein:

FIG. 1a illustrates an external view of a device constructed according to the present invention;

FIG. 1b is a block diagram of an internal view of a device constructed according to the teachings of the present invention;

FIG. 2a is a block diagram of a typical uniprocessor system with standard memory devices;

FIG. 2b is a block diagram of a system including devices constructed according to the teachings of the present invention;

FIG. 3a is a block diagram illustrating bus traffic with a standard memory device;

FIG. 3b is a block diagram illustrating bus traffic in a system employing a device constructed according to the teachings of the present invention;

FIG. 4 is a block diagram of a memory map of a system including a device constructed according to the teachings of the present invention;

FIG. 5a is a block diagram illustrating processor control signals according to the present invention; and

FIG. 5b is a block diagram illustrating processor startup of a device constructed according to the teachings of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The problems discussed in the background of the invention are addressed with the present invention by integrating a processor into a large video random access memory ("VRAM") in a single integrated circuit. Throughout this description, a device constructed according to the teachings of the present invention will be referred to, from time to time, as a smart video memory or a smart VRAM (video random access memory). These terms are used because a device constructed according to the teachings of the present invention appears externally as a random access video memory chip and may have the pinout of a dynamic random access video memory chip.

FIGS. 1a and 1b present external and internal views of a smart VRAM in accordance with the present invention. As shown in FIG. 1a, externally, a device 10 constructed according to the teachings of the present invention appears as a standard video memory device with a memory-like pinout, such as that of a TMS48C121 multiport video RAM, made by Texas Instruments Incorporated. Device 10 may have a pinout arrangement that is the same or substantially the same as standard video memory pinouts, or Device 10 may have a pinout arrangement that includes a standard video memory pinout plus additional pins, as will be discussed below. In either case the pins are to be arranged such that the Device 10 is directly accessible as a standard video memory device by external devices.

Device 10 includes, by way of example, 40 pins which provide equivalent inputs and outputs of a typical VRAM. Device 10 may also include other pins in addition to those of a standard video memory device, for additional functionality, as will be discussed below. It should be understood that the pinout illustrated in FIG. 1a is for example only, and the pinout of Device 10 may be arranged to correspond to any standard video memory pinout, and as discussed, may include pins in addition to those of standard video memories. A host CPU, such as an Intel 386 microprocessor, may access the device 10 as it would access a standard video memory device.

In a particular embodiment, a smart VRAM constructed according to the teachings of the present invention may have a pinout as shown in FIG. 1a. The following table provides the pin, or lead, nomenclature for the pinout as shown in FIG. 1.

______________________________________Pin Nomenclature Standard Mode Smart Mode______________________________________A0-A8 Address Inputs Address inputsCAS Column Enable Column EnableDQ0-DQ7 DRAM Data In-Out/ DRAM Data In-Out/ Write Mask Bit Write Mask BitSE Serial Enable Serial EnableRAS Row enable Row enableSC Serial Data Clock Serial Data ClockSDQ0-SDQ7 Serial Data In-Out Serial Data In-OutTRG Transfer Register/ Transfer Register/ Q Output Enable Q Output EnableW Write Mask Select/ Write Mask Select/ Write Enable Write EnableDSF Special function select Special function selectOSF Split-register Split-register Activity Status Activity StatusVcc 5-V Supply (TYP) 5-V Supply (TyP)Vss Ground GroundM/RESET No care Mode/ResetTC No care Task completionIG No care Interrupt Generate______________________________________

As shown in the table above, for a particular embodiment of the present invention, the device has 40 pins identical to a "standard" 132K by 8-bit VRAM device, with the three no-care pins used for special functions of the present invention, to be discussed. In a particular embodiment, the internal bus is 32 bits wide. The on-board processor has a 30-ns instruction cycle time, and the chip operates on a 5-V power supply. The on-board processor can also be powered and grounded through additional pins, or the standard power and ground pins. It should be understood that the above specifications are for a particular embodiment, and other specifications may be used without departing from the intended scope of the present invention. For example, a wider bus than 32 bits, such as a 64 bit or 128 bit wide internal bus may be used.

As shown in the block diagram of FIG. 1b, internally device 10 appears like a processor with a large on-chip video memory. In the illustrated embodiment, program and data reside in partitioned data storage, although program and data may reside in the same memory space of the data storage without departing from the intended scope of the present invention. A wide internal bus, inherently available inside memory devices, connects the processor with the memory. As shown in FIG. 1b, the internal bus may be 32 bits wide. The program memory 12 is coupled to instruction decoder 14. Instruction decoder 14 decodes instructions residing within program memory 12 and outputs control signals to a logic unit 16. Logic unit 16 is also coupled to program memory 12 and to data memory 18. Data memory 18 is also coupled to serial access memory ("SAM") 19.

Instruction decoder 14 and logic unit 16 represent the processor core integrated into a memory according to the present invention. Processor cores to be integrated may range from fairly limited processor cores, such as those including only an integer unit, to those including both fixed point and floating point multipliers. For example, a RISC-based integer unit (such as SPARC or MIPS) may be included as the processor core in the present invention. Typically, such integer units would occupy less than 10 percent of the area of a 16-Mbit VRAM. Thus, RISC cores are attractive for integration because of their relatively small size compared to other processor cores. Processor cores using hardware multipliers in addition to the integer unit may also be included. For example, a digital signal processor core, such as those used in the Texas Instruments TMS320C10-C50 digital signal processors may be integrated into smart memories according to the present invention.

As discussed above, program memory 12 and data memory 18 may occupy the same memory space or may be separately partitioned. Furthermore, these memories are parallel access memories and may comprise dynamic random access memories. A memory controller 20 is also coupled to logic unit 16. Memory controller 20 is used to ensure that external accesses to the memory of device 10 have priority over internal accesses. Thus, memory controller 20 freezes logic unit 16 during external accesses and then releases the logic unit 16 to resume processor execution after completion of the external access. External devices will have the highest memory access priority. Thus, for example, if a host processor tries to access the on-chip memory of a device constructed according to the teachings of the present invention while it is processing, then the on-chip processor will be halted.

Serial access memory 19 provides for serial access to memory 18. In the embodiment shown in FIG. 1a, serial access memory 19 comprises eight SAM registers, with each of these registers coupled to one of the serial I/O leads, SDQ0-SDQ7. Each of these registers are, for example, 256 bits wide. Serial access to memory 18 is obtained via SAM 19. In a particular embodiment, each of the serial access memory registers is coupled to each of the columns of memory 18, such that a selected row of memory 18 will be read from or written to one of the SAM registers, and serially through that SAM register's serial I/O lead.

FIG. 2a is a block diagram of a prior art uniprocessor system with two standard memory devices and two standard VRAMs. As shown in FIG. 2a, the CPU 22 operates to store and retrieve data from the memory devices 24, 26, 28, and 30 through the use of an address and data bus. As an example, CPU 22 may comprise a TMS 320 made by Texas Instruments Incorporated, while memory devices 24 and 26 may comprise 132K.times.8 bit VRAMs, and devices 28 and 30 may comprise 32K.times.8 RAMs. VRAMs 24 and 26 are coupled to digital to analog converters 25 and 27, respectively, which are coupled to monitors 29 and 31, respectively. These D/A converters and monitors allow for video display of the data within VRAMs 24 and 26.

FIG. 2b illustrates a system including two smart VRAMs 32 and 34 as shown in FIGS. 1a and 1b. As can be seen from FIGS. 2a and 2b, the standard memory devices shown in FIG. 2a have been replaced by devices constructed according to the teachings of the present invention without the need for additional hardware. Smart VRAMs 32 and 34 appear as typical video memory devices, and thus are connected as if they were such memory devices. Thus, such smart video memories can convert an existing uniprocessor system, such as a personal computer, into a powerful multiprocessor system without major system redesign. As shown in FIG. 2b, the two smart video memory devices may be used to execute tasks in parallel with operations performed by the CPU.

Because of the design of the present invention, significant advantages are realized to systems including smart memories. One such advantage is system throughput. System throughput increases because of the simultaneous execution of several self-contained tasks. For example, in a personal computer environment, one smart video memory may be executing a graphics application downloaded by a host CPU and preparing that data for output to a graphics display, while another smart video memory may be executing another downloaded graphics application on an image stored within that smart VRAM. These tasks are performed through the control of a controlling CPU. With the tasks distributed among the smart video memories as described above, the only task for the central CPU would be to move the data to and from the smart video memories, without having to perform any processing on the data within those smart memories.

Another advantage of the present invention is improved CPU to memory bandwidth. Instead of fetching raw data from the memory, processing that data, and writing the processed results back to the memory, the host CPU now fetches only the processed data or information from the memory. Traffic on the system bus is therefore reduced. FIGS. 3a and 3b illustrate an example of reduced traffic due to use of a smart VRAM constructed according to the teachings of the present invention. In certain graphics applications, vectors must often be multiplied by various matrices. For example, a vector A may be multiplied by a matrix B to result in a vector C. As shown in FIG. 3a, in a conventional prior art system a host CPU fetches the elements of matrix B (raw data), multiplies them with the elements of vector A, and writes the products back to memory. With a system using a smart VRAM constructed according to the teachings of the present invention, the CPU moves the elements of vector A to the smart memory 36 containing matrix B, and the smart memory 36 then calculates C by multiplying A and B, thus freeing the host CPU from this vector multiplication. For a vector size of 100 and the above example, the traffic on the system bus is reduced by a factor of 100 when a smart VRAM constructed according to the teachings of the present invention is used.

Another advantage of the present invention is that it can serve two separate functions. In the default mode, devices according to the present invention serve as standard video memory devices. However, as will be discussed below, they can also be switched into a "smart" mode and made to execute specific tasks by downloading appropriate software. In contrast, coprocessor cards in current computers physically occupy a slot. When idle, their dedicated memory is not available to the host CPU.

The present invention also allows ease of upgrading functionality in existing systems. Designing memory subsystems and adding them to existing processor systems is easier than designing and adding processor subsystems. Today's memories are standardized components, in stark contrast to processors, and thus devices constructed according to the teachings of the present invention, because they are pin-compatible with memory chips, may be easily integrated into existing systems. Furthermore, since the address space of a processor is typically populated with several memory devices, each time a smart VRAM is added to a system, not only is additional memory added, but also additional processing capability. Thus, as the computational needs of a system grow, the system can be easily and quickly scaled up by adding smart VRAMs constructed according to the teachings of the present invention. FIG. 4 illustrates a typical processor and memory system and its inherently parallel structure. Thus, smart video memories designed according to the present invention provide for parallel processing with minimum design change, since they can be added to systems just as standard memory devices are.

Another advantage of the present invention is increased processing rates because of the locality of the memory and wide internal bus structure. Since all of the data needed for a program being executed on a smart VRAM are on-chip, the processing speed is faster than if the data were off-chip. Furthermore, wide internal busses are more feasible inside a memory chip than across chip boundaries because of size and electrical characteristic considerations.

In a preferred approach, the present invention has two modes, "smart" and "standard". In the "smart" mode, the processor core is enabled to process data in the data memory 18, if instructed to begin processing. In the "standard" mode, the processing core is prevented from processing. The default operating mode is the "standard" mode. In the "standard" mode, the device operates as a standard video memory device. As shown in FIG. 5a, the host processor 38 of the system dynamically switches the operating mode by writing to a mode pin of the smart video memory 10. The mode pin may comprise a no care pin on a typical video memory device such as pin 13 in FIG. 1a. By using a mode pin, the operating mode of the device is guaranteed, and software bugs cannot inadvertently switch the mode. In another alternative, the mode pin could be used as an extra address pin. Thus, when addressed in one particular range, the smart video memory would function in the standard mode. When addressed in another range, it would function in the smart mode.

In another embodiment, the mode of a smart video memory device could be switched without the use of a mode pin. With this approach, a fixed memory location is allocated as an operating mode switch. For example, a particular location within data memory 18 of FIG. 1b can be reserved as a mode switch. The host processor can switch operating modes by addressing and writing fixed patterns to this memory location across address and data busses as shown in FIG. 5a. The smart processor senses the pattern, or sequence of patterns, and switches modes accordingly. Other alternatives for selecting the mode of the device that do not require an extra pin like a mode pin include write-per-bit type functions or other design-for-test ("DFT") functions.

The mode pin can also be used as a reset pin. Because a smart VRAM according to the present invention includes a processor, a reset function for the processor is needed. This reset can be accomplished through the mode pin--every time the mode is switched to "smart," a reset takes place. As an alternative embodiment, an additional reset pin can be used. Furthermore, the reset function may be accomplished without the use of pin signals, but by writing patterns to particular memory locations within the smart VRAM across address and data busses as shown in FIG. 5a, as discussed in connection with the mode switch. The reset function could be associated with the same memory location as the mode switch, or a separate memory location. FIG. 5a illustrates the reset pin in combination with the mode pin.

Once in the "smart" mode, the host processor may start and stop the processor on the smart VRAM by writing fixed patterns to a fixed "go" location as shown in FIG. 5b. If not in the "smart" mode, then the processor on the smart VRAM cannot begin processing, even if the "go" instruction has been received. A host CPU 38 addresses the go memory location 40 of smart VRAM 10 and writes the fixed "go" pattern to that location. The processor on the smart video memory device will then begin to execute, provided the device is in the smart mode. After the smart video memory has completed its task, it can signal the processor of its task completion through the TC pin. The TC pin, as shown in the above table and FIG. 5a, may comprise a no care pin of a standard memory device such as pin 15 in FIG. 1a. This TC pin may be connected to the interrupt line of a host CPU. It should be understood that the TC pin need not be used to signal task completion. For example, a particular memory location could be reserved as a status memory location within the smart VRAM. The host processor could poll this status memory location for a particular code indicating that a task has been completed by the smart VRAM through use of the address and data busses as shown in FIG. 5a. As another approach, the smart VRAM could have a reserved memory location for an estimate of the length of time required for completion of its task. The host CPU could read this memory location and then request the processed data after the estimated length of time has elapsed.

As shown in the preceding table and FIG. 5a, an interrupt generate signal is also provided. This signal may be accomplished through a pin such as a no care pin or an additional pin, or, as discussed in connection with the mode switch, through a "soft" signal, by writing appropriate codes to particular memory locations across address and data busses shown in FIG. 5a. The interrupt generate signal causes the processor of the smart VRAM to interrupt its current task and process an interrupt task. Upon completion of the interrupt task, the initial task is resumed. The ID or address of the interrupt task can be passed by the host processor along with the interrupt generate signal.

As shown in FIG. 5a, a serial data lead of smart VRAM 10 is coupled to monitor 29 through D/A 25. With this setup, video data is displayed on monitor 29 from the smart VRAM 10. The video data is serially output through the SAM 19 across a serial data lead.

For additional processing abilities, smart VRAM 10 may include bus request and bus grant signals, for use in connection with a bus arbitrator 42 as shown in FIG. 5a. With this capability, smart VRAM 10 can directly take control of the address and parallel data system bus to perform, for example, I/O functions, to provide for more complete parallel processing.

The data read from and written to the parallel DRAM memory of a smart VRAM by a host CPU is performed conventionally. The host CPU writes input data to the DRAM of the smart VRAM and reads data conventionally. If an 8-bit wide external bus is used with a 16-bit host CPU, for example, the processor will have to make two reads and writes to accomplish 16-bit data transfers.

Although the present invention has been described in detail, it should be understood the various changes, substitutions and alterations can be made without departing from the spirit and scope of the invention as defined solely by the appended claims.

Claims

1. A smart video memory, comprising:

data storage including a random access memory and a serial access memory;

a processor to execute instructions stored in said data storage and to read and write data in said data storage, said data storage and processor integrated in a single integrated circuit;

external leads coupled to said data storage and processor and extending from said single integrated circuit for externally connecting an external device to said data storage and processor, said external leads arranged such that the smart video memory is directly accessible as a standard video memory device by said external device while the processor is prevented from executing the instructions; and

at least one of said external leads comprising a serial data lead coupled to said serial access memory for serial data access, wherein one of said external leads comprises a mode lead for switching said processor between a smart mode and standard mode.

2. A smart video memory, comprising:

data storage including a random access memory and a serial access memory;

a processor to execute instructions stored in said data storage and to read and write data in said data storage, said data storage and processor integrated in a single integrated circuit;

external leads coupled to said data storage and processor and extending from said single integrated circuit for externally connecting an external device to said data storage and processor, said external leads arranged such that the smart video memory is directly accessible as a standard video memory device by said external device while the processor is prevented from executing the instructions; and

at least one of said external leads comprising a serial data lead coupled to said serial access memory for serial data access, wherein said data storage includes a predetermined memory location for storing mode information for switching said processor between a smart mode and a standard mode.

3. A smart video memory, comprising:

data storage including a random access memory and a serial access memory;

a processor to execute instructions stored in said data storage and to read and write data in said data storage, said data storage and processor integrated in a single integrated circuit;

external leads coupled to said data storage and processor and extending from said single integrated circuit for externally connecting an external device to said data storage and processor, said external leads arranged such that the smart video memory is directly accessible as a standard video memory device by said external device while the processor is prevented from executing the instructions; and

at least one of said external leads comprising a serial data lead coupled to said serial access memory for serial data access, wherein said data storage includes a predetermined memory location for storing information for causing said processor to start and stop executing instructions.