Signal processing device, image capturing device, network camera system and video system

Abstract

A reference image memory comprises seven reference bank memories each having a capacity which can store three reference macroblocks in the vertical direction. Of the seven reference bank memories, any three reference bank memories are adaptively allocated to motion estimation of full pixel precision, any other three reference bank memories are adaptively allocated to motion estimation of half pixel precision, and remaining one reference bank memory is adaptively allocated to spare transfer for image data of a reference macroblock used in motion estimation with respect to the next target macroblock. The reference bank memories are connected to a motion estimating circuit via a total of seven respective buses separately. The motion estimating circuit performs the motion estimation of full pixel precision, the motion estimation of half pixel precision, and the spare transfer for image data of a reference macroblock used in motion estimation with respect to the next target macroblock, in parallel.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a schematic whole configuration of a signal processing device according to a first example of the present invention.

FIG. 2 is a diagram for describing an operation of the motion estimating device of FIG. 1.

FIG. 3A is a diagram for describing parallel processing of motion estimation of full pixel precision and motion estimation of half pixel precision. FIG. 3B is a diagram for describing serial processing of motion estimation of full pixel precision and motion estimation of half pixel precision. FIG. 3C is a diagram for describing an operation in which only motion estimation of full pixel precision is performed.

FIGS. 4A and 4B are diagrams illustrating schematic whole configuration of image motion estimating devices which are signal processing devices according to a second example of the present invention. FIG. 4A is a diagram when different reference bank memories are used to perform motion estimation of full pixel precision and motion estimation of half pixel precision in parallel. FIG. 4B is a diagram when the same reference bank memory is used to perform motion estimation of full pixel precision and motion estimation of half pixel precision in series.

FIGS. 5A and 5B are diagrams illustrating schematic whole configuration of image motion estimating devices which are signal processing devices according to a third example of the present invention. FIG. 5A is a diagram illustrating allocation of banks in a reference image memory when a search range is set to be an initial range. FIG. SB is a diagram illustrating allocation of banks in a reference image memory when a search range is set to be an enlarged range.

FIG. 6 is a diagram for describing bidirectional prediction in which forward prediction and backward prediction are used in motion-compensated prediction of MPEG.

FIGS. 7A and 7B are diagrams illustrating schematic whole configuration of image motion estimating devices which are signal processing devices according to a fourth example of the present invention. FIG. 7A is a diagram illustrating allocation of banks in a reference image memory in a coding method employing only forward prediction. FIG. 7B is a diagram illustrating allocation of banks in a reference image memory in a coding method employing bidirectional prediction.

FIG. 8 is a diagram illustrating a schematic whole configuration of an image motion estimating device which is a signal processing devices according to a fifth example of the present invention.

FIGS. 9A and 9B are diagrams illustrating bank allocation of a reference image memory in the motion estimating device of FIG. 8. FIG. 9A is a diagram when a search range is set to be a normal range. FIG. 9B is a diagram when a search range is set to be an enlarged range.

FIG. 10 is a control flowchart of a motion estimation operation in the motion estimating device of FIG. 8.

FIG. 11 is a control flowchart illustrating a motion estimation operation of an image motion estimating device which is a signal processing device according to a sixth example of the present invention.

FIG. 12 is a diagram illustrating a schematic whole configuration of an image motion estimating device which is a signal processing device according to a seventh example of the present invention.

FIG. 13 is a control flowchart of a motion estimation operation in the motion estimating device of FIG. 12.

FIG. 14 is a control flowchart illustrating a motion estimation operation of an image motion estimating device which is a signal processing device according to an eighth example of the present invention.

FIG. 15 is a diagram illustrating a schematic whole configuration of an image motion estimating device which is a signal processing device according to a ninth example of the present invention.

FIG. 16 is a control flowchart of a motion estimation operation in the motion estimating device of FIG. 15.

FIGS. 17A to 17D are diagrams illustrating bank allocation of a reference image memory in the motion estimating device of FIG. 15. FIG. 17A is a diagram when a search range is set to be a normal range. FIG. 17B is a diagram when a search range is enlarged in the horizontal and vertical directions. FIG. 17C is a diagram when a search range is enlarged only in the horizontal direction. FIG. 17D is a diagram when a search range is enlarged only in the vertical direction.

FIG. 18 is a diagram illustrating a schematic whole configuration of an image motion estimating device which is a signal processing device according to a tenth example of the present invention.

FIG. 19 is a diagram illustrating a schematic whole configuration of a network camera including an image motion estimating device which is a signal processing device according to an eleventh example of the present invention.

FIG. 20 is a diagram illustrating a relationship between the number of channels to be coded and a request processing speed in the network camera of FIG. 19.

FIG. 21 is a diagram illustrating a schematic whole configuration of the motion estimating device of the eleventh example.

FIGS. 22A to 22D are diagrams illustrating bank allocation of a reference image memory in the motion estimating device of FIG. 19. FIG. 22A is a diagram when the number of channels is 1. FIG. 22B is a diagram when the number of channels is 2. FIG. 22C is a diagram when the number of channels is 3. FIG. 22D is a diagram when the number of channels is 4.

FIG. 23 is a diagram illustrating a schematic whole configuration of a network camera including an image motion estimating device which is a signal processing device according to a twelfth example of the present invention.

FIG. 24 is a state transition diagram illustrating state transition of control states in the network camera of FIG. 23, including a state in which a notification is issued to a requesting client.

FIG. 25 is a state transition diagram illustrating state transition of control states in the network camera of FIG. 23, including a state in which a notification is issued to another client.

FIG. 26 is a diagram illustrating a schematic whole configuration of an image capturing system including an image motion estimating device which is a signal processing device according to a thirteenth example of the present invention.

FIG. 27 is a diagram illustrating a sequence of inter-frame predictive coding.

FIG. 28 is a diagram illustrating the concept of a motion search of full pixel precision.

FIG. 29 is a diagram illustrating a schematic whole configuration of a conventional image motion estimating device.

FIG. 30 is a diagram for describing the conventional image motion estimating device of FIG. 29.

FIG. 31 is a diagram illustrating the concept of a motion search of half pixel precision.

FIGS. 32A and 32B are diagrams illustrating timing of a motion estimation process in the conventional image motion estimating device of FIG. 29. FIG. 29A is a diagram for describing an operation in which only motion estimation of full pixel precision is performed. FIG. 29B is a diagram for describing an operation in which motion estimation of full pixel precision and motion estimation of half pixel precision are performed in series.

Claims

1. A signal processing device for estimating a motion of a target macroblock image included in a target frame image by referencing a plurality of reference macroblock images included in a reference frame image, comprising: a target image storing section for storing the target macroblock image included in the target frame image;a reference image storing section for storing a plurality of reference macroblock images located in a first direction which is a direction in which the target macroblock image is processed and in a second direction perpendicular to the first direction around a reference macroblock image corresponding to the target macroblock image of the plurality of reference macroblock images included in the reference frame image, and having a plurality of reference bank sections which are physically separately divided areas each including a predetermined number of reference macroblock images located in the second direction; anda motion estimating section for performing motion estimation with respect to the target macroblock image in the target image storing section by referencing the plurality of reference macroblocks stored in the plurality of reference bank sections of the reference image storing section.

2. The signal processing device of claim 1, wherein the plurality of reference bank sections of the reference image storing section are adaptively allocated to at least: a full pixel precision reference bank section for storing a plurality of reference macroblock images for performing motion estimation of full pixel precision with respect to a target macroblock image;a spare transfer reference bank section for storing a plurality of reference macroblock images for the next cycle for performing motion estimation with respect to a target macroblock image in the next cycle of the target macroblock image; anda half pixel precision reference bank section for storing a plurality of reference macroblock images for performing motion estimation of half pixel precision with respect to a target macroblock image a predetermined number of cycles before the target macroblock image,wherein all of the reference bank sections included in the reference image storing section and the motion estimating section are connected via respective separate signal lines.

3. The signal processing device of claim 2, further comprising: a transfer control section for controlling transfer of the plurality of reference macroblock images stored in the plurality of reference bank sections of the reference image storing section,wherein the transfer control section performs a full/half pixel precision search parallel execution control to store the plurality of reference macroblock images for the next cycle into the spare transfer reference bank section while simultaneously transferring the plurality of reference macroblock images stored in the full pixel precision reference bank section and the plurality of reference macroblock images stored in the half pixel precision reference bank section to the motion estimating section.

4. The signal processing device of claim 3, wherein, instead of the full/half pixel precision search parallel execution control, the transfer control section performs a full/half pixel precision search serial execution control to store the plurality of reference macroblock images for the next cycle into the spare transfer reference bank section, and thereafter, transfer the plurality of reference macroblock images stored in the full pixel precision reference bank section as a plurality of reference macroblock images for half pixel precision to the motion estimating section, while transferring the plurality of reference macroblock images stored in the full pixel precision reference bank section to the motion estimating section.

5. The signal processing device of claim 3, further comprising: a bank configuration specifying section for specifying the number of reference bank sections used in the reference image storing section as a bank configuration,wherein the bank configuration specifying section receives information about required image performance, and changes the bank configuration of the reference image storing section, depending on the received required performance, andthe transfer control section switches the full/half pixel precision search parallel execution control and the full/half pixel precision search serial execution control, depending on the bank configuration of the reference image storing section changed by the bank configuration specifying section.

6. The signal processing device of claim 3, further comprising: a bank configuration specifying section for specifying the number of reference bank sections used in the reference image storing section as a bank configuration,wherein the bank configuration specifying section receives information about required image performance, and changes the bank configuration of the reference image storing section, depending on the received required performance, and when the received required performance is high, enlarges storage areas of the full pixel precision reference bank section, the spare transfer reference bank section, and the half pixel precision reference bank section.

7. The signal processing device of claim 5, wherein the information about required image performance is image quality of an image.

8. The signal processing device of claim 5, wherein the information about required image performance is a size of an image.

9. The signal processing device of claim 5, wherein the information about required image performance is a frame rate of an image.

10. The signal processing device of claim 3, wherein motion estimation of an image is performed with bidirectional prediction, the full pixel precision reference bank section, the spare transfer reference bank section, and the half pixel precision reference bank section of the reference image storing section are separated into forward prediction-dedicated sections and backward prediction-dedicated sections, andreference macroblock images are transferred to the motion estimating section from the forward prediction-dedicated full pixel precision reference bank section, spare transfer reference bank section and half pixel precision reference bank section, and the backward prediction-dedicated full pixel precision reference bank section, spare transfer reference bank section and half pixel precision reference bank section.

11. The signal processing device of claim 1, further comprising: a prediction precision detecting section for detecting prediction precision of motion estimation in a set range of an image by the motion estimating section; anda bank configuration specifying section for specifying the number of reference bank sections used in the reference image storing section as a bank configuration,wherein the bank configuration specifying section changes the bank configuration of the reference image storing section, depending on the prediction precision detected by the prediction precision detecting section, and when the prediction precision is low, increases the number of reference bank sections to be used.

12. The signal processing device of claim 11, wherein the set range of an image is one frame of the image.

13. The signal processing device of claim 12, wherein, depending on prediction precision in one frame of an image detected by the prediction precision detecting section, the bank configuration specifying section changes the number of reference bank sections in the reference image storing section to be used in motion estimation with respect to a frame next to the frame in which the prediction precision has been detected.

14. The signal processing device of claim 11, wherein the set range of an image is one block line in one frame of the image including a plurality of block lines located in the second direction, each block line including a plurality of macroblocks located in the first direction.

15. The signal processing device of claim 14, wherein, depending on prediction precision in one block line of an image detected by the prediction precision detecting section, the bank configuration specifying section changes the number of reference bank sections in the reference image storing section to be used in motion estimation with respect to a block line next to the block line in which the prediction precision has been detected.

16. The signal processing device of claim 1, wherein the motion estimating section calculates a motion vector for each target macroblock image, the signal processing device further comprises: a comparison section for comparing the motion vector for each target macroblock image in the set range of an image by the motion estimating section with a reference value; anda bank configuration specifying section for specifying the number of reference bank sections used in the reference image storing section as a bank configuration, andthe bank configuration specifying section changes the bank configuration of the reference image storing section, depending on a result of the comparison in the comparison section.

17. The signal processing device of claim 16, wherein the comparison section counts the number of motion vectors for target macroblock images exceeding the reference value, and the bank configuration specifying section, when the count value is larger than or equal to a set value, increases the number of reference bank sections used in the reference image storing section.

18. The signal processing device of claim 16, wherein the set range of an image is one frame of the image.

19. The signal processing device of claim 18, wherein the bank configuration specifying section changes the number of reference bank sections in the reference image storing section to be used in motion estimation with respect to a frame next to the frame including the target macroblock in which the motion vector has been detected, depending on a result of the comparison by the comparison section.

20. The signal processing device of claim 16, wherein the set range of an image is one block line in one frame of the image including a plurality of block lines located in the second direction, each block line including a plurality of macroblocks located in the first direction.

21. The signal processing device of claim 20, wherein, depending on a result of the comparison by the comparison section, the bank configuration specifying section changes the number of reference bank sections in the reference image storing section to be used in motion estimation with respect to a block line next to the block line including the target macroblock in which the motion vector has been detected.

22. The signal processing device of claim 16, wherein the comparison section compares the first direction component and the second direction component of a motion vector of each target macroblock image with a first direction reference value and a second direction reference value, respectively, and the bank configuration specifying section changes the bank configuration of the reference image storing section, depending on a result of the comparison of the first direction and the second direction in the comparison section.

23. The signal processing device of claim 1, further comprising: a bank configuration specifying section for specifying the number of reference bank sections used in the reference image storing section as a bank configuration.

24. An image capturing device comprising: the signal processing device of claim 23; andan image capturing section for capturing a moving image, and outputting the image signal to the signal processing device,wherein the image capturing section has a change detecting section for detecting a change in image position when the moving image is captured, anda portion of the bank configuration specifying section included in the signal processing device is also used as the change detecting section included in the image capturing section.

25. A network system comprising: a network camera having an image capturing section; anda plurality of image terminals for requesting distribution of a moving image captured by the image capturing section of the network camera, and each having an image display device for displaying the distributed moving image,wherein the network camera has the signal processing device of claim 23, andthe bank configuration specifying section included in the signal processing device changes the bank configuration of the reference image storing section, depending on the number of the image terminals which simultaneously issue the request for distribution of the moving image.

26. The network camera system of claim 25, wherein the signal processing device has the transfer control section of claim 3, and the transfer control section invariably performs the full/half pixel precision search parallel execution control using the bank configuration of the reference image storing section changed by the bank configuration specifying section.

27. The network camera system of claim 25, wherein the signal processing device has the transfer control section of claim 3, and the transfer control section switches and controls the full/half pixel precision search parallel execution control and the full/half pixel precision search serial execution control, depending on the bank configuration of the reference image storing section changed by the bank configuration specifying section.

28. The network camera system of claim 25, wherein, assuming that while the moving image is distributed to any of the plurality of image terminals, there is a request for distribution of a moving image from another image terminal, when the bank configuration specifying section cannot change the bank configuration of the reference image storing section in response to the request for distribution of the moving image from the other image terminal, the network camera notifies the other image terminal issuing the distribution request of that matter.

29. The network camera system of claim 28, wherein the other image terminal issuing the distribution request, when receiving from the network camera the notification indicating that the bank configuration of the reference image storing section cannot be changed, transmits to the network camera a request for distribution in which required performance of the moving image is lowered, and the bank configuration specifying section of the signal processing device of the network camera, when receiving from the other image terminal the request for distribution in which rthe equired performance of the moving image is lowered, tries to change the bank configuration of the reference image storing section, depending on the required performance.

30. The network camera system of claim 28, wherein, when the bank configuration specifying section cannot change the bank configuration of the reference image storing section, depending on the request for distribution of the moving image from the other image terminal, the network camera notifies the image terminal which is distributing the moving image of an inquiry of whether or not allocation of the reference bank section to the other image terminal issuing the distribution request with priority is permitted.

31. The network camera system of claim 30, wherein the image terminal which is distributing the moving image responds to the network camera, regarding the inquiry of whether or not allocation of the reference bank section to the other image terminal issuing the distribution request with priority is permitted, and the network camera performs the priority allocation when receiving, from image terminal which is distributing the moving image, a response which permits the allocation of the reference bank section to the other image terminal issuing the distribution request with priority.

32. A video system comprising: an image processing section including the signal processing device of claim 1 and for performing image processing;a sensor for outputting an image signal to the signal processing device of the image processing section; andan optical system for imaging light onto the sensor.

33. A video system comprising: an image processing section including the signal processing device of claim 1 and for performing image processing; andan A/D conversion section for receiving an image signal having an analog value, converting the image signal into a digital value, and outputting the digital value to the signal processing device of the image processing section.

34. A signal processing method for estimating a motion of a target macroblock image included in a target frame image by referencing a plurality of reference macroblock images included in a reference frame image, wherein the motion estimation includes first motion estimation for estimating a motion of the target macroblock image in units of a full pixel, and second motion estimation for estimating a motion of the target macroblock image in units of a half pixel,the signal processing method comprises: a first step of performing the first motion estimation; anda second step of performing the second motion estimation, and the first step and the second step are performed in parallel.

35. A signal processing device for estimating a motion of a target macroblock image included in a target frame image by referencing a plurality of reference macroblock images included in a reference frame image, wherein the motion estimation includes first motion estimation for estimating a motion of the target macroblock image in units of a full pixel, and second motion estimation for estimating a motion of the target macroblock image in units of a half pixel,the signal processing device comprises: a reference image storing section for storing a plurality of reference macroblock images located in a horizontal direction and a vertical direction around a reference macroblock image corresponding to the target macroblock image of the plurality of reference macroblock images included in the reference frame image; anda motion estimating section for performing the first motion estimation and the second motion estimation by referencing the plurality of reference macroblocks stored in the reference image storing section, andthe first motion estimation and the second motion estimation are performed in parallel in the motion estimating section.

36. The signal processing device of claim 35, wherein the reference image storing section is a built-in memory provided in the signal processing device.

37. The signal processing device of claim 36, wherein the built-in memory is an SRAM.

38. The signal processing device of claim 35, wherein, when a search range for motion estimation includes n reference image macroblocks in a processing direction, the first motion estimation with respect to an (m+n)-th macroblock and the second motion estimation with respect to an m-th macroblock are performed in parallel.