1. Field of the Invention
The present invention is in the field of software-based design tools for creating integrated circuits having active components operating in the electromagnetic fields of high frequency transmission signals.
2. Description of Related Art
The current level of stereoscopic camera design makes use of two discrete camera systems loosely coupled together to capture pairs of images. Extensive post processing is often required to present the two images to the viewer in a manner that his or her brain can mesh into an integrated 3D image.
Such systems are assembled in a specially constructed frame that mounts the two discrete cameras for stereoscopic use, thus they are at least twice as heavy as a single 2D camera, often weighing over 150 pounds. This is clearly too heavy for easy mobile use.
Image sensors on such units are not precisely matched and operate inconsistently causing differences in performance (i.e. differing color balance levels). Unmatched optical components also tend to cause unpleasant physical side effects for users, such as nausea and headaches. This is a direct result of the brain trying to correct differences in a multitude of visual factors, such as image size and position, at 30-60 Hz/sec.
Current stereoscopic 3D systems lack modular functionality and the optics of a system cannot be changed in the field. This creates a cumbersome and unwieldy system that is generally unsuitable for many film settings. Such systems require long setup times between each shoot. Most systems are not robust and cannot easily retain alignment of sensors and lenses nor are they symmetrical and cannot be used with a Steadicam system.
When two cameras are used, no common data stream is generated. Thus, additional processing power must be used to merge the two data streams before any post processing occurs. Existing systems do not support complete 3D meta-data, specifically inter-axial distance and convergence angle.
Existing systems are limited in the types of lenses supported by the systems. For example, existing systems often do not support wide-angle lenses. The demand for high resolution systems cannot be adequately met with the current level of technology as most systems cannot support high performance sensors up to 4K.
Traditional cameras have one or more audio inputs that require all audio feeds to be sent to the camera prior to them being recorded with the video signal. The embedded audio then plays back with the video for synchronous playback. Synchronization between camera and discreet audio recorded separately from the camera is obtained by creating a synchronous visual/audio cue such as film slate clap, or by means of electronic time-code by which both camera and audio recorder share the same time-code numbers often via direct connection or via “jam synch” whereby both camera and audio recorder are preloaded with time code numbers by means of a short term connection to establish synch numbers for internal time code generators.
The present invention is a built from the ground up stereoscopic 3D imaging system that utilizes a single set of electronics to power and control two lens/sensor assemblies. The system is designed to be lightweight, modular and portable. Depending upon accessories and attachments it weighs approximately 15-20 lbs.
The system is designed to function in a manner similar to the human eye. Precisely matched sensor and lens pairs aligned to a high degree of precision will be used to eliminate unpleasant visual side-effects, such as headaches and nausea, users may experience. Precision matched lens/sensors will also greatly reduce or eliminate the need for post processing image corrections.
The present invention is designed to be modular in nature. The body of the camera will be covered with a proprietary mounting system based upon the STANAG 4694 NATO accessory rail. The camera head unit housing the lenses, sensors and servos comprise a detachable module from the camera body and can be changed quickly in the field for other head units and allows for a wide variety of sensor and lens types to be quickly attached to the camera system. Similarly, a variety of camera backs are provided that can be changed in the field. A standard back module provides the interconnections to external storage devices, monitors, power supplies etc. A data-back module also comprises a removable storage module that can be used for local storage. This provides two important capabilities. One the ability to be untethered and two the ability to record at very high-speed which is not possible over the standard interfaces used when tethered to an external storage device.
All opto-mechanical features of the camera system are controlled by a common mechanism in order to ensure that the optics track accurately to a high degree of precision. In addition, the camera system will compile extensive Meta Data including 3D parameters. Meta data will be used during editing to ensure accurate 3D images are supplied.
The Meduza 3D1 stereoscopic camera system includes a unique lens mount. Traditional lens mounts apply torque moments and other physical stresses to the camera body when a lens is removed or attached. The unique “Kenji mount” developed for this system places all of these stresses on the lens which is being held by the user. This helps ensure that the alignment of the 3D optical system is not compromised by the replacement of a lens. This is especially important in-the-field.
The Meduza 3D1 stereoscopic camera system supports a wide variety of sensors. The limiting factors are for image bandwidth and power consumption. The Head units are designed so that new sensors can be readily adopted without a system redesign. To do this the sensor image output streams are run through a sensor control module that converts whatever data format is presented by the image sensor into a common pixel data format. This format is modular and can handle a bandwidth of up to 100 Gbits/sec per sensor in the current generation. The modularity allows for low cost lower performance sensors to be used with low cost sensor control module FPGAs and retain full compatibility with the rest of the system. Adoption of a new sensor is accommodated by a new carrier PCB that adapts the sensor to the carrier modules (“eyes”) and by new firmware for the Sensor Control Module FPGA that is written to describe the conversion of the interface formats.
There is no fundamental restriction as to the number of sensors/lens assemblies that a Head unit supports. Normal 2D heads with a single sensor are as viable as heads with 5 sensor/lens units.
The lens/sensor assemblies, called camera “eyes”, are attached to and move along an inter-axial rail. Motors are employed within the system, one to adjust the inter-axial (inter-ocular) distance and the one motor each to control the convergence angle of each camera eye. Within the eye modules are further motors to control the lens functions focus, iris and when appropriate zoom.
Lens settings are simultaneously adjusted, including the focus, iris, zoom, inter-axial distance and convergence angle. In the case of convergence angle adjustment, it is helpful to think of one lens “mirroring” the other (i.e. one head rotates left, the other right). In order to optimize all these functions and maintain accuracy down to the single pixel level the 3D1 camera system uses image processing techniques to create error signals that are used to help the servos systems maintain correct registration of the desired settings.
A typical camera system requires a number of different operators, each with different responsibilities. There is usually a cinematographer, an assistant cinematographer, a focus puller, a stereographer, as well as a director, all of which need to control different camera functions. Currently these operations have to be done sequentially and can take considerable time as the adjustments from one operator may affect the adjustments of another and multiple corrections may be necessary.
The Meduza 3D1 camera system provides a dynamic control and registration system in order to allow multiple camera functions to be performed simultaneously, even if these commands come from different users. This mechanism is performed via one or more wireless remote controls. Conflicts will undoubtedly occur as multiple users have access to the same control function. A system to resolve conflicts is integrated into the wireless control system. The rules by which the conflict resolution system functions are arbitrary and user programmable.
The camera may also be attached to secondary systems, for example a remote storage device that must also be controlled by the multiple remote controllers. The camera must then act as a “clearing house” for the commands and control which actions require control of the secondary systems and pass-on appropriate commands. For example, in the case of the remote storage device, typical commands are Start, Stop, Record, Erase and Playback.
Having gone to great lengths to ensure the best attainable image quality from the image sensors, no further processing or compression is performed by the camera system. This again is done to retain as high an image quality as possible. This requires a storage system that can store the RAW image data from the image sensors at a wide range of data rates based on the sensor resolution and frame rate. The 3D1 camera system can be equipped with an attached storage system capable of storing up to 100 Gbits/s to FLASH memory. The current density of NAND FLASH devices allows for a recording time around 4 minutes of high definition 1,000 frames per second video.
The camera is equipped with a complete positioning system that allows the precise location and orientation of the camera to be known at all times. GPS is used for base position and universal time-code and is augmented with 3-axis gyroscope, 3-axis accelerometer, 3-axis magnetometer, a barometer and a thermometer. This information is stored as metadata along with video whenever recording takes place. As such camera systems are often leased equipment and the location information can be reported back to a leasing agent or other supervisor. This is accomplished either via internet access if possible or via GSM cellular telephone built-in to the camera. For very remote use, an interface is also provided to an external satellite phone system. As a security measure, the system can be set up to require regular check-ins to the supervisor system. If the camera does not check-in, the camera will shut down and prevent further recording, which effectively renders the camera inoperable. This is done using rolling-code security keys similar to common garage-door openers. On a regular time-interval the camera “checks-in” and receives a new key code. If no new key code is received, because the camera failed to “check-in” for any reason then the camera shuts down. It is also capable of receiving, over the same system, a new key that will re-enable full system functionality. This allows the lessor to control the use and operation of the camera system if so desired.
Embodiments in accordance with aspects of the present invention are described below in connection with the attached drawings in which:
As illustrated schematically in
In the illustrated embodiment, the enclosure 120 comprises a first enclosure shell 150 and a second enclosure shell 152. The two enclosure shells may be identical as shown. Accordingly, the first enclosure shell is illustrated in more detail in
The two lens assemblies 234, 236 are movable horizontally along the upper and lower guide rails 240, 242. The horizontal movement of the two lens assemblies is controlled by a double-threaded screw 250. The right half of the double-threaded screw is formed with a conventional right hand thread that engages a threaded recess (not shown) at the rear of the right lens assembly. The left half of the double-threaded screw is formed with a left hand thread that engages a threaded recess (not shown) at the rear of the left lens assembly. The double-threaded screw is driven by a gear 252 that is driven by a lens spacing motor (not shown). When the motor turns the gear in a first rotational direction, the double-threaded screw causes the right lens assembly to move towards the right and causes the left lens assembly to move towards the left, thus causing the two lens assemblies to move farther apart away from the center of the front of the lens mounting assembly 230. When the motor turns the gear in a second rotational direction opposite the first rotational direction, the right lens assembly moves toward the left and the left lens assembly moves toward the right, thus causing the two lens assemblies to move towards each other at the center of the lens mounting assembly. When initially mounted on the double-threaded screw, the two lens assemblies are accurately positioned by substantially equal distances from the center of the lens mounting assembly. Accordingly, regardless of the direction of movement caused by the rotation of the gear, the two lens assemblies will always be positioned by substantially the same distance from the center of the lens mounting assembly.
As further shown in
Each lens mounting assembly 234, 236 supports a removable lens 270. Each lens is mounted in the respective lens mounting assembly by a low-torque threaded mounting interface. Each lens is electronically controlled in a conventional fashion to vary the focal length and the opening of the aperture. In preferred embodiments, the lens in the right lens assembly and the lens in the left lens assembly are manufactured as pairs that include optics that are selected to match so that the images produced by the left lens assembly and the right lens assembly are precisely matched.
The enclosure 120 houses electronic circuitry that controls the convergence of the two lens assemblies. The convergence control electronics, represented by a block diagram in
In
In the illustrated embodiment, the right lens assembly 234 is controlled in a similar manner by corresponding right control circuitry. In particular, the right control circuitry includes a right summing circuit 350. In the illustrated embodiment, the right summing circuit also receives a target convergence angle from the block 322 and receives a feedback signal from a right convergence angle sensor 354. The right summing circuit generates a difference signal that is provided as an input to a right loop compensation circuit 330. The right loop compensation circuit is also optimized to ensure loop stability. The right loop compensation circuit generates an output signal that controls a right motor drive 362, which controls the operation of the convergence motor in the right lens assembly. The convergence angle of the right lens assembly is measured by the right convergence angle sensor, which generates the feedback signal to the right summing circuit, as discussed above.
The convergence circuitry in
Both lens assemblies are placed upon a mechanical system that will allow translation and rotation. The translation of the lens assemblies is linear and varies the distance between the optical centers of the two lens assemblies. This distance is referred to herein as the inter-axial distance (analogous to the inter-ocular distance between human eyes). The rotation is the toe-in of the two lens assemblies such that they converge upon a common point in space (ROI) in front of the camera. This facilitates alignment to the convergence point by providing a direct connection between the two lens assemblies. The mechanical information provided by this system will be used in conjunction with optical data to ensure optimum alignment.
By measuring the amount of position difference between the left and right eye images in several regions of interest, as shown in
In an exemplary embodiment of the method, a Sobel Edge Operator is first applied to each point in the selected regions of interest (ROI) in both the Right and Left eye images corresponding to an equivalent time period. The output of this operation produces edge intensity images for the respective ROIs. Next, the edge intensity images in the Right and Left eye ROIs are compared to determine which sets of edge images are correlated. An efficient and proven way of tracking features across multiple video frames has been described by Jiambo Shi and Carlo Tomasi in “Good Features to Track” and is illustrated in the attached “Appendix_Shi-Tomasi.”
When correlated sets of edge images are identified in the ROIs, the relative horizontal and vertical separation of these can be measured. As illustrated in
In this way, the objects in the convergence zone can be continually tracked by applying Sobel edge operators and motion tracking algorithms to consecutive video frame ROIs, and measuring relative position differences between correlated image edges.
In order to closely emulate the human eye, the rotation of lens assemblies must occur about the Nodal Points of the lens/sensor assemblies. One skilled in the art of optics will know that the Nodal Point of an image capture system is the point at which light rays converge in front of the image plane. If rotation does not occur about the Nodal Point, a multitude of optical disparities can occur. Such discrepancies will cause unpleasant side-effects in the viewer, such as nausea and head and eye pain.
The method by which the lens/sensor assemblies rotate about their respective Nodal Points is as follows. A combination of adjusting convergence angle and the inter-axial distance are utilized to trigonometrically achieve a Nodal Point rotation. Since the Inter-axial (inter-ocular) distance is known, the Nodal Point of the lens/sensor assemblies at any given setting can be determined using simple trigonometry.
The translation towards the subject is very small and can be compensated for by a slight adjustment in focus. Subjectively, this difference may be so low as to be unnoticeable by the viewer and may not be included in production systems.
To effect a convergence rotation about the nodal point, the servo controls of inter-axial, convergence rotation and forward translation need to be coordinated. The parallax adjustment method is applied to all these servo mechanisms to ensure correct and precise convergence about the nodal point. This is an extension to the basic parallax method in which the servo loop controllers take into account the trigonometry involved in creating the rotation about the nodal point. So rather than simply rotating the lenses about the Actual Rotation point the error signal is fed into a calculation that applies the Pythagorean Theorem to create the rotation about the nodal point.
As the convergence of the lenses is changed, the focus and iris settings of the lenses may need to be changed. It is an operator selected function to leave the focus and iris settings untouched when changing convergence. This allows full artistic freedom for the camera user. However, it is also desirable to have the focus and iris track with the convergence. The desired focus point is often also the desired focus point. Also the iris, which affects the depth of focus, can be selectively tracked with the focus. For example, if the iris is left untouched then the furthest point in focus in a scene will shift as the convergence changes. This may be undesirable. If the focus and/or the iris need to track with convergence, they also receive the error signal.
As various changes could be made in the above constructions without departing from the scope of the invention, it is intended that all the matter contained in the above description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
The present application is a continuation of U.S. patent application Ser. No. 13/439,834, filed on Apr. 4, 2012, which claims the benefit of priority under 35 USC §119(e) to U.S. Provisional Application No. 61/471,211, filed on Apr. 4, 2011; to U.S. Provisional Application No. 61/471,689, filed on Apr. 4, 2011; and to U.S. Provisional Application No. 61/472,185, filed on Apr. 5, 2011, which are incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
61471211 | Apr 2011 | US | |
61471689 | Apr 2011 | US | |
61472185 | Apr 2011 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 13439834 | Apr 2012 | US |
Child | 13680803 | US |