The present disclosure is generally related to image processing.
Advances in technology have resulted in smaller and more powerful computing devices. For example, there currently exist a variety of portable personal computing devices, including wireless computing devices, such as portable wireless telephones, personal digital assistants (PDAs), and paging devices that are small, lightweight, and easily carried by users. More specifically, portable wireless telephones, such as cellular telephones and Internet Protocol (IP) telephones, can communicate voice and data packets over wireless networks. Further, many such wireless telephones include other types of devices that are incorporated therein. For example, a wireless telephone can also include a digital still camera, a digital video camera, a digital recorder, and an audio file player.
A text-based augmented reality (AR) technique is described. The text-based AR technique can be used to retrieve information from text occurring in real world scenes and to show related content by embedding the related content into the real scene. For example, a portable device with a camera and a display screen can perform text-based AR to detect text occurring in a scene captured by the camera and to locate three-dimensional (3D) content associated with the text. The 3D content can be embedded with image data from the camera to appear as part of the scene when displayed, such as when displayed at the screen in an image preview mode. A user of the device may interact with the 3D content via an input device such as a touch screen or keyboard.
In a particular embodiment, a method includes receiving image data from an image capture device and detecting text within the image data. The method also includes, in response to detecting the text, generating augmented image data that includes at least one augmented reality feature associated with the text.
In another particular embodiment, an apparatus includes a text detector configured to detect text within image data received from an image capture device. The apparatus also includes a renderer configured to generate augmented image data. The augmented image data includes augmented reality data to render at least one augmented reality feature associated with the text.
Particular advantages provided by at least one of the disclosed embodiments include the ability to present the AR content in any scene based on the detected text in the scene, as compared to providing AR content in a limited number of scenes based on identifying pre-determined markers within the scene or identifying a scene based on natural images that are registered in a database.
Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.
In a particular embodiment, the image capture device 102 includes a lens 110 configured to direct incoming light representing an image 150 of a scene with text 152 to an image sensor 112. The image sensor 112 may be configured to generate video or image data 160 based on detected incoming light. The image capture device 102 may include one or more digital still cameras, one or more video cameras, or any combination thereof.
In a particular embodiment, the image processing device 104 is configured to detect text in the incoming video/image data 160 and generate augmented image data 170 for display, as described with respect to
In a particular embodiment, the display device 106 is configured to display the augmented image data 170. For example, the display device 106 may include an image preview screen or other visual display device. In a particular embodiment, the user input device 180 enables user control of the three-dimensional object displayed at the display device 106. For example, the user input device 180 may include one or more physical controls, such as one or more switches, buttons, joysticks, or keys. As other examples, the user input device 180 can include a touchscreen of the display device 106, a speech interface, an echolocator or gesture recognizer, another user input mechanism, or any combination thereof.
In a particular embodiment, at least a portion of the image processing device 104 may be implemented via dedicated circuitry. In other embodiments, at least a portion of the image processing device 104 may be implemented by execution of computer executable code that is executed by the image processing device 104. To illustrate, the memory 108 may include a non-transitory computer readable storage medium storing program instructions 142 that are executable by the image processing device 104. The program instructions 142 may include code for detecting text within image data received from an image capture device, such as text within the video/image data 160, and code for generating augmented image data. The augmented image data includes augmented reality data to render at least one augmented reality feature associated with the text, such as the augmented image data 170.
A method for text-based AR may be performed by the image processing device 104 of
A particular illustrative embodiment of a use case is a restaurant menu. When traveling in a foreign country, a traveler might see foreign words which the traveler may not be able to look up in a dictionary. Also, it may be difficult to understand a meaning of the foreign words even if the foreign words are found in the dictionary.
For example, “Jajangmyeon” is a popular Korean dish, derived from the Chinese dish “Zha jjang mian”. It consists of wheat noodles topped with a thick sauce made of Chunjang (a salty black soybean paste), diced meat and vegetables, and sometimes also seafood. Although this explanation is helpful, it is still difficult to know whether the dish would be satisfying to an individual's taste or not. However, it would be easier for an individual to understand Jajangmyeon if the individual can see an image of a prepared dish of Jajangmyeon.
If 3D information of Jajangmyeon were available, the individual could see its various shapes and then have a much better understanding of Jajangmyeon. Text-based 3D AR system can help to understand a foreign word from its 3D information.
In a particular embodiment, text-based 3D AR includes performing text region detection. A text region may be detected within a ROI (region of interest) around a center of an image by using binarization and projection profile analysis. For example, binarization and projection profile analysis may be performed by a text recognition detector, such as a text region detector 122 as described with respect to
The text detector 120 is configured to detect text within image data received from the image capture device 102. The text detector 120 may be configured to detect text of the video/image data 160 without examining the video/image data 160 to locate predetermined markers and without accessing a database of registered natural images. The text detector 120 is configured to generate verified text data 166 and text region data 167, as described with respect to
In a particular embodiment, the AR content generator 190 is configured to receive the verified text data 166 and to generate augmented reality (AR) data 192 that includes at least one augmented reality feature, such as the AR feature 154, to be combined with the video/image data 160 and displayed as embedded within the augmented image 151. For example, the AR content generator 190 may select one or more augmented reality features based on a meaning, translation, or other aspect of the verified text data 166, such as described with respect to a menu translation use case that is illustrated in
In a particular embodiment, the tracking/pose estimation module 130 includes a tracking component 131 and a pose estimation component 132. The tracking/pose estimation module 130 is configured to receive the text region data 167 and the video/image data 160. The tracking component 131 of the tracking/pose estimation module 130 may be configured to track a text region relative to at least one other salient feature in the image 150 during multiple frames of the video data while in the tracking mode. The pose estimation component 132 of the tracking/pose estimation module 130 may be configured to determine a pose of the image capture device 102. The tracking/pose estimation module 130 is configured to generate camera pose data 168 based at least in part on the pose of the image capture device 102 determined by the pose estimation component 132. The text region may be tracked in three dimensions and the AR data 192 may be positioned in the multiple frames according to a position of the tracked text region and the pose of the image capture device 102.
In a particular embodiment, the renderer 134 is configured to receive the AR data 192 from the AR content generator 190 and camera pose data 168 from the tracking/pose estimation module 130 and to generate the augmented image data 170. The augmented image data 170 may include augmented reality data to render at least one augmented reality feature associated with the text, such as the augmented reality feature 154 associated with the text 152 of the original image 150 and text 153 of the augmented image 151. The renderer 134 may also be responsive to user input data 182 received from the user input device 180 to control presentation of the AR data 192.
In a particular embodiment, at least a portion of one or more of the text detector 120, the AR content generator 190, the tracking/pose estimation module 130, and the renderer 134 may be implemented via dedicated circuitry. In other embodiments, one or more of the text detector 120, the AR content generator 190, the tracking/pose estimation module 130, and the renderer 134 may be implemented by execution of computer executable code that is executed by a processor 136 included in the image processing device 104. To illustrate, the memory 108 may include a non-transitory computer readable storage medium storing program instructions 142 that are executable by the processor 136. The program instructions 142 may include code for detecting text within image data received from an image capture device, such as text within the video/image data 160, and code for generating the augmented image data 170. The augmented image data 170 includes augmented reality data to render at least one augmented reality feature associated with the text.
During operation, the video/image data 160 may be received as frames of video data that include data representing the image 150. The image processing device 104 may provide the video/image data 160 to the text detector 120 in a text detection mode. The text 152 may be located and the verified text data 166 and the text region data 167 may be generated. The AR data 192 is embedded in the video/image data 160 by the renderer 134 based on the camera pose data 168, and the augmented image data 170 is provided to the display device 106.
In response to detecting the text 152 in a text detection mode, the image processing device 104 may enter a tracking mode. In the tracking mode, the text detector 120 may be bypassed and the text region may be tracked based on determining motion of points of interest between successive frames of the video/image data 160, as described with respect to
The text region detector 122 is configured to locate a text region within the video/image data 160. For example, the text region detector 122 may be configured to search a region of interest around a center of an image and may locate a text region using a binarization technique, as described with respect to
The text recognizer 125 is configured to receive the video/image data 160 and the initial text region data 162. The text recognizer 125 may be configured to adjust a text region identified in the initial text region data 162 to reduce a perspective distortion, such as described with respect to
The text recognizer 125 may be further configured to access a dictionary to verify the proposed text data. For example, the text recognizer 125 may access one or more dictionaries stored in the memory 108 of
In a particular embodiment, the text recognizer 125 may include a perspective distortion removal component 196, a binarization component 197, a character recognition component 198, and an error_correction component 199. The perspective distortion removal component 196 is configured to reduce a perspective distortion, such as described with respect to
Text-based AR that is enabled by the system 100 of
Another drawback to marker-based AR schemes is that markers must be embedded in every scene in which augmented reality content is to be displayed. As a result, marker schemes are inefficient. Further, because markers must be pre-defined and inserted into scenes, marker-based AR schemes are relatively inflexible.
Text-based AR also provides benefits as compared to natural features-based AR schemes. For example, a natural features-based AR scheme may require a database of natural features. A scale-invariant feature transform (SIFT) algorithm may be used to search each target scene to determine if one or more of the natural features in the database is in the scene. Once enough similar natural features in the database are detected in the target scene, relevant information may be overlaid relative to the target scene. However, because such a natural features-based scheme may be based on entire images and there may be many targets to detect, a very large database may be required.
In contrast to such marker-based AR schemes and natural features-based AR schemes, embodiments of the text-based AR scheme of the present disclosure do not require prior modification of any scene to insert markers and also do not require a large database of images for comparison. Instead, text is located within a scene and relevant information is retrieved based on the located text.
Typically, text within a scene embodies important information about the scene. For example, text appearing in a movie poster frequently includes the title of the movie and may also include a tagline, movie release date, names of actors, directors, producers, or other relevant information. In a text-based AR system, a database (e.g., a dictionary) storing a small amount of information could be used to identify information relevant to a movie poster (e.g. movie title, names of actors/actresses). In contrast, a natural features-based AR scheme may require a database corresponding to thousands of different movie posters. In addition, a text-based AR system can be applied to any type of target scene because the text-based AR system identifies relevant information based on text detected within the scene, as opposed to a marker-based AR scheme that is only effective with scenes that have been previously modified to include a marker. Text-based AR can therefore provide superior flexibility and efficiency as compared to marker-based schemes and can also provide more detailed target detection and reduced database requirements as compared to natural features-based schemes.
Given the orientation of text, a text region may be found.
satisfying:
y
i
≦ax
i
+b (i=1,2, . . . N)
y
i
≧cx
i
+d (i=1,2, . . . N)
where:
In a particular embodiment, this condition may intuitively indicate that the upper line 504 and the lower line 506 are determined in a manner that reduces (e.g., minimizes) the area between the lines 504, 506.
After vertical bounds of text have been detected (e.g., lines that at least partially distinguish upper and lower bounds of the text), horizontal bounds (e.g., lines that at least partially distinguish left and right bounds of the text) may also be detected.
The left line 608 may be described by a third equation y=ex+f, and the right line 610 may be described by a fourth equation y=gx+h. Since there may be a relatively small number of pixels on left and right sides of the bounding box, slopes of the left line 608 and the right line 610 may be fixed. For example, as shown in
The bounding box or bounding region may correspond to a distorted boundary region that at least partially corresponds to a perspective distortion of a regular bounding region. For example, the regular bounding region may be a rectangle that encloses text and that is distorted due to camera pose to result in the distorted boundary region illustrated in
The bounding box or bounding region described with reference to
For the recognition of rectified characters, one or more optical character recognition (OCR) techniques may be applied. Because conventional OCR methods may be designed for use with scanned images instead of camera images, such conventional methods may not sufficiently handle appearance distortion in images captured by a user-operated camera (as opposed to a flat scanner). Training samples for camera-based OCR may be generated by combining several distortion models to handle appearance distortion effects, such as may be used by the text recognizer 125 of
In a particular embodiment, text-based 3D AR includes performing a dictionary lookup. OCR results may be erroneous and may be corrected by using dictionaries. For example, a general dictionary can be used. However, use of context information can assist in selection of a suitable dictionary that may be smaller than a general dictionary for faster lookup and more appropriate results. For example, using information that a user is in a Chinese restaurant in Korea enables selection of a dictionary that may consist of about 100 words.
In a particular embodiment, an OCR engine (e.g., the text recognizer 125 of
For example, the first character is “” in the binarized result and several candidates (e.g., ‘
’, ‘
’, ‘
’, ‘
’, ‘
’) are returned according to their confidence (illustrated as ranked according to a vertical position within the group 910, from a highest confidence value at top to a lowest confidence value at bottom).
A lookup operation at a dictionary 916 may be performed. In the example of ”, “
”, “
”, . . . “
”). A lookup process may be performed to find a corresponding word in the dictionary 916 for one or more of the candidate words. For example, when multiple candidate words may be found in the dictionary 916, the verified candidate word 918 may be determined according to a confidence value (e.g., the candidate word that has a highest confidence value of those candidate words that are found in the dictionary).
In a particular embodiment, text-based 3D AR includes performing tracking and pose estimation. For example, in a preview mode of a portable electronic device (e.g., the system 100 of
A tracking method can include extracting interest points and computing motions of the interest points between consecutive images. By analyzing the computed motions, a geometric relation between real plane (e.g., a menu plate in the real world) and captured images may be estimated. A 3D pose of the camera can be estimated from the estimated geometry.
In a particular embodiment, text tracking in text-based 3D AR differs from conventional techniques because (a) the text may be tracked in text-based 3D AR based on corner points, which provides robust object tracking, (b) salient features in the same plane may also be used in text-based 3D AR (e.g., not only salient features in a text box but also salient features in surrounding regions, such as the second set of representative interest points 1004), and (c) salient features are updated so that unreliable ones are discarded and new salient features are added. Hence, text tracking in text-based 3D AR, such as performed at the tracking/pose estimation module 130 of
A 3D AR system may operate on real-time video frames. In real-time video, an implementation that performs text detection in every frame may produce unreliable results such as flickering artifacts. Reliability and performance may be improved by tracking detected text. Operation of a tracking module, such as the tracking/pose estimation module 130 of
During initialization, the tracking module may be started with some information from a detection module, such as the text detector 120 of
The first image 1102 may be stored as a reference frame to enable tracking of the camera pose when an image processing system enters a tracking mode, as described with reference to
In particular circumstances, distortions may be introduced in the second image 1202 that were not present in the first image 1102. For example, the change in the camera pose may introduce distortions. In addition, points of interest detected in the second image 1202 may not correspond to points of interest detected in the first image 1102, such as points 1107-1207 and the points 1109-1209. Statistical techniques (such as random sample consensus) may be used to identify one or more flow lines that are outliers relative to the remaining flow lines. For example, the flow line 1217 illustrated in
Salient features that have lost their correspondences, such as the points of interest 1107 and 1109, may therefore be recovered using a windows-matching approach. As a result, text region tracking without use of predefined markers may be provided that includes an initial estimation of displacements of points of interest (e.g., motion vectors) and window-matching to recover outliers. Frame-by-frame tracking may continue until tracking fails, such as when a number of tracked salient features maintaining their correspondence falls below a threshold due to a scene change, zoom, illumination change, or other factors. Because text may include fewer points of interests (e.g., fewer corners or other distinct features) than pre-defined or natural markers, recovery of outliers may improve tracking and enhance operation of a text-based AR system.
The camera pose 1404 can be represented by a rigid body transformation composed of 3×3 rotation matrix R and 3×1 translation matrix T. Using (i) the internal parameters of camera and (ii) the homography between the text bounding box in the keyframe and a bonding box in the current frame, the pose can be estimated via following equations:
R
1
=H
1
′/∥H
1′∥
R
2
=H
2
′/λH
2′∥
R
3
=R
1
xR
2
T=2H3/′(∥H1′∥+∥H2′∥)
Where each number 1, 2, 3 denotes the 1, 2, 3 column vector of target matrix, respectively, and H′ denotes the homography normalized by internal camera parameters. After estimating the camera pose 1404, 3D content may be embedded into the image so that the 3D content appears as a natural part of the scene.
Accuracy of tracking of the camera pose may be improved by having a sufficient number of points of interest and/or accurate optical flow results to process. When the number of points of interest that are available to process falls below a threshold number (e.g., as a result of too few points of interest being detected), additional points of interest may be identified.
A number of points of interest (indicated as boxes) of the text character 1504 are highlighted in
An area 1512 around the second point of interest 1508 is enlarged to show details of the technique for identifying additional points of interest. The second point of interest 1508 may be identified as an intersection of two lines. For example, a set of pixels near the second point of interest 1508 may be checked to identify the two lines. A pixel value of a target or corner pixel p may be determined. To illustrate, the pixel value maybe a pixel intensity values or grayscale values. A threshold value, t, may be used to identify the lines from the target pixel. For example, edges of the lines may be differentiated by inspecting pixels in a ring 1514 around the corner p (the second point of interest 1508) to identify changing points between pixels that are darker than I(p)−t and pixels that are brighter than I(p)+t along the ring 1514, where I(p) denotes a intensity value of the position p. Changing points 1516 and 1520 may be identified where the edges that form the corner (p) 1508 intersect the ring 1514. A first line or position vector (a) 1518 may be identified as originating at the corner (p) 1508 and extending through the first changing point 1516. A second line or position vector (b) 1522 may be identified as originating at the corner (p) 1508 and extending through the second changing point 1520.
Weak corners (e.g., corners formed by lines intersecting to form approximately a 180 degree angle) may be eliminated. For example, by computing the inner product of the two lines, using an equation:
where a, b and pεR2 refer to inhomogeneous position vectors. Corners may be eliminated when v is lower than a threshold value. For example, a corner formed by two position vectors a, b may be eliminated as a tracking point when the angle between two vectors is about 180 degrees.
In a particular embodiment, homography of an image, H, is computed using only corners. For example, using:
x′=Hx
where x is a homogeneous position vectorεR3 in a key-frame (such as the key frame 1302 of
In another particular embodiment, the homography of the image, H, is computed using corners and other features, such as lines. For example, H may be computed using:
x′=Hx
l
T
=l′
T
H
Where l is a line feature in a key-frame, and l′ is its corresponding line feature in a current frame.
A particular technique may use template matching via hybrid features. For example, window-based correlation methods (normalized cross-correlation (NCC), sum of squared differences (SSD), sum of absolute differences (SAD), etc.) may be used as cost functions, using:
Cost=−COR(x,x′)
The cost function may indicate similarity between a block (in a key-frame) around x and a block (in a current frame) around x′.
However, accuracy may be improved by using a cost function that includes geometric information of additional salient features such as the line (a) 1518 and the line (b) 1522 identified in
Cost=α(d(l1,HTl1′)+d(l2,HTl2′))−β·COR(x,x′)
In some embodiments, additional salient features (i.e., non-corner features, such as lines) may be used for text tracking when few corners are available for tracking, such as when a number of detected corners in a key frame is less than a threshold number of corners. In other embodiments, the additional salient features may always be used. In some implementations the additional salient features may be lines, while in other implementations the additional salient features may include circles, contours, one or more other features, or any combination thereof.
Because the text, the 3D position of the text, and the camera pose information are known or estimated, content can be provided to users in a realistic manner. The content can be 3D objects that can be placed naturally. For example,
Image data may be received from an image capture device, at 1702. For example, the image capture device may include a video camera of a portable electronic device. To illustrate, video/image data 160 is received at the image processing device 104 from the image capture device 102 of
Text may be detected within the image data, at 1704. The text may be detected without examining the image data to locate predetermined markers and without accessing a database of registered natural images. Detecting the text may include estimating an orientation of a text region according to a projection profile analysis, such as described with respect to
Detecting the text may include adjusting a text region to reduce a perspective distortion, such as described with respect to
Detecting the text may include generating proposed text data via optical character recognition and accessing a dictionary to verify the proposed text data. The proposed text data may include multiple text candidates and confidence data associated with the multiple text candidates. A text candidate corresponding to an entry of the dictionary may be selected as verified text according to a confidence value associated with the text candidate, such as described with respect to
In response to detecting the text, augmented image data may be generated that includes at least one augmented reality feature associated with the text, at 1706. The at least one augmented reality feature may be incorporated within the image data, such as the augmented reality features 1606 and 1608 of
In a particular embodiment, the image data may correspond to a frame of video data that includes the image data and in response to detecting the text, a transition may be performed from a text detection mode to a tracking mode. A text region may be tracked in the tracking mode relative to at least one other salient feature of the video data during multiple frames of the video data, such as described with reference to
Image data may be received from an image capture device, at 1802. For example, the image capture device may include a video camera of a portable electronic device. To illustrate, video/image data 160 is received at the image processing device 104 from the image capture device 102 of
The image may include text. At least a portion of the image data may be processed to locate corner features of the text, at 1804. For example, the method 1800 may perform a corner identification method, such as is described with reference to
In response to a count of the located corner features not satisfying a threshold, a first region of the image data may be processed, at 1806. The first region of the image data that is processed may include a first corner feature to locate additional salient features of the text. For example, the first region may be centered on the first corner feature and the first region may be processed by applying a filter to locate at least one of an edge and a contour within the first region, such as described with reference to the region 1512 of
Image data may be received from an image capture device, at 1902. For example, the image capture device may include a video camera of a portable electronic device. To illustrate, video/image data 160 is received at the image processing device 104 from the image capture device 102 of
The image data may include text. A set of salient features of the text may be identified in a first frame of the image data, at 1904. For example, the set of salient features may include a first feature set and a second feature. Using
A mapping that corresponds to a displacement of the first feature set in a current frame of the image data as compared to the first feature set in the first frame may be identified, at 1906. To illustrate, the first feature set may be tracked using a tracking method, such as described with reference to
In response to determining the mapping does not correspond to a displacement of the second feature in the current frame as compared to the second feature in the first frame, a region around a predicted location of the second feature in the current frame may be processed according to the mapping to determine whether the second feature is located within the region, at 1908. For example, the point of interest 1107 of
Image data may be received from an image capture device, at 2002. For example, the image capture device may include a video camera of a portable electronic device. To illustrate, video/image data 160 is received at the image processing device 104 from the image capture device 102 of
The image data may include text. A distorted bounding region enclosing at least a portion the text may be identified, at 2004. The distorted bounding region may at least partially correspond to a perspective distortion of a regular bounding region enclosing the portion of the text. For example, the bounding region may be identified using a method as described with reference to
A pose of the image capture device may be determined based on the distorted bounding region and a focal length of the image capture device, at 2006. Augmented image data including at least one augmented reality feature to be displayed at a display device may be generated, at 2008. The at least one augmented reality feature may be positioned within the augmented image data according to the pose of the image capture device, such as described with reference to
An input image 2104 is received from a camera module 2102. A determination is made whether a current processing mode is a detection mode, at 2106. In response to the current processing mode being the detection mode, text region detection is performed, at 2108, to determine a coarse text region 2110 of the input image 2104. For example, the text region detection may include binarization and projection profile analysis as described with respect to
Text recognition is performed, at 2112. For example, the text recognition can include optical character recognition (OCR) of perspective-rectified text, as described with respect to
A dictionary lookup is performed, at 2116. For example, the dictionary lookup may be performed as described with respect to
In response to a lookup success, tracking is initialized, at 2118. AR content, such as translated text, 3D objects, pictures, or other content may be selected associated with the detected text. The current processing mode may transition from the detection mode (e.g., to a tracking mode).
A camera pose estimation is performed, at 2120. For example, the camera pose may be determined by tracking in-plane points of interest and text corners as well as out-of-plane points of interest, as described with respect to
When the current processing mode is not the detection mode when a subsequent image is received, at 2106, interest point tracking 2128 is performed. For example, the text region and other interest points may be tracked and motion data for the tracked interest points may be generated. A determination may be made whether the target text region has been lost, at 2130. For example, the text region may be lost when the text region exits the scene or is substantially occluded by one or more other objects. The text region may be lost when a number of tracking points maintaining correspondence between a key frame and a current frame is less than a threshold. For example, hybrid tracking may be performed as described with respect to
A camera module 2102 receives an input image and a determination is made whether a current processing mode is a detection mode, at 2106. In response to the current processing mode being the detection mode, text region detection is performed, at 2108, to determine a coarse text region of the input image. For example, the text region detection may include binarization and projection profile analysis as described with respect to
Text recognition is performed, at 2109. For example, the text recognition 2109 can include optical character recognition (OCR) of perspective-rectified text, as described with respect to
A camera pose estimation is performed, at 2120. For example, the camera pose may be determined by tracking in-plane points of interest and text corners as well as out-of-plane points of interest, as described with respect to
When the current processing mode is not the detection mode when a subsequent image is received, at 2106, text tracking 2129 is performed. Processing continues with camera pose estimation, at 2120.
A camera module 2102 receives an input image and text region detection is performed, at 2108. As a result of text region detection at 2108, text recognition is performed, at 2109. For example, the text recognition 2109 can include optical character recognition (OCR) of perspective-rectified text, as described with respect to
Subsequent to the text recognition, a camera pose estimation is performed, at 2120. For example, the camera pose may be determined by tracking in-plane points of interest and text corners as well as out-of-plane points of interest, as described with respect to
A camera module 2102 receives an input image and a determination is made whether a current processing mode is a detection mode, at 2106. In response to the current processing mode being the detection mode, text region detection is performed, at 2108, to determine a coarse text region of the input image. As a result of text region detection 2108, text recognition is performed, at 2109. For example, the text recognition 2109 can include optical character recognition (OCR) of perspective-rectified text, as described with respect to
Subsequent to the text recognition, a camera pose estimation is performed, at 2120. For example, the camera pose may be determined by tracking in-plane points of interest and text corners as well as out-of-plane points of interest, as described with respect to
When the current processing mode is not the detection mode when a subsequent image is received, at 2106, 3D camera tracking 2130 is performed. Processing continues to rendering at the 3D rendering module, at 2122.
Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processing device such as a hardware processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or executable software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in a non-transitory storage medium such as random access memory (RAM), magnetoresistive random access memory (MRAM), spin-torque transfer MRAM (STT-MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or a user terminal.
The previous description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the disclosed embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.
The present application claims priority from U.S. Provisional Patent Application No. 61/392,590 filed on Oct. 13, 2010 and U.S. Provisional Patent Application No. 61/432,463 filed on Jan. 13, 2011, the contents of each of which are expressly incorporated herein by reference in their entirety.
Number | Date | Country | |
---|---|---|---|
61432463 | Jan 2011 | US | |
61392590 | Oct 2010 | US |