Patent Application
20140333585
This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2013-099748, filed May 9, 2013, the entire contents of which are incorporated herein by reference.
Embodiments described herein relate generally to an electronic apparatus such as an information processing device, an information processing method, and a storage medium.
A projector that projects information generated by an electronic apparatus such as an information processing apparatus is widely used.
A method for pointing an arbitrary point in a projected image such as a document, image, or photograph projected to a screen by a projector includes a method for directly pointing the point by a pointer and a method for adding an image information such as a cursor image to a projection data supplied to a projector by an electronic device.
The method using a pointer needs the pointer device. The method for displaying a cursor image requires an operation for moving the cursor image.
A general architecture that implements the various features of the embodiments will now be described with reference to the drawings. The drawings and the associated descriptions are provided to illustrate the embodiments and not to limit the scope of the invention.
Various embodiments will be described hereinafter with reference to the accompanying drawings.
In general, according to one embodiment, an electronic apparatus is connectable to a projector that projects projection information onto a projection surface to create a projection image and a camera that captures the projection image. The electronic apparatus includes an object detector and a processor. The object detector is configured to detect an object between the projector and the projection surface based on the projection information and captured information from the camera. The processor is configured to perform an object emphasis processing for at least one of the projection information and the captured information.
A projection system 1 includes an electronic apparatus 101, that is, an information processing device such as a personal computer (PC), a projector 201, and a camera 301. The projector 201 projects a projection image corresponding to projection information as a signal output from the electronic apparatus 101 onto a white projection surface S such as a screen. The camera 301 captures the projection image projected onto the screen S by the projector 201. The projection surface S may include other materials such as a whiteboard without limiting to the screen. A user, an announcer, or a presenter (hereinafter, referred to as a user) may be located in a predetermined position, including either the left or the right side, with respect to the screen S. A user is not necessarily located in a position where a display screen (projected information) of the electronic apparatus 101 of a personal computer (PC) or the like is viewed. Windows of a plurality of applications may be overlapped on the display screen of the electronic apparatus 101. The camera 301 may be integrated with, for example, the electronic apparatus 101.
Image information (camera image) output from the camera 301 that captures the image projected onto the screen S is supplied to a position correction module 115. A projection information obtaining module (screen capturing module) 117 is also connected to the projection information synthesizing module 113, so that the projection information (such as an original image and a captured image) corresponding to the image projected by the projector 201 is obtained from synthesis information.
The projection information output from a projection information obtaining module 117 is also input to the position correction module 115. A position correction module 115 performs computation for identifying what kind of perspective transformation is to be performed for the camera image in order to match the original image. For example, local features such as speeded up robust features (SURF) are extracted from both the original image and the camera image, and cross-matching is performed for the extracted local features, so that a 3×3 homography matrix is estimated, for example, using a random sample consensus (RANSAC). That is, the image output from the position correction module 115 is a camera image (hereinafter, referred to as a perspective transformation image) obtained by performing perspective transformation using the homography matrix (block #1 of
The output of the position correction module 115 is supplied to a color correction module 121 and an exposure/white-balance controller 116. The output of the exposure/white-balance controller 116 is supplied to the camera 301 to control the camera 301. The output of the color correction module 121 is supplied to a dim-area detector 125 and a difference generator 123. The projection information output from the projection information obtaining module 117 is also input to the color correction module 121, the dim-area detector 125, and the difference generator 123.
The color correction module 121 corrects a color (or luminance) of the perspective transformation image to match that of the original image. For example, for each color (luminance) of overall pixels, if a color (luminance) of a pixel in a position (x, y) on a perspective transformation image has a value Ii (where a range of the value Ii is set to, for example, [0, . . . , 255]), and a color (or luminance) of a pixel in a position (x, y) of the original image has a value Ij (where a range of the value Ij is set to, for example, [0, . . . , 255]), an average m(Ij) for the values Ij on the original image corresponding to overall points (x, y) having a certain value Ii in a perspective transformation image is calculated, and a function f(Ii) that returns a color (or luminance) obtained by correcting the color Ii is established. If the number of values Ij is smaller than that of the values Ii, interpolation may be performed using the values f(Ii) for neighboring positions.
An image output from the color correction module 121 (hereinafter, referred to as a corrected image) is obtained by applying a function f for overall pixels of the perspective transformation image.
As a result, it is possible to cancel an influence of tint included in the camera image and the original image having a substantially white background. In many cases, the tint is resulted from the tint components of the background (slightly colored in an originally white image) generated in the projection image (block #2 in
The difference generator 123 calculates a difference image “Idiff(x, y)=|Ic(x, y)−Io(x, y)|” using a corrected image Ic(x, y) output from the color correction module 121 and the projection image (original image or captured image) Io(x, y) output from the projection information obtaining module 117. The dim-area detector 125 calculates a dim image “Idim(x, y)=threshold(Ic(x, y))” from the corrected image Ic(x, y) output from the color correction module 121.
The outputs of the dim-area detector 125 and the difference generator 123 are supplied to a shadow extracting module 127. The shadow extracting module 127 calculates a shadow image “Ishadow(x, y)=threshold (Idiff(x, y)*Idim(x, y))” from the output Idiff(x, y) of the difference generator 123 and the output Idim(x, y) of the dim-area detector 125. In addition, the functions of the dim-area detector 125 and the shadow extracting module 127 have different threshold values. A function “threshold (I, pred)” is a function that generates an image having a value “1” if a binarization condition pred (for example, a pixel value I is equal to or smaller than 50) is satisfied in a pixel (x, y). Otherwise, the function “threshold (I, pred)” generates a value “0”.
In such a processing, an object placed between the screen S and the projector 201 and a shadow thereof are detected as a value Idiff, and a dark area including the shadow is detected as a value Idim, so that a product between the values Idiff and Idim is used to extract a shadow (to detect a shadow) (block #3 in
The output of the shadow extracting module 127 is supplied to the standing position detector 131. The standing position detector 131 is used to determine which of the left or right side a user stands in with respect to the screen S. As illustrated in
The output of the standing position detector 131 is supplied to a fingertip detector 133. The fingertip detector 133 detects a fingertip of a user. The fingertip detector 133 calculates a fingertip position Pf(x, y) having a maximum value x satisfying the condition “Ishadow(x, y)>0” in a case where a user stands in the left side. In addition, the fingertip detector 133 calculates a ratio of the pixels satisfying the condition “Ishadow(x, y)>0” within a pixel range neighbor to the pixel Pf(x, y). In a case where this ratio is smaller than a threshold value, it means that the pixel Pf(x, y) is included in a thin image. As a result, the pixel Pf(x, y) can be detected as a fingertip.
The output of the fingertip detector 133 is supplied to a fingertip tracking module 135. The fingertip tracking module 135 outputs a final fingertip position using the past fingertip position information of a user. The fingertip tracking module 135 appropriately performs filtering with the past fingertip position information being added in order to remove a noise and outputs a final fingertip position Pfinal(x, Y). For example, the final fingertip position Pfinal(x, y) is obtained through filtering for removing a noise using a Kalman filter in which state variables (x, x′, y, and y′) are used (block #4 in
The output of the fingertip tracking module 135 is supplied to a gesture information generator 141. The gesture information generator 141 detects that the final fingertip position indicates substantially the same position for a certain time. The gesture information generator 141 moves the cursor to the final fingertip position Pfinal(x, y) (block #5 in
A gesture output device 143 performs an operation on an actual apparatus.
The information transmitted to projection overlapping information input device 145 is synthesized with the original image through an overlay and the like using the projection information synthesizing module 113 and is transmitted to the projector 201 as the projection information in the next frame. In this case, as illustrated in
The electronic apparatus 101 includes a control module (MPU) 103 that controls each elements described above, a ROM 105 that stores a program used in operation of the MPU 103, a RAM 107 serving as a work area in an actual processing, a non-volatile memory 109 that stores numerical data or applications, and the like.
Automatic distortion correction is performed in block 322 based on the projection information and the captured information subjected to the projection position correction. Ambient color correction is performed in block 324 based on the captured information subjected to the automatic distortion correction. Projection color correction is performed in block 326 based on the captured information subjected to the ambient color correction and the projection information subjected to the automatic distortion correction. Delay correction is performed in block 328 based on the projection information and the captured information subjected to the projection color correction. Blur correction is performed in block 330 based on the projection information and the captured information subjected to the delay correction. Shadow/bright-spot correction is performed in block 332 based on the projection information and the captured information after the blur correction. Fingertip detection (including prediction/stabilized shadow detection/detection position conversion) is performed in block 334 based on the projection information and the captured information subjected to the shadow/bright-spot correction. User interface (UI) operation management is performed in block 336 based on the projection information and the captured information subjected to the fingertip detection and a result of the target application detection performed in block 338. Based on the result of the UI operation management, a graphic user interface (GUI) is displayed in block 340, and target application operation is performed in block 342.
Hereinafter, a specific example of each processing of
<1. Environment>
In some cases, the operation of the projection system may not be stabilized, and time may be consumed for preparation depending on a user's environment. Description will now be made for a processing for improving such cases.
<1-1. Projection Position Correction>
This is a processing regarding the position correction module 115 of
It is possible to suppress display of an unnatural screen if it is determined whether or not the image projected by a user using the projector 201 has a property suitable for the projection position correction, and a calibration image suitable for correction is projected in the projection system side only when the image does not have a suitable property.
<1-2. Projection Color Correction>
This is a processing regarding the color correction module 121 of
It is possible to suppress display of an unnatural screen if it is determined whether or not the content projected by a user using the projector 201 has a property suitable for the projection color correction, and the calibration image suitable for correction is projected in the projection system side only when the content does not have the suitable property.
<1-3. Automatic Exposure Control>
This is a processing relating to the exposure/WB controller 116 of
In a case where the absolute value of the difference is greater than the threshold value θ, a target exposure value is obtained efficiently through a binary search. For this reason, in block 706, it is determined whether or not the average luminance value s is smaller than the target luminance value t. In a case where the average luminance value s is smaller than the target luminance value t, in block 708, the exposure is set to an intermediate point between the minimum settable value and the current value, and the processing from the determination of block 704 is repeated.
In a case where the average luminance value s is equal to or greater than the target luminance value t, in block 710, the exposure is set to an intermediate point between the maximum settable value and the current value, and the processing from block 704 is repeated.
As a result, it is possible to cause the average luminance value s to match the target luminance value t (within an error range θ) for a short time.
<1-4. Automatic WB Control>
This is a processing regarding the exposure/WB controller 116 of
In block 904, a representative WB value s of the image obtained after the WB value of the camera 301 is set to a certain value is calculated, and it is determined whether or not an absolute value of the difference between the representative WB value s and the target WB value t determined from the representative WB value m is smaller than the threshold value θ. In a case where the absolute value of the difference between the target WB value t and the representative WB value s is smaller than the threshold value θ, it is determined that an error is within an allowable range, and exposure control is terminated.
In a case where the absolute value of the difference is larger than the threshold value θ, a target WB value is obtained efficiently through a binary search. For this reason, in block 906, it is determined whether or not the representative WB value s is smaller than the target WB value t.
In a case where the representative WB value s is smaller than the target WB value t, in block 908, the WB value is set to an intermediate point between the minimum settable value and the current value, and the processing from determination of block 904 is repeated. In a case where the representative WB value is equal to or greater than the target WB value t, in block 910, the WB value is set to an intermediate point between the maximum settable value and the current value, and the processing from determination of block 904 is repeated. As a result, it is possible to match the representative WB value s and the target WB value t (within an error range θ) for a short time.
<1-5. Automatic Distortion Correction>
This is a processing regarding the color correction module 121 of
<1-6. Ambient Color Correction>
This is a processing regarding the color correction module 121 of
There is known a Phong reflection model as described below.
I
c
=k
a
i
a
+Σ{k
d(Lj,Nj)id+ks(Rj,Vj)αis}
Here, “Σ” denotes a summation of “j” over all light beams. kaia denotes an ambient term, kd(Lj, Nj)id denotes a diffuse term, and ks(Rj, Vj)αis denotes a specular term.
In order to simplify this model, if it is assumed that the material (k*) is constant, and ambient/diffuse/specular light sources other than the projector are constant, these are simplified to “id.” Specular reflection caused by the projector is neglected. Diffuse reflection caused by the projector is constant in a light source direction Lj and a normal direction Nj for each pixel position. In addition, if a light attenuation coefficient p in each pixel position is introduced into the diffuse reflection term in consideration of reduction of an ambient light amount to provide “r=kd(Lj, Nj)p,” it is possible to obtain “Ic=Ib+r·id.”
First, the luminance Ic(0) of the camera image obtained by projecting the luminance id=0 is observed. Then, the luminance Ic(v) of the camera image obtained by projecting the luminance id=v(>0). The observation result is illustrated in
The “id” is calculated from the light “Ic” observed by the camera 301.
I
c(0)=Ib
I
c(v)=Ib+r·id
I
b
=I
c(0)
r=(Ic(v)−Ic(0))/v
i
d=(Ic−Ic(0))v/(Ic(v)−Ic(0))
In order to prevent only a particular channel (color) from being saturated, scaling is performed such that each channel is saturated at “id=v.”
The “v” is set to a maximum luminance in order to prevent any channel of any pixel from being saturated. The maximum luminance v can be determined in a high speed through a binary search under a constant exposure amount.
As a result, it is possible to obtain a camera image having no influence from a color caused by ambient light and a diffuse reflection coefficient on a projection surface.
<1-7. Color Correction Direction>
This is a processing regarding the color correction module 121 of
Specifically, a function “f_{p→c}” representing how colors (r, g, b)p in the projection content image Ip match colors (r, g, b)c in the camera image Ic subjected to the position correction and the ambient color correction is calculated in advance. That is, a function “f1=(r, g, b)p→rc” representing a transformation “(r, g, b)p→{rc}” from the projection image to the camera image is obtained using a support vector regression (SVR) method by focusing on one (for example, r) of the colors r, g, and b obtained by projecting a certain pattern (such as a calibration image). Similarly, for other colors, a function “f2=(r, g, b)p→gc” and a function “f3=(r, g, b)p→bc” are obtained. A function f_{p→c} representing a transformation “(r, g, b)p→(r, g, b)c” is obtained based on the functions f1, f2, and f3. Such an operation is calibration. When a new projection content image Ip and a camera image Ic subjected to position correction and ambient color correction are obtained, the projection content image Ip′ subjected to color correction can be obtained as “Ip′=f_{p→c}(Ip)” using the function “f_{p→c}.”
<1-8. Local Adaptive Luminance Addition>
This is a processing regarding the projection information correction module 112 of
As illustrated in
<1-9. Operation Improvement in Neighboring Portion>
This is a processing regarding the projection information correction module 112 of
Specifically, the side where a user stands is detected through the same processing as that described in conjunction with
<2. User Interface>
Description will now be made for a processing for implementing diverse and various operations without limiting to a simple click gesture by manipulating a finger. The chapter <2. User Interface> is a processing regarding the gesture information generator 141 and the gesture output device 143 of
<2-1. Scroll>
It is conceivable that only one of fingertips or hands is detected regardless of a shape, and rest thereof is allocated to a left click operation. However, instead of the click gesture based on rest of a fingertip or a hand, a scroll gesture can be implemented with a velocity based on a relative shift amount by setting, as a start point, a gesture start point by moving a fingertip or a hand. In this case, if the gesture start point is displayed on a screen, it is possible to implement an operation easily recognizable by a user.
If an icon indicating a “start point” (circle in
A scroll velocity v is determined as “v=f(L) or v=f(L′)” by using a length L obtained by tying the two points or a length L′ obtained by vertically projecting the length L. The function f is set to, for example, “f(L)=min(upper limit, L*first constant),” or the like. The first constant may be either positive or negative or may be set by a user.
An application immediately under the start point is vertically scrolled depending on the scroll velocity v. In a case where the shadow tip is deviated from an operable range, or shadow detection is failed, the scroll mode is terminated, and the display of the “start point” also stops. In this case, in order to prevent the scroll mode from being terminated abruptly due to an instant detection failure and the like, the scroll mode may be terminated if the shadow tip is deviated from the operable range, or shadow detection is failed continuously for a period of time.
The scroll velocity v may be applied to a zoom instead of the scroll. In this case, similarly, the first constant may be either positive or negative or may be set by a user.
The scroll velocity v may be allocated to a horizontal scroll instead of a vertical scroll. In this case, the scroll velocity v is converted into a horizontal scroll velocity, the length L′ is obtained by projecting the length L in a horizontal direction, and the operable range is rotated by 90°.
In addition, the scroll velocity v may be allocated to both the horizontal and vertical scrolls. In this case, the vertical scroll velocity vx and the horizontal scroll velocity vy are individually calculated using the aforementioned method. The operable range may not be set or may be set to a circle within a certain range from the start point.
The scroll velocity v may be set as “scroll to an initial movement direction.” In this case, while the length L is short, the scroll velocity v is allocated to both vertical and horizontal scrolls or is not allocated to any scroll. As the length L is equal to or longer than a threshold value TL, the scroll velocity v is allocated to the vertical scroll if the line obtained by tying the start point and the current position is nearly vertical. Otherwise, if the line obtained by tying the start point and the current position is nearly horizontal, the scroll velocity is allocated to the horizontal scroll, so that an operable range suitable for each direction is set and displayed.
<2-2. Zoom>
It is conceivable that only one of the fingertips or hands is detected regardless of the shape thereof, and rest thereof is allocated to a left click operation. However, instead of the click operation caused by resting a fingertip or a hand, an intuitive zoom operation may be implemented by detecting a plurality of fingers from the shadow image and analyzing a relative distance change as a zoom ratio change. In this case, a user can make an easily recognizable gesture by displaying a distance between fingers at the start of the gesture and a distance between the current fingers.
If icons indicating the “start points” (circles in
A zoom ratio m is determined as “m=f(L, Li)” using the current length L between the two points and an initial length Li between the start points. The function f is set to, for example, “f(L, Li)=min(upper limit, L/Li*second constant).” The second constant may be arbitrarily set or may be set by a user.
An application immediately under the start point is zoomed out depending on the zoom ratio m. The zoom mode is terminated, and display of the “start point” is interrupted in a case where one or both of the shadow tips are deviated from an operable range, or detection of any one of the shadows is failed. In this case, in order to prevent the zoom mode from being terminated abruptly due to an instant detection failure, the zoom mode may be terminated when the shadow tip is deviated from the operable range, or shadow detection is failed continuously for a while.
<2-3. Drag>
It is conceivable that only one of fingertips or hands is detected regardless of the shapes thereof, and rest thereof is allocated to the left click operation. However, a drag start operation may be implemented by moving a fingertip or a hand instead of a click operation made by resting a fingertip or a hand. In this case, the drag may be terminated by hiding a finger or resting a finger once more.
A drag mode starts as it is detected that the shadow tip rests, or there is a confirmation operation described below. Alternatively, the drag mode may start only when the shadow has a particular shape. Alternatively, the drag mode may start only when a particular application is used, or in a particular application state. Alternatively, the drag mode may start only when it is allowed by a GUI object state immediately under a cursor.
A drag start event is transmitted in the shadow tip position at the start of the drag mode. In response to a cursor movement, a drag movement event is transmitted. When detection of a shadow is failed, the drag mode is terminated, and a drag termination event is transmitted. In this case, in order to prevent the drag mode from being terminated abruptly due to an instant detection failure or the like, the drag mode may be terminated when detection of a shadow is failed continuously for a while.
<2-4. Additional Transmission of Click or Key>
It is conceivable that only one of fingertips or hands is detected regardless of the shape thereof, and rest thereof is allocated to the left click operation. However, allocation to further various operations is possible with issuing an event such as a right click, a center button click, and key pressing depending on detection features of hands and like instead of a click operation made by resting a fingertip or a hand.
Various operations are implemented by detecting rest of the shadow tip, use of a particular application, a particular application state, or a confirmation operation described below and issuing an event such as a right click, a center button click, and key pressing depending on a shadow shape, an application type, or an application state as well as a click or the scroll/zoom/drag described above.
<2-5. Marker Drawing Shaping>
A user may emphasize any portion of the projection content by displaying a locus of the drag gesture described above and displaying a figure on the projection content of the projector 201. In addition, since a locus of a shadow tip may not be smooth, it is possible to perform drawing on a screen by shaping the locus into a straight line, a curve, or a circle without any necessity of a cautious operation.
A marker drawing mode starts if it is detected that a particular finger shape or a tool bar is selected, or there is a confirmation operation described below. Alternatively, the marker drawing mode may start only when the shadow has a particular shape. Alternatively, the marker drawing mode may start only when a particular application is used, or in a particular application state.
A system is controlled depending on the drag gesture described above. As illustrated in
In this case, a figure (such as a straight line or a circle) suitable at any time during a drag may be continuously shaped and may be confirmed when the drag is terminated. Alternatively, shaping may be performed when the drag is terminated without performing shaping during the drag. As a result, a user can emphasize any portion of the projection content. In addition, during the drag, in order to distinguish figures before and after the shaping, both figures before and after the shaping are displayed with a different color, and the figure after the shaping may remain when the drag is terminated.
<2-6. Finger Shape>
A presentation of a finger having a particular thickness or shape is detected. Based on the detection result, allocation is made to a click gesture (
Specifically, when a shadow tip is detected, a thickness or shape of a finger is determined using the following method.
(a) The maximum number n of successive shadow pixels in a vertical direction of the shadow is calculated in a position going back by x pixels from the shadow tip to the shadow origin. The thickness (or number) of fingers is estimated depending on which of ranges divided by one or a plurality of threshold values the number of pixels n is included.
(b) After the maximum number n of successive shadow pixels in a vertical direction of the shadow is obtained in a position going back by x pixels from the shadow tip to the shadow origin, an operation is performed. The number of fingers is estimated to “m/2,” where “m” denotes a pixel value switching frequency between “0” (non-shadow) and “1” (shadow).
(c) w*h pixels including the shadow tip are cut out, and a feature amount such as a Co-occurrence Histograms of Oriented Gradient (CoHOG) is extracted. Then, a suitable fingertip shape is estimated by applying a classifier such as a support vector machine (SVM) for a plurality of fingertip shapes registered in advance.
Depending on the determined fingertip shape, allocation is made to a click gesture (
<2-7. Finger Direction>
By detecting a finger inclination on a screen, allocation to processing or operations can be made depending on the inclination. For example, as illustrated in
Specifically, when a shadow tip is detected, a finger direction is estimated based on an inclination of a principal axis through a principal component analysis or by adapting a set of shadow positions (x, y) of w*h pixels including the shadow tip for a straight line using a least-squares method. Depending on the finger inclination, allocation is made to a click gesture (
<2-8. Gesture Type Change by Recognizing Grasped Object>
If a characteristic shadow is projected, or operation is performed by holding a dedicated pointer (
Specifically, several types of bar-like dedicated pointers whose tip or support projects a characteristic shadow are prepared. A characteristic shadow (
Depending on a device used in determination or a posture thereof, allocation is made to a click gesture (
<2-9. Distance-Based Operation>
If a change of a shadow size or a shadow dimming quantity (
Specifically, when a shadow tip is detected, w*h pixels including the shadow tip is cut out from a shadow image, an area where a shadow probability p is equal to or higher than a threshold value t1 is set to a shadow area a, an area where a shadow probability p is equal to or lower than a threshold value t2 is set to a non-shadow area b, and other portions are set to a dimming area c.
Using a shadow size (shadow area) a or a dimming area c, a distance f(a) or g(c) between the hand and the screen S is estimated. The estimated value may be substituted with the scroll amount or the zoom amount described in the chapter <2-1. Scroll> or <2-2. Zoom> (
<2-10. Locus-Based Operation>
If a locus of a finger (
Specifically, when a shadow tip is detected, this detection position (x, y) is recorded at all times.
If “t” denotes the current time, a sequence L (
For each element (x, y) of the sequence L, normalization is performed using a computation such as “(x, y):=((x−xt2)/maxL(x), (y−yt2)/maxL(y)),” and re-sampling is performed to obtain the number of elements N. In addition, it is determined which of a plurality of loci registered in advance matches using a classifier such as an SVM.
A registered processing is executed depending on the registered locus. For example, it is possible to reduce a phase that a user directly manipulates a PC by performing allocation to an infrequent functionality such as activation or inactivation of a particular application.
<2-11. Operation-Based Operation>
A cyclic operation such as reciprocation of a finger (
Specifically, when a shadow tip is detected, this detection position (x, y) is stored at all times.
If “t” denotes the current time, a sequence L (
For each element (x, y) of the sequence L, normalization is performed using a computation such as “(x, y):=((x−xt2)/maxL(|x|), (y−yt2)/maxL(|y|)),” and polling is performed to a histogram obtained by dividing a range “[−1, 1]×[−1, 1]” by “N*M.” In addition, it is determined which of a plurality of operations registered in advance matches using a classifier such as a SVM. In addition, a velocity characteristic obtained by performing polling for a derivative of the element (x, y) within a range of “[−1, 1]×[−1, 1]” on a X*Y-divided histogram may be used together.
A registered processing is executed based on the registered operation. For example, it is possible to reduce a phase that a user directly manipulates a PC by performing allocation to a slightly infrequent functionality such as page ejection or screen erasure.
<2-12. Change of Operation Allocation for Operation Target>
A click is transmitted without recognizing an operation target. However, it is possible to manipulate a plurality of applications having different operation systems by recognizing an operation target and changing user's poses set in advance and a display method on the issued event/screen (refer to
Specifically, a currently focused application is recognized when a shadow tip is detected, periodically at a certain time, or in response to a change of the cursor position. Alternatively, an application state is recognized. Alternatively, a GUI object state immediately under the cursor position is recognized.
When rest of a shadow tip or other operations are detected, an issued event or a display method on a screen is changed based on a predetermined matching relationship depending on an application recognized in advance, a state thereof, and a GUI object state.
The example of
In the case of the Web-browsing, a one-finger gesture corresponds to a click, a two-finger gesture corresponds to a scroll gesture, and a multi-touch gesture (one finger per each hand) corresponds to a zoom. In the Web-browsing, an operation is performed while referencing the Web information in a conference or the like, and a zoom-in or zoom-out operation can be appropriately made to see details of the information searched in the Web. In a case where reference materials are temporarily displayed in a lecture or the like, a presentation/operation can be rapidly performed in a simplified manner.
In a presentation of a PDF file (input using a simplified marker), a one-finger gesture corresponds to a marker, a two-finger gesture corresponds to a scroll, and a multi-touch gesture (one finger per each hand) corresponds to a zoom. In the PDF file presentation, a marking is appropriately added to an emphasis portion while reference materials are presented in a conference or the like. Therefore, in a large conference room, details of the reference materials can be viewed even by a distant audience through a zoom-in or zoom-out operation. In addition, in the field of education, a teacher or a student can select an answer or point out an important portion.
In the case of a web-based electronic teaching material used for game/education purposes as web contents for children, a one-finger gesture corresponds to a click, a two-finger gesture corresponds to a scroll, and a multi-touch (one finger per each hand) corresponds to a zoom. As a result, in kindergartens or lower classes of elementary schools, children can make an operation on Web-based simplified game contents or children English teaching contents to share a situation with children.
<3. Operation Timing Detection>
This is a processing regarding the fingertip detector 133 of
<3-1. Screen Touch Detection Based on Shadow Erasure>
In the aforementioned description, a click or the like is transmitted by resting a finger shadow. However, as illustrated in
Specifically, a shadow tip is detected (
As a hand approaches the screen S, the shadow area is obscured by a hand and is finally narrowed (
If it is detected that the shadow area is erased from the previous position (
Since an actual screen touch instant can be obtained, an operation timing becomes clear, and a operation feeling is improved.
<3-2. Vibration-Based Screen Touch Detection>
In the aforementioned description, a click or the like is transmitted by resting a finger shadow. However, a user input such as a click can be detected and transmitted by detecting a slight vibration on a screen generated when a hand touches the screen as illustrated in
Specifically, a motion on the content projected onto the screen is monitored. A user input is sensed by detecting an abrupt motion (
Since an actual screen touch instant can be obtained, an operation timing becomes clear, and an operation feeling is improved.
<3-3. Sound-Based Screen Touch Detection>
In the aforementioned description, a click or the like is transmitted by resting a finger shadow. However, if a microphone is installed in a screen touch position or in the vicinity of the screen as illustrated in
Specifically, a microphone is installed in a screen touch position or in the vicinity of the screen. If a steep increase of an acoustic pressure exceeding a threshold value set in advance or set based on a sound input by a user in a test stage is detected, it is possible to detect a screen touch of a user. As a touch point, the previous shadow tip position is used.
Since an actual screen touch instant can be obtained, an operation timing becomes clear, and an operation feeling is improved.
<3-4. Association with Remote Control>
In the aforementioned description, a click or the like is transmitted by resting a finger shadow. However, if a shadow is used to designate an operation position as illustrated in
Specifically, a shadow is used to designate an operation position (
A user holds a device that outputs an operation timing of a click or the like in a wired or wireless manner to a system. When a user desires to perform a confirmation operation, a user performs an input operation by pressing a button or manipulating a touch panel. For example, it is possible to user an application program of a smart phone to notify a system of a message “click at the current position” upon detection of a pressing of a “volume” button.
Since an instant at which a determination operation is to be performed can be specified, an operation timing becomes accurate, and an operation feeling is improved.
<4. Shadow Influence Reduction>
This is a processing regarding the fingertip detector 133 of
<4-1. Operation Position Transformation>
Typically, a tip indicated by a shadow is set as a cursor (
Specifically, an operation area is set based on a user standing position or a projector size or using an area set by a user in advance (
In this regard, an operation on the operation area is set as a cursor position after a transformation (such as an affine transformation) mapping the operation area to the entire screen (
<4-2. Laser Pointer>
A position on the projection content irradiated from a laser pointer may be used instead of the shadow position (
Specifically, in a shadow image creating sequence, if a difference Ic−Im between the camera image Ic subjected to correction and the projection content Im subjected to correction is larger than a threshold value θ, a bright spot of the laser pointer is detected instead of the shadow (
As a result, it is possible to perform an operation such that an influence on projection is insignificant without making a shadow on the projection content. In addition, it is possible to perform an operation even when a hand does not reach the entire screen or even in a distant position.
<4-3. Transparent Pointing Bar>
Typically, since a shadow is formed by a hand or a pointing bar, a large portion of projection contents may be blocked by a hand or a pointing bar. If a tip of the pointing bar is opaque as it is, to form a shadow while other portions are transparent, it is possible to perform an operation without making a shadow on the projection content as much as possible (
<5. Multiple Users>
This is a processing regarding the fingertip detector 133 of
Typically, an operation on a projector projection content is performed from any one of the left or right sides and is not simultaneously performed from two or more sides. However, if a shadow start point is searched around the projection content, and a tip of the shadow extending therefrom is determined as a fingertip, it is possible to allow a plurality of users to simultaneously perform operations from a plurality of directions (
Specifically, the shadow image is segmented into a plurality of areas without being connected through labeling to obtain shadow segments. A portion where each shadow segment adjoins a circumferential edge of the screen is referred to as an “origin.” For each shadow segment, a tip portion of the shadow is detected according to a method of the related art, and allocation is performed to a click or other gestures through a confirmation operation including rest of the shadow tip or other operations.
<6. Whiteboard>
This is a processing regarding the color correction module 121 of
<6-1. Drawing Content Incorporation>
In a case where a whiteboard is used as a projection surface, it may be difficult to use a drawing content of the whiteboard when projection is performed on the whiteboard.
If the “ambient color correction” described in the chapter <1-6> is performed at the time of system activation, the content (
Specifically, the “ambient color correction” is performed at the time of system activation by using a whiteboard (
In the “ambient color correction,” a diffuse reflection coefficient d and the ambient light i in each pixel within a projector projection range are obtained. Here, “d” indicates information on the content (
The stored drawing content may be printed or used as data from an application employing this system.
Accordingly, it is possible to conveniently utilize the whiteboard drawing content as electronic information by using the results of “position correction” and “ambient color correction” executed in advance without arranging, beforehand, a marker or the like, which is necessary in a method of the related art.
<6-2. Drawing Content Cancellation>
By mixing an image for canceling the drawing content (
An appearance color is set to “i*(first constant)+p*d*(second constant),” where “i” denotes ambient light in any position, “d” denotes a diffuse reflection coefficient, and “p” denotes a projector output. Therefore, the drawing target color c can be expressed as “c=i*(first constant)+p*d*(second constant),” and the projection target color p can be determined as “p=(c−i*(first constant))/p/(second constant).”
As a result, it is possible to project the projection content as if nothing has been written on the whiteboard and eliminate an effort of erasing the whiteboard. A user can easily use the system since there is no need to erase the whiteboard.
<7. Dynamic Calibration>
This is a processing regarding the MPU 103 of
<7-1. Camera Deviation Detection/Re-Calibration>
As a significant movement of an object in portions (
As a result, in a case where the camera is operated incautiously, a system is not malfunctioned suddenly, but can return to a normal state with a little effort.
<7-2. Illumination Change Detection/Re-Calibration>
As a significant change of the luminance value is detected in a portion (
As a result, in a case where an illumination condition changes, a system is not malfunctioned suddenly, but can return to a normal state with a little effort.
The various modules of the systems described herein can be implemented as software applications, hardware and/or software modules, or components on one or more computers, such as servers. While the various modules are illustrated separately, they may share some or all of the same underlying logic or code.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2013-099748 | May 2013 | JP | national |
