Patent Application
20250139996
The present invention is related to a method of identifying a whiteboard, especially a method of identifying a whiteboard based on image tokens.
A whiteboard is a white surface made of plastic or tempered glass that can be written repeatedly with a special whiteboard pen and can be wiped with a special whiteboard eraser. It is often used for classroom teaching, meeting discussions, or personal notes. From the mid-1990s, whiteboards began to be commonly used. Before that, almost only blackboards were common.
Whiteboards are usually more expensive than blackboards. Compared with the chalk dust left after writing on a blackboard, a whiteboard is easier to wipe than a blackboard because the whiteboard does not raise powder particles when wiping. In addition, the contrast of pictures or words written on the whiteboard is relatively high. However, because whiteboards tend to reflect light, people far away from the whiteboard may not be able to see clearly, so most schools still use blackboards.
A live video showing the whiteboard captured by a camera can solve the light reflection issue and the small whiteboard issue. Thus, a method of identifying a whiteboard to generate a plain image for students sitting far away from the whiteboard is desirable. The method includes blurring, sharpening, or mosaicking to filter the private information and show the desired information on the whiteboard. Sometimes, the camera is not set in front of the whiteboard, a mapping method to show the plain image of the whiteboard is desirable.
A method of identifying a whiteboard based on image tokens includes setting a plurality of image tokens on the whiteboard, obtaining an image including the whiteboard from a camera, employing hardware or software techniques to detect image shaking in order to prevent a display of non-relevant content or privacy-sensitive information, detecting image tokens of the whiteboard by using a machine learning model, applying token tracking to the image tokens to enhance stability and robustness of token detection, calculating coordinates of the image tokens, determining an optimal mapping matrix based on the coordinates of the image tokens, mapping the image based on the optimal mapping matrix to generate a mapped image of the whiteboard, and displaying the mapped image on a screen.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
In step S102, a plurality of image tokens are set on the whiteboard. The whiteboard can be a glass whiteboard, a mobile whiteboard, a projection screen, a mobile green board, or a 3M whiteboard film. The image tokens can be magnets, magic tapes, stickers, patterns such as symbols drawn by various pens (whiteboard pens or color pens), gestures, display of electronic screens or other plain images. The symbols drawn by various pens include but not limit to rectangular symbols, star symbols, diamond symbols, and line symbols. The number of the image tokens can be N, where N is an integer larger than 1. The positions of the image tokens can be but not limited to at the four corners of the whiteboard. For instance, the positions of image tokens can be near the four corners of the whiteboard. In step S104, an image including the whiteboard is obtained from a camera.
In step S105, the stability of the image can be detected by software such as an optical flow method. The system can also check whether the screen is shaking according to camera movement. The camera movement can be detected by hardware such as an accelerator, a gyroscope or another device. Detecting image shaking using hardware or software techniques can prevent a display of non-relevant content or privacy-sensitive information. In step S106, the image tokens of the whiteboard are to be detected in the image captured by the camera by the machine learning model. The machine learning model can be an artificial intelligence (AI) model, a deep learning model, a computer vision model, or a you only look once (YOLO) model.
In step S108, token tracking is applied to the image tokens to enhance stability and robustness of token detection. The token tracking is applied based on a center point based tracking system, an intersection over union based tracking system, or other tracking system. In step S110, the coordinates of the image tokens are calculated. If only M instead of N image tokens are detected (M<N), then a permission is requested to generate the remaining image tokens at or near the remaining corner automatically according to the M detected image tokens. If the automatic generation of image tokens is not allowed, then the remaining image tokens are requested to provide enough information for further processes. Likewise, if less than two image tokens are detected, then more image tokens are requested to make up the shortfall of image tokens. If M instead of N image tokens are detected (M>N), then a deletion of image tokens is requested to make sure the M detected image tokens correspond to N image tokens.
In step S112, an optimal mapping matrix can be determined based on the coordinates of the image tokens for image mapping. The image is mapped to generate a mapped image based on the optimal mapping matrix. In step S114, before displaying the mapped image on a screen, a post-processing such as blurring, sharpening, and/or mosaicking can be performed on the mapped image. The mapped image which may be post-processed is displayed on the screen for members far away from the whiteboard.
Step S302: set a camera to capture the image of the whiteboard;
Step S304: input an image of the whiteboard to the system;
Step S306: check whether the screen is shaking; if so, go to step S304; else go to step S308; and
Step S308: enter the tracking state.
In step S302, a camera is set to capture the image of the whiteboard. In step S304, an image of the whiteboard is inputted to the system. In step S306, the system checks whether the screen is shaking according to stability of the image. The stability of the image can be detected by software such as an optical flow method. The system can also check whether the screen is shaking according to camera movement. The camera movement can be detected by hardware such as an accelerator, a gyroscope or another device. If screen shaking is detected, step S304 is performed again to capture another image so as to check whether the screen is still shaking later on. If screen shaking is not detected, step S308 is performed to enter the tracking state.
Step S402: set a camera to capture the image of the whiteboard;
Step S404: input an image of the whiteboard to the system;
Step S408: check whether the screen is shaking; if so, go to step S410; else go to step S412;
Step S410: enter the reset state;
Step S412: check whether the image tokens are tracked; if so, go to step S414; else go to step S416;
Step S414: enter the stable state;
Step S416: detect and track image tokens of the whiteboard;
Step S418: calculate an optimal mapping matrix based on the image tokens and map the image based on the optimal mapping matrix to generate a mapped image of the whiteboard;
Step S420: display the mapped image on a screen; go to step S414.
In step S402, a camera is set to capture the image of the whiteboard. In step S404, an image of the whiteboard is inputted to the system. In step S408, the system checks whether the screen is shaking based on the software method or the hardware method as in step S306. If screen shaking is detected, go to step S410 to enter the reset state. If screen shaking is not detected, then in step S412, the system checks whether the image tokens are tracked. If the image tokens are tracked, go to step S414 to enter the stable state. If the image tokens are not tracked, step S416 is performed to detect and track the image tokens of the whiteboard by using a machine learning model. The machine learning model can be an artificial intelligence (AI) model, a deep learning model, a computer vision model, or a you only look once (YOLO) model.
In step S418, the coordinates of the image tokens are calculated. If only M instead of N image tokens are detected (M<N), then a permission is requested to generate the remaining image tokens at or near the remaining corner automatically according to the M detected image tokens. If the automatic generation of image tokens is not allowed, then the remaining image tokens are requested to provide enough information for further processes. Likewise, if less than two image tokens are detected, then more image tokens are requested to make up the shortfall of image tokens. If M instead of N image tokens are detected (M>N), then a deletion of image tokens is requested to make sure the M detected image tokens correspond to N image tokens. Then, an optimal mapping matrix can be determined based on the coordinates of the image tokens for image mapping. After that, the image is mapped to generate a mapped image based on the optimal mapping matrix. In step S420, a post-process such as blurring, sharpening, and/or mosaicking may be performed on the mapped image, and the post-processed mapped image or the mapped image is displayed on the screen.
Step S502: set a camera to capture an image of the whiteboard;
Step S504: input the image of the whiteboard to the system;
Step S508: check whether the screen is shaking; if so, go to step S510; else go to step S512;
Step S510: enter the reset state;
Step S512: determine an optimal mapping matrix based on the image tokens and map the image based on the optimal mapping matrix to generate a mapped image of the whiteboard; and
Step S514: display the mapped image on a screen; go to step S504.
In step S502, a camera is set to capture the image of the whiteboard. In step S504, an image of the whiteboard is inputted to the system. In step S508, the system checks whether the screen is shaking based on the software method or the hardware method as in step S306. If screen shaking is detected, go to step S510 to enter the reset state. If screen shaking is not detected, then in step S512, an optimal mapping matrix can be determined based on the coordinates of the image tokens for image mapping. After that, the image is mapped to generate a mapped image based on the optimal mapping matrix. In step 514, a post-process such as blurring, sharpening, and/or mosaicking may be performed on the mapped image, and the post-processed mapped image or the mapped image is displayed on the screen. Then step S504 is performed again to continue the routine.
In conclusion, a live video showing the plain image of the whiteboard captured by a camera can be implemented by applying the method of identifying the whiteboard based on image tokens. The method uses the state machine including the reset state, stable state, and tracking state, solving the problem of light reflecting from the whiteboard for far away audiences. Moreover, the mapping technique solves the skew image problem to always show the plain image of the front side of the whiteboard. The post-processing including blurring, sharpening, and mosaicking can filter the private information and show the desired information of the whiteboard, thus avoiding privacy-sensitive information.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
