Patent Application
20220019781
The present disclosure relates to a home user interface (UI) system for managing digital ink.
Digital ink generated from an electronic pen is data used to reproduce the trace made by the electronic pen, which is similar to the trace left on paper by a conventional pen used to perform handwriting. An example of digital ink that is an object-based data model residing in a computer is disclosed in U.S. Pat. No. 7,158,675, and an example of a serialization format used for digital ink is disclosed in U.S. Pat. No. 7,397,949.
Another type of digital ink data is known that goes beyond merely reproducing handwritten traces and that enables recording of “when, by whom, where, and in what situation” the handwriting has been performed by a person to leave the trace. For example, Japanese Patent No. 5886487 discloses digital ink that makes it possible to identify who has written stroke data that represents the trace. US Patent Application Publication No. 2016/0224239 discloses digital ink that allows acquisition of information as context data when stroke data is input, such as the author information, pen identification (ID), clock time information, and location information acquired by the global positioning system (GPS), and recording it as metadata.
In recent years, studies have been made to use digital ink to estimate “with what kind of thought or emotion a handwriting has been made.” For example, in Japanese Patent Laid-open No. 2010-131280, a method and a device are disclosed that ascertain and determine the mental, psychological, and physiological states of the writing person by quantifying time-space information of handwritten characters or pictures and extracting characteristic features. In International Patent Publication No. WO 2018/043061, an emotion estimation system is disclosed that associates the writing state, such as the writing pressure, with biological information corresponding to emotion, and derives the biological information corresponding to the emotion only from the writing state.
Such digital ink indicating “when, where, and how handwriting has been made” can be considered a trace of a writing person, i.e., history data of the person's action or emotion. Such digital ink can be combined with a personalized artificial intelligence (AI) to support realization of a more advanced life style.
For example, in US Patent Application Publication No. 2016/0259308, a smart home automation system is disclosed that senses an environment in a house or a person's motion and dynamically changes, based on the sensing detection result, a “policy” that characterizes operations of respective home-use devices.
By incorporating a handwriting sensor into a home-use device that is provided in a house or that forms part of the house, the “house itself” can be configured to function as an input-output device of digital ink. However, when each of multiple members of the house can access plural home-use devices, multiple home-use devices or multiple writing persons may coexist at the same time of digital ink input, and multiple pieces of written information may coexist at the same time of digital ink output. Thus, the following operational inconveniences may occur, for example: [1] a use opportunity may be lost when a home-use device is overlooked; [2] the identify of a writing person cannot be determined when one electronic pen is shared by multiple people; and [3] handwriting information fragmentally input from multiple home-use devices cannot be effectively utilized.
An aspect of the present disclosure is directed to providing a home UI system for managing digital ink, when a house itself is configured to function as an input-output device of digital ink, wherein the home UI system is capable of suppressing inconveniences due to multiple home-use devices and multiple writing persons coexisting at the same time of digital ink input, or effectively outputting handwriting information indicated by the digital ink.
A home UI system for managing digital ink in a first aspect of the present disclosure includes a plurality of state sensors capable of detecting a state in a house or a change in the state, a plurality of home-use devices that are provided in the house or that form part of the house and that each include a handwriting sensor capable of detecting handwriting by a person, one or more notification units that notify the person of existence of the home-use device or a detectable region of the handwriting sensor, and a controller that, responsive to a determination based on a detection result from one or more of the plurality of state sensors that a notification is necessary, instructs at least one of the notification units to perform the notification.
A home UI system for managing digital ink in a second aspect of the present disclosure includes a plurality of state sensors capable of detecting a state in a house or a change in the state, a home-use device that is provided in the house or that forms part of the house and that includes a handwriting sensor capable of detecting handwriting by a person, and an estimation unit that, responsive to a detection by the home-use device of a handwriting input, estimates a person who has inputted the handwriting based on a detection result from one or more of the plurality of state sensors.
A home UI system for managing digital ink in a third aspect of the present disclosure includes a plurality of home-use devices that are provided in a house or that form part of the house and that each include a handwriting sensor capable of inputting handwriting by a person, a storage device that stores handwriting data indicating a form of handwriting in such a manner as to associate the handwriting data with the home-use device, a terminal device configured to display an image or a video and associated with one or more of the plurality of home-use devices, and a controller that, in response to receiving a predetermined operation through the terminal device, acquires handwriting data of the home-use device associated with the terminal device from the storage device, and instructs the terminal device to display written-by-hand information indicated by the acquired handwriting data or content information specified by the acquired handwriting data.
A home UI system for managing digital ink in a fourth aspect of the present disclosure includes a plurality of state sensors capable of detecting a state in a house or a change in the state, a home-use device that is provided in the house or that forms part of the house and that includes a touch panel display capable of inputting and outputting handwriting by a person, a storage device that stores handwriting data indicating a form of handwriting in such a manner as to associate the handwriting data with the home-use device, and a controller that, in response to one or more of the plurality of state sensors detecting a predetermined state of a person who exists near the home-use device, acquires handwriting data associated with the home-use device from the storage device, and instructs the home-use device to display written-by-hand information indicated by the acquired handwriting data.
According to the first aspect of the present disclosure, the inconvenience due to coexistence of multiple home-use devices at the time of digital ink input can be avoided.
According to the second aspect of the present disclosure, the inconvenience due to coexistence of multiple writing persons at the time of digital ink input can be avoided.
According to the third and fourth aspects of the present disclosure, the handwriting information indicated by digital ink can be effectively output.
A home UI system that manages digital ink in the present disclosure will be described with reference to the accompanying drawings.
Configuration of Home UI System 10
Overall Configuration
The digital ink server 12 is a computer that carries out comprehensive control relating to input and output of digital ink, and is specifically configured to include a communication section 26, a control section 28, and a storing section 30.
The communication section 26 is an interface that transmits and receives an electrical signal to and from an external device. The control section 28 comprises an arithmetic processing device including a central processing unit (CPU) and a graphics processing unit (GPU). The control section 28 functions as a data acquiring section 32, a data analyzing section 34, and a device control section 36 (controller), by reading out a program stored in the storing section 30 and executing the program.
The storing section 30 comprises a non-transitory, computer-readable storage medium. Here, the computer-readable storage medium is a portable medium such as a magneto-optical disc, a read-only memory (ROM), a compact disc (CD)-ROM, or flash memory or a storage device such as a hard disk incorporated in a computer system.
The storage device 14 comprises a hard disk drive (HDD) or a solid state device (SSD), for example, and stores various pieces of data to be handled by the digital ink server 12. Specifically, the storage device 14 includes a database that accumulates handwriting information generated in the life within the house 100 (hereinafter, handwriting information DB 38) and a database that accumulates environmental information of the house 100 (hereinafter, environmental information DB 40).
The IoT hub 16 is a relay equipment that can bi-directionally communicate with each piece of IoT equipment that exists in the house 100. This allows each piece of IoT equipment to supply its own data to the digital ink server 12 through the IoT hub 16 and a network NW. The digital ink server 12 can instruct each piece of IoT equipment to carry out any necessary operation through the network NW and the IoT hub 16. In the example of this diagram, the IoT equipment includes the state sensor 18, the home-use device 20 or 22, and the terminal device 24.
The state sensor 18 is a sensor that detects a state in the house 100 or a change in the state. The state in the house 100 may be an internal state of the house 100 or may be the state of a person Hm who is present in the house 100. The internal state of the house 100 may include an environmental state such as the state of air, light, and sound, the arrangement or use state of the home-use devices 20 and 22, and so forth. The state of person Hm may include the position, posture, motion, physical condition, emotion, and so forth of person Hm.
Various kinds of state sensors 18 may be used according to the detection target, the room layout of the house 100, the installation cost, and so forth. For example, the state sensors 18 may be non-contact-type sensors including a camera, an illuminometer, a thermo-hygrometer, a sound sensor, an ultrasonic sensor, a motion sensor, and an infrared sensor, and may be contact-type sensors including a pressure sensor, a touch sensor, and a motion sensor. The state sensors 18 may be stationary-type sensors arranged in the house 100 and may be portable-type sensors (which may additionally be wearable-type or non-wearable-type) brought into the house 100.
The home-use devices 20 and 22 are devices that are provided in the house 100 or that form part of the house 100. As examples of the former, home appliances (including a terminal device to be described later), furniture, home accessories and equipment, and so forth may be provided as the home-use devices. As examples of the latter, the home-use devices may form a wall, a floor, a window, a pillar, and so forth, of the house 100. The home-use devices 20 and 22 are each configured to include a main unit 50 for executing its own functions and a handwriting sensor 52 that can detect handwriting made by person Hm. The handwriting sensor 52 is configured to include plural sensor electrodes for detecting the position of an electronic pen 54 or a finger. The electronic pen 54 is, for example, a stylus of an active capacitive system (AES) or an electromagnetic induction system (EMR).
The home-use device 22 further includes an indicator 56, in addition to the main unit 50 and the handwriting sensor 52, differently from the other home-use device 20. The indicator 56 is an indicating instrument that can output information that appeals to the five senses of person Hm, and may comprise a lamp, a speaker, a vibrator, or a display, for example. This allows the home-use device 22 to indicate its own existence or its detectable region.
The terminal device 24 may be a multi-function, multi-purpose device, such as a personal computer, a tablet, a smartphone, and an AI speaker, for example. The terminal device 24 may be a stationary device or may be a portable device. Hereinafter, devices that can inform person Hm through output of information will be collectively referred to as “notification devices 58” in some cases (the home-use device 22 and the terminal device 24 in the example of
As illustrated in
As illustrated in
Here, the “accessible range” means the range in which person Hm present in the range can visually recognize the home-use device 20 or 22 or the range in which person Hm in the range can touch the home-use device 20 or 22.
As illustrated in
The foregoing has described the configuration of the home UI system 10 in the present embodiment. The following describes operations of the home UI system 10 including a first, second, third, and fourth operations.
When the handwriting sensor 52 is incorporated into the home-use device 20 or 22, to improve the appearance of a product, the home-use device 20 or 22 may be designed such that the form of the handwriting sensor 52 is inconspicuous. In the case of arranging the home-use device 20 or 22 in a well-blended manner in the house 100, due to the camouflage effect (or the stealth effect), it may become difficult for a user to determine the existence of the handwriting sensor 52 or the position of the detectable region that blends in the ambient environment. A similar phenomenon may also occur when the inside of the house 100 is dark, or when the home-use device 20 or 22 is misarranged so that its detection surface is hidden.
Thus, the home UI system 10 carries out the “first operation” to timely notify the location of the home-use device 20 or 22 or the detectable range of the handwriting sensor 52 to a user who may be present. The first operation will be described below with reference to a flowchart of
In a step S11 in
In a step S12, the data analyzing section 34 carries out primary determination relating to whether or not a notification to be described later is necessary, by using the state data acquired in the step S11. Specifically, the data analyzing section 34 analyzes the state data from multiple perspectives or in a time-series manner by using various analytical methods including sensor fusion, and checks whether or not a predetermined motion by person Hm has been detected. Here, the “predetermined motion” means a motion that is a precursor of a writing operation on the home-use device 20 or 22. For example, a predetermined motion may be [1] a motion of approaching the front of a refrigerator and stopping there, or [2] a motion of stepping on a bathroom scale. The former corresponds to a “precursor” of a series of actions in which person Hm opens a door of the refrigerator, checks the content therein, closes the door of the refrigerator, and makes a note of food items that need to be additionally bought. The latter corresponds to a “precursor” of a series of actions in which a person gets on the bathroom scale, checks the measurement result, steps off the bathroom scale, and makes a note of the measurement result.
In a step S13, the data analyzing section 34 checks whether or not the determination result in the step S12 satisfies a primary determination condition. When the determination condition is not satisfied (step S13: NO), the operation returns to the step S11 and the steps S11 to S13 are sequentially repeated until the condition is satisfied. On the other hand, when the determination condition is satisfied (step S13: YES), the operation proceeds to the next step S14.
In the step S14, the data analyzing section 34 calculates plural features used for secondary determination to be described later from the state data acquired in the step S11 or state data additionally acquired. For example, when the state sensor 18 is a camera, “physical features” may be extracted including characteristics of the face, the height, the body shape, and the clothes of the writing person, or “motion features” may be extracted including the writing person's dominant hand and habits of writing. When the state sensor 18 is a body measuring instrument, the physical features including the weight, the body fat percentage, and the body composition of the writing person may be extracted. When the state sensor 18 is a microphone, the physical features including the voiceprint and the voice volume of the writing person may be extracted, and “emotional features” that represent the state of delight, anger, sorrow, and pleasure may be extracted.
In a step S15, the data analyzing section 34 carries out secondary determination relating to whether or not a notification to be described later is necessary, by using the features calculated in the step S14. For example, the data analyzing section 34 can determine whether or not a notification is necessary depending on whether or not plural individual conditions are all satisfied or by converting satisfaction and non-satisfaction of the plural individual conditions to a score. Alternatively, the data analyzing section 34 may carry out the determination with higher accuracy by using a determination implement 70 in which machine learning has been carried out.
Although the determination implement 70 is constructed by using the neural network in the above-described example, the method of machine learning is not limited thereto. For example, various methods including a logistic regression model, support vector machine (SVM), decision tree, random forest, and boosting may be used.
In a step S16, the data analyzing section 34 checks whether or not the determination result in the step S15 satisfies a secondary determination condition. Here, the data analyzing section 34 determines that the determination condition is satisfied when the determination value output from the determination implement 70 is larger than a threshold (for example, 0.7). On the other hand, the data analyzing section 34 determines that the determination condition is not satisfied when the determination value is equal to or smaller than the threshold. When the determination condition is not satisfied (step S16: NO), the operation returns to the step S11 and the steps S11 to S16 are sequentially repeated until the condition is satisfied. On the other hand, when the determination condition is satisfied (step S16: YES), the operation proceeds to the next step S17.
In the step S17, the device control section 36 selects at least one notification device 58 suitable to the state of person Hm, among one or more notification devices 58 usable in the house 100. Specifically, the device control section 36, by collating the sensor information 42 and the first device information 44, extracts the ID of the home-use device 20 or 22 regarding which the detectable range and the accessible range partially overlap (that is, the first device ID). The device control section 36 refers to the second device information 46 and extracts the ID of one or plural notification devices 58 corresponding to the first device ID (that is, the second device ID).
In a step S18, the device control section 36 instructs the notification device 58 selected in the step S17 to carry out the notification relating to the home-use device 20 or 22. Specifically, the device control section 36 sends out a command signal to instruct the notification toward the notification device 58 that is the transmission destination. The notification device 58 receives the command signal from the digital ink server 12 via the network NW and the IoT hub 16.
In a step S19, the notification device 58 notifies the existence of the home-use device 20 or 22 or the detectable range of the handwriting sensor 52 to person Hm who is present nearby, using the output form according to the output function of the notification device.
As illustrated in
Then, as illustrated in
Then, as illustrated in
Thereafter, the operation returns to the step S11 in
The digital ink server 12 may carry out, concurrently with the first operation, reinforcement learning for the determination implement 70 (
Similarly, the digital ink server 12 may carry out similar learning (that is, cleansing processing of conditions) also for the primary determination. For example, in the case of carrying out the primary determination depending on whether or not plural individual conditions are all satisfied, correction, deletion, and addition of the individual conditions may be carried out in such a manner that the overall percentage of correct answers becomes higher. Alternatively, in the case of carrying out the primary determination by converting the satisfaction and non-satisfaction of the plural individual conditions to a score, weighting factors of the individual conditions may be adjusted in such a manner that the overall percentage of correct answers becomes higher.
As described above, the home UI system 10 that manages digital ink includes the plural state sensors 18 capable of detecting a state in the house 100 or a change in the state, and the plural home-use devices 20 and 22 that are arranged in the house 100 or that form part of the house 100 and that each include the handwriting sensor 52 capable of detecting handwriting by person Hm. The home UI system 10 also includes one or more notification devices 58 (notification units) that notify the existence of the home-use device 20 or 22 or the detectable region of the handwriting sensor 52 to person Hm, and the device control section 36 (controller) that instructs at least one notification device 58 to carry out the notification when it is determined that the notification is necessary from a detection result of one or more state sensors 18.
In an notification method and an notification program that use the home UI system 10, the digital ink server 12 as a computer carries out an instruction step (S18 in
The data analyzing section 34 (determining part) may determine whether or not a notification is necessary by using the determination implement 70 that uses at least one feature indicating a detection result of the state sensor 18 on the input side and that uses the determination value indicating whether or not a notification is necessary on the output side. The device control section 36 may instruct the notification according to the determination result by use of the determination implement 70. The determination implement 70 may be configured to allow the learning parameter group defining the calculation rule to be updated by reinforcement learning.
For example, a situation is contemplated in which plural members of the house 100 share one electronic pen 54. In this case, it is not possible to use identification information (hereinafter, pen ID) of the electronic pen 54 to identify the actual writing person. Thus, the home UI system 10 may be configured to perform the “second operation” to estimate the writing person without a separate input operation. The second operation will be described below with reference to a flowchart of
In a step S21 in
In the step S22, the data acquiring section 32 acquires state data of the home-use device 20 or 22 that is the acquisition source of the handwriting data acquired in the step S21. Specifically, the data acquiring section 32 collates the sensor information 42 of
In a step S23, the data analyzing section 34 calculates features from the state data acquired in the step S22. This calculation is an operation similar to the calculation made in the step S14 in
In a step S24, the data analyzing section 34 estimates the writing person who has inputted handwriting by using the features calculated in the step S23. For example, when a reference value of the feature is stored regarding each member of the house 100, the data analyzing section 34 quantitatively evaluates the degree of similarity of the feature regarding each member, and estimates that the member with the maximum evaluation value is the writing person. Alternatively, the data analyzing section 34 may carry out estimation with higher accuracy by using a discrimination implement 80, in which machine learning has been carried out.
The input layer 82 is a layer to which a feature relating to the state in which writing has been carried out is input, and is composed of plural units. This feature is an input vector including at least one kind of component in “handwriting features” relating to the form of handwriting, “time features” relating to the writing timing, “place features” relating to the writing places, “physical features” relating to the body of the writing person, “motion features” relating to the motion of the writing person, and other features.
The output layer 86 is a layer that outputs a group of labels of members of the house 100, and is composed of five units in the example of
In a step S25, the digital ink server 12 supplies the handwriting data acquired in the step S21 to the storage device 14 together with identification information of the writing person estimated in the step S24 (that is, user ID). Therefore, the handwriting data is accumulated in the handwriting information DB 38 in the state of being associated with the proper user ID.
Thereafter, the operation returns to the step S21 in
As described above, the home UI system 10 that manages digital ink includes the plural state sensors 18 capable of detecting a state in the house 100 or a change in the state, and the home-use device 20 or 22 that is arranged in the house 100 or that forms part of the house 100 and that includes the handwriting sensor 52 capable of detecting handwriting by person Hm. The home UI system 10 also includes the data analyzing section 34 (estimation unit) that, when an input of the handwriting is detected by the home-use device 20 or 22, estimates the writing person who has inputted the handwriting by using a detection result of one or more of the state sensors 18.
In an estimation method and an estimation program that use the home UI system 10, the digital ink server 12 as a computer carries out an estimation step (S24 in
For example, a use case is considered in which the same person Hm writes down multiple ideas as they come to mind while living in the house 100, each time on the nearest home-use device 20 or 22. However, if multiple pieces of information are fragmentally accumulated, this may not lead to any new creation. Thus, the home UI system 10 may be configured to perform the “third operation” to more effectively present the written information that has been written down by person Hm in the past. The third operation will be described below with reference to a flowchart of
In a step S31 in
In the step S32, the data acquiring section 32 reads out desired handwriting data from the handwriting information DB 38. Prior to this reading-out, the data acquiring section 32 identifies the second device ID of the terminal device 24 by referring to the second device information 46 and acquires one or more first device IDs associated with the second device ID. Thereafter, the data acquiring section 32 reads out, from the handwriting information DB 38, the user ID of the owner of the terminal device 24 and at least part of handwriting data associated with the first device ID. In the case of partially reading out the handwriting data, the data acquiring section 32 may impose a search condition relating to the creation time, creation place, and so forth of the handwriting data, or may extract a predetermined ratio (amount) of the handwriting data at random.
In a step S33, the device control section 36 identifies past written information that is a display target from the handwriting data read out in the step S32. Here, the “written information” is written-by-hand information indicated by handwriting, or content information that is specified by handwriting within electronic content such as within an electronic book. For example, the former handwriting corresponds to a written-by-hand note or an annotated comment, and the latter handwriting corresponds to an annotation including an underline and an enclosing line.
In a step S34, the device control section 36 instructs the terminal device 24, from which the predetermined operation has been received in the step S31, to display the written information identified in the step S33. Specifically, the device control section 36 sends display data used to display the written information toward the terminal device 24 that is the transmission destination. The terminal device 24 receives the display data from the digital ink server 12 via the network NW and the IoT hub 16.
In a step S35, the terminal device 24 displays the written information on a display screen by using the display data supplied together with the instruction in the step S34. The terminal device 24 may display plural pieces of written information, in turn, sequentially in time-series order, or may display plural pieces of written information, in turn, in random order. In the displaying, various display forms may be used (for example, size of characters, color, position, orientation, whether or not display of a time stamp exists, and so forth), and various control methods may be used (for example, display time, display switching cycle, visual effects, and so forth).
The written-by-hand information 111 of
Person Hm, by visually recognizing the display, realizes that it is technically feasible to incorporate a touch sensor into a window, which leads to creation of a “home UI system” as a new business model, in which a “house itself” functions as an input-output device of digital ink.
After the step S35 in
As described above, the home UI system 10 that manages digital ink includes the plural home-use devices 20 and 22 that are arranged in the house 100 or that form part of the house 100 and that each include the handwriting sensor 52 capable of inputting handwriting by person Hm. The home UI system 10 also includes the storage device 14 that stores handwriting data that indicates the form of handwriting in such a manner that the handwriting data is associated with the home-use device 20 or 22. The home UI system 10 also includes the terminal device 24 that is configured to display an image or video and is associated with one or more home-use devices 20 and 22. The home UI system 10 includes the device control section 36 (controller) that, when receiving a predetermined operation through the terminal device 24, acquires from the storage device 14 handwriting data of the home-use devices 20 and 22 corresponding to the terminal device 24, and that instructs the terminal device 24 to display written-by-hand information indicated by the acquired handwriting data or content information specified by the acquired handwriting data.
In a display method and a display program that use the home UI system 10, the digital ink server 12 as a computer carries out an instruction step (S34 in
In particular, when the terminal device 24 displays plural pieces of handwritten information or plural pieces of content information in turn at random, it becomes possible to successively present combinations of pieces of fragmented information that person Hm may not think of. Thus, organic linkage of pieces of information in various perspectives may be readily formed in the brain of person Hm, which leads to further creative activity of person Hm.
Person Hm, by looking back on memories in the past, may obtain physical and mental effects such as psychological comfort, activation of the brain, and a feeling of belonging to home. The home UI system 10 may be configured to perform the “fourth operation” to trigger recollection of the content of writing in the past. The fourth operation will be described below with reference to a flowchart of
In a step S41 in
In a step S42, the data analyzing section 34 carries out primary determination relating to whether or not a notification is necessary by using the state data acquired in the step S41. Specifically, the data analyzing section 34 analyzes the state data from multiple perspectives or in a time-series manner by using various analytical methods including sensor fusion, and checks whether or not a predetermined state of person Hm has been detected. Here, the “predetermined state” may mean, for example, the state in which person Hm relaxes near the home-use device 20 or 22. Examples of such state include: [1] the state in which person Hm sits in a chair or sofa, [2] the state in which person Hm stands still, [3] the state in which person Hm is viewing the specific home-use device 20 or 22, and so forth.
In a step S43, the data analyzing section 34 checks whether or not the determination result in the step S42 satisfies a primary determination condition. When the determination condition is not satisfied (step S43: NO), the operation returns to the step S41 and the steps S41 to S43 are sequentially repeated until the condition is satisfied. On the other hand, when the determination condition is satisfied (step S43: YES), the operation proceeds to the next step S44.
In the step S44, the data analyzing section 34 calculates plural features used for secondary determination from the state data acquired in the step S41 or state data additionally acquired. The data analyzing section 34 may calculate the same features as in the case of the step S14 in
In a step S45, the data analyzing section 34 carries out secondary determination relating to whether or not displaying to be described later is necessary, by using the features calculated in the step S44. For example, the data analyzing section 34 can determine whether or not displaying is necessary depending on whether or not plural individual conditions are all satisfied or through converting satisfaction and non-satisfaction of the plural individual conditions to a score. Alternatively, the data analyzing section 34 may carry out the determination with higher accuracy by using a determination implement 90 in which machine learning has been carried out. This determination implement 90 can employ a configuration similar to that of the determination implement 70 illustrated in
In a step S46, the data analyzing section 34 checks whether or not the determination result in the step S45 satisfies a secondary determination condition. Here, the data analyzing section 34 determines that the determination condition is satisfied when a determination value output from the determination implement 90 is larger than a threshold.
On the other hand, the data analyzing section 34 determines that the determination condition is not satisfied when the determination value is equal to or smaller than the threshold. When the determination condition is not satisfied (step S46: NO), the operation returns to the step S41, and the steps S41 to S46 are sequentially repeated until the condition is satisfied. On the other hand, when the determination condition is satisfied (step S46: YES), the operation proceeds to the next step S47.
In the step S47, the data analyzing section 34 selects, based on the analytical result in the step S42, one home-use device 22 that is present near person Hm and that has a touch panel display. Alternatively, the data analyzing section 34 may select the home-use device 22 through collation of the sensor information 42 and the first device information 44 similarly to the case of the step S17 (
In a step S48, the data acquiring section 32 reads out handwriting data of the home-use device 22 selected in the step S47 from the handwriting information DB 38. Written-by-hand information indicated by handwriting is included in the handwriting data. In the case of partially reading out the handwriting data, the data acquiring section 32 may impose a search condition relating to the creation time, creation place, and so forth of the handwriting data, or may extract a predetermined ratio (amount) of the handwriting data at random.
In a step S49, the device control section 36 sends out display data including the handwriting data read out in the step S48 toward the home-use device 22 selected in the step S47. The home-use device 22 receives the display data from the digital ink server 12 via the network NW and the IoT hub 16.
In a step S50, the home-use device 22 displays written information on a display screen by using the display data supplied in the step S49. Similarly to the case of the step S35 (
The home-use device 22 carries out displaying various content, in turn, at a constant slow cycle as with the slow cycle of a pendulum clock. Person Hm, by visually recognizing the display which does not appear to have any special meaning, may become attached to the home-use device 22 and be triggered to remember memories of the house 100. Person Hm may feel the sense of belonging to the house 100.
After the step S50 in
The digital ink server 12 may carry out, concurrently with the fourth operation, reinforcement learning for the determination implement 90 (
As described above, the home UI system 10 that manages digital ink includes one or plural state sensors 18 capable of detecting a state in the house 100 or a change in the state, and the home-use device 22 that is arranged in the house 100 or that forms part of the house 100 and that includes a touch panel display capable of inputting and outputting handwriting by person Hm. The home UI system 10 also includes the storage device 14 that stores handwriting data that indicates the form of handwriting in a manner such that the handwriting data is associated with the home-use device 22. The home UI system 10 includes the device control section 36 (controller) that, when it is determined that displaying is necessary based on a detection result of one or more of the state sensors 18, acquires from the storage device 14 handwriting data corresponding to the home-use device 22 located near person Hm, and instructs the home-use device 22 to display written-by-hand information indicated by the acquired handwriting data.
In the display method and the display program that use the home UI system 10, the digital ink server 12 as a computer carries out, when it is determined that displaying is necessary from a detection result of one or more of the state sensors 18, acquiring from the storage device 14 handwriting data corresponding to the home-use device 22 located near person Hm and carrying out the instruction step (S49 in
In particular, when the home-use device 22 displays plural pieces of written-by-hand information, in turn, at random, it becomes possible for person Hm to successively look back on memories of the past, which would trigger physical and mental effects such as psychological comfort and activation of the brain. Furthermore, by carrying out display in a mode that seems not to have a special meaning, person Hm may experience the sense of being present and belonging in the house 100 to remember memories of the house 100. The data analyzing section 34 (determination unit) may determine whether or not displaying is necessary by using the determination implement 90 that uses at least one feature indicating a detection result of the state sensor 18 on the input side and that uses the determination value indicating whether or not displaying is necessary on the output side. The device control section 36 may instruct displaying according to the determination result by use of the determination implement 90. This determination implement 90 may be configured to allow the learning parameter group defining the calculation rule to be updated by reinforcement learning.
The present disclosure is not limited to the above-described embodiments and various modifications are possible. Further, various configurations may be selectively combined as long as technical contradiction is not created.
Regarding the step S15 (
Regarding the steps S12 and S15 (
Furthermore, the data analyzing section 34 may carry out the primary determination by using a relatively smaller number of state sensors 18 and carry out the secondary determination by using a relatively larger number of state sensors 18. This can alleviate the communication load on a link with the state sensors 18. Similarly, the data analyzing section 34 may carry out the primary determination by using determination processing, in which the amount of calculation per instance of processing is relatively smaller, and carry out the secondary determination by using determination processing, in which the amount of calculation per instance of processing is relatively larger. This can alleviate the calculation load on the digital ink server 12.
Regarding the step S19 (
Regarding the step S48 (
The various embodiments described above can be combined to provide further embodiments. All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-073696 | Apr 2019 | JP | national |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2020/006006 | Feb 2020 | US |
| Child | 17489472 | US |
