Patent Application
20190253580
This application relates generally to augmented reality for electronic images associated with document processing devices. The application relates more particularly to providing multifunction peripheral device users, device administrators or device technicians with targeted information relative to the multifunction peripheral.
Document processing devices include printers, copiers, scanners and e-mail gateways. More recently, devices employing two or more of these functions are found in office environments. These devices are referred to as multifunction peripherals (MFPs) or multifunction devices (MFDs). As used herein, MFPs are understood to comprise printers, alone or in combination with other of the afore-noted functions. It is further understood that any suitable document processing device can be used.
MFPs are complex devices that touch on three classes of individuals. First are device users who rely on the MFP for printing, scanning, faxing, or any other user function provided by the device. A second class of individuals includes device administrators which may include someone who allocates user permissions or performs accounting for device usage. Administrators may also include individuals who perform routine maintenance, like stocking of consumables such as ink, toner or paper. The third class of individuals is service technicians who must periodically come on site to maintain, diagnose or repair devices. Each class has its own needs for device information.
A user may wish to know if ink, toner or paper is depleted to select an alternative MFP, replenish the consumable themselves or notify an administrator to do so. An administrator may further wish to know a page count for paper based document processing jobs. A service technician may wish to know a location of an MFP needing servicing, as well as error conditions associated with an MFP. It will be appreciated that some individuals may fill two or more of these roles.
In accordance with an example embodiment of the subject application, a system and method for augmented reality for office machines such as MFPs includes a processor and memory and a data interface. Multifunction peripheral data is stored and corresponds to a location of an identified MFP. Device location data corresponds to a location of a portable data device such as a smartphone or other portable digital device. Relative location of the MFP and the portable data device is determined. An image overlay corresponding to a property of the at least one MFP is displayed on the smartphone or other portable digital device.
Various embodiments will become better understood with regard to the following description, appended claims and accompanying drawings wherein:
The systems and methods disclosed herein are described in detail by way of examples and with reference to the figures. It will be appreciated that modifications to disclosed and described examples, arrangements, configurations, components, elements, apparatuses, devices methods, systems, etc. can suitably be made and may be desired for a specific application. In this disclosure, any identification of specific techniques, arrangements, etc. are either related to a specific example presented or are merely a general description of such a technique, arrangement, etc. Identifications of specific details or examples are not intended to be, and should not be, construed as mandatory or limiting unless specifically designated as such.
MFP device interaction by users, administrators and technicians is typically done via a user interface, such as a touchscreen display. Different interfaces may be supplied to different users based on their login credentials. A user may walk up to an MFP to make a copy, only to find that the device is out of paper or out of toner. An administrator may need to login with their administrative credentials to obtain access to page counts or user logs. A technician, frequently employed by a third party service provider, may be unfamiliar with a particular premises. The technician may also be unaware as to a location of a particular device on the premises, such as a device for which a service call was placed.
Proliferation of less expensive and more powerful portable data devices has led to widespread adoption. Such devices include smartphones, tablet computers, laptop computers, notebook computers, smart watches, intelligent eyewear, or other portable digital devices. A high percentage of workers in the workforce own one or more if these devices. Many carry their device, such as their smartphone, with them throughout the day. Modern portable data devices frequently include an embedded digital camera and a display, which may be a touchscreen display. Cameras can generate high resolution still images and video. Images are captured by the camera as directed by a user viewing the camera output on the device display.
Use of cameras associated with portable data devices has evolved beyond capture of still or video images. Cameras can serve as an input to function as a barcode or QR code reader. Cameras can also provide input for identification, such as via facial or other biometric recognition.
It is possible to determine a location of a portable data device by several different or complementary means. One such means relies on receipt of global positioning system (GPS) signals into a GPS input on the device from which its position can be calculated. Another means of location is via a cellular data connection. Rough location can be determined by knowing a location of a cell tower to which a cell phone is connected. Further refinement can be achieved by knowing a particular sector wherein the cell phone is connected. A cell tower may radiate three, 120° sectors in a horizontal, circular pattern. Still further refinement can be made by knowing a signal strength between the cell tower and the cell phone to approximate a distance between them. Still further refinement can be made by triangulation methods using multiple cell towers.
Other means for device location may rely on similar properties associated with connection to a wireless hotspot, a near field communication (NFC) signal, a Bluetooth signal or a beacon signal, such as with iBeacon technology from Apple, Inc. or Bluetooth low energy (LE) beacons.
Still further refinement of location can be accomplished using triangulation methods using multiple wireless connections with any of the afore-noted systems. Augmented Reality (AR) is a new technology used in conjunction with smart phone or device such as Glass eyewear to project digital information onto images of the physical world. Games such as Pokmon Go helped to popularize augmented reality games on mobile devices. Augmented reality is also used by Google, LLC, and others to provide services such as information about famous sites, statues, translations, and mapping information.
A reality application user interface is applied herein to office devices such as MFPs such wherein user, administrators or service technicians are provided with augmented reality information for any relevant information, such as device status, error information, troubleshooting information, consumable levels, and device health at a glance when viewing the actual physical device via camera feed or Glasses feed. Further, augmentation and visualizations can be differentiated by user role or user proximity to device allowing a customized experience. Visualizations are suitably interactive allowing user selection, such as using a touchscreen function on an augment display to invoke additional information, troubleshooting information, help, and tutorials that take into account the actual physical components of the device.
Suitable augmented reality aspects include:
A cloud-based service application contains device location based on Location GPS (or cell triangulation) in conjunction with metadata and state information including but not limited to the following:
A suitable augmented reality developers kit such as TANGO (developed by Google) or ARKit (developed by Apple iOS), allows for creation of a mobile applications that incorporate augmented reality visualizations as applied to the real world and detect surfaces, walls, and other objects. When trained to recognize MFPs, MFP components, or MFP location using machine learning and image matching or positioning, a system will recognize the MFP and suitably request associated cloud or device information including but not limited to consumable status, state, IP address, counter information, error codes, etc., as noted above.
User interaction with the mobile visualizations is implemented to access additional information. For example, when an error is shown (such as a paper jam), a superimposed status display suitably includes a clickable object to invoke troubleshooting information such as a video or step by step instruction on fixing the problem by recognizing and superimposing instruction on the actual component of interest. This helps service technicians as well as administrators in maintaining a healthy and working device.
Additionally, the type of information projected can depend on a user's role. For example, if a logged-in user is a service manager, they may see error code history. If the logged-in user is an on-premise administrator, they may see toner status, and if the logged-in user is a meter manager, they may see only meter data.
Visualizations are suitably invoked depending on relative proximity to device. As determined by beacon, GPS, or other directional device location system, visualizations can change depending on proximity. For example, if a portable data device is 20 feet away, an arrow or other alert would show only if state of device is not okay. If there is a device error, a visual indicator can be used to differentiate it from the rest of the devices without necessarily providing details.
Suitable identification of a device can also be made with indicia directly on the device, such as with written device name, such as “Printer 1,” or other visual identifier, such as a barcode or a quick response (QR) code. Device identification is suitably decoded from a code itself or from a server based or self-contained lookup table.
In accordance with the subject application,
Wireless data connection with smartphone 120 is also suitably via one or more Wi-Fi hotspots, such as via hotspots 136, 140 and 144. Location of smartphone 120 can be made in accordance with connection with one or more hotspots 136, 140, or 144 as noted above.
MFP 104 is suitably provided with one or more wireless data exchange devices, such Bluetooth 148, NFC 152 or beacon 156 facilitating determination of proximity or distance between MFP 104 and smartphone 120. Interaction between devices in
Turning now to
Processor 202 is also in data communication with a storage interface 208 for reading or writing to a storage 216, suitably comprised of a hard disk, optical disk, solid-state disk, cloud-based storage, or any other suitable data storage as will be appreciated by one of ordinary skill in the art.
Processor 202 is also in data communication with a network interface 210 which provides an interface to a network interface controller (NIC) 214, which in turn provides a data path to any suitable wired or physical network connection 220, or to a wireless data connection via wireless network interface 218. Example wireless connections include cellular, Wi-Fi, Bluetooth, NFC, wireless universal serial bus (wireless USB), satellite, and the like. Example wired interfaces include Ethernet, USB, IEEE 1394 (FireWire), Lightning, telephone line, or the like. Processor 202 is also in data communication with user interface 219 for interfacing with displays, keyboards, touchscreens, mice, trackballs and the like.
Processor 202 can also be in data communication with any suitable user input/output (I/O) interface 219 which provides data communication with user peripherals, such as displays, keyboards, mice, track balls, touch screens, or the like.
Also in data communication with data bus 212 is a document processor interface 222 suitable for data communication with MFP functional units. In the illustrated example, these units include copy hardware 240, scan hardware 242, print hardware 244 and fax hardware 246 which together comprise MFP functional hardware 250. It will be understood that functional units are suitably comprised of intelligent units, including any suitable hardware or software platform.
Turning now to
Referring now to
From the forgoing, it will be understood that augmented reality provides quick and updated information that can target a user's particular needs. Information is suitably updated or modified relative to distance to a device or devices. For example, multiple available devices can be shown at the same time from a distance, and differences can be revealed as a user gets closer. In the example, as a user gets closer, they may be informed as to a number of jobs ahead of their job on the devices or the different print speeds of the devices to allow a user to make their selection.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the spirit and scope of the inventions.
