Patent Application
20220264271
A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings that form a part of this document: Copyright eBay, Inc., All Rights Reserved.
Mobile applications executing on portable computing devices (e.g., mobile phones, tablets, laptops, smart watches, and the like) are often streamlined versions of popular desktop applications or web services. These applications often feature larger graphics, simpler layout, and a more touch-friendly interface. Despite the focus on making these applications simpler, it is still possible for mobile users to have problems with the applications. These problems may include confusion over how to properly utilize the features of an application, application errors, or errors in communication between a mobile application and a network service (e.g., a network based marketplace).
In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.
The description that follows includes illustrative systems, methods, techniques, instruction sequences, and computing machine program products that embody illustrative embodiments. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that embodiments of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.
Disclosed in some examples is a system and method for allowing a customer support agent to view and/or control a mobile application from a remote terminal. A user encountering problems with a mobile application may trigger a client help application which allows a customer support agent to view, or in some examples, view and control, the application on the user's mobile device using a server help application. In one example embodiment, the customer support agent may converse with the user of the mobile application simultaneously over a voice connection on the mobile device.
The help system may allow the customer support agent to view the problem that the user is experiencing and, in one example embodiment, perform a particular action on behalf of the user. Since the customer support agent may also have access to any back-end data, the customer support agent may have a full-system view of the problem. For example, where the application on the mobile device is a client-server application, the customer support agent may be able to access the server-based portions of the application.
In one example embodiment, the system may have an emulator running at the remote terminal that may receive data from the mobile application and that may mimic the application state and/or screen display of the mobile application. This may allow the customer support agent to view the user's mobile application experience. The emulator may emulate the mobile application by obtaining state information from the mobile application running on the client device and/or by obtaining command information submitted to the mobile application running on the client device (by, for example, a user of the client device). In one example embodiment, the emulator may emulate the mobile application by processing screen shots from the client device to determine the state of the mobile application running on the client device and/or by processing screen shots received from the client device to determine inputs (e.g., commands) provided to the mobile application running on the client device.
In one example embodiment, the system may have an emulator running at the remote terminal that may receive data from the mobile application and that may mimic the application state and screen display of the mobile application. This may allow the customer support agent to view the user's mobile application experience.
The customer support agent may issue commands to the mobile application on the mobile device. The commands may include inputting information to various user input fields, executing actions on behalf of the user, and the like. Commands may be entered by the customer support agent using a variety of different techniques. For example, the remote terminal may specify that certain mouse clicks, keyboard keys, touch screen inputs, drop down menus, command line interfaces, button presses, and the like initiate certain commands to be executed by the mobile application.
In one example embodiment, the client help application may be part of the mobile application itself. For example, the client help application may be a module or component of the software that the user is having trouble with and for which help is desired. In one example embodiment, the help application may be deployed to the mobile device and may process the screen captures (or intercept the screen commands, or otherwise send data about the state of the device to allow for viewing of the user's mobile application experience), and may inject commands into the mobile device at the request of the customer service agent at the remote terminal without being integrated into the mobile application. For example, the client help application may be a separate application, may be integrated into the operating system of the mobile device, and the like. The client help application may allow for helping users with multiple different applications. For descriptive convenience, the client help functionality will be referred to herein as the client help application regardless of whether it is part of another application or a stand-alone application.
Activating remote support using the client help application may be initiated in a variety of ways. In one example embodiment, various buttons and user input dialogues may be embedded in the mobile application to enable users to launch the remote help features. The user may access the help functionality via a gesture, a universal “hot spot” on the screen (e.g., holding a finger on a particular spot on the screen for more than a predefined time period to initiate help, or similar methods), a particular input into an accelerometer on the device (e.g., shaking the device in a certain manner or fashion), voice input (e.g., speaking “help” into the speaker of the device), automatically upon an application error, selection from a menu, launching an application, or the like.
In addition to being able to view the screen of the user and control the screen, the application may also send application state information and device state information to the remote terminal for viewing by the customer agent. Example device state information may include a memory dump (e.g., a stack or heap dump or trace), a list of the various processes running in memory on the mobile device (and associated data about those processes such as CPU utilization, memory allocation, memory addresses, execution parameters, and the like), operating system information, wireless data information (e.g., bandwidth, carrier, connection type, or the like), display screen information (e.g., resolution, etc.), device identifier (e.g., UDID), network traces, Wireless network information (e.g., cell or WiFi), or any other device information. Example application state information may include user preference information (e.g., user's language, a preferred network marketplace online site, user identification information, locale settings, time zone settings), application state information, application memory contents, the application's program counter, and the like.
The above example embodiments may be beneficially employed in a variety of mobile application environments. One example environment may include an online marketplace application. For example, the eBay mobile application developed by eBay, Inc. of San Jose, Calif., which allows users to buy and sell items on the eBay online marketplace from their mobile device.
In one example embodiment, the services of a customer support agent may be secured and the establishment of a voice communication session may be obtained automatically as part of establishing a connection between the customer support agent and the user. In one example embodiment, the voice session may be on the same mobile device as the application for which the user is requesting assistance. The server help software negotiates the screen capture parameters and captures a screen shot (or other data which allows for the creation of a representation of the screen of the application or of the mobile device) from the client help software and displays it to the customer support agent (operation 2020). The server help application may receive input from the customer support agent indicating that the customer support agent wishes to perform an action on the mobile device (operation 2030). In response to receiving the input from the customer support agent, the server help application may send the command to the client help application and the client help application may execute the command (operation 2040). For example, the client help application may emulate certain keyboard keystrokes and mouse actions to execute the command.
At operation 2050, the server help application may receive a screen update from the client help application. The server help application may update the screen for the customer support agent (operation 2060). The server help application method 2000 may repeat operations 2030-2060 while the session is active. Once the session is terminated, the server help application method 2000 may terminate. In one example embodiment, operations 2030-2040 may be executed in serial or parallel with operations 2050-2060. For example, the screen updates for the remote terminal and the command execution for the mobile application may be performed simultaneously or nearly simultaneously.
It will be appreciated that various confirmation prompts and security techniques (e.g., encryption) may be utilized to prevent an attacker from utilizing these techniques for improper purposes.
A control module 3070 may coordinate the activities of the client help application 3040 and may be responsible for establishing a connection with the server help application 3100.
The client help application 3040 may communicate with a server help application 3100 executing on one or more remote terminals (that are a component of assistance server 3090) over a network 3080. The mobile device 3010 may access the network 3080 in a variety of ways including wired connections (such as Ethernet or dial-up modem), wireless connections (2G, 3G, 4G cellular, WiFi, Bluetooth, or the like), and the like. The network 3080 may be or include portions of one or more of: the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a cellular network, a Plain Old Telephone (POTS) network, and the like.
The server help application 3100 may include a control module 3110 which may facilitate the establishment of a connection with the client help application 3040 and may provide the simulated display of the mobile device 3010 to one or more customer service agents at one or more remote terminals (assistance server) 3090. The control module 3110 may receive screen information (or other information which may be used to reproduce the display of the mobile device 3010 at the remote terminal (assistance server) 3090) from the client help application 3040 and instruct the output module 3130 to display the information. The control module 3110 may also receive input through the input module 3120 from one or more of the customer service agents which utilize one or more of the features of the server help application 3100. The control module 3110 may process this input and execute one or more instructions in response. As described above, in one example embodiment, these instructions cause the execution of one or more commands on the mobile device 3010. The control module 3110 may pass the commands to the action module 3060 of the client help application 3040 for execution via the output module 3130. An input module 3120 and the output module 3130 communicate with client help application 3040 over network 3080.
An Application Program Interface (API) server 4014 and a web server 4016 are coupled to, and provide programmatic and web interfaces respectively to, one or more application servers 4018. The application servers 4018 host one or more marketplace applications 4020 and payment applications 4022. The application servers 4018 are, in turn, shown to be coupled to one or more database servers 4024 that facilitate access to one or more databases 4026.
The marketplace applications 4020 may provide a number of marketplace functions and services to users that access the networked system 4002. For example, the marketplace applications 4020 may provide one or more background images. The payment applications 4022 may likewise provide a number of payment services and functions to users. The payment applications 4022 may allow users to accumulate value (e.g., in a commercial currency, such as the U.S. dollar, or a proprietary currency, such as “points”) in accounts, and then later to redeem the accumulated value for products (e.g., goods or services) that are made available via the marketplace applications 4020. While the marketplace and payment applications 4020 and 4022 are shown in
Further, while the network-based system 4000 shown in
The web client 4006 accesses the various marketplace and payment applications 4020 and 4022 via the web interface supported by the web server 4016. Similarly, the programmatic client 4008 accesses the various services and functions provided by the marketplace and payment applications 4020 and 4022 via the programmatic interface provided by the API server 4014. The programmatic client 4008 may, for example, be a seller application (e.g., the TurboLister application developed by eBay Inc., of San Jose, Calif.) to enable sellers to author and manage listings on the networked system 4002 in an off-line manner, and to perform batch-mode communications between the programmatic client 4008 and the networked system 4002. In other examples, the client machines 4010, 4012 may be buyer machines which access the networked system 4002 to buy products.
Also shown in
In one example embodiment, a request for a remote assistance session may be received (operation 5004). For example, a request for a remote assistance session may be received from the client machine 4010 or the client machine 4012. The remote assistance session may be granted and initiated (operation 5008). For example, a connection between the assistance server 4042 and the requesting client machine may be established via the network 4004, and a thread corresponding to one remote assistance session may be created.
A test may be performed to determine if a captured screen shot has been received (operation 5012). For example, a captured screen shot may be received from web client 4006. If a captured screen shot has not been received, the method may repeat operation 5012; otherwise, the received screen shot may be processed and may be displayed to the customer support agent (operation 5016). In addition, optical character recognition may be performed on the captured screen shot to identify the current state of the client application.
Based on the information provided by the captured screen shot, one or more actions may be determined (operation 5020). For example, if a user has not properly entered user identification information, an email address and the like, the customer support agent may determine that the correct information should be entered on behalf of the user. In one example embodiment, the assistance server 4042 may automatically determine a course of action. For example, the assistance server 4042 may determine that an email address entered by a user is incorrect and may determine that the correct email address should be entered on behalf of the user.
In one example embodiment, one or more commands to execute the determined action(s) may be generated and issued. For example, commands corresponding to certain keyboard keystrokes and mouse actions for entering an email address may be generated and sent to the client machine 4010.
In one example embodiment, a request for a remote assistance session may be received from a user (operation 5040). For example, a request for a remote assistance session may be received from a user at the client machine 4010 or the client machine 4012. The request for a remote assistance session may be issued (operation 5044). For example, a request for a remote assistance session may be issued to the assistance server 4042.
A test may be performed to determine if a remote assistance session has been granted (operation 5048). For example, a remote assistance session may be granted by the assistance server 4042. If a remote assistance session has not been granted, the method may repeat operation 5048; otherwise, a screen shot may be captured (operation 5052) and sent to the assistance server 4042 (operation 5056).
A test may be performed to determine if a command has been received or a screen change has been detected (operation 5060). If a screen change has been detected, the method 5050 may proceed with operation 5052. If a command has been received, for example, from the assistance server 4042, the command may be executed (operation 5064). The method 5050 may then proceed with operation 5060.
Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.
In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.
Accordingly, the term “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.
Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple of such hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).
The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules.
Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors may be distributed across a number of locations.
The one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).
Example embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Example embodiments may be implemented using a computer program product, for example, a computer program tangibly embodied in an information carrier, for example, in a machine-readable medium for execution by, or to control the operation of, data processing apparatus, for example, a programmable processor, a computer, or multiple computers.
A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
In example embodiments, operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations can also be performed by, and apparatus of example embodiments may be implemented as, special purpose logic circuitry (e.g., a FPGA or an ASIC).
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments deploying a programmable computing system, it will be appreciated that both hardware and software architectures merit consideration. Specifically, it will be appreciated that the choice of whether to implement certain functionality in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware may be a design choice. Below are set out hardware (e.g., machine) and software architectures that may be deployed, in various example embodiments.
The example computer system 7000 includes a processor 7002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory 7004 and a static memory 7006, which communicate with each other via a bus 7008. The computer system 7000 may further include a video display unit 7010 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 7000 also includes an alphanumeric input device 7012 (e.g., a keyboard), a user interface (UI) navigation (or cursor control) device 7014 (e.g., a mouse), a disk drive unit 7016, a signal generation device 7018 (e.g., a speaker) and a network interface device 7020.
The disk drive unit 7016 includes a machine-readable medium 7022 on which is stored one or more sets of data structures and instructions 7024 (e.g., software) embodying or used by any one or more of the methodologies or functions described herein. The instructions 7024 may also reside, completely or at least partially, within the main memory 7004, static memory 7006, and/or within the processor 7002 during execution thereof by the computer system 7000, the main memory 7004 and the processor 7002 also constituting machine-readable media.
While the machine-readable medium 7022 is shown in an example embodiment to be a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions 7024 or data structures. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions (e.g., instructions 7024) for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present inventive subject matter, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, including by way of example, semiconductor memory devices (e.g., Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
The instructions 7024 may further be transmitted or received over a communications network 7026 using a transmission medium. The instructions 7024 may be transmitted using the network interface device 7020 and any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a LAN, a WAN, the Internet, mobile telephone networks, Plain Old Telephone (POTS) networks, and wireless data networks (e.g., WiFi and WiMax networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such software.
Although the present inventive subject matter has been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader scope of the inventive subject matter. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof, show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.
Such embodiments of the inventive subject matter may be referred to herein, individually and/or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments.
Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.
All publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.
In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended; that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” and so forth are used merely as labels, and are not intended to impose numerical requirements on their objects.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.
This application is a continuation of U.S. patent application Ser. No. 16/790,660, filed on Feb. 13, 2020, which is a continuation of U.S. patent application Ser. No. 16/380,412, filed on Apr. 10, 2019 (issued as U.S. Pat. No. 10,609,531), which is a continuation of U.S. patent application Ser. No. 15/903,078, filed on Feb. 23, 2018 (issued as U.S. Pat. No. 10,278,040), which is a continuation of U.S. patent application Ser. No. 15/645,737, filed on Jul. 10, 2017 (issued as U.S. Pat. No. 9,936,332), which is a continuation of U.S. patent application Ser. No. 15/255,629, filed on Sep. 2, 2016 (issued as U.S. Pat. No. 9,706,332), which is a continuation of U.S. patent application Ser. No. 14/075,582, filed on Nov. 8, 2013 (issued as U.S. Pat. No. 9,436,428), which claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 61/724,180 filed Nov. 8, 2012. Each of the aforementioned applications is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61724180 | Nov 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16790660 | Feb 2020 | US |
| Child | 17734288 | US | |
| Parent | 16380412 | Apr 2019 | US |
| Child | 16790660 | US | |
| Parent | 15903078 | Feb 2018 | US |
| Child | 16380412 | US | |
| Parent | 15645737 | Jul 2017 | US |
| Child | 15903078 | US | |
| Parent | 15255629 | Sep 2016 | US |
| Child | 15645737 | US | |
| Parent | 14075582 | Nov 2013 | US |
| Child | 15255629 | US |
