Patent Application
20180335913
The present description relates generally to implementing drag and drop functionality on touchscreen electronic devices.
Drop and drop gestures enable moving or copying data from a source application to a destination application. For example, a user may drag a representation of a photo from a first application and drop the representation of the photo into a second application. The data corresponding to the photo may then be copied or moved from the first application to the second application in response to the dropping.
Certain features of the subject technology are set forth in the appended claims. However, for purpose of explanation, several embodiments of the subject technology are set forth in the following figures.
The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and can be practiced using one or more other implementations. In one or more implementations, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.
The subject system provides an architecture that enables drag and drop functionality, with security features, on touchscreen electronic devices.
The network environment 100 includes an electronic device 110, a server 120, and a server 122 in which the server 120 and/or the server 122 may be included in a group of servers 130. The network 106 may communicatively (directly or indirectly) couple, for example, the electronic device 110 with the server 120 and/or the server 122 and/or the group of servers 130. In one or more implementations, the network 106 may be an interconnected network of devices that may include, or may be communicatively coupled to, the Internet. For explanatory purposes, the network environment 100 is illustrated in
The electronic device 110 may include a touchscreen and may be, for example, a portable computing device such as a laptop computer that includes a touchscreen, a smartphone that includes a touchscreen, a peripheral device that includes a touchscreen (e.g., a digital camera, headphones), a tablet device that includes a touchscreen, a wearable device that includes a touchscreen such as a watch, a band, and the like, any other appropriate device that includes, for example, a touchscreen, or any electronic device with a touchpad. In one or more implementations, the electronic device 110 may not include a touchscreen but may support touchscreen-like gestures, such as in a virtual reality or augmented reality environment. In one or more implementations, the electronic device 110 may include a touchpad. In
The electronic device 110 may implement the subject system to provide drag and drop functionality via touchscreen. For example, the electronic device 110 may implement the example drag and drop architecture that is discussed further below with respect to
The server 120 and/or the server 122 may be part of a network of computers or the group of servers 130, such as in a cloud computing or data center implementation. The server 120, the server 122, and the group of servers 130 may store data, such as photos, music, text, web pages and/or content provided therein, etc., that may be accessible on the electronic device 110. In one or more implementations, the electronic device 110 may support a drag and drop operation that involves dragging and dropping a representation of a data item that is physically stored on the server 120 or the server 122 or one or more servers from the group of servers 130, such as an image file, text, a sound file, a video file, an application, etc.
As illustrated in
The touch input in the source application 230 may be identified by the electronic device 110 as the initiation of a drag session for a drag and drop operation based on one or more factors, such as, for example, the duration of the touch input. In an example, the touch input may correspond with a long touch gesture in which the touch input is held for at least a period of time on the touchscreen 210 of the electronic device 110. Upon detection of the long touch gesture associated with the initiation of the drag session, the electronic device 110 may cancel (or forgo processing) other current touch inputs in the source application 230 that are received by the electronic device 110. In an example, a hierarchy for touch inputs may prioritize the long touch gesture for initiating the drag session over other types of touch gestures that may be received during the drag session and the electronic device 110 may delay the processing of these other touches until detection of a touch release that corresponds with dropping the item into the destination application 240. In another example, the hierarchy for touch inputs may prioritize a long touch or press gesture such that this gesture overrides another gesture. Further, although
In one or more implementations, only the source application 230 may be displayed when the drag session is initiated and/or for the duration of the drag session. For example, the image 220 can be dragged from the source application 230 to a shortcut or other representation of the destination application 240. In one or more implementations, the destination application 240 may not be executing when the drag session is initiated and is executed or launched in response to the image 220 being dropped onto the representation of the destination application 240. In another example, the destination application 240 may be launch by hovering the image 220 over the representation of the destination application. In yet another example, the destination application 240 may be launched by another selection or touch input to launch the destination application 240.
As illustrated in
While the drag session is occurring and active, the source application 230 and/or the destination application 240 are responsive and may be interacted with, by the user, in other ways. In particular, the electronic device 110 supports multi-touch inputs by allowing other touch inputs to occur during the drag session. These other touch inputs may be received by a background process, as part of a drag and drop architecture, which is described in more detail by reference to
As illustrated in
As mentioned above, during the drag session, the source application 230 and/or the destination application 240 are responsive and may be interacted with by the user in other ways based on provided multi-touch support from the electronic device 110. As shown in
If the destination application 240 satisfies any security and/or data access policies associated with the data item corresponding to the image 220, the image 220 is allowed to be dropped into the destination application 240, such as into the document 250, and the data item corresponding to the image 220 is transferred from the source application 230 to the destination application 240. An example process of performing a data transfer as part of a drag and drop operation is discussed further below with respect to
In one or more implementations, a user can drag and drop (or perform another gesture such as a flick gesture) a representation of an application (e.g., an application shortcut) over or near a given portion of the display (e.g. a picture-in-picture area) provided in the touchscreen 210 of the electronic device 110, and the application will launch with the application window being displayed in the given portion of the display, such as, for example, in conjunction with the concurrent display of one or more other applications in one or more other portions of the display.
In one or more implementations, while the aforementioned data transfer is being performed, the source application 230 and/or destination application 240 may be interacted with, by the user, and/or placed into the background without disrupting the data transfer. Further, another different application may be launched or opened while the data transfer is being performed. The normal operations of the electronic device 110 therefore are supported while the data transfer is being performed without negatively impacting the data transfer and/or system performance.
In one or more implementations, the source application 230 and/or destination application 240 may implement a table or collection view, and therefore the image 220 may be inserted into a table or collection view of the destination application 240. An example drag and drop operation involving a table view is discussed further below with respect to
In one or more implementations, instead of dragging an image 220 corresponding to a data item from the source application 230 to the destination application 240, the user may drag a text selection from the source application 230 to the destination application 240. An example of performing a drag and drop operation for text selection is discussed further below with respect to
In one or more implementations, the electronic device 110 provides an animation for indicating that the image 220 is now part of the drag session, which may be provided after the detection of a specific gesture by the electronic device 110. In an example, the gesture may be a long touch gesture where the user presses down a finger on a representation of a data item (e.g., the image 220) and holds the finger there for a predetermined period of time. After the detection of this gesture, the electronic device 110 may, for example, perform an animation showing the image 220 lifting up from the user interface of the source application 230. The electronic device 110 may customize this animation to include any other type of animation for indicating that the image 220 is part of the drag session. In an implementation, the source application 230 may implement, specify, or provide the animation.
In one or more implementations, the electronic device 110 provides an animation for indicating that the image 220 is being dropped into the destination application 240, which may be provided after the detection of the completion of the drag gesture (e.g., when the user's finger 270 is lifted from the touchscreen 210 of the electronic device 110). For example, the electronic device 110 may perform an animation showing the image 220 scaling down and/or fading out from the user interface of the destination application 240. The electronic device 110 may customize this animation to include any other type of animation.
In one or more implementations, the drag and drop operation may be canceled based on a detected input (or set of inputs) from the user indicating a cancelation of the drag session and/or based on the electronic device 110 determining that the drop operation cannot be completed when the user lifts their finger to perform the drop operation (e.g. due to one or more security constraints that are discussed further below). For example, the electronic device 110 may detect that the user released the drag gesture within the source application 230 without moving the image 220 over to the destination application 240. In another example, the electronic device 110 may detect that the user moved the image 220 over to the destination application 240 but then moved the image 220 back to the source application 230 and released the drag gesture over the source application 230. After detecting the cancelation of the drag session, the electronic device 110 may provide an animation for indicating that the drag session is canceled. For example, the electronic device 110 may, perform an animation showing the image 220 scaling up and/or flying out from the user interface of the source application 230. The electronic device 110 may customize this animation to include any other type of animation.
In one or more implementations, the subject system allows for multiple data items to be dragged concurrently as part of the same drag session. The data items may be selected together when the drag session is initiated and/or additional items may be added to a drag session in progress. For example, while the user's finger 270 is dragging the image 220, the user may select, using another gesture such as a tap from another finger, the image 225 from the source application 230 to be included in the drag session. Thus, the subject system supports multi-touch input where multiple touch inputs, such as multiple fingers, may be concurrently detected. Further, the subject system allows for removing an item (s) from the same drag session. For example, the image 225 that was added to the same drag session may also be removed. The subject system also supports gestures for interacting with data items included in the same drag session. For example, the electronic device 110 may detect that the user performed a pinch to zoom in gesture, and may zoom in multiple items that have been included as part of the drag session. In addition, the electronic device 110 may detect that the user performed a pinch to zoom out gesture, and may zoom out multiple items that have been included as part of the drag session. The electronic device 110 can support other types of touch inputs for interacting with an item in a drag session. For example, the electronic device 110 may detect touch inputs from two fingers of the user which will enable the item to be rotated and/or scaled based on the two finger touch inputs.
The subject system provides support for including multiple data items as part of a drag session.
As illustrated in
For including another data item in a drag session, the electronic device 110 may detect a second initial touch input based on the user's finger 290 touching the touchscreen 210 of the electronic device 110. For example, a touch input may be a tap gesture detected based on the user's finger 290 touching the image 225 in the image library of the source application 230. Responsive to the second touch input, the electronic device 110 may add the image 225 as part of the drag session. Although a tap gesture is described as an example, the source application may utilize any gesture or type of touch input for adding a data item to an existing drag session. In this manner, multiple data items may be added to the same drag session prior to the user dragging the items to the destination application 240. The multiple items include in the drag session may be arranged in different ways. In an example, the items may be arranged in a stack in which the items are organized in a fan pattern or in a grid, or the items may be arranged substantially adjacent to each other.
In one or more implementations, the electronic device 110 may provide an animation to indicate that another data item has been added to the drag session. For example, the added item can be animated to move toward an existing drag session item or item stack.
As illustrated in
As illustrated in
If the destination application 240 satisfies any security and/or data access policies associated with the data items corresponding to the image 220 and the image 225, the images 220 and/or 225 are allowed to be dropped into the destination application 240, such as into the document 250, and the data items corresponding to the images 220 and/or 225 are transferred from the source application 230 to the destination application 240. If the destination application 240 does not satisfy a security or data access policy associated with the data item corresponding to the image 220 and/or the data item corresponding to the image 225, the drop operation may be cancelled in part, or in full.
In one or more implementations, the electronic device 110 may support multiple drags sessions that occur simultaneously while processing other touch inputs. For example, the user may launch another application during the multiple drag sessions and/or interact with another application without negatively impacting system performance. Further examples of multiple drag sessions in connection with multiple drag and drop operations are described below.
As illustrated in
In
The electronic device 110 may detect a first drag gesture caused by the user's finger 270 dragging across the touchscreen 210 of the electronic device 110. In this example, the drag gesture drags the image 220, as part of the first drag session, from the source application 230 to the destination application 240.
The electronic device 110 may detect a second drag gesture caused by the user's finger 290 dragging across the touchscreen 210 of the electronic device 110. In this example, the drag gesture drags a representation of the image 282, as part of the second drag session, from the source application 235 to the destination application 240.
As illustrated in
Similarly, the electronic device 110 detects the completion of the second drag gesture when the user's finger 290 is lifted from the touchscreen 210 of the electronic device 110. When the electronic device 110 detects the completion of the second drag gesture, the electronic device 110 determines whether the destination application 240 satisfies any security and/or data access policies associated with the data item corresponding to the image 282.
If the destination application 240 satisfies any security and/or data access policies associated with the data items corresponding to the image 220 and the image 225, the images 220 and/or 225 are allowed to be dropped into the destination application 240, such as into the document 252, and the data items corresponding to the images 220 and/or 282 are transferred from the source application 230 and/or the source application 235 to the destination application 240. In the example of
As illustrated in
For including a data item corresponding to the image 225 in a second drag session associated with the same source application 230, the electronic device 110 may detect a second initial touch input based on the user's finger 290 touching the touchscreen 210 of the electronic device 110. For example, a touch input may be a long touch gesture detected based on the user's finger 290 touching the image 225 for a predetermined period of time. Responsive to the touch input, the electronic device 110 may include the image 225 as part of a second new drag session associated with the source application 230.
The electronic device 110 may detect a first drag gesture caused by the user's finger 270 dragging across the touchscreen 210 of the electronic device 110. In this example, the drag gesture drags the image 220, as part of the first drag session, from the source application 230 to the destination application 240.
The electronic device 110 may detect a second drag gesture caused by the user's finger 290 dragging across the touchscreen 210 of the electronic device 110. In this example, the drag gesture drags the image 225, as part of the second drag session, from the source application 230 to a different destination application 245 (e.g., a presentation application or any application).
As illustrated in
Similarly, the electronic device 110 detects the completion of the second drag gesture when the user's finger 290 is lifted from the touchscreen 210 of the electronic device 110. When the electronic device 110 detects the completion of the second drag gesture, the electronic device 110 determines whether the destination application 245 satisfies any security and/or data access policies associated with the data item corresponding to the image 225.
If the destination application 240 satisfies any security and/or data access policies associated with the data item corresponding to the image 220, the image 220 is allowed to be dropped into the destination application 240, such as into the document 255, and the data item corresponding to the image 220 is transferred from the source application 230 to the destination application 240.
If the destination application 245 satisfies any security and/or data access policies associated with the data item corresponding to the image 225, the image 225 is allowed to be dropped into the destination application 245, and the data item corresponding to the image 225 is transferred from the source application 230 to the destination application 245.
As illustrated in
In
The electronic device 110 may detect a first drag gesture caused by the user's finger 270 dragging across the touchscreen 210 of the electronic device 110. In this example, the drag gesture drags the image 220, as part of the first drag session, from the source application 230 to the destination application 240.
The electronic device 110 may detect a second drag gesture caused by the user's finger 290 dragging across the touchscreen 210 of the electronic device 110. In this example, the drag gesture drags a representation of the image 282, as part of the second drag session, from the source application 235 to the second destination application 245.
As illustrated in
Similarly, the electronic device 110 detects the completion of the second drag gesture when the user's finger 290 is lifted from the touchscreen 210 of the electronic device 110. When the electronic device 110 detects the completion of the second drag gesture, the electronic device 110 determines whether the second destination application 245 satisfies any security and/or data access policies associated with the data item corresponding to the image 282.
If the destination application 240 satisfies any security and/or data access policies associated with the data item corresponding to the image 220, the image 220 is allowed to be dropped into the destination application 240, such as into the document 256, and the data item corresponding to the image 220 is transferred from the source application 230 to the destination application 240.
If the destination application 245 satisfies any security and/or data access policies associated with the data item corresponding to the image 282, the image 282 is allowed to be dropped into the destination application 245, and the data item corresponding to the image 282 is transferred from the source application 235 to the destination application 245.
In the example of
The drag and drop architecture 300 includes a drag and drop manager 310 which is configured to manage a drag session corresponding to a drag event between the source application 230 and the destination application 240. Multiple drag sessions are supported by the drag and drop architecture 300. In particular, the drag and drop manager 310 is configured to manage a separate drag session corresponding to a drag event between the source application 235 and the destination application 245. For explanatory purposes, the discussion below references the drag session associated with the source application 230 and the destination application 240; however, the discussion below can also apply with respect to the drag session associated with the source application 235 and the destination application 245.
In one or more implementations, the drag and drop manager 310 may be implemented as a user interface (UI) process, such as an application or daemon running on the electronic device 110 that has system-level privileges, includes a render context associated with the application that enables the application to draw or render over any user interface displayed on the touchscreen, and also allows the application to create drag sessions associated with drag events. In an example, the render context associated with the application (e.g., the drag and drop manager 310) is a transparent full-screen layer that sits on top of any user interface displayed on the touchscreen.
In one or more implementations, the drag and drop manager 310 manages drag item previews during the drag session. For example, while being dragged, an item from a source application may be provided as a preview corresponding to a graphical representation of the item. For example, depending on the type of item, the preview of the item may be a thumbnail image, a video clip, or any other appropriate graphical representation. As the item is being dragged from the source application into the destination application, the preview of the item presented in the destination application may be a different graphical representation than the preview in the source application. The drag and drop manager may provide an animation that transitions or morphs the different representations as the item is being dragged from the source application over to the destination application and vice-verse. Further, each preview in the source application 230 and/or the destination application 240 may be dynamically generated such that a preview transitions from a first type of graphical representation into a second type of graphical representation and so on while being presented within that application. In another example, the drag and drop manager 310 may not provide a preview for an item during a drag session, or only provide a preview when the item is dragged in the source application only or the destination only.
In an implementation, the drag and drop manager 310 may utilize a portal for providing the preview of the item in the destination application 240. A portal refers to a pixel-by-pixel reference to a GUI object specified by the source application 230 that enables the drag and drop manager 310 to access and process the specified GUI object for providing the preview of the item in the destination application 240. A portal therefore is not a replication of the application's specified GUI object. Instead, a portal “points to” or “references” the application's GUI object, such as in a render tree. Examples of a portal of this type are disclosed in more detail in U.S. Provisional Patent Application entitled “Core Animation Portals,” having Attorney Docket No. P34669US1, and filed on May 16, 2017, which is hereby incorporated by reference in its entirety for all purposes.
With reference to
As mentioned above, multiple drag sessions are supported by the drag and drop manager 310. A drag event associated with the source application 235 and the destination application 245 as described before by reference to
The drag and drop architecture 300 includes a touch event manager 320 that is configured to manage touch events when received through the drag and drop architecture 300. The touch event manager 320 may be implemented as a background process (e.g., daemon) executing on the electronic device 110 and is configured to receive all touch inputs coming into the subject system. The touch event manager 320, for example, can detect an initial touch input indicating a start of a drag event in a given source application (e.g., a long touch gesture in the source application 230 or 235) and forward the touch input to the drag and drop manager 310 for processing and creating a new drag session and its associated drag session identifier. Upon detection of the long touch gesture associated with the initiation of the drag session, the touch event manager 320 may cancel (or forgo processing) other current touch inputs in the source application that are received. In an example, a hierarchy for touch inputs may prioritize the long touch gesture for initiating the drag session over other types of touch gestures that may be received during the drag session and the touch event manager 320 may delay the processing of these other touches until detection of a touch release that corresponds with dropping the item into the destination application 240. In another example, the hierarchy for touch inputs may prioritize a long touch or press gesture such that this gesture overrides another gesture.
The touch event manager 320 can receive a request from the drag and drop manager 310 to generate a copy of the drag event in the form of a specialized drag event, or a detached drag event, that coexists with the drag event. The specialized drag event is provided to the drag and drop manager 310, which is further configured to manage the specialized drag event and receive new drag touches through the specialized drag event (e.g., corresponding to the new drag session). These new drag touches may be associated with the drag session identifier of the drag session. In one or more implementations, the touch event manager 320 provides an interface that utilizes interprocess communication (IPC) for the drag and drop manager 310 to supply touch inputs associated with a particular drag session identifier.
During the drag session, when touch inputs are received that have been flagged with the drag session identifier, the touch event manager 320 is also configured to perform hit testing of the drag event to determine whether the destination application 240 (or any destination application) is configured and/or authorized to receive the item from the drag event. Hit testing, as mentioned herein, refers to an operation for determining whether a location of a current touch input on the touchscreen of the electronic device 110 (or any electronic device) intersects a corresponding application on the screen. In at least an implementation, each touch input is hit tested to determine a corresponding application as a potential destination application for a dropped item. The hit testing may be rate limited to mitigate potential performance issues to the drag and drop architecture 300 and/or the electronic device 110. In an implementation, the touch event manager 320 may determine, based on the drag session identifier, respective locations of all touches for the drag session, compute a centroid of the locations of the touches, and perform hit testing on the location of the centroid to determine a potential destination application for a dropped item.
In one or more implementations, the touch event manager 320 associates a new unique identifier with a touch path of a given drag event and verifies that newly received touch inputs that are part of the drag event match existing touch inputs known to the touch event manager 320 using the new unique identifier. During the drag session, the touch event manager 320 may forward touch inputs flagged with the drag session identifier to the drag and drop manager 310, the source application 230 and/or the destination application 240. In one or more implementations, when receiving a touch input associated with the drag session identifier, the destination application 240 requests an XPC connection (e.g., a type of interprocess communication mechanism with sandbox features) with the drag and drop manager 310. The XPC connection may provide a sandbox environment, limiting the type of accessible information, for the destination application 240 to communicate with the drag and drop manager 310. For example, the drag and drop manager 310 will not release any data regarding one or more representations of the item until the drag event has ended.
Upon a touch release by the user's finger 270 at the end of drag gesture (indicating the end of the drag event) shown in
Using the touch event manager 320 to deliver touch events to the drag and drop manager 310 and the destination application 240 (or any destination application) advantageously enables the drag and drop manager 310 and the destination application 240 (or any destination application) to receive touch events with minimal latency (e.g., at effectively the same time). Further, synchronization with other touch events received by the destination application 240 (or any destination application) is provided to enable the destination application 240 to behave normally for any additional touch inputs that are not part of the drag event.
The drag and drop manager 310 may be configured to notify the destination application 240 that the drag event has ended and that the item may be dropped into the destination application 240, if accepted for receipt by the destination application 240. In one or more implementations, the destination application 240 may indicate to the drag and drop manager 310 that it wishes to receive the dropped item. For example, the drag and drop manager 310 may receive a request from the destination application 240 for information corresponding to the item associated with the drag event. The request may include, for example, the drag session identifier.
The drag and drop manager 310 may request additional information regarding the item from the source application 230, such as a list of available representations of the item. For example, each representation may be a different digital version of the item and the list may be sorted in an order of fidelity or level of quality. The drag and drop manager 310 may receive the additional information corresponding to the item from the source application 230 and may provide the additional information corresponding to the item to the destination application 240. The destination application 240 can then utilize the received additional information to initiate a data transfer for a particular representation of the item, which is discussed further below with respect to
The drag and drop manager 310 controls the flow of information to any destination application, and any request by the destination application may not be fulfilled until the drag and drop manager 310 has determined that the request should be allowed.
In one or more implementations, as a security feature the drag and drop architecture 300 may provide the destination application 240 with minimal or no information regarding the item being dragged until after the drag and drop architecture 300 verifies that the destination application 240 is authorized/configured to receive the item being dragged, and/or after the destination application 240 accepts receipt of the item being dragged. Accordingly, during the drag session the destination application 240 may be aware that an item is being dragged over it, but the destination application 240 may not have access to any specific information regarding the item being dragged.
As described in
As illustrated in
The drag and drop manager 310 sends, in response to the detected touch release, a message to the second application, the message including information describing multiple different representations of the item (406). For example, the source application 230 may provide a list including multiple representations of the image 220 in varying degrees of fidelity or quality (e.g., original image, PDF, PNG, JPG, plain text, etc.). Each representation in the list can be indicated in the form of a uniform type identifier (UTI), which is a respective text string to uniquely identify a given class or type of item. In one or more implementations, a destination application (e.g., the destination application 240) is responsible for providing UTI conformance information to indicate which representations of the item that the destination application may accept. An XPC connection established between the drag and drop manager 310 and the destination application 240 may be utilized to send the message including information describing the representations of the item.
To provide additional flexibility, in one or more implementations, the drag and drop architecture 300 provides a service that maps a file to a file type as specified in a corresponding UTI. A given application can utilize the service to register a new UTI or file type, and the application can also extend an existing UTI or tile type (e.g., to associate with other file(s)) using the service.
The drag and drop manager 310 receives, from the second application, a request for a representation of the item from among the multiple different representations (408). Although a single requested representation is discussed, it is appreciated that the destination application 240 may also request multiple representations of the item. In an example, the destination application 240 may choose a particular representation with the highest degree of fidelity provided by the source application 230. In another example, the representation of the item that is requested depends on context (e.g., a destination application that will be graphically rendering the received representation may choose a different representation than another destination application that will not be graphically rendering the received representation).
The drag and drop manager 310 sends, to the first application, the request for the representation of the item (410). The drag and drop manager 310 initiates a data transfer of the representation of the item from the first application to the second application (412). In an implementation, during the drag session, the drag and drop manager 310 may have a connection with the source application 230. However, the connection is not provided to the destination application 240 until after the drag session is completed in order to prevent an unauthorized or unintended data transfer from occurring, a security feature which is enabled by the drag and drop architecture 300 of
After the drag and drop manager 310 detects that the touch release has occurred, and the destination application 240 has requested the representation of the item, the drag and drop manager 310 may provide the connection (or an extension to an endpoint of the connection) to the destination application 240 to perform the data transfer of the representation of the item. In an implementation, as a security feature, the drag and drop manager 310 may also set a timeout for the data transfer to be completed and enforce the timeout (e.g., by closing the connection and/or stopping the data transfer) to prevent a long data transfer from occurring. The drag and drop manager 310 may detect when the data transfer is completed, and may tear down the connection between the source application 230 and the destination application 240 at that time.
During the data transfer, the drag and drop manager 310 may provide a placeholder preview that is a graphical representation indicating or showing a progression of the data transfer. After the data transfer is complete, the representation of the item may be displayed replacing the placeholder preview. An animation may be implemented that morphs the placeholder preview into the representation of the item.
Additionally, one or more other implementations for performing the data transfer may be provided. While the data transfer is occurring, the source application 230 may be placed into the background and over time could be restricted from accessing resources provided by the drag and drop architecture 300 which would negatively impact the data transfer. Further, data transfers for large files or files located on the network 106, e.g. on the server 120, may take a long period of time to complete.
To alleviate such issues, for example, a file provider may be provided to handle the data transfer. A file provider may be an extension (e.g., a non-UI background process or daemon) that provides files or data, and can be used to open documents from other containers (e.g., where files or data are stored, such as locally or on the server 120). In an implementation, the file provider may be included in a sandbox environment as a security feature.
The drag and drop manager 310 receives a request for a representation of an item (422). The drag and drop manager 310 sends a request for the representation of the item to the source application 230 (424). In response to the request for a particular representation of the item, the source application 230 can provide, to the drag and drop manager 310, a URL or link to a file provider that fulfills a data transfer of the representation of the item. The drag and drop manager 310 receives this URLor link to the file provider that fulfills a data transfer of the representation of the item (426). The drag and drop manager 310 can then send, to the destination application 240, this URL or link to the file provider (428). Subsequently, the destination application 240 can request the file or data from the file provider via the URL. Advantageously, the drag and drop manager 310 may end the drag session at this time and rely on the file provider to complete the data transfer for the requested data. The file provider, in response to the request from the destination application 240, can initiate a data transfer to the destination application 240 for the requested representation of the item.
In one or more implementations, the drag and drop manager 310 may create an access control list (ACL) in a database of the file provider to indicate a specific pair of processes (e.g., the source application 230 and the destination application 240) that share and/or can access specific file(s). The file provider may then check the ACL after receiving the request from the destination application 240 for that specific file to ensure that access is permitted to the file. In an implementation, a file coordination process (e.g., daemon or background process) may be provided to monitor the state of the data transfer, and notify when transfer is complete, or when one of the processes crashes, to the file provider. Once notified that the data transfer is complete, the file provider removes the ACL from the file provider database.
In one or more implementations, support is provided for editing in place a remote (or cloud) file or document that is located over the network 106 (e.g., on the server 120 or otherwise not local to the electronic device 110). This example provides coordination with the file provider to upload changes back to the server 120, or to a network or cloud of computers that includes the server 120. The destination application 240 can request to open the cloud file in place. The destination application 240 may receive a reference to the requested cloud file and make changes via the reference to the cloud file. The file provider may collect these changes and upload the changes back to the cloud to update the cloud file with the changes.
In one or more implementations, the drag and drop architecture 300 supports policy creation for excluding a destination application from receiving any dropped items and/or from receiving one or more specific types of dropped items. For example, data may be prevented from moving between a particular source application and a particular destination application. Further, certain policies may be provided to enable allowing drag and drop between only managed apps (e.g. in an enterprise configuration). In an implementation, such policies can be enforced by the drag and drop manager 310 based on information indicating that the source and destination applications are in the same managed configuration. Further, the source application may control how much and which metadata (e.g., information related to representations of an item) to expose to other apps with respect to a drag event. For a third party application, an entitlement check using an entitlement key and code sign may be provided to determine access to a dropped item.
The drag and drop architecture 300 provides support for adding and/or removing an item from an existing drag session. As mentioned above, each drag session may be assigned a drag session identifier, and each item included in a drag session may also be assigned its own unique identifier. The following discussion relates to an example process 430 for adding an item to an existing drag session, and an example process 440 for removing an item from an existing drag session.
During a drag session, the drag and drop manager 310 receives a request for adding an item to a drag session (432). The request for adding the item to the drag session may be from the source application 230 in
The drag and drop manager 310 determines an identifier of the added item (434). In an implementation, each item provided by the source application 230 is associated with a unique identifier, and this unique identifier may be included as part of the request from the source application 230 to add this item to the drag session.
The drag and drop manager 310 associates the identifier of the added item to the drag session (436). In an implementation, the drag session may include a data structure (e.g., an array or list, etc.) corresponding to one or more items that are included in the drag session. The drag and drop manager 310 may insert or include the identifier of the added item in such a data structure.
During a drag session, the drag and drop manager 310 receives a request for removing an item from a drag session (442). The request for removing the item to the drag session may be from the source application 230 in
The drag and drop manager 310 determines an identifier of the item requested for removal (444). In an implementation, each item provided by the source application 230 is associated with a unique identifier, and this unique identifier may be included as part of the request from the source application 230 to remove this item from the drag session.
The drag and drop manager 310 disassociates the identifier of the item requested for removal from the drag session (446). In an implementation, the drag session may include a data structure (e.g., an array or list, etc.) corresponding to one or more items that are included in the drag session. The drag and drop manager 310 may remove the identifier of the item in such a data structure.
The drag and drop manager 310 detects, for a drag and drop event, a touch release at an end of a drag gesture for dropping an item from a first application into a second application, the item corresponding to data (502). The first application may be the source application 230 and second application may be the destination application 240 where the image 220 is dropped into as shown in
The drag and drop manager 310 determines, using a data access policy, whether the second application has access to data corresponding with the item (504). The data corresponding to the item may be, for example, and image file, a video tile, a sound file, etc. For example, the data access policy may only allow access to the data corresponding to the item between managed applications. The drag and drop manager 310 therefore determines that the source application 230 and the destination application 240 are both managed applications (e.g., under the same enterprise configuration), and allows access to the item.
In an implementation, a drag and drop operation may be enabled only between apps having views associated with a same account type. In particular, depending on the current view of the destination app, a determination is made whether the view is associated with a managed or unmanaged account, and then determining whether drag events will be sent to the destination app based on whether item being dragged is associated with a same type of account and/or is being dragged from a view associated with the same type of account.
If the drag and drop manager 310 determines that the second application has access to the data corresponding with the item (504), the drag and drop manager 310 allows the drag and drop event to be completed (506). For example, the drag and drop manager 310 may facilitate the data transfer of the data corresponding with the item, such as is discussed above with respect to
If the drag and drop manager 310 determines that the second application does not have access to the data corresponding with the item (504), the drag and drop manager 310 denies completion of the drag and drop event (508). The access may be denied when both the source application 230 and the destination application 240 are not managed applications and/or the item being dragged is not associated with same type of account and/or is being dragged from a view not associated with the same type of account. After denying completion of the drag and drop event, the drag and drop manager 310 may end the drag session associated with the drag and drop event.
As illustrated in
The electronic device 110 may detect an initial touch input (e.g., a long touch gesture) based on a user's finger 270 touching the touchscreen 210 of the electronic device 110 that initiates a drag session. For example, a touch input may be detected based on a user's finger 270 touching text 234 displayed in the source application 235. While the user's finger 270 is still in contact with the touchscreen 210, the electronic device 110 may detect another touch input (e.g., a swipe gesture) selecting the text 234 to be part of the drag session, in some instances, a second touch input may not be required to select certain types of text. For example, the touch input corresponding to the long touch gesture may be utilized to select a hyperlink or URL provided in the source application 235.
In one or more implementations, the electronic device 110 may provide a platter graphical element (“platter”) corresponding to a graphical representation of the selected text. The platter, when provided for display, may blur the background as it is being dragged during the drag session. Different representations of the platter may be provided. For example, the platter may be presented with rounded corners, with a shadow, without a shadow, without a border, and/or further customized in any way. In addition, in example in which the selected text may include a large number of selected characters, the platter may truncate a number of characters for providing as part of the presented platter.
As illustrated in
As illustrated in
The drag and drop manager 310 detects a text selection gesture for text provided in a first application (602). For example, the text selection gesture may include one or more text characters provided in the first application. The drag and drop manager 310 selects the text characters for dragging based on the text selection gesture (604). In one or more implementations, if a selection only includes whitespace and one or more images, the drag and drop manager 310 can filter out the whitespace and only the images are selected to be dragged, or if the selection only includes whitespace the drag and drop manager 310 may decide to not select any text characters for dragging. Further, for a selection that includes text characters with markup information, the text characters may selected with markup or without markup by the drag and drop manager 310 depending on implementation.
The drag and drop manager 310 detects a drag gesture of the selected text characters from the first application into a second application (606). In an example, the first application can add to or modify the selected text that is being dragged such that the modified text will be dragged into the second application. For example, the first application may reverse the text or apply some other transformation to the selected text (e.g., transforming the selected text to an image).
The drag and drop manager 310 detects a touch release at an end of the drag gesture for dropping the selected characters at a position within displayed content in the second application (608). The drag and drop manager 310 inserts the selected text characters at the position within the displayed content in the second application (610).
In one or more implementations, the subject system may display a cursor above or below the text that is being dragged. The cursor may be used to indicate the precise location within the content of the second application where the text selection will he dropped. When the cursor is being displayed above the dragged text selection and the text selection is dragged to the bottom of the screen, the system may adaptively flip the cursor to the bottom of the text selection. Similarly, when the cursor is being displayed below the dragged text selection and the text selection is dragged to the top of the screen, the system may adaptively flip the cursor to the top of the text selection.
As illustrated in
The electronic device 110 detects a drag gesture based on the user's finger 270 moving across the touchscreen 210 of the electronic device 110. The drag gesture may include moving the item 720 to a position within the table view of the list of items provided in the destination application 740. As illustrated in
Upon completion of the drag gesture, the item 720 has been moved (or copied) from the source application 730 and inserted into the table view of the destination application 740 with the items 750, 752, 754, and 756. In particular, the item 720 has been inserted between the item 754 and the item 756 in in the list of items. The drag session has now completed. In one or more implementations, when inserting the item into the table view, the electronic device 110 may implement the state reconciliation of the process discussed below with respect to
In one or more implementations, if the item corresponds to a large file or if the item corresponds to a file being transferred from a remote location, such as the server 120, a placeholder image may be inserted into the location within the table view where the item was released until the transfer or the download of the file is completed. The placeholder image may indicate to the user that the file is still being transferred or downloaded. For example, the placeholder image may be, and/or may include, a progress bar that indicates the progress of the transfer or download.
In one or more implementations, the destination application may determine that the position within the table view at which the user dropped the item is not an appropriate position for inserting the item. For example, the table view may be sorted by a particular factor, such as alphabetically, and the user may have dropped the item at a position within the table view that is not consistent with the alphabetical sorting. In this instance, the destination application may retarget the item to an appropriate position within the table view, and the item may automatically move, and be inserted into, the appropriate position within the table view. In an implementation, during the retargeting of the item, a graphical representation of a “hole” at the position within the table view that the item would be inserted into may he displayed in the destination application, and when drop occurs the item would be animated as moving to the hole at the position within the table view.
As illustrated in
The electronic device 110 detects a drag gesture based on the user's finger 270 moving across the touchscreen 210 of the electronic device 110. The drag gesture may include moving the item 820 to a position within the collection view of items provided in the destination application 840. As illustrated in
Upon completion of the drag gesture, the item 820 has been moved from the source application 830 and inserted into the collection view of the destination application 840 with the items 850, 852, 854, and 856. In particular, the item 820 has been inserted in the same position as the item 854 was previously at in
The drag and drop manager 310 identifies an item being dragged over a first application, the first application including a table of items (or a collection of items), the items being arranged in the table in a first arrangement, and each item in the table corresponding to a graphical representation of a file (902). The drag and drop manager 310, responsive to the identifying, copies the first arrangement to generate a second arrangement that corresponds to an initial arrangement of the items in the table when the item is first identified as being dragged over the first application (904).
The drag and drop manager 310 updates the first arrangement to reflect changed positions of the items in the table resulting from, for example, another item being inserted into the table while the item is being dragged over the first application (906). For example, if a copy or download operation had previously been initiated with respect to the table view, additional items may be inserted as they are copied or downloaded, such as from the server 120. In one or more implementations, changed positions of the items may also result from, for example, an item being deleted from the table.
The drag and drop manager 310 updates the second arrangement to reflect changed positions of items within the initial arrangement of the items resulting from the item being released for insertion into a location within the table, the second arrangement being exclusive of the another item (908). For example, when the item is released for insertion into a location or position within the table, the positions of the items surrounding the insertion location may change to create space to insert the item.
The drag and drop manager 310 merges the updated second arrangement with the updated first arrangement to reconcile the changed positions resulting from the item being inserted into the table at the location based on the initial arrangement with the changed positions resulting from the another item being inserted into the table (910). In this manner, any updates to the table view that occurred during the drag session can be reconciled with the position where the item was inserted, e.g. based on the initial arrangement.
The bus 1008 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 1000. In one or more implementations, the bus 1008 communicatively connects the one or more processing unit(s) 1012 with the ROM 1010, the system memory 1004, and the permanent storage device 1002. From these various memory units, the one or more processing unit(s) 1012 retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The one or more processing unit(s) 1012 can be a single processor or a multi-core processor in different implementations.
The ROM 1010 stores static data and instructions that are needed by the one or more processing unit(s) 1012 and other modules of the electronic system 1000. The permanent storage device 1002, on the other hand, may be a read-and-write memory device. The permanent storage device 1002 may be a non-volatile memory unit that stores instructions and data even when the electronic system 1000 is off. In one or more implementations, a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) may be used as the permanent storage device 1002.
In one or more implementations, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as the permanent storage device 1002. Like the permanent storage device 1002, the system memory 1004 may be a read-and-write memory device. However, unlike the permanent storage device 1002, the system memory 1004 may be a volatile read-and-write memory, such as random access memory. The system memory 1004 may store any of the instructions and data that one or more processing unit(s) 1012 may need at runtime. In one or more implementations, the processes of the subject disclosure are stored in the system memory 1004, the permanent storage device 1002, and/or the ROM 1010. From these various memory units, the one or more processing unit(s) 1012 retrieves instructions to execute and data to process in order to execute the processes of one or more implementations.
The bus 1008 also connects to the input and output device interfaces 1014 and 1006. The input device interface 1014 enables a user to communicate information and select commands to the electronic system 1000. Input devices that may be used with the input device interface 1014 may include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output device interface 1006 may enable, for example, the display of images generated by electronic system 1000. Output devices that may be used with the output device interface 1006 may include, for example, printers and display devices, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information. One or more implementations may include devices that function as both input and output devices, such as a touchscreen. In these implementations, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Finally, as shown in
Implementations within the scope of the present disclosure can be partially or entirely realized using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. The tangible computer-readable storage medium also can be non-transitory in nature.
The computer-readable storage medium can be any storage medium that can be read, written, or otherwise accessed by a general purpose or special purpose computing device, including any processing electronics and/or processing circuitry capable of executing instructions. For example, without limitation, the computer-readable medium can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. The computer-readable medium also can include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.
Further, the computer-readable storage medium can include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, the tangible computer-readable storage medium can be directly coupled to a computing device, while in other implementations, the tangible computer-readable storage medium can be indirectly coupled to a computing device, e.g., via one or more wired connections, one or more wireless connections, or any combination thereof.
Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be realized as executable or non-executable machine code or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code. Further, instructions also can be realized as or can include data. Computer-executable instructions also can be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those of skill in the art, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without varying the underlying logic, function, processing, and output.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits, such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions that are stored on the circuit itself.
Those of skill in the art would appreciate that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, components, methods, and algorithms have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. Various components and blocks may be arranged differently (e.g., arranged in a different order, or partitioned in a different way) all without departing from the scope of the subject technology.
It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
As used in this specification and any claims of this application, the terms “base station”, “receiver”, “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” means displaying on an electronic device.
As used herein, the phrase “at least one of” preceding a series of items, with the term “and” or “or” to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase “at least one of” does not require selection of at least one of each item listed; rather, the phrase allows a meaning that includes at least one of any one of the items, and/or at least one of any combination of the items, and/or at least one of each of the items. By way of example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; any combination of A, B, and C; and/or at least one of each of A, B, and C.
The predicate words “configured to”, “operable to”, and “programmed to” do not imply any particular tangible or intangible modification of a subject, but, rather, are intended to be used interchangeably. In one or more implementations, a processor configured to monitor and control an operation or a component may also mean the processor being programmed to monitor and control the operation or the processor being operable to monitor and control the operation. Likewise, a processor configured to execute code can be construed as a processor programmed to execute code or operable to execute code.
Phrases such as an aspect, the aspect, another aspect, some aspects, one or more aspects, an implementation, the implementation, another implementation, some implementations, one or more implementations, an embodiment, the embodiment, another embodiment, some implementations, one or more implementations, a configuration, the configuration, another configuration, some configurations, one or more configurations, the subject technology, the disclosure, the present disclosure, other variations thereof and alike are for convenience and do not imply that a disclosure relating to such phrase(s) is essential to the subject technology or that such disclosure applies to all configurations of the subject technology. A disclosure relating to such phrase(s) may apply to all configurations, or one or more configurations. A disclosure relating to such phrase(s) may provide one or more examples. A phrase such as an aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to other foregoing phrases.
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration”. Any embodiment described herein as “exemplary” or as an “example” is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the term “include”, “have”, or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term “comprise” as “comprise” is interpreted when employed as a transitional word in a claim.
All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. § 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for”.
The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more”. Unless specifically stated otherwise, the term “some” refers to one or more. Pronouns in the masculine (e.g., his) include the feminine and neuter gender (e.g., her and its) and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.
The present application claims the benefit of U.S. Provisional Patent Application Ser. No. 62/507,199, entitled “DRAG AND DROP FOR TOUCHSCREEN DEVICES,” filed May 16, 2017, which is hereby incorporated herein by reference in its entirety and made part of the present U.S. Utility Patent Application for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 62507199 | May 2017 | US |
