Patent Application
20240036962
Typically, products and services deployed in a computing environment touch upon, inhabit, or interrelate with numerous disparate locations, devices, and/or entities that pertain to the product or service. Each of these disparate locations, devices, and/or entities may inhabit a compartmentalized ecosystem that is distinct from—and effectively walled off from—any of the other compartmentalized ecosystems operating in the computing environment. For example, an organization that develops a software product may inhabit a development ecosystem having a set of development tools and databases that are organized and optimized for the software development process. A user(s) of the software product may correspond to a set of computing systems, machines, or server devices configured for the efficient execution of the developed software product. Further, an administrator of the software product may correspond to a set of computing systems configured with operational or monitoring tools that may be used to administer the software product. Additionally, a customer relationship personnel may correspond to a set of systems or tools to manage the interactions with the users (e.g., using CRM systems, billing systems, etc.) in relation to maintaining sales and/or customer relationships.
Similarly, an organization providing services instead of products may have certain equivalent configurations focused on enabling the delivery of the services as compared to a product. The configurations may correspond to a geographical location for service delivery, scope of services delivered, and the like. In certain cases, users of the service may be either internal to the organization or may be operating in an environment external to the organization.
Embodiments of the invention provide an approach to automatically cluster events generated by a plurality of distinct ecosystems deployed in a connected environment in relation to a product. In operation, the clustered events may be correlated, and one or more of the correlated events that trigger a response may be identified as an actionable item. Based on a correlation between the actionable items, a hierarchical product parameter responsible for the actionable item may be identified. The hierarchical product parameter is an operational factor in a hierarchy of operational factors associated with the product, where the hierarchy of operational factors span over the plurality of distinct ecosystems operating in relation to the product. Subsequently, one or more entities across the distinct ecosystems may be notified of the actionable item, to initiate a required action. Further, techniques to detect operational factors associated with the product deployed in the distinct ecosystems are disclosed.
Further details of aspects, objects, and advantages of the invention are described below in the detailed description, drawings, and claims. Both the foregoing general description and the following detailed description are exemplary and explanatory, and are not intended to be limiting as to the scope of the invention.
The drawings illustrate the design and utility of various embodiments of the invention. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by reference numerals throughout the figures. In order to better appreciate how to obtain the above-recited and other advantages and objects of various embodiments of the invention, a detailed description of the present subject matter will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the present subject matter will be described and explained with additional specificity and detail through the use of the accompanying drawings.
Certain organizations may develop and deploy products and/or services in a connected computing environment. The connected computing environment in relation to a particular product and/or service of the organization includes one or more distinct ecosystems inhabited by user-end entities and developer-end entities of the particular product and/or service of the organization. Each of the one or more distinct ecosystems, while operating, produces a stream of data pertaining to events that describe an activity or change in the ecosystems. The events may generally be unique to each distinct ecosystem but may reference some entities that span the distinct ecosystems. The product and/or service and one or more entities associated with the product and/or service are common across all ecosystems and the events reference the common components of the product and/or service either directly or indirectly. The terms products and services have been used interchangeably throughout the specification.
Since the distinct ecosystems are typically isolated from one another, it is normally not possible to adequately and effectively perform analysis to obtain meaningful data that spans across the boundaries of the distinct ecosystems. Moreover, the stream of data pertaining to events pertaining to events generated by the distinct ecosystems would likely correlate to an extremely high volume of data. Therefore, any organization that seeks to analyze data from across the distinct ecosystems would be inundated with a high volume of events coming from many diverse sources.
Conventional approaches taken by software or service organizations face immense struggles to effectively identify events, extract value from the events, and respond efficiently to the events. For example, one conventional approach is to use a manual process to analyze data streams pertaining to activities occurring at the distinct ecosystems, where a significantly high level of human effort is required to be devoted to individually identify and analyze the activities in order to identify events and act on them appropriately. The manual process is, however, both costly and reactive rather than proactive, and tends to suffer from the inability of humans to be able to accurately and efficiently process huge amounts of data. The analysis based on the conventional approach thus tends to be incomplete, inaccurate, or just plain wrong in many cases. For instance, events such as test failures, customer login problems, service timeouts in data centers, etc. can be captured from the distinct ecosystems and could have a common root cause, where the root cause would possibly remain undetected if the human-based manual analysis approach is taken to analyze the events. Additionally, identification of one or more entities at the distinct ecosystems that may require a modification related to the correction of the root cause is difficult.
The present subject matter discloses techniques to automatically cluster events generated by a plurality of distinct ecosystems and to correlate them to identify product parameters across the distinct ecosystems responsible for an event. The plurality of distinct ecosystems may exist in relation to a product deployed in a connected environment. The present subject matter functions in two stages: a setup stage and a runtime stage. At the setup stage, the events received from the distinct ecosystems may be correlated to obtain an event correlation map. The event correlation map may provide a coherent converged picture of a data stream pertaining to events captured from the various distinct ecosystems. The event correlation map may include events correlated to form incidents representing the event necessary for indication to a user-end entity and/or a developer-end entity operating in one or more distinct ecosystems. The incidents are automatically triaged to identify tickets and the severity of the tickets is ascertained from an object associated with the incidents. The tickets refer to one or more incidents that may require certain cosmetic modifications. Further, the tickets that may reference underlying defects with the product or service are converted into issues. The incidents, tickets, and issues, referred to collectively as items are provided onto the event correlation map. Each item of the event correlation map may be associated with a plurality of objects providing a description of the activities leading to the generation of the item.
At the runtime stage, one or more of the items which trigger a response may be identified as an actionable item. Based on a correlation between the actionable items, a hierarchical product parameter responsible for the actionable item may be identified. The hierarchical product parameter may be understood as a parameter associated with a specific level of granularity in a hierarchy of operational factors associated with the product. The operational factors associated with the product may span over the plurality of distinct ecosystems operating in relation to the product deployed in the connected environment. The operational factors may be divided amongst user-end entities and/or developer-end entities responsible for causing them or fixing them. A person authorized to undertake an action at one or more computing systems at the specific user-end and/or developer-end entities may be notified of the actionable item. Upon being notified, the person may initiate one or more responses required to address the actionable item. Therefore, the intensive task of identifying actionable items and appropriately addressing the actionable items is made simpler and more efficient. In addition, the actionable items are assigned a priority level for action at the specific user-end and/or developer-end entities to allow critical actionable items to be addressed on priority. The priority level may be determined by the scope and level of severity determined based on one or more objects associated with the actionable items, explained in detail with respect to figures.
In another exemplary implementation, the present subject matter provides techniques to detect and interact with entities inside an organization. At the setup stage, one or more entities deployed in a connected environment with respect to a product of the organization may be auto-detected. The entities may be at least one of the user-end entities and developer-end entities in relation to one or more distinct ecosystems inhabited by such entities. Further, operational factors associated with the product deployed in the distinct ecosystems may be automatically detected by extracting information related to how the operations of the entities are executed and synthesizing the extracted information. A hierarchy of the operational factors may be obtained and one or more hierarchical product parameters in relation to the operational factors may be obtained for each specific ecosystem.
At the runtime stage, the present subject matter provides techniques to obtain visualization of the hierarchy of operational factors and hierarchical product parameters. Further, each hierarchical product parameter may be ranked for prioritization by the specific user-end and/or developer-end entities authorized for undertaking an actional in relation to the each hierarchical product parameter. While certain embodiments described below may pertain to products or services specifically directed to software, the present subject matter is not limited in its applicability to just software and is indeed applicable to any type of product or service.
The present subject matter addresses a significant problem of product lifecycle management, which is that the developer-end entities and user-end entities remain greatly disconnected from the operations conducted within the distinct ecosystems they inhabit. By removing barriers between the developer-end entities and user-end entities, code on the developer-end may be connected to the operations at the user-end. Consequently, in an example implementation, the developer-end entities may efficiently address production issues and user interactions on the user-end entities. Therefore, the entities are no longer required to work across silos of data generated by distinct ecosystems to meaningfully communicate and collaborate among each other in the connected environment. Accordingly, the inefficiencies in the product lifecycle management may be addressed.
The present subject matter is further described with reference to
The user station 102 may be communicably coupled to a processing unit 106, a memory 108, and a data store 110. The processing unit 106 enables the user station 102 to run at least one operating system and other applications and services. The processing unit 106, amongst other capabilities, may be configured to fetch and execute computer-readable instructions stored in the memory 108. The processing unit 106 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. The functions of the various elements shown in the figure, including any functional blocks labelled as “processing unit”, may be provided through the use of dedicated hardware as well as hardware capable of executing machine-readable instructions.
The functions, performed by the processing unit 106, may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processing unit” should not be construed to refer exclusively to hardware capable of executing machine readable instructions, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing machine readable instructions, random access memory (RAM), non-volatile storage. Other hardware, conventional and/or custom, may also be included.
The interface may include a variety of machine-readable instructions-based interfaces and hardware interfaces that allow the user station 102 to interact with different components. Further, the interface may enable the user station 102 to communicate with entities, for example, the entities operating in the various distinct ecosystems, web servers, and external repositories. The interface may facilitate multiple communications within a wide variety of networks and protocol types, including wireless networks, wireless Local Area Network (WLAN), Radio Access Network (RAN), satellite-based network, and the like.
The memory 108 may be coupled to the processing unit 106 and may, among other capabilities, provide data and instructions for generating different requests. The memory can include any computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic random-access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.
The operation of the unified analysis platform 104 involves analysis of large amounts of data streams captured from the distinct ecosystems to identify events occurring at the distinct ecosystems for processing by the unified analysis platform 104. The unified analysis platform 104 implements autonomous clustering of the events to generate items of interest to one or more entities operating in distinct ecosystems. The items of interest may be an event necessary for notification to the one or more operating in a plurality of distinct ecosystems as well as autonomous classification and communication between these items.
Any number or type of events generated by distinct ecosystems may be acted upon by embodiments of the invention. For example, the connected environment 100 includes a development ecosystem 112 having developer entities 114 and generating developer data 116, a user ecosystem 118 comprising user entities 120 and generating user data 122, an operation ecosystem 124 comprising operator entities 126 and generating operations data 128, a Customer Relationship Management (CRM) ecosystem 130 comprising CRM/Revenue-based entities 132, and generating CRM data 134. For the sake of understanding, the development ecosystem 112 and operation ecosystem 124 may be collectively referred to as developer-end ecosystems, and the user ecosystem 118 and CRM ecosystem 130 may be collectively referred to as user-end ecosystems.
In some embodiments, the unified analysis platform 104 uses machine-learning based techniques to automatically process the data and events generated by the distinct ecosystems to generate an event correlation map. In an example, the event correlation map may be generated by clustering and correlating the data generated by each distinct ecosystem in the connected environment 100. The unified analysis platform 104 stores both raw data generated by each distinct ecosystem and the event correlation map illustrating properly categorized data at the data store 110, so that the processing unit 106 can be used to accurately identify actionable items. The actionable items may be one or more items of interest on the event correlation map that may require a response trigger. In this way, a correlated view of the data pertaining to distinct ecosystems can be obtained, despite the fact that the data may be originated from numerous entities operating in the distinct ecosystems, and despite the fact that the collected data may correspond to an immense amount of data that may be captured.
The disclosed techniques, therefore, allow organizations to focus their efforts on the actionable items in their product and anticipate and trigger a required response for the actionable items before the actionable item may be reported. The present subject matter may be applicable to any organization that develops products that are delivered to end users in a manner that events captured during the development, deployment, and usage of the product may aid in monitoring the health of the product. The present subject matter outlines an application of the inventive method for the Software-as-a-Service (SaaS) industry, but it is not limited to this industry.
Embodiments of the present subject matter operate by capturing data streams from the distinct ecosystems within the connected environment 100 through a series of triaging stages. Objects associated with the captured data streams may be “funneled” through the series of triaging stages to generate the event correlation map containing meaningful sets of analyzed data. The raw data generated by the distinct ecosystems are likely to correspond to an extremely large volume of data, which individually by themselves may not provide enough cross-hierarchy context to be meaningful. The large volume of raw data, when analyzed in a unified way across the distinct ecosystems, may be triaged into increasingly smaller and more meaningful sets of data that can be usefully provided to developers, users, managers, and/or any other party that can benefit from a unified cross-hierarchy look at events and data within the connected environment 100.
In an example, the connectors 202 may provide bi-directional communications between the unified analysis platform and one or more entities operating in the distinct ecosystems. For example, the connectors 202 may be used with respect to entity CRUD (“create, read, update, delete”) operations. The rover(s) 204 may be used to implement pure collection activities in a single direction from the entities operating in the distinct ecosystems. For example, the rover(s) 204 may collect logs, alerts, jobs, and usage data from the entities operating in the distinct ecosystems. The APIs 206 may be used to implement programmatic access to the entities operating in the distinct ecosystems and pull data from the entities operating in the distinct ecosystems, for example, from developer-end entities.
The items collected by the data capturing tools may pertain to one or more events 208 indicative of an activity at the entities operating in the distinct ecosystems. The events 208 may be triaged in different levels to generate one or more items of interest. In an example, each level includes a triaging logic to generate increasingly meaningful and smaller sets of items, referred to as items of interest.
The events 208 may progress through a series of triaging logics applied to cluster and classify the events 208 into the one or more items of interest. The series of triaging logics may be implemented using machine-learning based techniques. For example, at a first level, events of interest (e.g., which are correlated across multiple distinct ecosystems) may be clustered into event clusters. Further, the event clusters may be classified to generate one or more incidents, such as incident 210-1, 210-2 . . . 210-N, collectively referred to as incidents 210. Incidents 210 may correspond to one or more events from the event clusters occurrence of which may be important for notification to the one or more entities operating in the distinct ecosystems. In a subsequent level, incidents that require human or machine interactions may be clustered into incident clusters, using the triaging logic. Further, the incident clusters may be classified to generate a ticket 212. The ticket 212 may be associated with a severity level indicative of how urgently an action on the ticket 212 may be required. Ticket 212 with a severity level beyond a threshold level may be acted upon by another triaging logic to create an issue 214. For example, an issue 214 may be created if a production and/or service change is deemed extremely necessary based upon the severity level of the ticket 212. One or more of the incidents, tickets, and issues that require a response trigger may qualify as an actionable item. In an example, the response trigger may include requirement of a notification to one or more authorized entities operating in the distinct ecosystems related with the actionable item.
It may be understood that steps of the method 300-1 may be performed by programmed computing devices and may be executed based on instructions stored in a non-transitory computer readable medium. The non-transitory computer readable medium may include, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media. The method 300-1 may be performed by a unified analysis platform 104 in communication with the processing unit 106, memory 108, and data store 110 communicably coupled with a user station 102.
At step 302, an event correlation map is extracted in relation to a plurality of events generated by a plurality of distinct ecosystems. In an example, an event is indicative of occurrence of an activity at one or more entities operating in the plurality of distinct ecosystems.
At step 304, a plurality of actionable items based on the event correlation map is identified. In an example, an actionable item is a response trigger associated with the one or more events from amongst the plurality of events.
At step 306, correlations among the plurality of actionable items are deduced.
At step 308, a hierarchical product parameter associated with each of the plurality of actionable items is identified based on the correlations. In an example, the hierarchical product parameter is an operational factor associated with the product within a hierarchy of operational factors.
At step 310, an entity from amongst the one or more entities operating in the plurality of distinct ecosystems in relation to the identified hierarchical product parameter is determined. In an example, the determined entity is authorized to undertake an action in relation to the identified hierarchical product parameter associated with each of the plurality of actionable items.
At step 312, an indication of each set of the identified hierarchical product parameter and the associated actionable items therefor is transmitted to the determined entity.
It may be understood that steps of the method 300-2 may be performed by programmed computing devices and may be executed based on instructions stored in a non-transitory computer readable medium. The non-transitory computer readable medium may include, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media. The method 300-2 may be performed by components of the conceptual schematic described in relation to
At step 314, one or more programming files are received from one or more entities operating in a plurality of distinct ecosystems.
At step 316, one or more constructs and metadata associated with the one or more constructs are extracted from the one or more programming files.
At step 318, the one or more constructs and the metadata are correlated to obtain one or more operational factors associated responsible for each construct.
At step 320, one or more entities operating across the plurality of distinct ecosystems associated with each operational factor are identified. In an example, the one or more entities may be identified based on the correlation between the one or more constructs and the metadata.
As is evident, central to various embodiments of the present subject matter are the products operating in distinct ecosystems and events generated at each distinct ecosystem.
For the sake of explanation,
The four quadrants are illustrated to include a set of concentric rings (depicted with dotted lines) which represent stages for triaging for events generated by the discrete ecosystems. A first ring 402 encompasses the greatest area of the connected environment 400, and represents event data generated by the discrete ecosystems. From the first ring 402, the events may be triaged using a triaging logic to form a second ring 404 representing incidents, which correspond to the next stage from events. A third ring 406 corresponds to the tickets, which is the next stage from the incidents. A fourth ring 408 corresponds to the issues, which is the next stage from the tickets.
As previously noted, the lower left quadrant represents the development ecosystem. The development ecosystem inhabits entities used by and interfaced with software development activities that are used to “build” a software product. Each level in an event correlation map (depicted by dotted concentric rings) may involve an entity responsible for the generation of the event and an object descriptive of the activity performed by the entity leading to the event. At an event level, a non-exclusive list of entities generating the event, referred to as event entity, may include service registry tools, security compliance analysis, continuous deployment tools, continuous integration tools, and/or version control systems. Examples of such entities that produce event data for the development ecosystem may include, GitHub, Gitlab, Git, CircleCL, Jenkins, Artifactory, and/or service registries (e.g., with regards to Kubernetes, consul, DNS, etc.). At an incident level, entities, referred to as incident entity, in the development ecosystem may include entities associated with, for example, integration engineers, project maintainers, and/or build engineers. Examples of incident objects are merge requests, fork/branch, pipeline/workflow milestones, job/stage completions, package creation, and/or artifact promotion/distribution. At a ticket level, entities, referred to as ticket entity, in the development ecosystem may be associated with, for example, test engineers. Examples of ticket objects are test failures, repo cloning, repo branching, pipeline updates, workflow tuning, job anomalies, security scan failures, artifact promotion, and/or canary rollout failures. At an issue level, entities, referred to as issue entity, in the development ecosystem may be associated with, for example, the actual developers themselves. Examples of issue objects are product bugs and/or unit test bugs. At a component level, a unified analysis platform, such as the unified analysis platform 104, may autonomously learn about components based at least in part upon, for example, directory structures for the code and/or unit test directory location. At a service/microservice level, the unified analysis platform 104 may autonomously learn about services based at least in part on, for example, service registries and/or load balancer proxies.
During an analysis stage, clustering and/or classification and/or communications operations may be performed upon the events for the development ecosystem. With regards to clustering, one or more of the following may be performed: (a) deduplication of incidents, (failed) workflows, and/or jobs; (b) correlation between upstream changes (e.g., in Git) and downstream failures (e.g., CI/CD failures); (c) anomaly detection; and/or (d) correlating code and test suites with components and (micro)services. With regards to classification, one or more of the following may be performed: (a) attribution; (b) prioritization; and/or (c) recommendation (e.g., garbage collection tickets and issues, promote tags to fields and demote fields to tags, hush noisy incidents, and detect alert spam). With regards to communications, one or more of the following may be performed: (a) context sensitive collaboration; (b) real time notification; and/or (c) cross-funnel propagation.
The lower right quadrant represents the operations ecosystem. The operations ecosystem inhabits entities used by and interfaced with activities that are used to enable “operation” of the product. At the event level, a non-exclusive list of entities, referred to as event entity, generating the event data for the operations ecosystem may include image repositories, cloud monitoring tools, service mesh, service registry tools, observability tools, and/or infrastructures (e.g., Kubernetes). Examples of event entities for the operations ecosystem may include public cloud outages, network provider outages, service registries, and/or logs and log analytics. With respect to the operations ecosystem, the incident entity may include, for example, network security engineers, cloud operations engineers, and/or site reliability engineers. Examples of incident objects are artifact promotion, service uptime/performance, mesh upgrade failures, network breaches, A/B testing data, canary/staged rollout data, DDoS attack information, and/or observability anomalies. A ticket entity may be entities associated with, for example, DevOps engineers. Examples of ticket objects are artifact promotion/deployment failures, ingress/regress CRUD, mesh upgrade failures, canary rollout failures, and/or A/B testing tasks. An issue entity may include entities associated with, for example, the actual developers themselves. Examples of issue objects are product scale and performance bugs, upgrade bugs, feature rollout bugs, security holes, service-to-service compatibility bugs, and/or API gateway bugs. At the component level, the unified analysis platform 104 may autonomously learn about, for example, correlations between user features and source code components/teams for clustering and classification of issues. At the service/microservice level, the unified analysis platform 104 may autonomously learn about, for example, correlations between customer features and service/teams for classification of tickets and issues.
During an analysis stage, clustering and/or classification and/or communications operations may be performed upon the events for the operations ecosystem. With regards to clustering, one or more of the following may be performed: (a) deduplication of incidents, DDoS events, observability tool alerts, and/or service restarts (b) correlation between upstream artifact promotions and downstream service failures (c) anomaly detection; and/or (d) correlating code and test suites with components and (micro)services. With regards to classification, one or more of the following may be performed: (a) attribution; (b) prioritization; and/or (c) recommendation (e.g., garbage collection tickets and issues, promote tags to fields and demote fields to tags, hush noisy incidents, and detect alert spam). With regards to communications, one or more of the following may be performed: (a) context sensitive collaboration; (b) real time notification; and/or (c) cross-funnel propagation.
The upper right quadrant represents the CRM ecosystem. The CRM ecosystem inhabits entities responsible for providing “support” to the operation of the product. At the event level, a non-exclusive list of entities, referred to as event entity, generating the event data for the CRM ecosystem may include content delivery networks, social media, application stores, mobile applications, and/or a client browser. Examples of the event entities for the CRM ecosystem may include browsers (such as Chrome, Safari, Firefox), app stores (such as Apple App Store, Google Play), social media (such as Facebook, Instagram, Reddit, Twitter), and/or content delivery networks (such as Akamai, CloudFlare, CloudFront). An incident entity may be associated with, for example, Li support engineers and/or a marketing analyst. Examples of incident objects pertain to client app/browser crash, app downloads, feature tracking, social comments, and/or chatbot requests. A ticket entity may be associated with, for example, L3 support engineers, sales engineers, and/or product manager. Examples of ticket objects pertain to a request for enhancement, a new feature request, regression in behavior/usability/performance/reliability, entitlement (e.g., licensing), API throttle, and/or user provisioning. An issues may be associated with, for example, the actual developers themselves. Examples of issue objects pertain to a request for enhancement, and/or regression in behavior/usability/performance/reliability. At the features level, the unified analysis platform 104 may autonomously learn about, for example, correlations between product features and code components for the classification of tickets and issues. At the capabilities level, the unified analysis platform 104 may autonomously learn about, for example, correlations between product capabilities and running services for the classification of tickets and issues.
During the analysis stage, clustering and/or classification and/or communications operations may be performed upon the events generated by the CRM ecosystem. With regards to clustering, one or more of the following may be performed: (a) deduplication of incidents, clickstreams, social media alerts, cascading failures, cloud outages, and/or feature request tickets (e.g., RFEs) (b) correlation between upstream artifact promotions and downstream customer tickets, cloud and network outages, and/or new feature rollouts; (c) anomaly detection; and/or (d) correlating code and test suites with features and capabilities. With regards to classification, one or more of the following may be performed: (a) attribution; (b) prioritization; and/or (c) recommendation (e.g., garbage collect tickets and issues, promote tags to fields and demote fields to tags, hush noisy incidents, and detect alert spam). With regards to communications, one or more of the following may be performed: (a) context sensitive collaboration; (b) real time notification; and/or (c) cross-funnel propagation.
The upper left quadrant represents the user/customer ecosystem, which inhabits entities responsible for actual working on the software product. At the event level, a non-exclusive list of entities, referred to as event entity, generating event objects for the user/customer ecosystem may include an organic clickstream, a billing system, payment gateways, website analytic tools, and/or CRM systems. An incident entity generating incident objects may be associated with, for example, finance managers, account managers, and/or product managers. Examples of incident objects pertain to shopping carts, paywalls, data for shipping/registration/payment/billing/invoicing, onboarding, retention, upgrades, purchase orders, upsell/cross-sell data, and/or customer references. A ticket entity generating ticket objects may be associated with, for example, finance managers, account managers, and/or product managers. Examples of the ticket objects pertain to data for churn/attribution/cancellation, credit card bounce, feature/API rollout, and/or catalog updates. An issue entity generating issue objects may include, for example, revenue recognition bugs, design enhancements, and/or usage of third-party APIs. At the features level, the unified analysis platform 104 may autonomously learn about, for example, correlations between product features and code components for the classification of tickets and issues. At the capabilities level, the unified analysis platform 104 may autonomously learn about, for example, correlations between product capabilities and running services for the classification of tickets and issues.
During the analysis stage, clustering and/or classification and/or communications operations may be performed upon the event data for the user/customer ecosystem. With regards to clustering, one or more of the following may be performed: (a) deduplication of incidents, clickstreams, provisioning events, cascading failures, collection days, and/or large customer onboarding; (b) correlation between upstream feature/catalog/partner/API rollout and downstream customer tickets; (c) anomaly detection; and/or (d) correlating code and test suites with features and capabilities. With regards to classification, one or more of the following may be performed: (a) attribution; (b) prioritization; and/or (c) recommendation (e.g., garbage collection tickets and issues, promote tags to fields and demote fields to tags, hush noisy incidents, and detect alert spam). With regards to communications, one or more of the following may be performed: (a) context sensitive collaboration; (b) real time notification; and/or (c) cross-funnel propagation.
The product or service 500 is the origin entity that the hierarchy of operational factors is based upon. The product or service 500 is defined as a unit of profit and loss, identity, onboarding and contracting, collectively referred to as billing/chargeback 510. The product or service 500 can have one or more capabilities 502-1, 502-N, collectively referred to as capabilities 502, which is the core unit a user-end entity interacts with. The product or service 500 may have an Application Programming Interface (API) namespace in case delivered as a software product or service and can have associated service level agreements (SLA). A simple way to think about a capability 502 is that the capability 502 may provide entities (noun) and associated activities (verbs) that can be performed on the entities being provided. Each capability 502 can have configurable items that may be instantiated as one or more features 504-1, 504-N, collectively referred to as features 504. A feature 504 is specific to a capability 502 and is not commonly directly interacted with by the user-end entity. Using the noun/verb reference, as taken above, a feature may define/provide traits (adjectives) specific activities at an entity.
The capabilities 502 and/or features 504 may link to backend constructs, termed microservices 506-1, 506-N, collectively referred to as microservices 506, and components 508-1, 508-N, collectively referred to as components 508, most commonly, if the product/service 500 is a software product or service. A microservice 506 is a unit of deployment and may be an operational factor which composes a feature 504 or capability 502. In an example a feature 504-N may be composed by microservice 506-1. In another example implementation, a capability 502-N may be composed by a combination of microservices 506-1 and 506-N. At a further level in the hierarchy of operational factors is a component 508. The component 508 may be meant to be part of the entity, but not the entity as a whole.
In an example, data pertaining to activities undertaken at one or more entities operating in the distinct ecosystems in a connected environment may be collected. The data may be obtained from a user 602 operating in relation to a user-end entity through a browser 608. The user 602 may also be providing information through an application 610 being executed at the user-end entity. The user 602 may also interact with the unified analysis platform through Application Programming Interfaces (APIs) 612. Above-mentioned are exemplary conduits through which the user 602 may interact with the unified analysis platform, for example to provide information and/or to operate the unified analysis platform. Further, the user 602 may interact with the unified analysis platform to, for example, query the unified analysis platform, obtain notification communication, perform analytics, etc. In some embodiments, the browser 608 and application 610 may also communicate to the unified analysis platform using the APIs 612. For example, the application 610 may include hooks to publish events into the unified analysis platform, e.g., using the APIs 612.
Further, data generated by one or more entities operating in the connected environment is captured into the unified analysis platform using any suitable data capturing tools. In one example implementation, rover(s) 604 and connector(s) 606 may be implemented to capture data from entities operating in the connected environment. For example, connector(s) 606 may be used to collect data regarding development activities, events, and entity data from a development ecosystem. In addition, rover(s) 604 may be implemented on user ecosystem to gather real-time events that occur based upon user activities. As such, one or more databases that may be populated by the connector(s) 606 are part of a bulk store 624 in the system architecture. The events, incidents, tickets, and issues are stored as part of the object data store 626 in the system.
The rover(s) 604 may be implemented for each possible entity operating in the distinct ecosystems. The rover(s) 604 may collect events from the one or more entities into a centralized database (not shown) either by pulling through API calls from the unified analysis platform or by pushing from the one or more entities by adding a few lines of code that sends the data directly to the centralized database.
In parallel to rover(s) 604 and to supplement the data in events, connectors 606 pull in information about various systems of records that are used by the entities operating at each distinct ecosystem. The system of records are de-duplicated in an entity database (not shown) of the system architecture. The entity database also records the relationships between these entities.
In some embodiments, the rover(s) 604 may operate with a “pull” approach, which provides a one-way path to obtain information into the unified analysis platform and operates as a data-feed into the platform. This approach to obtaining information works very well with entities from which data just need to be consumed without necessarily needing synchronization in the other direction, such as a log file or message store. In contrast, the connector(s) 606, is bi-directional which synchronizes data between entities in the distinct ecosystems at the unified analysis platform, and which permits information to be published in both directions. The connector(s) 606 therefore requires a greater level of access, which may necessitate the establishment of privileges, creation of an ongoing session, and/or permission to create or update entities. To illustrate the different scenarios in which the rover(s) 604 may be used instead of the connector(s) 606, or vice versa, consider a software development organization that may use a software source control system to manage its coding projects. If there is a desire to merely publish events to the unified analysis platform, then a one-way pull component such as the rover 604 may be used to capture information from that source code control system. On the other hand, the user 602 may desire the ability to have the unified analysis platform be able to open, update or close actionable items that are created in the source control system, based upon analysis that occurred using the unified analysis platform. In such case, the connector 606 may implement the two-way communications that may allow the actionable items that may be stored in the object data store to be created, updated, or closed.
The system architecture includes a platform front-end 614 enabling the entities operating in the distinct ecosystems to interface with the unified analysis platform. The platform front-end 614 handles requests and information from external locations, and sends responses back to the external locations. The platform front-end 614 captures event information from the rover(s) 604, connectors 606, and users 602, and directs the captured information to the appropriate processing and/or storage location within the unified analysis platform 104. In an example implementation, the processing locations may be one of machine learning engine 632 and stream processing engine 634. Further, in the example implementation, the storage locations may be one of the bulk store 624, object data store 626, and analytics data store 628.
In the platform front-end 614, caching tier(s) 618 may include one or more cache components (e.g., memory, SSDs) to cache data received from the entities operating in the distinct ecosystems in an in-bound direction and/or communications intended to be sent to the entities operating in the distinct ecosystems in an out-bound direction. The caching tier 618 is useful, for instance, to efficiently interact with data without necessarily going back to the entity which was the source of that data, for each subsequent interaction of the unified analysis platform with the data. By way of example, in case there is an actionable item in the object data store 626, a notification of which may be required to be sent to the entities operating in the distinct ecosystems. Rather than constantly accessing that actionable item from the object data store 626, the information can be instead cached in the caching tier 618.
The notification services 620 provide notification to the entities operating in the distinct ecosystems, e.g., using the operation of the connector 606 to update the entities operating in the distinct ecosystems, direct information to the browser 608 or application 610 for the user 602, or provide a push notification, for example, notifications about updates to actionable items. The notification may not only be to developer-end entities (e.g., engineers or support personnel for a software development company), but may also be made to the user-end entities (the end-user of the product created by the software developer that is the user of the unified analysis platform).
One or more web service tier(s) 616 may be employed to act as API gateways to handle API requests and to handle both content subject matter and routing functionality. For example, with regards to context, the web service tier(s) 616 will add context to requests with augmentation, e.g., by pulling the appropriate JavaScripts or HTML pertinent to a given request. With regards to routing, the web service tier(s) 616 may send captured data to the appropriate processing and storage locations of the system architecture. For instance, events and logs are sent to the stream processing engine 634, large data items such as attachments are sent to/from the bulk store 624, reporting items are sent to/from the analytics data store 628, primary I/O items are sent to/from object data store, and search/indexing information are sent to/from an indexing module 622.
The machine learning engine 632 refers to a collection of machine learning modules that are listening to various events generated by the entities operating in the distinct ecosystems and interacting with the object data store 626. The machine learning modules maintain their transient state using an in-memory database 642, referred to as In-mem database 642.
Overall, the machine learning engine 632 is responsible for three primary tasks, namely clustering 636, classification 638, and notification 640. Clustering 636 refers to the aggregation of multiple similar events or objects to a single meta-event or meta-object that captures some underlying commonality among these events or objects. Classification 638 refers to the assignment of an attribute of an event based on analysis of past assignments by the machine learning engine 632 or one or more entities in the distinct ecosystems authorized to undertake an action on the one or more entities. Finally, notification 640 refers to the act of sending out an indication to an entity that has registered interest in a certain class of events. However, in some embodiments, the indication is only sent when a condition is fulfilled. The condition may be said to be fulfilled when the event exceeds some threshold in terms of the rate of the event or the attributes of the event that would indicate an anomaly that is worth investigating.
Object data store 626 refers to a database of objects. The schema of the objects is described later in the present specification. In an example, the object may relate to one or more items generated at the entities operating in the distinct ecosystems. In another example, the object may relate to one or more actionable items identified from the items generated at the entities, wherein the actionable items may refer to items on an event correlation map that require a response trigger. A CDC (Change Data Capture) arrow from object data store 626 to a stream processing engine 634 refers to a process that captures incremental changes from the object data store 626 and puts them on a logical queue for further processing using streaming algorithms at the stream processing engine 634.
A CDC+ component 630 is one of the consumers of the CDC data stream. The CDC+ component 630 performs multiple functions with the received CDC data stream. The link between CDC+ component 630 and the indexing module 622 refers to the process of creating an index of the objects in the object data store 626 for faster retrieval. As objects change, the CDC+ component 630 incrementally updates the information in the index. The arrow from the CDC+ Component 630 to the bulk store 624 refers to the process of archiving some or all of the objects in the object data store 626 for long-term storage. As objects are changed or updated, the updates are propagated to the object data store 626. The arrow from CDC+ component 630 to the analytics data store 628 refers to the process of using incremental updates of the object data store 626 to maintain the results of expensive analytics queries without re-executing them from scratch.
The CDC+ component 630 also allows for storage of enhanced version history which can be used for a variety of use cases including audit history, change invalidation/roll back and the ability to compose a snapshot of the unified analysis platform and/or object at a specific point in time. With these composable snapshots, the unified analysis platform can restore an object or data to a specific point in time for troubleshooting or roll back.
The embodiments of the present subject matter are compatible with all event types that are relevant for the functioning of a product/service. The event types may include, but not restricted to, (a) Development: Build, test, and deploy events from tools such as GitHub, GitLab, CircleCI, Artifactory, Envoy, and Istio; (b) Operations: Operate, and manage events from tools such as DataDog, Splunk, AppDynamics, NewRelic, Prometheus; (c) User: Customer engagement, including omni-channel communication (voice, video, email, chat), and events from social media, client browsers; and/or (d) CRM: Customer journey events including clickstream events from tools such as Salesforce, Pendo, Gainsight, Mixpanel, Adobe CDP, Clickstream and Segment.
The collected events may be clustered into incidents that are classified as good or bad. No human intervention is required for creating incidents. The presence of multiple similar events that occur close in time and or have similar content is sufficient to help create the incidents. The incidents are presented to one or more entities operating in distinct ecosystems authorized to undertake an action in relation to the incidents. Based on feedback by the one or more authorized entities, the notion of similarity may be tuned. The present subject matter is compatible with any notion of similarity, and the present subject matter may be implemented using any suitable similarity algorithms, such as for example, Bloom filters, vector space embeddings or other approximate matching algorithms such as edit distance, but it is not restricted to them.
Incidents are further clustered into tickets that are given a priority and assigned to the one or more authorized entities operating in the distinct ecosystems in relation to the tickets. Some embodiments of the present subject matter implement ticket clustering by considering an evidence trail, performing correlation between different types of incidents, and/or performing prioritizations. With regards to an evidence trail, the creation of a ticket is based on the clustering of multiple incidents which in turn are created from events. The generated ticket presents a clustering history as a justification of the ticket. The most relevant information from the events that are part of the ticket is included in the ticket to make triaging easier. For example, deployment related tickets may include snippets of logs, traces and alerts which are related. With regards to correlation between different types of incidents, incidents that are created in different parts of the software organization are correlated together when a ticket is created to ensure that it is assigned to the correct person and given the appropriate priority. For example, issues that are created from fatal errors in the operations ecosystem may be linked to login delay issues in the development ecosystem and these may in turn link to errors in testing that are in turn linked to code commit events. Thus, an assignment of a ticket to a developer-end entity can be explained in terms of the correlation between incidents. With regard to prioritization, any suitable type of prioritization may be used. For example, prioritizations may be based at least in part on revenue impacts. The correlation of incidents from the distinct ecosystems allows the determination of the impact of the incidents to entities operating in relation to the distinct ecosystems and based on billing data from the CRM entities, a revenue number is assigned to the ticket. This allows the priority to be directly linked to the software organization's revenue.
Finally, after a root cause analysis is performed by a triaging logic on the tickets, some of the tickets are combined into issues. An issue represents a recurring long-term failing in a product or service offered by the organization that needs a systematic redesign. The events, incidents, tickets, and issues that require a response trigger from the one or more entities operating in distinct ecosystems may be referred to as an actionable item.
At step 704, the data is added to a queue for further processing. In an example, the data is placed into one or more event queues for consumption by the machine learning engine for further processing. In the system architecture discussed in
At step 706, an event correlation map is generated based on the processing of the data. In an example, the processing may be performed using machine learning-based techniques. The processing may be performed by implementing a series of triaging logics implemented at different levels to funnel the data into more meaningful sets of analyzed data. The raw event data from the multiple distinct ecosystems correspond to an extremely large volume of data, which is analyzed in a unified way across the distinct ecosystems, and is used to form increasingly smaller and more meaningful sets of data.
At each granular level of the triaging, clustering 708 and classification 710 actions are performed upon the data. The clustering and classification actions may occur at multiple granular levels. While the
The general idea is that event data from multiple distinct ecosystems may be received from entities inhabited by the distinct ecosystems, where clustering analysis is performed in a unified manner across that collected data such that cross-ecosystem analysis is performed upon that data. With regards to triaging, triaging logic is implemented at different levels of granularity, and each level of granularity reduces the overall data to a smaller and more meaningful set of data for the next level of granularity, and a different triaging logic may be used at each level of granularity.
At step 712, a plurality of actionable items may be identified based on the event correlation map. In an example, the transmission of an indication of the actionable item is performed to send out one or more messages to the one or more entities operating in distinct ecosystems. Each type of notification may be specific to a given entity that has registered interest in a certain class of events. For example, a first type of notification may be sent to the developer-end entities with regards to issues pertaining to software coding issues. A second type of notification may be sent to a user-end entity, who may be actually utilizing the deployed product.
In particular, one or more rovers 802-1, 802-N, collectively referred to as rover(s) 802, may collect data streams for information such as log data or stack traces that are provided for analysis to one or more incident model(s) 806. In contrast, one or more connector(s) 804-1, 804-N, collectively referred to as connector(s) 804 collects data streams for information such as CRUD events or pipelines that may be provided for analysis to one or more entity model(s) 808. In this embodiment, the connector(s) 804 are more focused on the entities, rather than the events that those entities are producing, e.g., for developer-end entities and user-end entities. As such, the information from the connector(s) 804 would be processed with respect to the entity model(s) 808. In some cases, a connector(s) 804 may include a rover(s) 802 as part of it, and thus the connector(s) 804 may produce a first stream of data from its rover(s) 802 for the incident model 806, and a second stream of event data for the entity model 808.
The data stream then undergoes both clustering and classification. The clustering operations will use a respective model to perform grouping operations on data objects based upon whether the data either looks similar or does not look similar to each other. Clustering may be operated at different granularity levels. For example, clustering may occur at the incident level using incident model(s) 806, at the entity level using entity model(s) 808, for classification using classification model 812, for ticket identification using ticket model 818, or to identify issues using the issue model 822.
The clustered data may then be further analyzed and augmented with more contextual information by performing classification upon the clustered data. The classification operations may, for example, classify the data into problem groupings or non-problem groupings. The classification operations may produce entity classifications that are stored in an entity database 810, incident classifications that are stored in an incident database 816, ticket classifications that are stored in tickets database 820, and issue classifications that are stored in an issue database 824.
Communications and/or notifications may occur at various points within an event correlation map. For example, a UI/API 826 may be used to communicate information to API users, or any other person in relation to the one or more entities operating in the distinct ecosystems. One or more auto-complete operations may be performed to automatically complete or update various external systems, fields, or entities. In addition, data and analysis results may be used to populate a knowledge base 830.
Regarding details of the processing by the ML architecture and the ML models, the incident models 806 read from multiple data streams pertaining to events and output to an incident stream. The incident models 806 learn to find the similarity between events of a certain kind so that they may be combined into an incident. In an example, the incident models 806 are responsible for clustering within data pertaining to a single ecosystem. Some examples may include: (a) reading test failure events over multiple test pipelines or within a single pipeline to determine that there is an test environment failure incident; (b) reading user tracking events from multiple users to determine that there is an incident whereby many users are seeing delays in logins; (c) reading operational alert events over time to determine that there is an incident whereby a service is not performing optimally; and/or (d) noticing a spike in the number of operational alerts above usual levels.
Similarities between events are learned by looking at events that occur closely in time and that are close in related objects. The incident model(s) 806 gets positive feedback when an incident is made part of a ticket or when an incident is not dismissed by an entity in relation to a set of hierarchical product parameters associated with the incident.
The incident model(s) 806 are trained automatically from the event streams that an entity is subscribing to in each of the distinct ecosystems. A specialized tokenizer can be used for each event stream to extract the most relevant tokens. An embedding layer is used to map the tokens in each event type to vectors. The embedding layer learns the pattern of tokens in each event and the sequence and co-occurrence of events to learn appropriate embeddings. This training is done without any manual intervention in some embodiments. An embedding model is produced that is saved persistently and relevant portions of it are cached as events are processed. A kNN (k nearest neighbor) indexing layer provides the ability to quickly find similar events.
The entity model(s) 808 reads from various systems of records associated with each entity in the one or more distinct ecosystems that provide information about the raw entities. The entity model(s) 808 deduplicates repeat mentions of entities across these systems of records and learns their relationships which are then all written out to the Entities Database 810. Some of the information that may be learned include: (a) which developers work in the dev organization; (b) what code repositories are owned by the dev organization; (c) which code files are part of these repositories; (d) which developer owns which repository/files; (e) which repository/file provides what code component; (f) what services are deployed; (g) who are the users and which entities do they belong to; (h) what products/capabilities/features are provided by the developer-end entities; (i) which user is signed up for which product/capability/feature; and/or (j) which product/capability/feature relies on what code component and which service is it deployed on.
The classification model 812 may also be referred to in some embodiments as an ML entity reference resolution model. The ML entity reference resolution model reads the incidents database 816 as well as the entity database 810 to identify which entities are referenced in each incident. Some examples may include: (a) which product/capability/feature is referenced in a user tracking incident at the user ecosystem; (b) which code component is referenced in a test failure incident; and/or (c) which ops service is referenced in an alert log incident.
The ticket model 818 reads from the incidents database 816 and determines whether a ticket should be created out of an incident. The ticket model 818 also determines whether some of the incidents should be linked together in the ticket. Some of the linkage examples may include: (a) determining that an incident related to slow logins captured from user tracking events in an event ecosystem is linked to the incident created from alert events captured from an operations ecosystem; and/or (b) determining that multiple test failure incidents are related to a single code commit. The ticket model 818 model is trained by identifying similar incidents that occur over a long period of time and also by manual intervention.
The issue model 822 links together issues that are related to a common underlying root cause in the product. The issue model 822 learns to distinguish between tickets caused by transient environment failures versus those caused by inherent flaws in the product itself. While tickets may describe problems being experienced by a single user-end entity, an issue may span all/multiple user-end entities and references the underlying product.
Correlation of events across the distinct ecosystems allows the assignment of appropriate severity and priority. Correlation across distinct ecosystems also allows for appropriate attribution of the actionable items identified from the events.
AutoComplete 828 may be implemented using an auto-completion model (not shown) that tracks all the fields that a user may enter in tickets and issues and learns to fill in these fields. The auto-completion model also learns to detect fields that are deprecated and should not be filled in as well as fields that are required to be filled in conjunction with other fields. The autocompletion model also clusters issue and tickets, and provides recommendations for similar issues and tickets to be linked. The autocompletion model also links appropriate incidents and events that may provide additional context for the ticket or issue. For example, log files for errors encountered in an operations ecosystem.
The Knowledge base 830 may be populated using a knowledgebase model (not shown) that identifies knowledgebase articles that are most relevant to an actionable item and suggests these to a unified analysis platform trying to identify actionable items from an event correlation map. The knowledgebase model autonomously creates knowledgebase articles based on common frequently occurring remedial steps that are undertaken by authorized entities in the one or more distinct ecosystems.
Some embodiments pertain to an object model to implement the invention. In the below description of the object model, the term “TechCompany” refers to an organization providing a product in relation to which a plurality of distinct ecosystems may operate in a connected environment, described herein. In the object model, various TechCompany terms may be employed. The term “dev org” refers to an entity operating in the connected environment. In the case of the entity being TechCompany itself, “dev org” will be its own dev org, and that of the user's “user org” would each be their own dev org. The term “dev oid” refers to the unique ID assigned to the dev org. In some embodiments, the TechCompany is assigned to a dev oid=0. The term “dev consumers” refers to the TechCompany's direct users. These are part of the Dev-org. The term “dev cid” refers to the unique ID assigned to the dev org's consumer. The term “dev cloud” refers to network connecting developer-end ecosystems operating in relation to the product and/or service.
The term “user org” refers to the TechCompany's consumer's consumer (account), which will be on boarded onto TechCompany by the dev org. The term “user oid” refers to the unique ID assigned to the user org. The term “user consumer” refers to the TechCompany's consumer's users, and are part of the user org. The term “user cid” refers to the unique ID assigned to the user org's consumer. The term “user cloud” refers to network connecting user-end ecosystems operating in relation to the product and/or service.
The term “rover” refers to a log/event collector which provides logs/events to the TechCompany platform. The rover can collect both machine or user generated items, and can inhabit the “Dev cloud”, “User cloud”, “TechCompany cloud” or external. A “connector” is used to synchronize entities between ecosystems. Connectors may synchronize objects bi-directionally in addition to creating objects in the source ecosystem.
The term “operational factor” refers to a component or piece of the hierarchy of operational factors associated with the product and/or service deployed in the connected environment. Exemplary operational factors include a particular capability, feature, microservice, component, and the like. The term “hierarchical product parameter” refers to an operational factor in the hierarchy of operational factors for a particular ecosystem of the connected environment. The term “actionable item” refers to any item of interest that triggers a response to be undertaken by one or more entities in the distinct ecosystems. Exemplary actionable items include: an event, an incident, a ticket, or an issue.
An “event” is a granular item of the data streams generated by the system, user, application, rover, etc., pertaining to entities operating in the distinct ecosystems in relation to a product. The data streams are commonly not of interest by themselves, however certain data in the data streams may be of interest (e.g., test failure, etc.), and may thus be termed as events. Exemplary events include: (a) alert; (b) stack trace; (c) test fail/pass; (d) system generated event (e.g., user CRUD, ticket creation, etc.).
An “incident” is used to group events by space and time into a clustered unit called an incident. Incidents are things of interest to entities deployed across ecosystem in a connected environment and are important for notification, however any action may not be necessary. For example, consider a critical error that occurs for a failed test. To gather context, one can look at relevant events leading up to the failure (e.g., user actions, logs, etc.) and cluster these together into an incident. In essence, an incident allows the system to gather context by combining relevant and related events and memorializing them with an incident (artifact).
Various terms in an object model may pertain to work management. The term “project” may pertain to an ephemeral/medium term series of work that exists as part of a product or service capability. For things that are longer running or require coordination/collaboration, it is preferable to create a project vs an unconnected issue. Objects that require phasing should be run as part of a project. For example, for a new release/version, the project may pertain to: (a) having associated issues for new features/bugs; (b) having associated marketing collateral, such as product collateral, website/digital updates, and/or documentation. The project may also pertain to the development of a new capability.
The term “issue” pertains to something requiring structural modifications, including modifications at backend constructs. For example, an issue may require back-office work (people writing code, etc.), and may relate to a new feature, a bug fix, etc. Some exemplary characteristics may include, for instance, duration of days/weeks/months, involving writing code (if development product), and/or having a higher impact/risk potential. Examples may include a new feature for a capability or a bug fix for a feature.
The term “ticket” pertains to something requiring cosmetic modification. For instance, the tickets may require front-office work. The tickets may be triaged to see if additional work/investigation is required. A ticket can be escalated/linked to an issue. Characteristics of the ticket may include some or all of: (a) Duration of minutes/hours; (b) Can be minor configuration/test changes; (c) Can be a minor activity (e.g., reset password); (d) Lower impact/lower risk potential. Examples may include a customer support ticket, or an auto generated ticket from ML.
The term “task” is a granular item of action performed by an authorized person and may be an action associated with resolution of a ticket or an issue. Examples may include peer review, Unit tests, etc.
Various terms in the object model may pertain to work management. With respect to a development project, the term “development repository” refers to a repository in a development ecosystem such as a source code repository. Examples include GitHub, GitLab, Git, etc. The term “version” pertains to a specific version cut from a branch at a specific time (commit). The version can be a release version or not. The term “branch” refers to a git branch that can be cut from another branch. The term “CI/CD” refers to Continuous integration/continuous deployment, which refers to the development methodology in which the development lifecycle and associated testing/push to product are much more automated/streamlined. Common tools have automated workflows/pipelines to handle testing and deployment. The term “test ecosystem” refers to an ecosystem where code is deployed, and tests are run.
Various terms in the object model may pertain to a hierarchy of operational parameters (e.g., for a product or service). A product is the highest level of the hierarchy of operational parameters and the parent of related capabilities and features (a hub or peak). Products typically have the following characteristics: (a) Units of revenue (have P&L); (b) Basis for contracts; (c) Level at which users are onboarded. A “capability” refers to a sub-unit of a product and is most commonly associated with the services a product provides. Capabilities typically have the following characteristics: (a) Services; (b) Unit of API scope; (c) Unit of licensing/pricing. An example is a TechCompany's work management service (ticketing, issues, etc.). A “feature” is a sub-unit of a capability and is typically something that the users can control via a flag or something they can experience indirectly. Features typically have the following characteristics: (a) Can be controlled via flags (e.g., on/off); (b) May or may not be visible to end user. Examples may include a TechCompany's work management services or automatic ticket creation, automated issue prioritization, etc.
At a physical level, a “microservice” can correspond to a generalized service or more granular “micro” service. The microservice typically directly maps to a Product's capability or feature which can be composed of one of more microservices. Microservices are a unit of functionality that is meant to be deployed independently and reused by multiple systems. Microservices typically have the following characteristics: (a) expose an API; (b) may be directly interacted with via an API/UI by the end customer; (c) may be published in a service registry/load balancer (Envoy, etc.). A “component” refers to a unit of functionality that is designed to be part of a system or application. A component corresponds to a library that can be used by multiple microservices. Components typically have the characteristics of support for various services though not commonly interacted with by the end customer. An “environment” may correspond to the client's operational environment (e.g., browser).
Operational factors relate to a piece of a product or service that has a lifecycle (ideation, prioritization, development, deployment, discovery, adoption, deprecation), and can be recursively made of smaller operational factors. Events and actionable items can be specific to any type of operational factors. An instance of an operational factor in a specific ecosystem of a plurality of distinct ecosystems may be referred to as a hierarchical product parameter.
The operational factors may have two core categories: user-end related operational factors and developer-end related operational factors. User-end related operational factors relate to hierarchical product parameters determining how the product is expressed, integrated, or consumed in the user-end ecosystem of a connected environment. User-end related operational factors may be purchased and interacted with by internal and external consumers. User-end related operational factors are primarily of three basic types: Product, Capability and Features. Further, internal facing User-end related operational factors like infrastructure services provided to support the product development and external facing user-end related operational factors which are consumed by an external user.
The code, service or components built or used by the developer are called developer-end related operational factors. A developer-end related operational factors may be a “runnable”, like a microservice or an executable. Runnable parts often expose APIs. A developer-end related operational factor may alternatively be a “linkable”, like a library or a component, which other developer-end related operational factors may depend on and use. Although the related operational factors often provide the functionality the entities of user-end ecosystems ultimately consume, they are often abstracted and unknown to the entities of user-end ecosystems.
Further, information gained through correlation with metadata being gathered from various systems may include:
The acquisitioned data including groups of information gained through parsing of code and information gained through correlation with metadata gathered from various systems may be collectively referred to as programming files. The data extraction engine 1006 clones the programming files into temporary storage subsystems, goes over the repositories, directories, and files inside the code (crawling), and uses specific detection functions to analyze the different code parsing sources of information.
As distinct functions are used for each type of programming file, the logic of specific tools for each function can be used to analyze the programming files to gain information. A discovery engine 1008 may utilize detection functions to analyze the programming files to identify one or more constructs related to the programming files by detecting how code is parsed inside the entities operating in the distinct ecosystems of the connected environment and specifically how and which code files are used in the generation of binaries and their deployment. Through the API Description files, the discovery engine 1008 can also analyze how APIs are exposed to end clients. The data extraction engine 1006 then stores all collected information in an object data store 1014 for later analysis.
The data extraction engine 1006 may acquire data from three main input sources. The input sources are described above as metadata gathered from various entities. The first input source is code tools 1002 which are summarized in the information gained through parsing of code section above. Code tools 1002 may include build tools, deployment definition files, deployment process files, Application Programming Interface (API) descriptor files and code files. When extracting data from the code tools 1002, the data extraction engine 1006 also has access to the GitHub-related metadata such as the location, timestamps and commit-hashes of their GitHub commits. For example, data extraction engine 1006 can detect a build tools file and store the location and content of the build tools file, as well as the scope of code files the build tool files concerns. The data extraction engine 1006 may also detect the existence of an API descriptor file such as an OpenAPI file at one or more entities operating in the distinct ecosystems, for example, a user-end entity.
The second information source is metadata 1004 from dynamic sources, which are discussed in the metadata from various systems section above. The metadata 1004 includes data from Continuous Integration and Deployment tools, package artifactories, as well as compute deployments and networking deployments. The Continuous Integration and Deployment tools contain information about the deployment workflow of binaries and their images, and how these deployments are handled by package artifactories and the compute deployments. Through the networking deployment, images routed to API endpoints may be inferred.
In a second stage, the discovery engine 1008 loads the stored programming files, constructs and metadata, from the object data store 1014, accumulates and correlates their information to identify and discover operational factors and then identifies hierarchical product parameters across distinct ecosystems for the operational factors. Similar to the data extraction engine 1006, the discovery engine 1008 only runs in the context of the product. The discovery engine 1008 is used to extract information from the noisy data obtained by the data extraction engine 1006 by combining different signals to identify high-value signals. In an example, combining different signals to identify high-value signals means that the discovery engine 1008 links input sources across distinct ecosystems by detecting that a build tool is used in a deployment definition file and that this deployment definition file is itself used by a deployment process file. The discovery engine 1008 then accumulates, correlates, and combines the constructs extracted from each of the programming files to determine the one or more user-end related operational factors and the one or more developer-end related operational factors.
Alternatively, the discovery engine 1008 can also screen over API description files and networking files to identify the API operations described in these files, what parameters may be used by the discovery engine and how the parameters belong to similar features. The discovery engine 1008 then uses this information to bundle the API information into hierarchical product parameters that are produced by the API operations (and offered to the user-end related entities). Based on the above accumulation and correlation, the discovery engine 1008 hands the information over to an operational factors manager 1010, which saves the user-end related operational factors and the developer-end related operational factors in the object data store 1014. The user-end related operational factors and the developer-end related operational factors may hereinafter be collectively referred to as hierarchical product parameters.
In a fourth step, entities 1012 in relation to the hierarchical product parameters of the connected ecosystems may communicate with the operational factors manager. In this interaction, entities 1012 may create hierarchical product parameters, delete hierarchical product parameters, combine hierarchical product parameters, and add information to existing hierarchical product parameters for transfer of information in a hierarchy of operational factors. Through such interaction, entities 1012 not only generate the correct hierarchical product parameters for their organization but also provide the unified analysis platform with labeled training data and thus allow the unified analysis platform to learn and identify hierarchical product parameters in an improved manner.
User-related operational factors cannot simply be detected from programming files, as they are abstract entities that may describe an organization's business model. The present subject matter discloses techniques to utilize Application Programming Interface (API) descriptor files 1102 containing structured information about the user-end entities API operations, and the information which nested API parameters, tags, descriptions and links are used for which API operation is extracted, by the data extraction engine 1006 through a specialized Parser function. The discovery engine 1008 then extracts and vectorizes the information using an information extraction and vectorization model 1108. For this vectorization step, the information extraction and vectorization model 1108 applies specialized Natural Language Processing functions that encode each API operation into a vector space, where similar API operations (as expressed by their attached information) have a similar encoding.
Another input source is data pertaining to websites 1106. In an example implementation, API descriptor files 1102 may determine which websites 1106 to scrape through to obtain the descriptive API file information and additional references throughout clients. The scraped websites contain the organization's API documentation, which is usually available online such that one or more entities deployed in the connected environment have a point of reference. The online API documentation typically contains text data stating which API paths and operations the organization exposes and which input and output parameters are related to an operation. The API documentation often also contains an implicit notion of how the API operations can be clustered, for example because different API operations occur under the same header. Besides simple text data, the information related to how the API operations can be clustered can also be conveyed in tables or images and website data to detect the user-related operational factors for a product deployed in the connected environment. In an example implementation, Natural Language Processing (NLP) Models and Visual Recognition models 1110 may extract the API operations from the websites 1106 and may transform the information into vectorized form. The NLP models and visual recognition models 1110 models use the data from websites 1106 as input and output a set of API operations and their vector encoding.
A third form of information comes from Network deployment files 1104. Network deployment files 1104, such as Kubernetes files, inform how API endpoints are routed to backend services and their images. The information from network deployment files 1104 may be obtained and vectorized, again using specialized NLP models to process the received information (in this case especially API path information) and vectorize it. Through the Network deployment files 1104, the unified analysis platform also has access to frequency, co-occurrence, and recency data, which is also used to identify related API operations.
In the next stage, a combination model 1112 combines the input information. In an example implementation, the combination model 1112 may be implemented by the discovery engine 1008. To do so, the combination model 1112 identifies duplicate received API operations and paths from the input sources and then combines the separate data sets. As the input sources were all mapped to vector spaces with the same embedding dimensions, and because the separate NLP models have all been trained to yield similar embeddings, all operations may simply be mapped to the same vector space in this stage.
In the next stage, a feature based clustering model 1114 groups the vectorized API operations into features. The feature-based clustering models may have the capability to be fine-tuned to follow a desired level of feature granularity and then identify the existence of a feature as a combination of API operations according to this level of feature granularity.
Next, an accumulation model 1116 accumulates information from the included operations to the cluster level—that is, from the API operations to the feature level. At this stage, the gained (non-vectorized) information may be simply combined for each API operation into a common data model and then accumulate summary information at the operation level. At this stage, the user-related operational factors 1118 may be identified and stored at an object data store.
As the identification of user-related operational factors 1118 is executed on a repeated basis and users have the ability to augment user-related operational factors 1118, their actions are saved such that the newly discovered user-related operational factors 1118 do not overwrite operational factors identified by manual intervention at the object data store.
The techniques for identification of developer-related operational factors involve three main input sources. A first input source may be version control providers 1202 that may deliver code files, build tools and deployment definition and deployment process files as summarized in information gained through parsing code, as well as the files describing Continuous Deployment and Integration workflows, as discussed in relation to
The input data exists may be received structured, semi-structured and unstructured form. The information from the input data may be extracted through specialized functions performed by an information extraction model 1208.
The information extracted on the different steps in the package creation and deployment process is next used to create a knowledge graph using a knowledge graph creation model 1210. The knowledge graph has the programming constructs pertaining to build and deployment information as its nodes and connections between different deployment processes as its edges. The graph is constructed using syntactic knowledge about the different tools that are also used to extract the information from multiple input sources.
In a next step, the knowledge graph is enriched using a knowledge graph enrichment model 1212. Assuming that the information up to this stage is not complete, and there are missing edges between existing nodes. In an example, the missing edges may appear when the information extraction model 1208 extracts information related to a code, build or deployment tool that is not yet supported by unified analysis platform, when they use the tool in an atypical way that is not captured by the knowledge extraction algorithms. In another example, the missing edges may appear when the information extraction model 1208 does not have access to the source of information (either because the source does not grant access, or because the integration tool or missing or because it is not possible to extract this information). As behavior of the one or more entities operating the developer-end ecosystem and the steps used by the entities follow a probabilistic behavior—or at least can be captured by probabilistic models, and because it may be inferred that information is missing—and even which piece of information is missing—as the known syntactic workflow rules suggest this, the knowledge graph enrichment model 1212 has the ability to enrich the graph at this stage.
Next, the knowledge graph may be used by a Knowledge Graph Deduction model 1214 to identify the existence of developer-related operational factors 1216. The developer-related operational factors 1216 can be of different types—such as runnables and linkables—to embody the different purpose of software packages (e.g. runnables are the operational factors that actually expose a microservice through code and its deployed images, while linkables take the role of utilized libraries). To identify the existence of developer-related operational factors 1216, the node information may be passed along the constructed knowledge graph and use syntactic and probabilistic rules to identify developer-related operational factors 1216 being exposed by a node.
Similar to the user-related operational factors, the identified runnables and linkables, as well as the links between these two developer-related operational factor types, are then uploaded to an object data store. Therefore, manual augmentation actions to add the specified information to the developer-related operational factor may be avoided.
Since the Network Environment 1304 is used to discover both developer-related operational factors 1308 and user-related operational factors 1310, the operational factors that are part of the same network environment may be linked across the distinct ecosystems. For example, if the network environment 1304 uses a backend name that was grouped with a developer-related operational factor 1308 (Runnable), and a connected API Path that was grouped with a user-related operational factor 1310, a link between the developer-related operational factors and user-related operational factors 1312 may be identified.
Developer-related operational factors are determined using the information for specific coding languages and deployment tools. Initially, a data extraction engine 1006 detects a coding language, such as Golang, and its correlated build tools, like bazel (these build tools inform where Golang binaries and container images are created). Using the build tools, files inside a specific repository and their contribution to a specific executable can be mapped. As this analysis does not define a single executable for each repository, but is instead location focused, mono repositories that produce multiple executables may be accounted for.
Next, deployment definition tools 1406 such as docker containers that use a specific build tool 1404 are detected, such that a build tool 1404 to deployment definition file 1406 link can be established. These files are then processed by detected deployment process files 1406 such as CircleCi, which has a macro to publish a docker image to a package artifactory 1408.
A CircleCi deployment is published to a package artifactory 1408 like ECR, which stores the list of published images. While the link between a deployment and an entry in ECR is not exact, timestamps, commit-ids and names allow for a heuristic-based identification. Using the ECR package artifactory 1408, a compute deployment 1410 such as AWS Lambda or a Kubernetes container can be identified which uses the image—for example, the image ID referring to the ECR image, which allows for a linkage of compute deployments 1410 and the code used.
For detected Kubernetes containers, the product, deployment, and service they belong to may be inferred. If the Kubernetes containers are part of a Kubernetes service which is a REST API object, a host and a port number may also be exposed. The API-exposing Kubernetes Service may be mapped to a load balancer, which has a routing prefix URL that maps to the Kubernetes service.
The API operation specified in a Swagger3 API specification file 1418 can be mapped to the API paths exposed by the load balancer. The Swagger3 file specifies additional information for each API operation that may be used to cluster the operations and map generated features that each map to a set of clustered API operations. Using additional user information, hierarchical product parameters related to features and capabilities may be generated, for example in a Trails View, and the features may be grouped to the hierarchical product parameters.
In certain cases, the operational factor identification may require further access to various data sources in a connected environment to make inferences and identify operational factors, such as the product's actual code, or their AWS and Kubernetes specifications. Since such information may be considered confidential, the aforementioned schematic is further configured to correctly identify developer-related operational factors and link them to their user-related operational factors, as explained under.
Users of the product may link an account for their favored version control system, such as GitHub, to the one or more entities operating in the connected environment. In such cases, a unified analysis platform receives live information from the version control system, such as information on their Branches on Pull Request. Branches actions and Pull Requests can be associated with actionable items. Git Snap-in allows a system for operational factor identification to better detect developer-related operational factors while they are being developed and link them to their related user-related operational factors. This enables linking developer-related operational factors and user-related operational factors faster and detect if events recorded in a version control system like GitHub qualifies as an actionable item based on in-app actions.
A discovery engine, such as the discovery engine 1008 discussed in relation to
The operational factors 1602, associated with a particular entity operating in the connected environment, identified using operational factor identification engine 1604 may help ideate new hierarchical product parameters, using an ideation engine 1606, and prioritize associated actionable items, using prioritization engine 1610. Further, user engagement with and usage of user-related operational factors, specifically feature-related hierarchical product parameters of the operational factors, identified by usage and adoption tracking engine 1612 may be used to determine usage and revenue metrics that provide more information to related developer-end entities, and thus help in identification of the operational factors 1602. The actionable items may be prioritized by the prioritization engine 1610 and translated into code and deployments 1608—which serves as a source for further identification of operational factors by the operational factor identification engine.
All metrics associated with the operational factors, such as the above-described usage metrics, but also metrics on deployments and actionable items associated with the entities operating in the connected environment, are presented in a Trails interface, which provides an intuitive way for the entities to interact with their hierarchical product parameters and gain insights into the architecture associated with the operational factors they may be authorized with.
DevUsers mainly engage with the hierarchical product parameters in two forms: performing actions on actionable items (issues or tickets) that may occur at a user-end entity operating in the connected environment. Through these actions, the user-related operational factors a DevUser interacts with—and by consequence also which linked developer-related hierarchical product parameters acted upon may be identified.
Further, a Code engagement data 1706 may be extracted to track which entities work on code that form a assigned hierarchical product parameter 1714 related to a new actionable item 1710. For example, code engagement data 1706 may be extracted through data on GitHub commits 1702 and therefore, the GitHub IDs which have committed to the new actionable item 1710 may be identified. Further, GitHub Pull Requests (PRs) 1704 may be tracked to determine their engagements such as, the GitHub IDs that have created, commented on, or reviewed the PR. Since it is known which DevUser IDs are associated with which GitHub IDs, and which files constitute developer-related hierarchical product parameters, the code engagement of DevUsers and developer-related hierarchical product parameters, as well as their linked user-related operational factors may be identified.
DevUsers and their relationships with operational factors may be used in different ways. An important such function is the autosuggestion of DevUsers when creating an actionable item. When a new actionable item 1710 is identified, related hierarchical product parameters 1714 have to be assigned to a user-related operational factor. Since it may be already identified which DevUsers have worked on this user-related operational factor, a set of the assigned hierarchical product parameter 1714 and the associated actionable item 1710 therefor may be transmitted to a DevUser ranking engine 1716 that auto-suggests the responsible DevUser 1718 for the assigned hierarchical product parameter 1714. In an example implementation, the set of the assigned hierarchical product parameter 1714 and the associated actionable item 1710 therefor includes an actionable item description 1712. For further context, the DevUser ranking engine 1716 also receives the existing actionable items 1708 with the operational factors as an additional input.
A changed hierarchical product parameter 1802, wherein the changed hierarchical product parameter is a developer-related hierarchical product parameter, (for the sake of explanation) may be linked to other developer-related hierarchical product parameters 1804-1, 1804-2, 1804-N (for example when a Runnable uses a Library). The developer-related hierarchical product parameters 1804-1, 1804-2, 1804-N are linked to user-related hierarchical product parameters 1806-1, 1806-2, 1806-3, 1806-N (when a user-related hierarchical product parameters is serviced by a runnable) and user-related hierarchical product parameters 1806-1, 1806-2, 1806-3, 1806-N can be linked to other entities 1808-1, 1808-2, 1808-3, 1808-4, 1808-N (for example when a feature is a child of a capability). These links are automatically generated by a discovery engine, explained in respect to
Since a system of the claimed subject matter knows how hierarchical product parameters are linked, these links may be used to provide an Impact Analysis that informs entities which other developer-related hierarchical product parameter and user-related hierarchical product parameters will be affected by a change of another hierarchical product parameter. While classic Impact Analysis only asks which other code bases or features will be affected by a change in the code base, the system can also evaluate the impact of a change on a hierarchical product parameter that is used as a provider for other hierarchical product parameters. Therefore, the full impact on the product deployed in a connected environment can be evaluated.
Since relationships between one more entities and the hierarchical product parameters they consume are known—both on an individual and an aggregated level (explained in detail with respect to
To suggest new actionable items for a product deployed in a connected environment, metrics associated with the hierarchical product parameters 1902 and associated hierarchical product parameters description 1904 are generated by a metric generation engine 1906. The metrics may be, for example, usage metrics (how often the hierarchical product parameters are used; how many and which entities use the hierarchical product parameters) 1906-1, revenue metrics (what is the revenue a user-end entity authorized for the hierarchical product parameter currently generates) 1906-2, cost metrics 1906-3 (what cost does the product currently occur for an associated developer-end entity or user-end entity), and Quality metrics 1906-4 (how often does this hierarchical product parameter malfunctions).
For each hierarchical product parameter 1902, the metric generation engine 1906 tracks the following metrics:
Further, data on the effects (in terms of the above metrics) of previous modifications, obtained using impact analysis engine 1908 is used as a second source of input. The data on effects for example shows that a specific modification of a hierarchical product parameter resulted in a 10% increase in retention for this hierarchical product parameter by a user-end entity or decreases the latency of a hierarchical product parameter by 15%.
The two data sources are passed to an ideation engine 1910. The ideation engine 1910 may use a generative Machine Learning based model to suggest probable actionable items 1912 to enhance the hierarchical product parameters. The output of the ideation engine 1910 is then presented in the form of probable actionable items 1912 that an organization that has developed the product deployed in the connected environment may use for its next development cycle planning.
Prioritization may be explained herein with respect to the issue 2004-2 (an example of identified actionable items generated by one or more entities 2002-1, 2002-N operating in a connected environment for a product) that can be linked to other issues 2004-1—such as issues 2004 they are dependent on—as well as tickets 2006. Tickets 2006 can be filed when an entity 2002 requests a change of a hierarchical product parameter they are interacting with, and multiple tickets 2006 can be associated with a single issue 2004-2. Besides information from the issue-ticket hierarchy, the issues 2004 may also contain information related to their assigned hierarchical product parameters 2008, authorized entity 2010 associated with entities 2002 operating in the connected environment, their priority level 2012, and the issue description 2014. Issues 2004 are further subscribed with metrics, such as usage metrics, revenue metrics, dependency metrics, or quality metrics that can offer further insight.
All this information related to each issue 2004 is forwarded to a prioritization engine 2016. The prioritization engine 2016 may be a Machine Learning-based engine that prioritizes the issues through ranking, adjusting the order based on the potential and importance associated with an issue (for example, the potential revenue or number of entities that can be gained with this issue; the revenue or number of entities lost while not completing this issue), workload (based on the description and subtasks, the prioritization engine 2016 may compute a time till completion estimate), its dependencies (if an issue depends on another issue, they must be completed in the correct order), and the entities authorized for the issue (entities should work at a limited amount of issues in a single time period). The prioritization engine 2016 then outputs a ranked list of prioritized issues 2018. Further, upon identification of a subsequent issue (not shown), the issue may again be forwarded to the prioritization engine 2016 to update the ranked list of prioritized issues 2018 based on impact of the subsequent issue.
A microservice for the deployed product does not stand on its own. Instead, it occurs along with data pertaining to code, build files, test files, deployment files, and interaction with entities deployed in the distinct ecosystems. Therefore, when the importance of a microservice for the overall company (general ranking) may be determined, information pertaining to a group of people such as developers, product managers, management, and executives (grouped ranking) or an individual DevUser (user-specific ranking), may be taken into account. For example, microservices that are connected to all of these follow-up tools can gain a higher level of importance and therefore gain a higher level of visibility. Such different sources of information may be used to establish closeness-relationships between different microservices and other operational factors and use them for individual rankings. For instance, if it is known that a product manager is interested in a hierarchical product parameter that a specific entity group interacts with and it is also known that this entity group interacts with a hierarchical product parameter, then the second operational factor can attain a higher ranking score for this product manager.
Ranking of hierarchical product parameters and resulting individualization is important whenever users only want to see hierarchical product parameters relevant to them. Therefore, individualization is needed to limit the depth (e.g., number of hierarchical product parameters from each operational factor shown on screen) of the visual parts output information.
Besides simply ranking operational factors, the metrics associated with hierarchical product parameters may also be ranked. When ranking different metrics by their relevance to a given entity, the category of the metric and a representation of the metrics' importance/relevance is determined by a level of abnormality of the metric score.
Individualization depends on two aspects: entity-based aspects and class-based aspects. Class aspects are, for example, the type of a entity 2102 (e.g. a developer is interested in different hierarchical product parameters than a product manager and wants to view different metrics), but also the hierarchical product parameters' importance for an entire organization for which the product is deployed in a connected environment, while entity-based aspects, such as the action history 2104 related to a log of actions undertaken by the entity, actionable items 2106 that may have been worked upon by the entity have worked on or are currently working on and a entities viewing history 2108 provide information on an individual level.
Using these aspects, an Individualization Engine 2114, which is a recommender system that attributes relevance score to the hierarchical product parameters and the associated metrics, ranks them according to those scores and outputs a concise and personalized view of a hierarchical product parameters architecture for a particular entity and associated metrics through a list of ranked hierarchical product parameters 2116. The hierarchical product parameters to be ranked are obtained from a hierarchical product parameter data store 2110, and the associated metrics may be obtained from a metrics data store 2112. This view of the hierarchical product parameters architecture for the particular entity and associated metrics is then displayed at a trails user interface 2118. Based on the interaction of the entity 2120 with the shown hierarchical product parameters and metrics the entity's interests can be better learnt, and predictions may be improved. The architecture of the Individualization Engine 2114 may include a standard Feed-Forward Deep Learning architecture that transforms an n×1-dimensional vector input into a single score variable that summarizes the user's interest in the hierarchical product parameters.
The Trails User Interface 2118 is a graphical user interface that shows hierarchical product parameters, hierarchy of operational factors related thereto, and dependencies, as well as additional information on these hierarchical product parameters, such as their metrics or attached items.
Additional information on the hierarchical product parameters, such as their metrics, is accumulated in an insight generation engine 2212, processed and displayed through different layers. Users can select which layers to view, using a layer selection engine 2210, and the layers modify the graphical display, for example by changing the size or color of a hierarchical product parameter box. The full set of the displayed hierarchical product parameters and the layers form the Trails UI 2208. Users can also select to add some of the actionable items directly to the trails UI 2208, for example by opening a ticket inside Trails. To enhance the insight-generation and layer-selection experience, the individualization engine 2214 may be used to select which items should be displayed in a default view and trains the layer selection engine through feedback gained from user interactions with the layers.
According to one embodiment of the present subject matter, the computing system 2400 performs specific operations by the processor 2414 executing one or more sequences of one or more instructions contained in main memory 2406. Such instructions may be read into main memory 2406 from another computer readable/usable medium, such as static storage device 2408 or disk drive 2410. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the present subject matter are not limited to any specific combination of hardware circuitry and/or software. In one embodiment, the term “logic” shall mean any combination of software or hardware that is used to implement all or part of the invention.
The term “computer readable medium” or “computer usable medium” as used herein refers to any medium that participates in providing instructions to the processor 2414 for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive 2410. Volatile media includes dynamic memory, such as main memory 2406. A data store 2420 may be accessed in a computer readable medium using a data interface 2412.
Common forms of computer readable media includes, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, a paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.
In an embodiment of the present subject matter, execution of the sequences of instructions to practice the present subject matter is performed by a single computing system 2400. According to other embodiments of the present subject matter, two or more computing systems 2400 coupled by communication link (e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions required to practice the invention in coordination with one another.
Computing system 2400 may transmit and receive messages, data, and instructions, including program, i.e., application code, through communication link and communication interface 2416. Received program code may be executed by processor 2414 as it is received, and/or stored in disk drive 2410, or other non-volatile storage for later execution.
In the foregoing specification, the present subject matter has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present subject matter. For example, the above-described process flows are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.
The present application claims the benefit of priority to U.S. Provisional Application No. 63/393,641, which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63393641 | Jul 2022 | US |
