Patent Application
20240386500
The present disclosure relates to machine learning process and organizational database systems, and in particular relates to a system and method that links a machine learning algorithm with a run-time API service in order to provide a suitable run-time determination for a prospective user.
In service industries, including financial services, it is important to obtain, store and manage knowledge related to prospective clients, which are referred to as “leads”. One objective of the lead management process is to match a prospective lead with the most appropriate recipient (advisor, salesperson, etc.) in an organization. Traditionally, lead management is handled by the individuals responsible for client prospecting. These individuals typically obtain information concerning leads by buying clients lists, receiving referrals, and creating web pages that collect information. In the case of large organizations, leads can also be assigned to sales personnel by assignment from a central process. While these processes have some effectiveness, the matching between the lead and advisor is based on fairly generalized rules and/or human-biased decisions that often result in sub-optimal or unsuccessful lead matching.
Machine learning techniques can aid in the matching process as such techniques can process a large number of relevant features and attributes, related to both clients and potential advisors, and determine an optimal matching solution. It is found, however, that there are circumstances which make simply incorporating data into a machine learning ecosystem infeasible, either due to data incompatibility an overly large amount of batch data or other factors. More generally, beyond the client/advisor matching process, there are instances in which a machine learning ecosystem cannot be effectively used due to such incompatibilities.
What is therefore needed is a system and method that can leverage machine learning ecosystem capabilities while accommodating data sets that cannot be directly used during as inputs to the learning phase of a machine learning processing model.
According to one aspect, the present disclosure describes a computer-implemented method of providing recommendations for matching leads to financial advisors using a machine learning model. The method, executed by one or more hardware processing units of a computing system, comprises training the machine learning model using known attributes of the leads and advisors to determine optimal lead/advisor matches and to output an initial recommendation list of lead/advisor matched pairs, serializing the machine learning model in a platform independent manner, providing access to the serialized machine learning model to an end user device via an application program interface, deserializing the machine learning model, receiving preferences from the end user regarding a preferred financial advisor in natural text form, converting the natural text received from the end user into a form adapted to be used as input to the machine learning model, executing the deserialized machine learning model at the end user device using the received preferences to determine a list of advisor recommendations tailored for the end user, filtering the list of advisor recommendations using business rules and protocols which disallow certain lead/advisor pairings, outputting a representation of the filtered recommendations at the end user device for viewing by a particular end user; and transmitting a selection from the filtered recommendations from the end user device for further training of the machine learning model with respect to the particular end user.
According to a further aspect, the present disclosure describes a computer-implemented method of providing recommendations for matching leads to financial advisors using a machine learning model, the method, executed by one or more hardware processing units of a computing system. The method comprises training the machine learning model using known attributes of the leads and advisors to determine optimal lead/advisor matches and to output an initial recommendation list of lead/advisor matched pairs, receiving preferences from the end user regarding a preferred financial advisor in natural text form converting the natural text received from the end user into a form adapted to be used as input to the machine learning model, identifying financial advisor attributes in the converted natural text; and executing the deserialized machine learning model at the end user device using the financial advisor attributes to determine a list of advisor recommendations tailored for the end user.
The present disclosure describes a machine learning-query-on-demand system and method that provides an outfacing interface to prospective leads and other users, allowing users to input, refine requests and obtain match (e.g., recommendation) determinations in real time. Importantly, users can further refine proposed advisor matches without direct connection to the lead-management ecosystem. This service enables virtually unlimited scale of client/advisory recommendations at near real-time speeds.
At the UI 110, the user is prompted to enter any criteria/preferences that they may have for a preferred advisor. The orchestration layer receives the preference information and then accesses a rules engine 145, which can be implemented using a query management tool, to apply a set of rules to the criteria and preferences. The rules can include conditions for removing certain advisors as candidates for matches based on preset criteria. As one example, a lead may prefer a lead to be located within a certain geographical proximity. Based on application of these criteria, the rules engine 145 returns a second list of lead/advisory pairs that match the criteria and preferences of the user. In parallel, the orchestration layer 140 takes the raw scored lead-advisor paring list 125 and applies further existing business protocols 150 that also can suppress out advisors from the list that do not meet a set of propriety business rules, resulting in a refined pairing list. For example, a business protocol might exclude a particular advisor on the basis that the selected advisor is already responsible for a threshold number of clients. The refined list is combined by the orchestration layer 140 with the second list generated by the via the rules engine 145. This combination removes any advisors that do not meet the criteria of the preferences, rules and business protocols. After the recommendations have been completed, a finalized list 160 is passed with corresponding management system information describing the advisor to the UI 110. At the UI 110, a predefined number of top advisor recommendations are presented to the lead in a visualization (an example is shown in
Upon selection of an advisor by the lead, the lead-advisor pair that is selected along with additional, applicable data is passed to a lead internal database management system 170 (referred to as “LeadIQ”) which, in response, assigns the lead data record to the selected advisor. Assignment within the LeadIQ system 170 enables the selected advisor to be alerted of the selection and to view the available data concerning the lead in existing advisor platforms (e.g., customer relationship management (CRM), next-best-action (NBA) platforms). The LeadIQ system 170 also triggers a real-time data application, which in some implementations can be in the form of an API service that delivers key data elements of the lead's information to specific advisor platforms, allowing the advisors to view and take action on the data in real-time.
The data sources 212-220 are input into a strategic feed 225 which is database consisting of a number of centralized, standardized, and cultivated tables that are accessible to all platforms in the ecosystem and comprise the governing data source for leads in the ecosystem. A party master hub 230 is a database which is used to store critical production attributes of current clients and leads. The party master hub 230 also serves as a centralized database for all downstream systems and is coupled to the strategic feed 225. A central coverage platform 235, coupled to the strategic feed 225, is a prospect management and control system that is configured to prevent the access to lead and advisor data to unauthorized personnel. PMCC services 240 are operatively connected to the central coverage platform and include enterprise search services that enable downstream systems to perform coverage central checks in real time.
The client and lead genome 136 (shown in
The orchestration engine 140 receives raw model scores, business rules, lead preferences, and hosted advisor information data, organizes the combined information, and outputs the finalized recommendation to end-user Interface 120. As previously noted, the LeadIQ 170 is a platform that centralizes lead assignment to advisors as well as lead-FA relationship monitoring and management. Advisor information cards 260 are provisioned by a management application that hosts published content for downstream application use. Advisor information cards 260 are published for use in the lead user interfaces as additional information presented for decision making. Further tools of the ecosystem that operate in connection with the LeadIQ platform include a next-best-action/3D advisor hub 265 that provides tools to advisors for monitoring, management, and engagement of clients and leads, a customer relationship management and cloud platform 270, and end user platforms 275 which can host or provide an interface to the MLAS service for advisor recommendations.
Machine learning pipeline models are end-to-end machine learning algorithms that typically incorporate data inputs, processing layers, and outputs. The pipeline models can also combine different machine learning and artificial intelligence techniques and can add boosting algorithms. More specifically, a particular pipeline model can include one or more supervised and unsupervised ML algorithms such as, but not limited to, artificial neural networks (e.g., deep learning networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs)), support vector machines (SVMs), k-means and Naïve Bayes algorithms, as well as boosting algorithms such as Adam, to name a few.
A pipeline model created using any of the model platforms 320 is exported into a common portable serialization format called a “bundle” 330. A bundle 330 is a graph-based portable file format that is used for serializing and deserializing pipeline model data structures. In the serialization process, structures developed in a particular higher-level programming context such as Python or C++ are converted into lower-level bytes that are not specific to a particular programming context. Bundles thereby provide a common serialization format for a large set of ML feature extractors and algorithms that can be exported and imported across the different model platforms 320 and simplify the sharing of the results of a training pipeline. In some implementations, as shown, an MLeap library 340 is used to create the bundle 330, and to store a model pipeline in a JSON (Java Script Object Notation) or Protobuf-based (Protocol Buffer) format. MLeap 340 is an open-source execution engine that includes program modules for serializing and deserializing machine learning pipeline model code. In some implementations, the Bundle 330 is serialized to a directory or a zip file and is serialized as either pure JSON, pure Protobuf or in a mixed mode (part JSON, part Protobuf).
In a preferred implementation, the bundle 330 stores all of the information of a pipeline model object in several distinct files. A first file, referred to as “bundle.json”, stores basic meta data concerning the serialization including a root/directory which contains a root transformer of the pipeline model. A second file, referred to as “model.json”, stores detailed information of the pipeline model in json format. A third file, referred to as “node.json” stores information concerning pipeline model features (e.g., estimates, transformers) that are employed in specific ML platforms such as Apache Spark.
After being bundled, the pipeline model can be readily deserialized back into a model pipeline configured for one of the ML modeling platforms (e.g., Spark, Scikit Learn, TensorFlow). The deserialized bundle is agnostic with respect to any content specific to a particular platform. A deserialized pipeline model is downloadable to an end user at a UI 110 through a restful machine learning service application program interface (MLAS API) 350. The user can employ a deserialized model for execution of a pipeline model using data structures referred to as “leapframes” which are passed as distributed data for performance of real-time operations. Real time execution is implemented by the downloaded pipeline model at the user end point 110. For purposes of the lead/advisor application, an end user can execute the pipeline model described in
The MLAS API 350 acts as a calling service for ML functions with respect to the end user UI 110. The MLAS API 350 includes a machine learning as a service (MLAS) library which includes program modules that are used to execute the interactive environment so that the end user can enter instructions and execute tasks on the ML model. For example, the ML library includes modules for performing bundle deserialization, conversion of generic input parameters and schema fields into a format (e.g., leapframes) suitable for an MLeap library. The ML library also includes models for triggering the ML pipeline model for determinations and returns output in a readable format. The MLAS API 350 generates an input data set (and input schema) as a map list which is an array of strings paired with objects (string, object), calls a run model method using an ML bundle path, the input data set, and input schema, and obtains results from execution of the ML pipeline model as another map list (string, object). Through its capabilities the MLAS API 250 provides an interface for real time operations with one or more machine learning pipeline models trained on a distributed platform or even on a standalone learning-based model such as python scikit. In this manner, the MLAS is agnostic of any platform as the bundle can be ported to any platform even in the cloud such as GCP (Google Cloud Platform), AWS (Amazon Web Services), etc. This allows real-time access by all prospective leads, current clients and advisors, and enables advisor recommendations to be determined and accessed rapidly in real time.
In further embodiments of the present disclosure, a large language model is incorporated in the matching process architecture and intelligent advisor recommendation and advisor ecosystem discussed above and illustrated in
Due to the variety of possible natural language inputs, user control over the raw text inputs can create varying results. If the user provides a description that is too vague, the LLM system 510 may not be able to process the text to achieve useful results. In some implementations, chat bots or other prompt request tools can be used to prompt the lead to enter further details on their preferred FA attributes. There are also restrictions on data items that can be used to match a lead with an advisor. If a lead requests attributes for a Financial Advisor that constitute restricted criteria, then these attributes are excluded and not processed by the matching algorithm. The restricted input would be caught in the Query Management Tool stage of the process, where the exclusion rules are stored.
The system and method disclosed herein enable organizations with large legacy services with large, disparate databases to provide a computation-light machine learning interface that utilizes the organizational information to provide real time machine learning services, including, without limitation, advisor recommendation and other predictions and determinations, without requiring execution of large batch training and other computation-heavy algorithmic processes. Importantly, a number of different databases as well as specific business protocols and rules are integrated seamlessly into the machine learning service, enabling an organization to have secure and granular control over the output of the machine learning service.
The MLAS service is hosted by a provider network that is coupled to processing and storage resources. A user can access the MLAS service, using a computing device, over one or more public or private networks using one or more APIs to access data to execute instructions and store data to storage resources of the provider network. The public or private networks can include an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. Communication over the network can be enabled via wired or wireless connections and combinations thereof.
Examples of user devices include personal computers (PCs), cell phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, electronic book readers, wearable electronic devices (e.g., glasses, wristbands, monitors), and the like. In some embodiments, the user may interact with the provider network via a web portal provided through the web server and application server. These services can be implemented by one or more electronic devices that can be co-located or located in different geographic locations. Each of the one or more electronic devices includes an operating system that implements executable program instructions and a computer-readable medium storing instructions that, when executed by a processor of the device, allow the device to perform its intended functions.
More generally, the methods and processes described herein are performed by multiple computing devices (e.g., user devices, physical servers, workstations, storage arrays, cloud computing resources, etc.) that communicate and interoperate over one or more networks to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-transitory computer-readable storage medium or device (e.g., solid state storage devices, disk drives, etc.). The various functions disclosed herein may be embodied in such program instructions, or may be implemented in application-specific circuitry (e.g., ASICs or FPGAs) of the computer system. Where the computer system includes multiple computing devices, these devices can be, but need not be, co-located. The results of the disclosed methods and tasks can be persistently stored by transforming physical storage devices, such as solid-state memory chips or magnetic disks, into a different state. In some embodiments, the computer system may be a cloud-based computing system whose processing resources are shared by multiple distinct business entities or other users.
The methods and algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor device, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of a non-transitory computer-readable storage medium.
It is to be understood that any structural and functional details disclosed herein are not to be interpreted as limiting the systems and methods, but rather are provided as a representative embodiment and/or arrangement for teaching one skilled in the art one or more ways to implement the methods.
It is to be further understood that like numerals in the drawings represent like elements through the several figures, and that not all components and/or steps described and illustrated with reference to the figures are required for all embodiments or arrangements.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising”, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Terms of orientation are used herein merely for purposes of convention and referencing and are not to be construed as limiting. However, it is recognized these terms could be used with reference to a viewer. Accordingly, no limitations are implied or to be inferred.
Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having,” “containing,” “involving,” and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.
While the disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the disclosed invention. In addition, many modifications will be appreciated by those skilled in the art to adapt a particular instrument, situation or material to the teachings of the disclosure without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention includes all embodiments falling within the scope of the appended claims.
The present application is a continuation of commonly owned and assigned U.S. patent application Ser. No. 18/404,289 entitled, “System and Method for a Machine Learning Service Using a Large Language Model, filed on Jan. 4, 2024, which, in turn, is a continuation in part (CIP) of commonly owned and assigned U.S. patent application Ser. No. 18/320,752, entitled “System and Method for a Machine Learning Service” and filed on May 19, 2023.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18404289 | Jan 2024 | US |
| Child | 18753202 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18320752 | May 2023 | US |
| Child | 18404289 | US |
