Techniques for benchmarking pairing strategies in a task assignment system

Abstract

Techniques for benchmarking pairing strategies in a task assignment system are disclosed. In one particular embodiment, the techniques may be realized as a method for benchmarking pairing strategies in a task assignment system comprising determining first and second pluralities of historical task assignments paired using first and second pairing strategies, respectively, during a first period, determining a value attributable to each task of the first plurality of historical task assignments and the second plurality of historical task assignments during a second period after the first period, determining a difference in performance between the first and second pairing strategies based on the value attributable to each task during the second period, and outputting the difference in performance between the first pairing strategy and the second pairing strategy for benchmarking at least the first pairing strategy and the second pairing strategy.

Description

FIELD OF THE DISCLOSURE

This disclosure generally relates to benchmarking pairing strategies and, more particularly, to techniques for benchmarking pairing strategies in a task assignment system.

BACKGROUND OF THE DISCLOSURE

A typical task assignment system algorithmically assigns tasks arriving at the task assignment center to agents available to handle those tasks. At times, the task assignment system may have agents available and waiting for assignment to tasks. At other times, the task assignment center may have tasks waiting in one or more queues for an agent to become available for assignment.

In some typical task assignment centers, tasks are assigned to agents ordered based on the order in which the tasks are created, and agents receive tasks ordered based on the time when those agents became available. This strategy may be referred to as a “first-in, first-out,” “FIFO,” or “round-robin” strategy.

Some task assignment systems may use a “performance-based routing” or “PBR” approach to ordering the queue of available agents or, occasionally, tasks. PBR ordering strategies attempt to maximize the expected outcome of each task assignment but do so typically without regard for utilizing agents in a task assignment system uniformly.

When a task assignment system changes from using one type of pairing strategy (e.g., FIFO) to another type of pairing strategy (e.g., PBR), overall task assignment system performance will continue to vary over time. It can be difficult to measure the amount of performance change attributable to using alternative pairing strategies because the amount of performance or value attributable to a given task assignment may not be realized until a later time (e.g., months or years after the initial task assignment).

In view of the foregoing, it may be understood that there is a need for a system that enables benchmarking of alternative task assignment strategies (or “pairing strategies”) to measure changes in performance attributable to the alternative task assignment strategies over time.

SUMMARY OF THE DISCLOSURE

Techniques for benchmarking pairing strategies in a task assignment system are disclosed. In one particular embodiment, the techniques may be realized as a method for benchmarking pairing strategies in a task assignment system comprising determining a first plurality of historical task assignments paired using a first pairing strategy during a first period, determining a second plurality of historical task assignments paired using a second pairing strategy during the first period; determining a value attributable to each task of the first plurality of historical task assignments and the second plurality of historical task assignments during a second period after the first period, determining a difference in performance between the first and second pairing strategies based on the value attributable to each task during the second period, and outputting the difference in performance between the first pairing strategy and the second pairing strategy for benchmarking at least the first pairing strategy and the second pairing strategy, wherein the difference in performance demonstrates that the first pairing strategy optimizes performance of the task assignment system as compared to the second pairing strategy.

In accordance with other aspects of this particular embodiment, the task assignment system may be a contact center system, and the first plurality of tasks may be a first plurality of contacts and the second plurality of historical task assignments may be a second plurality of contacts.

In accordance with other aspects of this particular embodiment, the first pairing strategy may be a behavioral pairing strategy.

In accordance with other aspects of this particular embodiment, the second pairing strategy may be a FIFO strategy or a performance-based routing strategy.

In accordance with other aspects of this particular embodiment, the task assignment system may cycle among at least the first and second pairing strategies at least once per hour.

In accordance with other aspects of this particular embodiment, the difference in performance may be adjusted for the Yule-Simpson effect.

In accordance with other aspects of this particular embodiment, the difference in performance may be weighted according to a relative difference in size between the first and second pluralities of historical task assignments.

In accordance with other aspects of this particular embodiment, the first pairing strategy may assign a task to an agent during the second period irrespective of determining which of the first pairing strategy or the second pairing strategy had assigned a corresponding task during the first period.

In accordance with other aspects of this particular embodiment, the method may further comprise generating a report of statistically fair assignment of tasks during the second period irrespective of which of the first pairing strategy or the second pairing strategy had assigned any corresponding tasks during the first period.

In another particular embodiment, the techniques may be realized as a system for benchmarking pairing strategies in a task assignment system comprising at least one computer processor communicatively coupled to and configured to operate in the task assignment system, wherein the at least one computer processor is further configured to perform the steps in the above-described method.

In another particular embodiment, the techniques may be realized as an article of manufacture for benchmarking pairing strategies in a task assignment system comprising a non-transitory processor readable medium and instructions stored on the medium, wherein the instructions are configured to be readable from the medium by at least one computer processor communicatively coupled to and configured to operate in the task assignment system and thereby cause the at least one computer processor to operate to perform the steps in the above-described method.

The present disclosure will now be described in more detail with reference to particular embodiments thereof as shown in the accompanying drawings. While the present disclosure is described below with reference to particular embodiments, it should be understood that the present disclosure is not limited thereto. Those of ordinary skill in the art having access to the teachings herein will recognize additional implementations, modifications, and embodiments, as well as other fields of use, which are within the scope of the present disclosure as described herein, and with respect to which the present disclosure may be of significant utility.

BRIEF DESCRIPTION OF THE DRAWINGS

To facilitate a fuller understanding of the present disclosure, reference is now made to the accompanying drawings, in which like elements are referenced with like numerals. These drawings should not be construed as limiting the present disclosure, but are intended to be illustrative only.

FIG. 1 shows a block diagram of a task assignment system according to embodiments of the present disclosure.

FIG. 2A shows a table of benchmarking data according to embodiments of the present disclosure.

FIG. 2B shows a table of benchmarking data according to embodiments of the present disclosure.

FIG. 2C shows a table of benchmarking data according to embodiments of the present disclosure.

FIG. 3 shows a flow diagram of a benchmarking method according to embodiments of the present disclosure.

FIG. 4 shows a flow diagram of a benchmarking method according to embodiments of the present disclosure.

DETAILED DESCRIPTION

A typical task assignment system algorithmically assigns tasks arriving at the task assignment center to agents available to handle those tasks. At times, the task assignment system may have agents available and waiting for assignment to tasks. At other times, the task assignment center may have tasks waiting in one or more queues for an agent to become available for assignment.

In some typical task assignment centers, tasks are assigned to agents ordered based on the order in which the tasks are created, and agents receive tasks ordered based on the time when those agents became available. This strategy may be referred to as a “first-in, first-out,” “FIFO,” or “round-robin” strategy.

Some task assignment systems may use a “performance-based routing” or “PBR” approach to ordering the queue of available agents or, occasionally, tasks. PBR ordering strategies attempt to maximize the expected outcome of each task assignment but do so typically without regard for utilizing agents in a task assignment system uniformly.

When a task assignment system changes from using one type of pairing strategy (e.g., FIFO) to another type of pairing strategy (e.g., PBR), overall task assignment system performance will continue to vary over time. It can be difficult to measure the amount of performance change attributable to using alternative pairing strategies because the amount of performance or value attributable to a given task assignment may not be realized until a later time (e.g., months or years after the initial task assignment).

In view of the foregoing, it may be understood that there is a need for a system that enables benchmarking of alternative task assignment strategies (or “pairing strategies”) to measure changes in performance attributable to the alternative task assignment strategies over time.

FIG. 1 shows a block diagram of a task assignment system 100 according to embodiments of the present disclosure. The description herein describes network elements, computers, and/or components of a system and method for benchmarking pairing strategies in a task assignment system that may include one or more modules. As used herein, the term “module” may be understood to refer to computing software, firmware, hardware, and/or various combinations thereof. Modules, however, are not to be interpreted as software which is not implemented on hardware, firmware, or recorded on a non-transitory processor readable recordable storage medium (i.e., modules are not software per se). It is noted that the modules are exemplary. The modules may be combined, integrated, separated, and/or duplicated to support various applications. Also, a function described herein as being performed at a particular module may be performed at one or more other modules and/or by one or more other devices instead of or in addition to the function performed at the particular module. Further, the modules may be implemented across multiple devices and/or other components local or remote to one another. Additionally, the modules may be moved from one device and added to another device, and/or may be included in both devices.

As shown in FIG. 1, the task assignment system 100 may include a task assignment module 110. The task assignment system 100 may include a switch or other type of routing hardware and software for helping to assign tasks among various agents, including queuing or switching components or other Internet-, cloud-, or network-based hardware or software solutions.

The task assignment module 110 may receive incoming tasks. In the example of FIG. 1, the task assignment system 100 receives m tasks over a given period, tasks 130A-130m. Each of the m tasks may be assigned to an agent of the task assignment system 100 for servicing or other types of task processing. In the example of FIG. 1, n agents are available during the given period, agents 120A-120n. m and n may be arbitrarily large finite integers greater than or equal to one. In a real-world task assignment system, such as a contact center, there may be dozens, hundreds, etc. of agents logged into the contact center to interact with contacts during a shift, and the contact center may receive dozens, hundreds, thousands, etc. of contacts (e.g., calls) during the shift.

In some embodiments, a task assignment strategy module 140 may be communicatively coupled to and/or configured to operate in the task assignment system 100. The task assignment strategy module 140 may implement one or more task assignment strategies (or “pairing strategies”) for assigning individual tasks to individual agents (e.g., pairing contacts with contact center agents).

A variety of different task assignment strategies may be devised and implemented by the task assignment strategy module 140. In some embodiments, a first-in/first-out (“FIFO”) strategy may be implemented in which, for example, the longest-waiting agent receives the next available task (in “L1” or agent-surplus environments) or the longest-waiting task is assigned to the next available agent (in “L2” or task-surplus environments). Other FIFO and FIFO-like strategies may make assignments without relying on information specific to individual tasks or individual agents.

In other embodiments, a performance-based routing (PBR) strategy may be used for prioritizing higher-performing agents for task assignment. Under PBR, for example, the highest-performing agent among available agents receives the next available task. Other PBR and PBR-like strategies may make assignments using information about specific agents but without necessarily relying on information about specific tasks or agents.

In yet other embodiments, a behavioral pairing (BP) strategy may be used for optimally assigning tasks to agents using information about both specific tasks and specific agents. Various BP strategies may be used, such as a diagonal model BP strategy or a network flow BP strategy. These task assignment strategies and others are described in detail for the contact center context in, e.g., U.S. Pat. No. 9,300,802 and U.S. patent application Ser. No. 15/582,223, which are hereby incorporated by reference herein.

In some embodiments, a historical assignment module 150 may be communicatively coupled to and/or configured to operate in the task assignment system 100 via other modules such as the task assignment module 110 and/or the task assignment strategy module 140. The historical assignment module 150 may be responsible for various functions such as monitoring, storing, retrieving, and/or outputting information about agent task assignments that have already been made. For example, the historical assignment module 150 may monitor the task assignment module 110 to collect information about task assignments in a given period. Each record of a historical task assignment may include information such as an agent identifier, a task or task type identifier, outcome information, or a pairing strategy identifier (i.e., an identifier indicating whether a task assignment was made using a BP pairing strategy or some other pairing strategy such as a FIFO or PBR pairing strategy).

In some embodiments and for some contexts, additional information may be stored. For example, in a call center context, the historical assignment module 150 may also store information about the time a call started, the time a call ended, the phone number dialed, and the caller's phone number. For another example, in a dispatch center (e.g., “truck roll”) context, the historical assignment module 150 may also store information about the time a driver (i.e., field agent) departs from the dispatch center, the route recommended, the route taken, the estimated travel time, the actual travel time, the amount of time spent at the customer site handling the customer's task, etc.

In some embodiments, the historical assignment module 150 may generate a pairing model or similar computer processor-generated model based on a set of historical assignments for a period of time (e.g., the past week, the past month, the past year, etc.), which may be used by the task assignment strategy module 140 to make task assignment recommendations or instructions to the task assignment module 110. In other embodiments, the historical assignment module 150 may send historical assignment information to another module such as the task assignment strategy module 140 or the benchmarking module 160.

In some embodiments, a benchmarking module 160 may be communicatively coupled to and/or configured to operate in the task assignment system 100 via other modules such as the task assignment module 110 and/or the historical assignment module 150. The benchmarking module 160 may benchmark the relative performance of two or more pairing strategies (e.g., FIFO, PBR, BP, etc.) using historical assignment information, which may be received from, for example, the historical assignment module 150. In some embodiments, the benchmarking module 160 may perform other functions, such as establishing a benchmarking schedule for cycling among various pairing strategies, tracking cohorts (e.g., base and measurement groups of historical assignments), etc. The techniques for benchmarking and other functionality performed by the benchmarking module 160 for various task assignment strategies and various contexts are described in later sections throughout the present disclosure. Benchmarking is described in detail for the contact center context in, e.g., U.S. Pat. No. 9,712,676, which is hereby incorporated by reference herein.

In some embodiments, the benchmarking module 160 may output or otherwise report or use the relative performance measurements. The relative performance measurements may be used to assess the quality of the task assignment strategy to determine, for example, whether a different task assignment strategy (or a different pairing model) should be used, or to measure the overall performance (or performance gain) that was achieved within the task assignment system 100 while it was optimized or otherwise configured to use one task assignment strategy instead of another.

In some task assignment systems, techniques for benchmarking task assignment strategies may primarily consider an instant outcome for each historical task assignment. For example, in a sales queue of a contact center, conversion rate may be measured by tracking whether contacts (e.g., callers) make a purchase during their interactions with agents. A benchmarking module or another component may track which contacts were paired with one pairing strategy (e.g., BP) as opposed to an alternative pairing strategy (e.g., FIFO or PBR). The benchmarking module may determine the relative performance of BP over the alternative pairing strategy or strategies by comparing the relative conversion rates of each pairing strategy.

In some task assignment systems, a value (e.g., monetary value or other compensation) may be ascribed to the relative performance. For example, the value may be based on cost of acquisition, amount of sale, average revenue per user (ARPU), etc. This value may be used to determine compensation to be paid to a vendor or other third-party provider of the optimized task assignment strategy. For example, the compensation may be a percentage of the value attributable to the optimized task assignment strategy.

However, there may be several shortcomings to relying primarily on instant outcomes for each historical task assignment. First, a value based on ARPU does not capture actual lifetime revenue attributable to the task assignment over time. For example, in a sales queue of a contact center for subscription services (e.g., cellphone or cable television subscriptions), the ARPU for a given subscriber may be based on the assumption that the tenure of an average subscriber is 24 months. However, relatively fickle subscribers may cancel their subscription after a shorter period, resulting in actual lifetime revenue lower than expected revenue based on ARPU and expected tenure, while relatively loyal subscribers may maintain their subscriptions for a longer period, resulting in actual lifetime revenue higher than expected revenue based on ARPU and expected tenure. Thus, a benchmark based on instant outcomes may over- or underestimate the relative performance of alternative task assignment strategies depending on whether subscribers acquired from one strategy or another tend to result in actual lifetime revenue higher or lower than expected revenue.

Second, for task assignment systems in which a vendor is compensated based on ARPU and expected tenure for a relative performance in a given period (e.g., day, week, month), the entirety of the estimated value would be due or otherwise collectible following the instant outcome. For example, in a sales queue of a contact center for subscription services (e.g., cellphone or cable television subscriptions), even though the subscriber may only owe a fraction of the value attributable to the optimized task assignment strategy to the operator at the time, the vendor may be entitled to compensation months or years before the value has been realized by the operator based on the estimated value of the outcome. The timing may lead to cash flow or budgeting concerns for the operator.

As described in detail below, these shortcomings may be overcome by tracking and measuring the actual performance of historical task assignments over time rather than primarily considering instant outcomes. These techniques are sometimes referred to as “cohort tracking” or “cohort modeling” because a new cohort or group of historical task assignments may be tracked for each time period in which these techniques for benchmarking may be applied. Measurements taken for cohort tracking may facilitate measuring actual value and performance over time, and these measurements may also enable the generation of behavioral pairing models optimized for value over time instead of primarily optimizing for instant outcomes. Thus, the optimizations enabled by the task assignment strategy may be better aligned with the operator's long-term goals such as increased ARPU, increased customer loyalty/tenure/satisfaction, decreased costs, increased internal rate of return on acquisition costs, etc.

FIGS. 2A-C show tables of benchmarking data 200A-C, respectively, according to embodiments of the present disclosure. The following section with reference to FIGS. 2A-C describes an example of cohort tracking in the context of a contact center system.

In this highly simplified hypothetical, there are a total of ten customers identified as A-J. Each customer has a 12-month subscription. The contact center cycles between two contact assignment strategies, BP and FIFO, with each strategy used for 50% of the contact interactions. In other environments, there may be an arbitrarily large number of customers, with varying subscription options and durations, different benchmarking techniques for cycling among various contact assignment strategies may be used, and shorter or longer cohort tracking periods and durations (e.g., monthly for five years; weekly for ten years, yearly for eight years, etc.).

As shown in benchmarking data 200A (FIG. 2A), in Year 1, each customer A-J calls to discuss their contract renewal. Customers A-E were paired using BP, and customers F-J were paired using FIFO. During Year 1, for each of the pairing strategies, four of the five customers chose to renew their contracts (customers A-D for BP and customers F-I for FIFO), and one customer for each of the pairing strategies chose not to renew (customer E for BP and customer J for FIFO, both shown with strikethrough typeface). In Year 2, customers E and J are no longer customers, and they do not call. The remaining customers A-D and F-I each call to discuss their contract renewal again, and customers C and F chose not to renew. This process continues in Years 3-6: In Year 3, customers H and G chose not to renew; in Year 4, customers A and I chose not to renew; in Year 5, customer D chose not to renew; and, in Year 6, only customer B, who chose to renew again, remains as a customer.

In some embodiments, as in this example, it does not matter whether a member of the cohort was paired using BP in one year and FIFO the next, or vice versa. Each customer interaction may be treated independently without regard for which pairing strategy handled assignment of the customer for prior interactions.

Whereas FIG. 2A shows the contact interactions for each pairing strategy for each year, FIGS. 2B and 2C show the evolution of the Year 1 and Year 2 cohorts, respectively, for each year of measurement.

As shown in benchmarking data 200B (FIG. 2B), the Year 1 cohort (or “Y1 cohort”) includes customers A-D for BP and customers F-I for FIFO. Assuming that each customer's renewal has the same value, and BP and FIFO had an equal number of contact interactions, the relative performance gain of BP over FIFO for Year 1 is 0 (four customers for BP less four customers for FIFO). After Year 2, the Year 1 cohort may be measured again: Customers A, B, and D remain in the Y1 cohort for BP after Year 2, and customer G, H, and I remain in the Y1 cohort for FIFO after Year 2. Again, the relative performance difference is 0 (three customers for BP less three customers for FIFO). After Year 3, customers A, B, and D still remain in the Y1 cohort for BP, but only customer I remains in the Y1 cohort for FIFO. Now, the relative performance gain of BP over FIFO with respect to the Y1 cohort is +2.

In these embodiments, it does not matter to the Y1 cohort whether a customer is paired with BP or FIFO in a subsequent measurement period. For example, in Year 2, customer B was paired with FIFO and chose to renew, so it remains in the Y1 cohort for BP.

Similarly, after Year 2, the Year 2 cohort is determined based on the callers who renewed in Year 2. As shown in benchmarking data 200C (FIG. 2C), the Year 2 cohort (or “Y2 cohort”) includes customers A, G, and H for BP, and customers B, D, and I for FIFO. In these embodiments, it does not matter to the Y2 cohort whether a customer was paired with BP or FIFO in a prior period. For example, in Year 1, customer B was paired with BP and became part of the Y1 cohort for BP.

The relative performance gain for the Y2 cohort after Year 2 is 0 (three customers for BP less three customers for FIFO). After Year 3, only customer A remains in the Y2 cohort for BP, and customers B, D, and I still remain in the Y2 cohort for FIFO, so the relative performance of BP over FIFO with respect to the Y2 cohort is −2.

After Year 2, taking both the Y1 and Y2 cohorts into account, the relative performance difference is 0 for Y1 cohort plus 0 for the Y2 cohort, for a net performance difference of 0 after Year 2. After Year 3, the relative performance differences are 2 for the Y1 cohort plus −2 for the Y2 cohort for a net difference of 0.

In each period, a new cohort may be defined. For example, after Year 3, a Y3 cohort may be defined, with a relative performance difference of 0. When determining the net performance difference for each year, each cohort may be taken into account. Thus, after Year 3, the net performance difference accounting for all of the Y1, Y2, and Y3 cohorts is 2 for the Y1 cohort plus −2 for the Y2 cohort plus 0 for the Y3 cohort for a net difference of 0.

The net differences after Year 4 (counting the Y1-Y4 cohorts), Year 5 (counting the Y1-Y5 cohorts), and Year 6 (counting the Y1-Y6 cohorts) are 0, 3, and 3, respectively.

In some embodiments, the relative performance difference may be weighted based on the actual number of calls assigned using each pairing strategy. For example, in Year 6, BP assigned one call (customer B), but no calls were assigned using FIFO. Consequently, the weighted performance difference for the first year of the Y6 cohort may be 0.

In some embodiments, it may take several measurement periods to “ramp-up” the benchmark. For example, after Year 1, there is only one cohort (the Y1 cohort) and one year's worth of measurement data to compute the benchmark. After Year 2, there are two cohorts and two years' worth of measurement data, and so on. Moreover, in a real-world example, additional callers (e.g., customers K-Q, etc.) would subscribe and renew for the first time in subsequent years.

In some embodiments, a customer may only be tracked for a limited duration (e.g., five years, ten years, etc.). If this example had been limited to five-year tracking, then customer B′s renewal in Year 6 would only be relevant to the Y2-Y6 cohorts and would no longer be tracked for the Y1 cohort. Subsequently, the net (or “cumulative”) performance determined after Year 6 would only look to the performance differences measured for the Y2-Y6 cohorts. In this example, the +1 for the Y1 cohort would be disregarded, and the cumulative difference would be the sum of −1 for Y2, 1 for Y3, 1 for Y4, 1 for Y5, and 0 for Y6 (weighted), for a total of 2 instead of 3.

As described in the example above, benchmarking with cohort tracking accounts for the value of a customer over time. For example, customer B had a relatively long tenure, and the Y1 cohort for BP was credited for this tenure six times (and counting), the Y2 cohort was credited for this tenure 5 times (and counting), and so on. In contrast, customer A had a relatively short tenure, and the Y1 cohort for BP was credited for this tenure three times, and the Y2 cohort was credited 2 times.

The example above also shows that the same customer can appear in multiple cohorts and for multiple pairing strategies. Indeed, in some examples, it may be possible for a customer to appear in the same cohort multiple times, for one or more pairing strategies, depending on how many times the customer called during the given measurement period. Because each task (here, contact interaction) is effectively randomly assigned to a pairing strategy for each interaction, the cohort tracking may be configured as described above to credit the pairing strategy responsible for the outcome during the given period and each subsequent period for which the customer gives value/revenue to the contact center operator. Occasionally a successful or unsuccessful instant outcome in one period may effectively “cancel out” successful outcomes in earlier periods, but the overall cumulative net performance effectively measures the relative performance of one pairing strategy over another over time.

In some embodiments, a benchmarking module 160 or similar component may cause the task assignment system to cycle between BP and an alternative (e.g., baseline or incumbent) strategy over time periods, such as 50% BP on and 50% BP off, 80%/20%, etc. These time periods may be relatively short, e.g., switching at least once every half hour, hour, etc.

In some embodiments, the benefit (e.g., value attributable to relative performance gain) of BP may be measured by tracking BP on cohorts and BP off cohorts over a longer period (e.g., each month for 60 months, each year for ten years, etc.). The month (or, e.g., year) in which the cohort is established may be referred to as a “base month” and each month for the following five years (or, e.g., ten years) may be referred to as “measurement months.” In any given period (e.g., 12 months), there will be 12 base months, and each of those base months will have 60 follow-on measurement months.

In some embodiments, the basis for measuring a value or benefit (e.g., revenue) in each measurement month associated with a base month may be the tracking of all tasks (e.g., customer interactions) associated with BP on for a base month and those with BP off for a base month. In each measurement month following a base month, the average revenue from customer interactions in the BP on cohort may be compared with the average revenue from customer interactions in the BP off cohort (as established in the relevant base month). The difference in average revenue between the two cohorts may be weighted (e.g., multiplied) by the number of tasks/interactions from BP on in the base month to determine the value or benefit attributable to BP in a given measurement month.

In some embodiments, a BP vendor may invoice a fee for providing the value or benefit attributable to BP. The fee may be a pay-for-performance fee as a function of the value or benefit delivered.

In some embodiments, it may be possible that a cohort contains unknown customers, in which case these unknown customers may be treated as providing the same value or benefit as known customers that did not make any changes to their accounts.

In some embodiments, it may be possible that the BP vendor and/or operator may no longer want or be able to continue tracking cohorts, in which the remaining value or benefit that would have been attributable to BP for the remaining measurement months may be estimated. In some contexts, the remaining value or benefit may be used to compute a final fee for the BP vendor.

In some embodiments, the net performance may be determined (e.g., adjusted or corrected) to account for various statistical phenomena such as the Yule-Simpson effect. See, e.g., U.S. Pat. No. 9,692,899, which is hereby incorporated by reference herein.

In some embodiments, the pairing strategy may be blind to a contact's inclusion in earlier cohorts. Ignoring information about a contact's status in earlier cohorts eliminates a risk of bias in the pairing strategy. For example, a nefarious pairing strategy could, hypothetically, be optimized to make intentionally poor pairing choices to eliminate contacts from earlier cohorts associated with an alternative or underlying pairing strategy (e.g., “BP Off” or FIFO).

In some embodiments, a report may be generated showing statistically fair treatment of all contacts irrespective of their presence in earlier cohorts associated with either the “on” or “off” pairing strategies.

FIG. 3 shows a flow diagram of a benchmarking method 300 according to embodiments of the present disclosure.

Benchmarking method 300 may begin at block 310. At block 310, a first plurality of historical task assignments paired using a first pairing strategy during a first period may be determined. In the example described above, this may be the customer interactions for customers A-D who are part of the Year 1 cohort for the BP strategy.

Benchmarking method 300 may proceed to block 320. At block 320, a second plurality of historical task assignments paired using a second pairing strategy during the first period may be determined. In the example described above, this may be the customer interactions for customers F-I who are part of the Year 1 cohort for the FIFO strategy. In some embodiments, block 320 may be performed prior to, or concurrently with, block 310.

Benchmarking method 300 may proceed to block 330. At block 330, a value attributable to each task of the first and second pluralities of historical task assignments during a second period after the first period may be determined. In the example described above, this may be the values associated with each customer interaction in Year 2 associated with a customer that was part of the Year 1 cohort for BP or FIFO.

Benchmarking method 300 may proceed to block 340. At block 340, a difference in performance between the first and second pairing strategies may be determined based on the value attributable to each task during the second period. In the example described above, this may be the difference in performance associated with the Y1 cohort after Year 2.

Benchmarking method 300 may proceed to block 350. At block 350, the difference in performance may be outputted. In some embodiments, the outputted difference in performance may be combined (e.g., cumulatively) with one or more additional differences in performance measured for other periods. In the example described above, this may be the cumulative difference in performance associated with the Y1 and Y2 cohorts after Year 2.

After block 350, benchmarking method 300 may end.

FIG. 4 shows a flow diagram of a benchmarking method 400 according to embodiments of the present disclosure.

Benchmarking method 400 may begin at block 410. At block 410, a first base cohort of a first plurality of historical task assignments may be determined for at least two pairing strategies. In the example described above, this may be the customer interactions associated with the Y1 cohort for BP and FIFO after Year 1.

Benchmarking method 400 may proceed to block 420. At block 420, a first performance difference between the at least two pairing strategies may be determined after a first measurement period. In the example described above, this may be the performance difference associated with the Y1 cohort for BP and FIFO after Year 2.

Benchmarking method 400 may proceed to block 430. At block 430, the first performance difference may be outputted. In some embodiments, benchmarking method 400 may end.

In other embodiments, benchmarking method 400 may proceed to block 440. At block 440, a second base cohort of a second plurality of historical task assignments may be determined for the at least two pairing strategies for a second base period. In some embodiments, the second base period may correspond to the first measurement period associated with block 420. In the example described above, this may be the customer interactions associated with the Y2 cohort for BP and FIFO after Year 2.

Benchmarking method 400 may proceed to block 450. At block 450, a second performance difference between the at least two pairing strategies may be determined after a second measurement period based on the first and second base cohorts. In the example described above, this may be the cumulative performance difference associated with the Y1 and Y2 cohorts for BP and FIFO after Year 3. In some embodiments, the second performance difference may include a greater number of additional, intermediate performance differences associated with additional cohorts. In the example described above, the cumulative performance difference after Year 6 included (intermediate) performance differences associated with the cohorts from each of Years 1-6.

Benchmarking method 400 may proceed to block 460. At block 460, the second performance difference may be outputted.

After block 460, benchmarking method 400 may end.

At this point it should be noted that techniques for benchmarking pairing strategies in a task assignment system in accordance with the present disclosure as described above may involve the processing of input data and the generation of output data to some extent. This input data processing and output data generation may be implemented in hardware or software. For example, specific electronic components may be employed in a behavioral pairing module or similar or related circuitry for implementing the functions associated with benchmarking pairing strategies in a task assignment system in accordance with the present disclosure as described above. Alternatively, one or more processors operating in accordance with instructions may implement the functions associated with benchmarking pairing strategies in a task assignment system in accordance with the present disclosure as described above. If such is the case, it is within the scope of the present disclosure that such instructions may be stored on one or more non-transitory processor readable storage media (e.g., a magnetic disk or other storage medium), or transmitted to one or more processors via one or more signals embodied in one or more carrier waves.

The present disclosure is not to be limited in scope by the specific embodiments described herein. Indeed, other various embodiments of and modifications to the present disclosure, in addition to those described herein, will be apparent to those of ordinary skill in the art from the foregoing description and accompanying drawings. Thus, such other embodiments and modifications are intended to fall within the scope of the present disclosure. Further, although the present disclosure has been described herein in the context of at least one particular implementation in at least one particular environment for at least one particular purpose, those of ordinary skill in the art will recognize that its usefulness is not limited thereto and that the present disclosure may be beneficially implemented in any number of environments for any number of purposes. Accordingly, the claims set forth below should be construed in view of the full breadth and spirit of the present disclosure as described herein.

Claims

1. A method for benchmarking pairing strategies in a contact center system, the method comprising: determining, by at least one computer processor configured to perform task assignment operations in the contact center system, a first cohort, wherein the first cohort comprises a first plurality of contacts paired using a first pairing strategy during a first time period and a second plurality of contacts paired using a second pairing strategy during the first time period;determining, by the at least one computer processor, a second cohort, wherein the second cohort comprises a third plurality of contacts paired using the first pairing strategy during a second time period and a fourth plurality of contacts paired using the second pairing strategy during the second time period, wherein the second time period is subsequent to the first time period;determining, by the at least one computer processor, a first relative performance difference between the first pairing strategy and the second pairing strategy for the first cohort during the second time period;determining, by the at least one computer processor, a second relative performance difference between the first pairing strategy and the second pairing strategy for the second cohort during the second time period;determining, by the at least one computer processor, a net performance difference for the first pairing strategy and the second pairing strategy based on the first relative performance difference and the second relative performance difference; andoutputting, by the at least one computer processor, the net performance difference for the first pairing strategy and the second pairing strategy for benchmarking the first pairing strategy and the second pairing strategy.

2. The method of claim 1, wherein determining the first relative performance difference further comprises: determining, by the at least one computer processor, a first value associated with the first plurality of contacts paired using the first pairing strategy;determining, by the at least one computer processor, a second value associated with the second plurality of contacts paired using the second pairing strategy; andproviding, by the at least one computer processor, a difference between the first value and the second value as the first relative performance difference.

3. The method of claim 2, wherein the first value is weighted based on a first percentage of time that the first pairing strategy was used during the first time period, and wherein the second value is weighted based on a second percentage of time that the second pairing strategy was used during the second time period.

4. The method of claim 1, wherein the first time period and the second time period comprise corresponding lengths of time.

5. The method of claim 1, wherein at least one contact paired in the first cohort is in the second cohort.

6. The method of claim 5, further comprising assigning, by the at least one computer processor, the at least one contact paired in the first cohort to the third plurality of contacts or the fourth plurality of contacts independent of which pairing strategy of the first pairing strategy and the second pairing strategy that the at least one contact paired in the first cohort was associated with during the first time period.

7. The method of claim 1, wherein the net performance difference is further based on relative performance differences of at least one third cohort, wherein the at least one third cohort comprises a fifth plurality of contacts paired using the first pairing strategy during a third time period and a sixth plurality of contacts paired using the second pairing strategy during the third time period, and wherein the third time period is within a threshold amount of time from the first time period.

8. A system for benchmarking pairing strategies in a contact center system, the system comprising: at least one computer processor configured to perform task assignment operations in the contact center system, wherein the at least one computer processor is further configured to: determine a first cohort, wherein the first cohort comprises a first plurality of contacts paired using a first pairing strategy during a first time period and a second plurality of contacts paired using a second pairing strategy during the first time period;determine a second cohort, wherein the second cohort comprises a third plurality of contacts paired using the first pairing strategy during a second time period and a fourth plurality of contacts paired using the second pairing strategy during the second time period, wherein the second time period is subsequent to the first time period;determine a first relative performance difference between the first pairing strategy and the second pairing strategy for the first cohort during the second time period;determine a second relative performance difference between the first pairing strategy and the second pairing strategy for the second cohort during the second time period;determine a net performance difference for the first pairing strategy and the second pairing strategy based on the first relative performance difference and the second relative performance difference; andoutput the net performance difference for the first pairing strategy and the second pairing strategy for benchmarking the first pairing strategy and the second pairing strategy.

9. The system of claim 8, wherein the at least one computer processor is further configured to determine the first relative performance difference by: determining a first value associated with the first plurality of contacts paired using the first pairing strategy;determining a second value associated with the second plurality of contacts paired using the second pairing strategy; andproviding a difference between the first value and the second value as the first relative performance difference.

10. The system of claim 9, wherein the first value is weighted based on a first percentage of time that the first pairing strategy was used during the first time period, and wherein the second value is weighted based on a second percentage of time that the second pairing strategy was used during the second time period.

11. The system of claim 8, wherein the first time period and the second time period comprise corresponding lengths of time.

12. The system of claim 8, wherein at least one contact paired in the first cohort is in the second cohort.

13. The system of claim 12, wherein the at least one computer processor is further configured to assign the at least one contact paired in the first cohort to the third plurality of contacts or the fourth plurality of contacts independent of which pairing strategy of the first pairing strategy and the second pairing strategy that the at least one contact paired in the first cohort was associated with during the first time period.

14. The system of claim 13, wherein the net performance difference is further based on relative performance differences of at least one third cohort, wherein the at least one third cohort comprises a fifth plurality of contacts paired using the first pairing strategy during a third time period and a sixth plurality of contacts paired using the second pairing strategy during the third time period, and wherein the third time period is within a threshold amount of time from the first time period.

15. An article of manufacture for benchmarking pairing strategies in a contact center system, the article of manufacture comprising: a non-transitory processor readable medium; andinstructions stored on the medium;wherein the instructions are configured to be executed by at least one computer processor configured to perform task assignment operations in the contact center system and thereby cause the at least one computer processor to: determine a first cohort, wherein the first cohort comprises a first plurality of contacts paired using a first pairing strategy during a first time period and a second plurality of contacts paired using a second pairing strategy during the first time period;determine a second cohort, wherein the second cohort comprises a third plurality of contacts paired using the first pairing strategy during a second time period and a fourth plurality of contacts paired using the second pairing strategy during the second time period, wherein the second time period is subsequent to the first time period;determine a first relative performance difference between the first pairing strategy and the second pairing strategy for the first cohort during the second time period;determine a second relative performance difference between the first pairing strategy and the second pairing strategy for the second cohort during the second time period;determine a net performance difference for the first pairing strategy and the second pairing strategy based on the first relative performance difference and the second relative performance difference; andoutput the net performance difference for the first pairing strategy and the second pairing strategy for benchmarking the first pairing strategy and the second pairing strategy.

16. The article of manufacture of claim 15, wherein the at least one computer processor is further caused to determine the first relative performance difference by: determining a first value associated with the first plurality of contacts paired using the first pairing strategy;determining a second value associated with the second plurality of contacts paired using the second pairing strategy; andproviding a difference between the first value and the second value as the first relative performance difference.

17. The article of manufacture of claim 16, wherein the first value is weighted based on a first percentage of time that the first pairing strategy was used during the first time period, and wherein the second value is weighted based on a second percentage of time that the second pairing strategy was used during the second time period.

18. The article of manufacture of claim 15, wherein the first time period and the second time period comprise corresponding lengths of time.

19. The article of manufacture of claim 15, wherein at least one contact paired in the first cohort is in the second cohort.

20. The article of manufacture of claim 19, wherein the at least one computer processor is further caused to assign the at least one contact paired in the first cohort to the third plurality of contacts or the fourth plurality of contacts independent of which pairing strategy of the first pairing strategy and the second pairing strategy that the at least one contact paired in the first cohort was associated with during the first time period.