Patent Application
20060078868
This invention relates to a system, method, and service for automated product and/or service design and/or analysis of learning programs. More specifically, the invention relates to determining and analyzing the effect of one or more product and/or service attributes on voluntary acceptance decisions for those products/services, particularly in the domains of education and training.
Although historical and cultural influences have associated learning with children, scientific investigation tracks it from before birth through the end of life, while the spread of adult education and training programs attest to the increasing social and economic value accorded it after childhood. Engaged participation, practice and problem-solving facilitates much of adult learning. Learners will participate in a learning activity if they have sufficient motivation to do so—if the factors that attract them to the learning experience or its outcome outweigh the ones that repel them. When competing learning alternatives are available, learners will choose the ones that maximize the attractive factors and minimize the negative ones.
In both formal and informal corporate training situations, many factors influence how attracted employees are to a learning program. Especially if participation is voluntary, employees have to weigh the benefits of the program against the demands of their job and their personal life.
Typically, before a learning program is launched within an enterprise, there is considerable effort devoted to gauging the potential success of the program. If the program is to be provided by a vendor, there is some process by which to compare the merits and cost of the different vendors, such as a bid process. External authorities provide feature lists which help compare products or services offered by different vendors. For example, EduTools http://www.edutools.info/course/index.jsp is a Web site that provides assistance to higher education institutions with a decision making process for choosing the best course management system for their needs. The site has product reviews, which include over 40 product features and provide automatic comparison by features.
Various consulting organizations such as Eduworks http://www.eduworks.com/ and Chief Learning Officer magazine http://www.clomedia.com/sourcebook/details.cfm?id=74 provide guidance for how to choose the best learning program for a given customer situation. Typically consulting includes an evaluation of the current learning programs and technologies in the corporation, an assessment of these against business objectives and goals, a set of meetings or workshops to discuss and distill these, and a resulting set of recommendations regarding strategy, architecture, technology, content development, procedures, etc. In evaluating or designing a particular learning program, these consulting agencies look at factors such as the quality of the learning experience, its alignment with corporate objectives, its operational feasibility (cost, available resources, etc), which are all essential to predicting effectiveness.
As more learning takes place online, learners become empowered to make their own decisions about their learning paths and select learning programs that best correspond to their needs. This shift of responsibility and choice from the employer to the employee underscores the importance of and motivates the need to identify and measure factors that contribute to or inhibit a successful online experience.
There are quite a few studies in the open literature which list factors that determine learning effectiveness. For example, Cashion & Palmieri provide a list of 11 factors that constitute a quality online learning experience and rank them in order of importance for determining this quality. (Cashion, J. and Palmieri, P. 2002 The Secret is the Teacher: The Learner's View of Online Learning. National Center for Vocational Education Research, Leabrook, Australia). The factors are: flexibility (24%), responsive teachers (15%), materials and course design (14%), access to resources (9%), online assessment and feedback (7%), increase in information technology (IT) skills (6%), learning style (6%), interaction with other students (5%), communication (5%), ease of use (3%), and hybrid mix of face-to-face and online learning (3%).
Muilenburg & Berge list categories which are perceived by learners to be barriers to online learning: administrative structure; organizational change; technical expertise, support, and infrastructure; social interaction and program quality; faculty compensation and time; threat of technology; legal issues; evaluation effectiveness; access; and student-support services. (Muilenburg, L. Y. and Berge, Z. L. 2001. Barriers to distance education: A factor-analytic study. The American Journal of Distance Education. 15(2): 7-22.)
Outside of the learning domain proper, work has been done in collecting the factors that determine the gravitation of employees to voluntary information technology (IT) programs deployed in the enterprise. One study in particular (Venkatesh, V., Morris, M., Davis, G., and Davis, F. “User Acceptance of Information Technology: Toward a Unified View”, MIS Quarterly, V27 n3, pp 425-478, Sep. 2003) has integrated eight previously established models into one unified model to predict the “individual acceptance of information technology”. The model was empirically tested and then cross validated and explained 79% of the variance in observed IT usage. The model includes 3 factors that determine gravitation to IT deployments: performance expectancy (how will this help me with my job?), effort expectancy (how difficult will this be to use?) and social influence (what will others think about my use of this technology?). In addition, the authors include 2 direct determinants of usage behavior and several other moderating influences.
The above cited references are herein incorporated by reference in their entirety.
Services that provide automatic feature comparisons of products do not tailor the comparison to the specific conditions of the customer. Without assessing the relevance of each feature to the particular conditions of the enterprise, the value of these rigorous product comparisons to determine the potential success of a learning program is limited. Consulting agencies do relate their analysis to the particular conditions of their customers, but they do not systematically measure the motivation the learners will have to engage in the programs being evaluated. They may employ such known techniques as focus groups, to get an intuitive sense of the learners' perspective, or suggest a process of incentives to encourage employee participation, but they do not employ a systematic and rigorous method to assess the “gravitation” learners will have towards a proposed learning program. The learner perspective is not systematically broken down to the many factors that contribute to it. As a result, it could well happen that a learning program that seems effective before deployment is still unsuccessful because learners are not motivated to experience it.
State-of-the-art studies of predictors and inhibitors of online learning experiences (as mentioned above) list factors and in some cases even rank them in order of importance, but fail to arrange them into an analytic model that allows a systematic scoring of each factor and an overall score of expected effectiveness for the total learning deployment. This lack of an analytic model has the following consequences: 1) it is not clear how to measure the presence or absence of each factor, or if present—to what degree, since there are no clear set of measures associated with a factor, or a precise methodology for how to estimate it 2) it is not clear how to combine the contribution of each factor into an overall score for the predicated effectiveness of a learning deployment 3) it is not clear what corrections should be made, i.e. what factors should be changed, in order to have a favorable effectiveness expectation 4) there is no combination of factors as they are perceived by learners with factors as they are perceived by the learning providers or administrators to provide an overall model.
It is our belief that failing to systematically and accurately gauge the learner's expected attraction to a particular program before it is invested in can result in a less effective deployment. The Venkatesh et al. study on user acceptance of IT does provide an analytic model, but it is not applied to learning per-se, rather to acceptance to other kinds of IT deployments, such as databases, accounting systems or online calendaring. We believe that some factors influencing learning will be the same (e.g., how will the technology improve performance on the job) but many others are irrelevant or missing. In addition, the Venkatesh et al. study is limited in several ways: 1) it is based on interviews conducted with users, taking into account the user perspective, but fails to correlate it with the provider or administrator perspective. We believe that the prior art fails to provide this correlation, or the identification of areas in which there is no good correlation between these perspectives, which indicates how the particular customer situation should be modified to improve the expected effectiveness of the learning program. 2) The model is not granular enough—it identifies generic factors that predict IT use across many industries and many applications. We believe that in order to be an effective consultancy tool, the model needs to be sensitive to the particular industry 3) In order to best predict the effectiveness of a learning program, the model needs to be continuously updated and learn from case studies. Venkatesh et al used case studies to cross-validate their model, but did not establish a system by which each case study, with precise weighting of many factors and sub-factors, actually serves to refine the model. 4) Aggregated models such as Venkatesh et al that are constructed based on pooling of data across hypothesized or presumptively similar variables do not bear the standard of evidence of an analysis built wholly out of empirical data collected within a uniform context.
An aspect of this invention is an improved system, method, and service method for providing a systematic measure of attractiveness of a learning program to one or more prospective users.
An aspect of this invention is an improved system, method, and service method for providing a product and/or service provider one or more systematically obtained measures of learning product/service attractiveness to a prospective user.
An aspect of this invention is an improved system, method, and service method for providing a learning product and/or service provider one or more systematically obtained measures of a learning product/service attractiveness to a prospective user that are used to identify barriers to successful deployment of the learning product/service.
An aspect of this invention is an improved system, method, and service method for providing a product and/or service provider a redesign of the product/service using one or more systematically obtained measures of learning product/service attractiveness to one or more prospective users.
An aspect of this invention is an improved system, method, and service method for providing a redesign of a learning product and/or service using one or more systematically obtained measures of product/service attractiveness and product/service feedback to provide one or more prospective users a more attractive product/service.
An aspect of this invention is an improved system, method, and service method for providing consulting services to design and/or redesign product and/or services using one or more systematically obtained measures of product/service attractiveness to one or more prospective users.
An aspect of this invention is an improved system, method, and service method for providing consulting services to design and/or redesign product and/or services using one or more systematically obtained measures of the product/service to identify aspects of the product/service to change in order to improve attractiveness to one or more prospective users.
The present invention is a computer system, method, program product, and service method for evaluating, designing, and/or redesigning a voluntary program, product, and/or service (program). The invention systematically determines the attractiveness of the voluntary program, preferably a learning program, to one or more (voluntary) end users by determining one or more variables. Each of the variables defines one or more aspects of the (learning) program. An assessment value is associated with each of the variables. The assessment value is a combination of two or more importance assessments given by one or more of the users for each of the respective aspects. A provisioning value is also associated with each of the variables. The provisioning value is a combination of two or more availability assessments given by one or more stake holders for the respective aspect. Then an evaluation process determines a measure of a difference between the assessment value and the respective provisioning value for one or more of the respective variables. The evaluation process also provides a report of the measure with the respective aspects. In an alternate embodiment, the invention includes an aggregation process that combines two or more of the measures to obtain a program measure. The program measure indicates an attractiveness of the learning program to the users. Alternative embodiments of the invention are service methods for providing consulting services to evaluate, design, or redesign product and/or services provided to users.
The foregoing and other objects, aspects, and advantages will be better understood from the following non limiting detailed description of preferred embodiments of the invention with reference to the drawings that include the following:
The end users 125 each provide two or more importance assessments that are combined into an importance or assessment value 210 (see
A provisioning value 220 (see
Alternative ways of surveying (300, 300P) information from the users 125 and stake holders 130 include: a face-to-face interview, an interview form, an on-line form, a conference call, and a focus group.
The databases 170 store one or more of the variables for one or more evaluations. Each variable defines one or more aspects of the learning program/service. The databases 170 also may store the importance assessments, importance values, provisioning values 220, availability assessments, and/or comparisons between the importance values 210 and provisioning values 220 (e.g., such as the difference between the importance and provisioning values).
An evaluation process (200, 500), in alternate preferred embodiments described in
In preferred embodiments, the users 125 may include any one or more of the following: a soldier, an employee, a university student, a customer, an elementary school student, a high school student, a retired person, an e-learning student, a continuing education student, a web user, and a person with a special interest.
A user 125 can also be an ad hoc user who is not officially continuing education or is not officially an e-learning “student”, but rather, a person (like a web user) who wants to learn how to do a one time or special purpose task. For example, an ad hoc user might want to learn how to build a deck and might access a web site of a material supplier like Home Depot in order to learn building techniques. Thus the invention 100 could be used to design a web site or an e-learning presentation and/or format that is appealing to the needs of such an ad hoc or specialized user.
In preferred embodiments, the stake holder 130 may include one or more of the following: an e-learning provider, a publisher, an aggregator, a corporate officer, a government, a government agency, a university, an e-learning institution, a corporation, a community college, an online university, an online high-school, an online elementary school, a certification program, and an industry association.
In one preferred embodiment of the invention, services are provided to the end users 125 and/or the stakeholders 130. In an example of this embodiment, a consultant 190 would use the invention to determine the most effective way to increase the attractiveness of the learning program/service to the user with the minimum cost to the stakeholder. The consultant/service provider 190 might also recommend changes to the learning program/service that increase the attractiveness to the user 125 and/or reduce the cost to the stake holder 130. In alternative embodiments, the consultant/service provider 190 would design, re-design, or change the learning program/service and/or implement such modifications.
Thus the consultant 190 or service provider 190 would use the invention 100 to provide recommendations to the stake holder 130. The consultant could use the invention 100 to design, re-design, and/or change the stake holder's learning program/service. Alternatively, the consultant would evaluate existing and/or proposed learning systems to determine what needs to be added, deleted, or modified to make the learning program/service more accessible to the targeted users 125. The consultant 190 would also use the system 100 to determine what needs to be added, deleted, or modified to make the learning program/service less costly and/or more convenient for the stake holder 130 to make the learning program/service available to the user 125. Therefore, in some embodiments, these recommendations and learning system designs, re-designs, and/or changes would also be output 160 of the system 100.
In a preferred embodiment, the invention 100 uses an evaluation process 200 further described in
In an alternative preferred embodiment, the invention includes an aggregation process 240 (see
There are alternative preferred formats for the output 160. Preferred outputs include an evaluation report that associates one or more measures with the respective aspects. One preferred output 160 provides a ranking of the program aspects by (variable) measure. This is can be done with standard ranking algorithms.
In providing a consulting service, the consultant 190 often makes recommendation to modify or modifies the learning program/service to optimize the program/service effectiveness. This is accomplished by providing program aspects that are most attractive to the users with the minimum cost to the stake holder 130. In some preferred embodiments, the consultant optimizes the program effectiveness by decreasing the measured difference for one or more of the aspects in order to increase the attractiveness of the learning program/service to the users and/or decrease the cost to the stake holder 130. Therefore, the learning program/service might be modified (or proposed to be modified) for aspects when the assessment value is high and the provisioning value is low and when the assessment value is low and the provisioning value is high.
An alternative preferred output format 160 pre-selects certain of the program aspects/variables. For example, the aspects with high assessment values and/or the aspects with low provisioning values might be pre-selected. In this example, the consultant 190 and/or stake holder 130 would know which aspects are most attractive to the users 125 (the ones with high assessment values) and which are least costly to provide (low provisioning values). If the invention identifies an aspect with a high assessment value and a low provision value that is not in the learning program/service, the stake holder 130 and/or consultant 190 becomes aware of a way to increase the attractiveness of the learning program/service at a low cost. In alternative embodiments, this information (pre-selected assessment values and provisioning values) can be ranked.
In a preferred embodiment, assessment values 210 are obtained by asking individual users 125 to fill out a survey 300, exemplified in
The results of the surveys—importance values assigned by each users—are captured in Data 280 and stored in the database 170. The importance values from individual users in Data 280 can be combined to yield assessment values 210 for each variable. In one preferred embodiment, the importance values are averaged (arithmetic mean) to yield assessment values 210. Other known methods can be used to combine the importance values.
Similarly, provisioning values are obtained from providers or stakeholders 130. In the preferred embodiment, provisioning values 220 are obtained by asking the stake holders to fill out a survey, exemplified in
An evaluation step 230 compares the assessment value (U) and the provisioning value (P). In a preferred embodiment, the evaluation step 230 compares these values by calculating a difference between the assessment value (U) and the provisioning value (P) of each variable to obtain a measure (here a difference measure) 250 and outputs 160 a set of one or more measures 234. One such measure, a difference measure, subtracts the provisioning value from the assessment value to obtain the difference:
Difference Measure=U−P (250)
This will provide the difference in absolute terms. A variant on the difference measure is to make the measure weighted, rather than absolute, by multiplying the difference by the assessment value:
Weighted Difference Measure=Difference*U=(U−P)*U (250)
This weighted difference takes into account the importance users attach to each variable, so that differences in highly important variables are greater (ignoring sign) than differences in less important variables.
Other methods for establishing weights for weighted differences 234 can be used in addition, or instead of, the above weighting scheme. Weights can be determined on the basis of historical weights, available in the database 170. For example, weights may be used that were established for assessments of the attractiveness of prior learning programs and/or services, especially if the prior programs/services are determined to be similar to the program/service currently being assessed. Weights can also be assigned a-priori based on the knowledge and expertise of the service provider 130 or consultant 190 (e.g., the program variable/aspect disconnected availability of the program/service is known to be more important for mobile employees than program variable/aspect available bandwidth). From our findings there are common assessment variable weightings based on the goals of the program/service and the profile of the learners/audiences that relate to the business or industry involved (e.g., higher/continuing education, financial services training, healthcare services training, etc.). Weights can be predetermined values. Finally, the weighted difference 234 can be adjusted or normalized by using constants, in conventional ways.
Another embodiment of measure 250 is where the measure multiplies the respective assessment and provisioning values for each variable to obtain an aspect measure.
In a preferred embodiment, the measures 250 (e.g. difference measures 250) for each variable obtained in the evaluation 230 are aggregated in the Aggregation process 240 to obtain an overall program measure 270. Any known aggregation method can be used, such as the closeness of two vectors in a multi-dimensional vector-space, often used in information retrieval. (See “The Vector Space Model Tutorial Presentation”, available at http://www.scit.wlv.ac.uk/˜jphb/cp4040/mtnotes/1, which is herein incorporated by reference in its entirety.) The aggregation in this case will compute the cosine of the angle existing between two vectors—one vector comprised of all the assessment values and the other vector comprised of all of the provisioning values.
In some embodiments, the program measure 270 serves as input to the service method described in
In alternative embodiments, the aspects or variables of the learning program/service can be ranked in a ranking step 235 according to the results of the evaluation 230. For example, from highest to lowest weighted difference. Other factors can be used to define other ranking methods, or added to further refine the rank of the variables. For example, the variables are ranked by the cost it will take to decrease their weighted differences, from lowest cost to highest cost. This ranking can be done to all of the variables evaluated in 230, or to a pre-selected set only.
Finally, a report 260 is issued 160 detailing the aggregated evaluation obtained in 240. The purpose of the report is to highlight the provisioning of variables that should be addressed to either increase the attractiveness of the learning program/service to the users or to decrease the cost of provisioning.
In preferred embodiments, note that the surveys 300 and 300P are identical, except for Column 330—end users enter relevance values but stakeholders enter accessibility values. Variables may be just listed in a flat list, or as shown in
In a preferred embodiment, the variables 340 are categorized in one or more of the following factors 310: quality, value, and access. Examples of the quality factor 310 include one or more of the following components 345: production values, individualization, and end user support. Examples of the value factor 310 include the following components 345: measurement, incentive, time, and performance. Examples of the access factor 310 include one or more of the following components 345: technology, cost, awareness, time, mobility, and selection.
In some embodiments, the Access components define a learner's ability to get to a desired or needed learning experience, and include components such as technology, cost and awareness. Access components are the most tangible and most measurable. The Quality components define a learner's experience during the learning event or process. Quality components are more subjective but can be measured with the help of content and instructional design guidelines. The Value components define the learner's perception of outcomes of the learning experience. Value cannot be measured, but is assessed by learners subjectively.
The table below gives some non limiting examples of factors 310, components 345 for each factor 310, and variables/aspects relating to each component 345. There is also a description of each example component/variable and how a high user (stake holder) rating and a low user (stake holder) rating would be interpreted.
All the points below vector 430, in area 460, represent variables where the assessment value provided by the user is lower than the provisioning value provided by the learning program/service. Any variable in area 460 is a potential candidate for reducing its provisioning value in order to decrease the cost of the program/service without losing attractiveness to the user. For example, point 470 represents a variable with a big difference between the assessment value and the provisioning value. A way of measuring or visualizing the difference is to draw a horizontal line between a point in area 460, for example point 470, and a point on the vector 430 that has the same U value, 475. This difference is negative—subtracting the P value of 470 from the ideal P value of 475. Thus the sign (+/−) indicates if it's a gravitational difference or a cost saving difference.
Users 125, stakeholders 130, and consultants 190 can use the representation described in 400 in order to determine which variables could be adjusted.
The consultant 190 will first determine variables or aspects of the program 501 that is being evaluated. This is done by associating 510 assessment values 210 with variables and associating 520 provisioning values 220 with variables. This associating will be done using techniques in the respective steps 210 and 220 above. However, the consultant 190 might use or add variables that the consultant 190 considers relevant. These relevant variables might come from the consultant's experience or from databases 170 that the consultant has developed in past engagements, e.g., historical data.
The consultant's motivation is to provide suggestions to the stake holder and/or user to improve the program/service. Typically this includes suggestions, designs, re-designs, and/or modifications to improve the program/service attractiveness to the user and/or to reduce the cost to the stake holder.
Therefore, the output 160 of the invention for the consultant 190 might have particular emphasis on how to improve the learning program/service. For example, the invention output 160 might be used as input to methods that increase attractiveness to the user 580 and/or decrease cost 590 to the stake holder (and/or user).
Another goal of the consultant 190 might be to improve the historical database 170 with the information developed under the study of the current learning program/service. For example, to build an improved database 170, data from the learning program/service under evaluation are collected and stored.
If the data collected for the current engagement match the format of the historical database 170, the data can be combined with the historical data in the database. If the data collected for the current engagement do not match the format of the historical database, possibly changes to the model relating data to the measures of attractiveness might be required.
Analysis of the weightings in the database 170 can provide useful insight to the consultant. For example, the weight determined from an historical database can provide baseline ranking and/or weights for program aspects, particularly for programs/services in similar domains or industries, e.g., corporate training. Relative values of weights might give an indication of “biggest gap”—which factor is the outcome most sensitive to. Importance to an industry, program type, or business goal of a particular program aspect might be related to the weighting across the data in the database 170.
In many situations, the consultant 190 uses the invention where the individual user 125 is given the freedom to choose whether or not to participate in the learning program/service. Therefore, the consultant needs to determine what causes the user 125 to choose the learning program/service, e.g., what is attractive to the user. Therefore, while the invention is primarily used to make learning programs more attractive to the user, the same invention 100 could be used to make any choice, e.g., a product purchase choice, more attractive to the user.
