Chapter 25: Measurement of R&D and for Executive and Shareholder Use

History of Measuring R&D
In-Process / R&D Activity Metrics
Output / R&D Impact Metrics
Detailed List of 50 Metrics with Definitions, Limitations, Uses, Options, Variations, References
Output Metrics for Assessing R&D Technology Assets
Outcome / Governance Metrics and Lack of Correlation
Visual Display of Metrics for Projects’ Progress and R&D Dashboard
Appendix 1: Competitive Intelligence Metrics
Appendix 2: Intellectual Property Metrics
Appendix 3: Business Level Metrics
Appendix 4: Selected Benchmarking of R&D Metrics
Reference and Selected Bibliographic Information

——————————————————————————————-

History of Measuring R&D

As R&D executives cope with tight financial resources and demands for increasing new revenues, the pressure to measure their organizations increases. Metrics allow R&D leaders to optimize R&D’s productivity and justify to the CFO and CEO their returns on continued funding. There are actually rich sources of appropriate R&D metrics for leaders to select from. By understanding the recent history of such metrics, leaders can use them to set the context for discussions with their CEO. Additionally, it has become evident that managing any form of intellectual capital utilizes common measurement elements. Consequently, R&D leaders are in an excellent position to help the leaders of other corporate functions develop metrics for their respective organizations.

Historically, R&D management has experienced four waves of productivity improvement. Each one has brought improved understanding of how to pick R&D projects (effectiveness) and how to best carry them out (efficiency). Likewise, measures associated with evaluating and improving the effectiveness and efficiency of R&D performance have changed too.

In the early 1990s, companies faced strong global competition and were under pressure to sustain their level of shareholder return against very successful global new product development systems. For example, Asian products captured large shares of profitable growth markets and built entire new ones. For all CTO’s, this made it imperative to learn how to improve a company’s R&D organization, and make oneself a better R&D leader.

1. Re-engineering Drives Incremental R&D
The first wave of change occurred around 1990 and was driven by a re-engineering initiative that swept through all corporate cost centers. CTO’s strove to reengineer all unnecessary costs out of their R&D organizations. Smaller, division-sponsored, research programs replaced large corporate R&D budgets. The focus was on improving incremental R&D and thereby boosting the company’s next -quarter earnings. New or modified metrics were needed to measure results of the new focus. Key questions asked at this time included: When does the corporation receive its money’s worth from its R&D dollars? What size organization is required so that only value-adding functions are left? What are the minimum financial targets that should be considered for a product to be introduced? This was clearly an accountant’s view of R&D.

For this short-term focused, incremental type of R&D project, new product ideas were most likely found by listening closely to customers’ stated needs for new features. For process research, scanning the literature for proven new technology to adopt quickly was a successful strategy. Project selection for this class of development was done effectively by using return on investment (ROI) calculations because most of the numbers were known for such short-term work. Class A, B and C customer lists ensured that key relationships were supported, and risk versus capability charts helped to select R&D project portfolios. Project work was characterized by a “just do it” mentality since incremental research was for the most part easy to define. In knowledge management jargon, researchers and the CTO “knew they knew what to do.”

Developing metrics for R&D organizations was a big issue during this era. The work of Szakonyi, Robb and Leet was built upon by the Industrial Research Institute’s committee and subcommittee work (1-3). From these efforts, it was found that the best overall metric for the outcome produced by incremental R&D was to measure divisional financial performance tracked monthly on a control chart. Output metrics included divisional products mapped on a Value Chart, which maps the relative costs of products and services produced by differing technologies versus the relative performance (quality, service level, unique design features) of the same technologies. In-process results were shared with executives using monthly progress reports. The parties with the most concern and influence over R&D results during this period were divisional general managers and their business teams.

2. Next-Generation R&D Fills the Gap
By the mid-1990s, the reengineering phase had passed and most companies were productive when it came to incremental research. But at the corporate office, gaps were found between what the divisions said they could produce in revenues and the financial returns the CEO had promised “The Street.” Next-generation products and services were needed to fill the growing gap.

For everyone directly involved in corporate R&D, the pressure was on to find and develop new product ideas quickly. The new strategy was to listen and to understand an entire industry, as opposed to the wants and needs of individual customers, in an effort to find good next -generation ideas to develop. Mapping technology trends and forecasting technology platform evolution developed as methodologies to address these needs. Sanderson, Meyer and Cooper each led teams of authors who addressed these issues via either de novo approaches or by borrowing from traditional marketing research programs (4-6). Options came into their own as selection criteria. Portfolios were studied and selected based on their risk-to-reward profiles.

Now the quality aspects of doing R&D effectively became the issue. The key questions being asked were: What are the best methods by which to manage R&D? How do we know if our methods are delivering results? Which technology can best assist us in creating sustainable product lines? During this period, “quality” in product design and R&D also became buzzwords.

To execute next-generation projects efficiently, stage-gate processes were employed and resources were tapped worldwide as the East European countries began participating in the growing scientific community. Project and portfolio progress was tracked with R&D gate reviews. These were expanded as the decade progressed to include cross-functional membership, and extended past product introduction to include follow-up business reviews. Work since the early 2000s has also seen the utilization of post-launch IP gate reviews.

As this period evolved, the quality driver moved onto, “How does R&D provide the best service to the operating divisions, as well as service to customers in the form of new products and services?” Key questions became: How do we shorten the R&D cycle time? How can we access the latest technology globally? The driving person moved from an internally looking Quality Director to the external viewpoint of the VP of Marketing.

“Knowing that we didn’t know” and behaving accordingly characterized such next-generation research. The overall outcome metric was change in market share of the businesses, tracked quarterly on control charts. At the output level, Use Charts and platform road maps showed progress towards company “gap filling.” At the in-process level, projects were tracked by mapping progress in sales and marketing milestones versus milestones for research and manufacturing. Senior group vice presidents, who needed their sales targets matched to systematic introduction of successive next-generation platforms, were now directly concerned with R&D.

3. New Business Demand Fuels Breakthrough Innovation
But of course Wall Street is never satisfied and CEOs kept promising even bigger returns. This was especially so as dot-com companies overtook the market capitalization of companies that had built their businesses over many decades. Filling the need to develop whole new highly valued breakthrough businesses fell mostly to R&D leaders. Christensen, Leifer, and Downes published books on the dilemmas facing companies in need of radical innovation and killer applications (7-9). Key questions demanding answers were: How do we get the most from R&D to spur business growth? How do we access tacit knowledge from around the world? How do we acquire resources on-demand and just-in-Time? The person needing answers to these questions migrated as well, from the Marketing VP to the General Manager, COO and CEO.

We were now clearly in the space characterized by “not knowing what we didn’t know.” However, here again professional society networks and committees shared information so that successful patterns of behavior could be recognized and built upon. Companies found that the odds of finding business ideas are improved when new ventures are utilized. Typically, these are external relationships that are proactively commissioned or in which small equity stakes provide glimpses into future market needs and services. Selection is accomplished by a combination of instinct and experience. Portfolios are mapped using risk and strategic fit. Managing-the-fuzzy-front-end developed as a methodology for internal project management, whereas external R&D and ventures were found appropriate for faster moving fields.

Tracking breakthrough business progress was done with portfolio reviews conducted by venture teams looking at all aspects of the developing market, business, research, and intellectual property. The overall outcome metric became the change in a company’s market capitalization tracked quarterly as a moving average. At the output level, first- year sales of new businesses, tracked again as a moving average, was utilized. The in-process metric was typically the commercialization or “round of funding” progress displayed on venture portfolio maps. The key stakeholder became the company CEO, and a strong relationship between this individual and the R&D leader was critical for continued personal and organizational support.

4. Dot-com Crash Brings On Leveraged R&D
But, of course the world is an ever-changing place. With the collapse of the dot-com’s, 9/11, and the financial scandals that followed, all business came under intense pressure to hold revenues and profits. It is no surprise then that the world has come full circle. With increased knowledge and maturity, reengineering costs out of R&D returned. In the words of P&G’s CTO at the time, Gil Cloyd, “We must find ways to do less with less and get more.” The pressure was certainly on from corporate boards and CEOs, with the CFO at the CEO’s shoulder.

This last challenge is as tough or tougher to manage than the previous waves, and a lot remains to be learned about effective R&D leadership in this environment. Although some of the past behavior that generated success will continue as best practices, it is unlikely that all the lessons from previous waves of change will remain inviolate.

To meet the challenge of more with less, some CTO’s are using more external R&D resources to provide the diversity of thought, flexibility in project staffing, just-in-time development, Agile and Lean product development methods, and lowered total cost. Others are looking to expand the impact of internal R&D discoveries via aggressive “carrot” out-licensing programs to other non-competing, and in some cases competing, companies.

For example, IBM aggressively asked potential infringers to license IBM technology, generating for IBM over $1.5 billion per year in increased before-tax revenues. Dow Chemical and DuPont donated tens of millions of dollars in R&D’s excess intellectual property to universities, using the donation to return more money to their shareholders. These one-time benefits have run out however and more sustaining revenue models replaced these initial one-time hits.

“Carrot licensing” is a dominant form of R&D value transfer utilized extensively in the pharmaceutical industry. Biotech and medical device manufactures target their R&D programs to produce salable/licensable results for other companies to exploit. Procter and Gamble, for instance, puts all of its technology up for license to others, and at the same time was aggressive in seeking 50 percent of its new technology from other companies to fuel its own growth. This effort generated medium term results, but it is now known that the approach was short-sighted, leaving P&G with a new product development gap. Long-term metrics showing investment in new-to-the-world technologies are necessary for long-term corporate sustainability and value.

Over the time periods discussed metrics have evolved. As one might expect, the ability to measure outcomes to the satisfaction of the various stakeholders changed with their level of sophistication and business understanding. Earlier, when reengineering was done from an accountant’s perspective, metrics could be segmented by consequence, timing and activities. This was an outgrowth of the accountant’s view of the world and “activity-based-accounting” concepts in vogue at that time. What this required of R&D was segmentation of the R&D activities by activity type. This was easiest for incremental R&D, wherein activities and timing could be defined accurately enough to meet the financial standards. But for long-term research, these efforts lead to frustration. The trick was to segment activities to a level of detail that could be tracked without such a heavy overhead investment of time that people were “spending as much time recording their activities as doing the activity itself.”

Consequence Metrics vs. Time Period of Measurement

The “Consequence Metrics vs.Time Period of the Measurement“ figure shows consequence- based metrics segmented by timing. It was developed by a subcommittee of the IRI’s Research-on-Research committee in the early 1990s and provides a good set of measures for incremental R&D activities aimed primarily at a stakeholder with a financial or quality perspective. The subcommittee found that these reduced set of measures best described the linkage between R&D activities, R&D outputs and company outcomes. These metric have stood up over the decades as the best ones to use for measuring incremental innovation. For details of the metrics, see Appendix 1 at the end of this Chapter.

Activity Metrics

The “Activity Metrics” figure shows some of the activity-based metrics later developed by the same group, segmented by who was affected (consequence) and what was measured (an item internal or external to the R&D organization). This advance in measurements aided new stakeholders with quality or service interests. With these metrics they could better understand the R&D organization’s performance. However, embodied in this viewpoint was the belief that everything had to be measured, and that it was all of equal importance all of the time. R&D leadership instinctively understood that any benefits of improved communication and rapport with the Quality and Marketing VPs that resulted from these measures were offset by the slower rate of learning that such metrics imposed on their organizations. All things are not equally important all the time! These metrics do provide good examples of what to measure when the Quality and Service questions become paramount to executives.

Hierarchy of Metrics

By the mid-1990s, as the focus turned to filling the gap in corporate new product and service pipelines, another metrics concept emerged on the scene. From an R&D perspective it came through the Environmental Health and Safety departments where the concept of a hierarchy of metrics evolved. This concept was derived from the work of Abraham Maslow, replacing his human needs with metrics associated with health and safety.

Maslow stated needs as a pyramid with physiological at the bottom, rising through security/safety, love/belonging, competence/prestige/esteem, self-fulfillment, to curiosity/need to understand at the top. Implicit in this hierarchy was the concept that what was measured was what was important to the organization at the time. When a metric’s utility fell off as the R&D environment or organization’s maturity changed, the metrics switched to others more appropriate to the new situation. The focus was “measure just what you need” in order to learn and improve your processes and results. As you improve, you move your measurement focus up the metrics hierarchy. This continually lightened the old load, even as new metrics were implemented.

EH&S directors further segmented their version of Maslow’s hierarchy into those metrics important to an R&D organization’s principal stakeholders: the laboratory, corporation, and society as a whole. Examples of laboratory metrics at the lower levels include: Physiological–Does the lab meet minimum regulatory compliance requirements? Security and Safety–Will the lab be the safest and healthiest environment for me this year? Love and feelings of belonging–Is everyone part of the EH&S effort? Do my co-workers care for my safety? What was also unique about this work is shown in the “Metrics Balance” figure, which depicts constant R&D laboratory performance gradients as a function of level and stakeholder’s interest.

The key insight from this view of metrics was that to have a truly effective EH&S program you always had to make sure the organization could “safely breathe” (literally) before you worried about toxic waste for society. You had to get the laboratory in order before you could focus on the company or society. But at the same time you could not focus on the highest planes of a safe laboratory and ignore unsafe products that put the company and/or society at risk. Plotting an organization’s changing EH&S awareness and performance on this grid gave the director a real-time view of what interventions to make for learning, and a way to explain EH&S’s contributions to the lab director.

This concept was later applied at a higher level by the quality movement. The insight for Quality was that you don’t need to spend time and money measuring a capable, in-specification, in-control process at the in-process level. Outcome metrics will do fine here. In such an environment, in-process controls represent organizational inefficiency versus best-practice. If the process drifts from the ideal state, however, then the in-process metrics are appropriately reinstituted until the reflexive “breathing” resumes.

Technology Value Pyramid

Understanding the hierarchy concept led an IRI subcommittee in the late 1990s to define and organize an elaborate set of over 200 metrics into a Technology Value Pyramid (TVP). It was one of the most complete and comprehensively laid out set of R&D metrics available. The “Technology Value Pyramid” figure is a high level depiction of the metrics available. The details of this hierarchy of metrics are in Appendix 2 of this chapter. This set of measures was extremely valuable to R&D leaders in selecting appropriate metrics tied to their organization’s needs for learning, improvement and auditing.

Fuzzy Front End Metrics

As R&D organizations moved in the late 1990s to focus on true breakthrough innovation– not just in technology per se but also in unique business models associated with the new products and services– models like the TVP, as comprehensive as they were, were leaving a gap. This led to formation of an IRI subcommittee on metrics for the fuzzy front end.

It is characteristic of environments such as the fuzzy front end of research that the qualitative nature of the measures demands descriptive anchored scales to create a ranking. Over 20 companies participated in the validation stage of the subcommittee’s investigation. From an initial list of more than 90 brainstormed measures, the team developed 20 anchored scale metrics.

The 20 final metrics were grouped into eight areas of the innovation process. Because these were a mix of “apples and oranges” measures, one way in which the results were displayed is shown in the “Fuzzy Front End Metrics” figure. Of most value was comparing the patterns derived from companies in similar industries, or of one company over time. Even with an anchored scale, it is the context of other companies or one company over time that produces the insight needed to both improve and persuasively sell R&D’s early ideation stage work to senior management. Hard measure forecasting revenues and profits from FFE efforts were ineffective in correctly forecasting the future (too much guesswork in the variables), so these anchored scales were found to be best practice. Note however that it requires a high-level of thought or mental processing capability in the CEO and his or her team to use these metrics. CEOs with less processing capability do not get much from such metrics, and so with them careful selection of the Technology Value Pyramid metrics is a compromise solution.

The most recent trend in R&D is the extensive use of external R&D capabilities. Procter & Gamble, for example, has gone on record as moving toward a 50/50 split between internal and external resources. Xerox led the way using spinout ventures, Intel the use of minority funding of start-ups, and Cisco the use of acquisitions to supplement internal R&D and meet new business growth targets. Solid metrics for these activities have not been agreed upon. Outcome metrics related to standard financial performance measures of such investments in the case of Cisco and Intel are easy to utilize. In the case of Xerox and P&G, the metric is improved company performance, but attributing the return to the presence of external interactions is harder to quantify.

Today a small number of TVP metrics are in place for organizations utilizing external R&D resources. These include the percentage of new products coming from technology obtained externally within the last five years and the ratio of external to internal R&D spending. Which metrics will ultimately provide the best insight for improvement and/or audit is still undetermined.

Metrics CTOs Use

Given that CTO’s are responsible for a portfolio mixed with incremental, next- generation, and breakthrough programs, several questions are raised: Which metrics are actually in use today? How have things changed since the TVP subcommittee conducted its survey in 1997? To answer these questions, 24 IRI Board members and ROR subcommittee chairpersons were queried to determine which metrics were actually in their performance objectives for the year.

The survey presented the 22 metrics from 1997 that had been voted on for their importance and asked which ones were being used now. The survey also asked which metrics not on the list were being used. Of the 22 “important” metrics identified in 1997, only six showed up in most respondents’ 2002/2003 performance objectives. The top two “most important” 1997 metrics– strategic alignment and financial return–again made the top of the list across-the-board. Other metrics being used by most people were the projected value of the R&D pipeline, use of project milestones, development pipeline milestones achieved, and quality of the technology plan. No new metrics were reported to be in use by a significant portion of the respondents. Thus metrics can be viewed as quite dependent on the R&D Game Type the company is pursuing and the level of thought of the CEOs team.

Thus as R&D metrics moved from the quantitative (accountant’s viewpoint) to the qualitative (CEO’s viewpoint), another attribute of metrics evolved as well. Quantitative metrics are intrinsically transferable from one context to another. By this I mean that any accountant can measure the cost of a product, service, apiece of manufacturing equipment or a new computer and it will have the same meaning for everyone. However, when the ROI of the same list of items is calculated, then the picture starts to cloud.

The ROI of a new product or service is easiest to calculate for commodities where prices and costs are relatively stable and predictable. It is harder for next-generation products and hardest for breakthrough items. There are two reasons for this: 1) The future is uncertain, and 2) the price or value of intangible items is dependent on their context. For most business concepts, the future behavior is the easier of the two to model financially. Metrics accounting for future value and for option value are now available and accepted (12,13). What is still an issue for corporate functions dealing with intangible intellectual capital, however, is how to account for the change in value of an item with respect to its context or use.

For example, what is the value of a patent? This is a dollar amount that the IP and licensing organizations need to know for their quantitative metrics. What is the value of an R&D scientist? This is a number some HR organizations are now calculating to use with their quantitative metrics.

The answer to both questions is “it depends”; it depends on the use to which the patent or person is put. A patent can “sit on the shelf”, be used to prevent a competitor from competing with a company’s product or service, or be licensed to create a new revenue stream. In the latter case, the value could be high to a direct competitor, less to someone applying the art in a different field of use, and still less to a university building a portfolio of art in a specific area. What value should the licensing group assign the patent in each case?

We have been dealing with this class of problem for decades in R&D. The outputs of our laboratories can go into immediate products, launched within months, or they can be the scientific building blocks of breakthrough products to come. Depending on the viewpoint of the stakeholder and metric, greatly different values can be assigned.

Looking at the metrics contained within the Technology Value Pyramid (TVP) makes it possible to resolve the dilemma of the IP, Licensing, and HR departments. If you replace the words “research and development” with “IP department,” “Licensing department,” “HR department,” “Quality department,” etc. one obtains a set of metrics that make sense for these departments to consider using. Going on the website and browsing the metrics in more detail brings this point home even more strongly. This concept of adopting the underpinnings of R&D metrics for measuring other intangible assets is being adopted by some licensing and intellectual asset management departments with both interest and success.

What was created for use in R&D can be thus leveraged by other corporate functions. The key learning here is that all intangible intellectual capital assets have the same characteristics and metrics. In the same way, all tangible manufacturing (commodity- like) assets also have the same characteristics and metrics underpinnings as noted earlier. Because of this, R&D leaders can help other colleagues and organizations in their companies to quickly develop metrics for their area that make sense and provide consistency across functions. This will enhance executive effectiveness, efficiency, teamwork, and rapport.

Different Perspectives on Metrics

The “Metrics Perspectives” figure illustrates one way of looking at the different perspectives that metrics must provide. An immediate insight is that one is “looking at an elephant” as the seven blind men did in the fable. The key, as the various IRI metrics committees discovered, is to use metrics in a hierarchical (Maslow) manner; i.e., use those metrics that provide the most value for learning and change at any point in time, constantly altering them as the environment changes and stakeholders mature. This sounds hard until one remembers that the TVP and new Fuzzy Front End metrics make the selection and implementation process easy. A few outcome metrics for all functions, like strategic alignment, financial return and projected value of the R&D (IP, licensing, HR, etc.) pipeline offer executive teams a long-term time sequence perspective.

R&D metrics continue to progress in their level of sophistication. This has been a pattern observed over the last several decades and is a continually evolving process.

The best approach is to select a few cross-functional outcome metrics that can be applied consistently across your own company’s senior R&D, IP, Licensing, and HR functions. Augment the metrics with shorter-term internal metrics for organizational learning and improvement that change with environmental (context) conditions.

In-Process or R&D Activity Metrics

The first level of tracking projects is the activity or in-process metrics. Reporting such metrics helps teams and management track and measure their progress. When ideas are generated, they have to be selected and reviewed. Each idea will have a number of untested assumptions that have to be transformed into knowledge. Each team’s goal is to focus and test only the assumptions that are relevant to their innovation stage. This is true for both stage gate and agile / lean projects. As teams identify their assumptions and start running experiments, reporting KPI’s helps them see how much progress they are making in turning assumptions into knowledge, thus meeting the goals of their innovation stage.

Activity metrics for teams can include the number of ideas generated, the number of ideas selected, the number of ideas reviewed and the number of assumptions identified for testing. Once the teams start running experiments, the number of experiments being run, the number of customer conversations taking place, the number of customer observations and the number of usability tests can be counted. When the team begins testing solution ideas, they can count the number prototypes or minimum viable products (MVP) built, and the number of customers exposed to each product type or MVP. If the team is developing a software product and is using design sprints and hackathons, they can also track the number of these events, the number of people who participated, and the number prototypes that were created during each design Sprint.

Examples of activity / in-process metrics are:

Number of ideas generated
Number of ideas chosen
Number of assumptions
Number of experiments
Number of customer conversations
Number of customer interviews
Number of customer observations
Number of prototypes developed
Number of MVPs built
Number of hackathons held
Number of design sprints
Number of partnerships and collaborations
Number of projects in the innovation pipeline
Number products in each innovation stage
Number of ideas submitted for investment decisions
Number of investment decisions made
Number projects moving between each innovation stage / quarter
Average amount spent per stage
Number of projects by innovation type (incremental/core, adjacent/next-generation, transformational/radical)
Number of employees trained
Number of startup partnerships

In-Process Metrics for Assessing R&D Knowledge Building, Fuzzy Front End or New Concept Development Model Projects

Examples of anchored scales for rating “Knowledge Building”, “Fuzzy Front End (FFE)” or “New Concept Development Model (NCD)” projects on their pathway to technical and commercial success follow. An Overview of the Metrics for each of the FFE Activities can be summarized as:

Engine
1. Internal Resources & Capabilities
2. Motivation and Innovation Culture
3. Communication of Strategy
Influencing Factors
4. External Resources Availability (Science, People, $)
5. Legal Hurdles (Patents & Regulations)
6. Public Opinion
Opportunity Identification
7. Degree of Potential Innovation
8. Number & Variety of Approaches
9. Knowledge of Un met Needs
Opportunity Analysis
10. Alignment with Business Strategy
11. Quality of Extremal Information – Technical Plan
12. Timeliness
Idea Genesis and Enrichment
13. Number and Variety of Approaches for Idea Generation & Enrichment
14. Effectiveness, Efficiency, and Quality of Germination and Enrichment Processes
15. Quality of Ideas
Idea Selection
16. Effectiveness, Efficiency, Quality, and Speed of Processes
17. Value of Ideas (Fit with Strategy, Goodness)
Concept Definition
18. Clarity of Value Proposition
19. Clarity and Quality of Technical Path
Output from NCD Model
20. Output from NCD Model

The Detailed Anchored Scales for measuring performance on each of the Activity Metrics are as follows:

1. Internal Resources & Capabilities: Availability of necessary resources (diversity, competencies, production, third parties), presences and availability of entrepreneurs/champions and gatekeepers, effectiveness of sponsors and champions.
Anchoring Statements:
1-5. Resources for Front End activities are actively planned for, provided, and supported. All employees are expected to spend some time exploring unassigned ideas that may benefit the company. Project champions are formally trained to help them improve their effectiveness. Core competencies are managed towards achieving diverse skill sets needed to address future needs. Learnings from all projects become part of a knowledge management system.
1-4. Some resources are dedicated to Front End activities. Some employees are assigned to spend time exploring unassigned ideas that may benefit the company. Value of championing projects is recognized and encouraged. Core competencies are actively managed to assure availability. Learnings from successful and cancelled projects are formally analyzed and documented.
1-3. Some resources are available for Front End activities. A few employees are encouraged to spend time exploring unassigned ideas. A few projects have champions. Core competencies are recognized but not actively maintained. Reasons why projects succeeded and failed are discussed.
1-2. The need for Front End work is recognized but resources are not easy to get. Must justify in writing and seek formal approval to explore new ideas. Project champions are considered too pushy and self-centered. Core competencies are not recognized. Projects are documented but no analysis of success and failures.
1-1. Technical support and incremental development consume all available resources. Exploring new ideas outside of assigned projects are openly discouraged. There is no history of project champions in the company. Core competencies are not valued. No formal documentation of projects.

2. Motivation and Innovation Culture: Reward system in place/feedback to individuals, upper management articulation of need for new ideas and “holy grails,” encouragement by management for risk taking.
Anchoring Statements:
2-5. Risk taking is part of corporate culture and is expected behavior. Senior executives personally acting as champions of high-risk projects. Personal wealth creation is possible through sharing of profits from innovation. Individuals in organization encourage each other to be creative and innovative. In-house grants available to pursue new ideas with no strings attached.
2-4. Risk taking is recognized as necessary for company’s success and openly supported. Senior executives routinely emphasize need for technical innovation by everybody in organization. Significant part of compensation and bonuses are tied to Front End activities. Most in the organization are personally excited about their contributions to innovation. In-house grants frequently given to help explore new ideas.
2-3. Risk taking is often talked about and encouraged but rarely acted on. Senior management likes to project the image of being supportive of risky projects. Significant financial rewards are sometimes given for technical innovations. It seems that always the same individuals are active in higher risk projects. Funding for high risk efforts are available but must be rigorously justified.
2-2. Risk taking tolerated but not encouraged. Senior management appears reluctantly supportive of risky projects. Token financial awards are sometimes given for technical innovation. Some individuals are involved in “secret” high-risk projects. Funding for risky projects must be obtained from outside sources.
2.1. Failed efforts openly criticized. Senior management known to be skeptical of the need for innovation. No formal financial incentives for innovation. Individuals avoid becoming involved in risky projects. No in-house financial support of any kind for high-risk projects.

3. Communication of Strategy: Clear communication of strategy and vision by upper management and effective communication throughout the organization.
Anchoring Statements:
3-5. Vision reflected in all actions by senior management. Stretch goals drive company strategic planning. Business and innovation plans are integrated into one plan and supported by all. Everybody’s behavior is consistent with company’s innovation goals.
3-4. Vision part of company culture. Innovation strategy is linked with business and strategic plans. Business and R&D managers openly support each other’s plans. Everybody in organization can describe what the company’s innovation goals are.
3-3 . Most of the employees know what the company vision is. Innovation targets mentioned briefly in business and strategic plans. Some technology plans are coordinated with business plans. General aspects of the innovation goals are understood by some in the organization.
3-2. Vision communicated during public appearances by senior management. The need for innovation formally recognized in strategic plans. Efforts are made to align innovation and business strategies. Everybody in organization has heard that there are innovation goals.
3-1. Company vision not well understood by employees. No mention of technical goals in strategic plans. Business and R&D managers often talk about the need to align strategies. Not much interest in innovation goals.

4. External Resources Availability (Science, People, $): Budget, manpower and commitment from top senior management for pursuing external new technology idea generation; research, development, and/or acquisition.
Anchoring Statements:
4-5. Same as statement 4-4 plus the following condition exists; Platform researchers have a budget to retain outside expert scientists. On a quarterly basis, these scientists evaluate internal and external platform research projects for sound science. These expert scientists also are retained as company spokes persons in the event independent third parties are needed to address public relations or government inquiries concerning a fuzzy front-end technology.
4-4. Same as statement 4-3 , plus the following condition exists; An internal technology scout continuously identifies candidate technologies that may fit into existing platforms or outside platforms, but are aligned with the strategic business plan. This is accomplished through the use of web based technology transfer sites, university sites, and attendance at technology shows. Technologies identified are evaluated for fit with the strategic business plan using an agreed to set of criteria and sent to platform leaders or the Director of R&D who then assign project managers to pursue projects.
4-3. The Vice-President or Director of R&D has overall responsibility for budget, execution and delivery of new technologies from outside resources. Platform leaders and the Vice-President or Director of R&D develop a budget for current platforms and an additional discretionary budget for exploratory pursuits that may generate new platforms, all of which are aligned with the corporate strategic business plan. The budget includes funds for technology idea generation, research, concept testing, and travel. Platform leaders manage the budget and prepare quarterly update reports to appropriate financial officer. Platform leaders sometimes retain outside consultants to hold outside expert panel idea generation sessions, continuously identify ideas from outside university or government researchers, and pursue research projects or programs with outside university, government, or industrial research laboratories. Researchers from each platform are assigned as project managers to monitor projects for adherence to budgets, timetables and work plans. Internal legal counsel works with project managers to secure confidential disclosure agreements, letters of intent, project contracts, joint development agreements, and joint research agreements etc. with external researchers or external sources of technology. Legal counsel and researchers have received some training in global negotiation skills to efficiently and effectively reach agreement with the other party regarding intellectual property ownership, royalties, regulatory requirements, criteria of success and other contractual details.
4-2. The Vice-President or Director of R&D has overall responsibility for budget, execution and delivery of new technologies from outside resources. On a case-by-case basis, this officer approves funding for platform and non-platform research based on priority related to near term product development needs. Platform researchers identify external research sources through literature reports or patents or by attending technical meetings. Projects are not closely monitored for adherence to work plans, timetables, or budget. Joint ventures are discouraged. Internal legal counsel reviews and approves research contracts. Negotiations may drag on because of poor training or overworked legal counsel.
4-1. The company has a philosophy to conduct all research and development internally.

5. Legal Hurdles (Patents & Regulations) : Availability of necessary resources (budget for legal opinions and searches on a global basis) and presence and availability of internal patent agents, intellectual property and regulatory legal counsel, and scientists with legal backgrounds, effectiveness in interpreting complex technical-legal issues, product/processes that are subject to food, drug, cosmetic or environmental regulations, and products/processes that may have partial or complete patent coverage by an outside party.
Anchoring Statements:
5-5. Computerized processes are used to scan all products, ideas and processes for compliance to all applicable regulations and potential patent infringements. Staff with regulatory responsibilities are frequently trained and tested for compliance to regulations (new and existing). Domestic and international legal counsel (internal or external) continuously search for and evaluate new regulations and recently issued patents and/or patent applications to avoid regulatory action and or lawsuits that could result in deleterious financial and for public relation results. Products/processes found to be out of regulatory compliance are documented and rectified in a punctual manner with full disclosure to senior management. Intranet based continuous resolution of patent offense and defense issues by an intellectual property committee composed of internal patent attorneys/patent agents, new technology development director, global and regional business directors, VP of R&D, and invited internal scientists. Outside patents that may present an infringement issue are reviewed with detailed searches for prior art in the patent or scientific literature, evaluated for patent life, breath of coverage, and prior art quality and quantity to determine risk and the need to either declare it invalid, avoid the patent or license it from the inventors.
5-4. Manual processes are used to scan all products, ideas and processes for compliance to all applicable regulations and potential patent infringements. Staff with regulatory responsibilities are routinely trained and tested for compliance to regulations (new and existing). Legal reviews are conducted at outside firms to semiannually evaluate new regulations to avoid regulatory action and or lawsuits. Products/processes found to be out of regulatory compliance are documented and rectified in a punctual manner with full disclosure to senior management. Monthly resolution of patent offense and defense issues by an intellectual property committee composed of internal patent attorneys/patent agents, new technology development director, global and regional business directors, VP of R&D, and invited internal scientists. Outside patents that may present an infringement issue usually resolved by avoidance or licensing.
5-3. Manual processes are used to scan products and/or processes just before commercialization for compliance to all applicable regulations and potential patent infringements. Staff with regulatory responsibilities are occasionally trained and tested for compliance to regulations (new and existing). Legal reviews are conducted at outside firms to annually evaluate new regulations to avoid regulatory action and or lawsuits. Products/processes found to be out of regulatory compliance are documented and rectified in a punctual manner with full disclosure to senior management. Quarterly meeting of an intellectual property committee composed of internal patent attorneys/patent agents, new technology development director, regional business directors, VP of R&D, and invited internal scientists review new ideas for fast track patent applications, create a strategy for filing a variety of patent applications to surround a fuzzy front end technology, patent applications for nationalization, nationalized patents for discontinuing annuity payment (cost savings), and new competitor patent applications and issued patents for action plans. Outside patents that may present an infringement issue are reviewed with detailed searches for prior art in the patent or scientific literature (conducted outside or internally), evaluated for patent life, breathe of coverage, and prior art quality and quantity. Typically, outside patents that may present an infringement issue are avoided by reformulation or changing a process.
5-2. Informal and occasional checks of products and/or processes just before commercialization are done for compliance to all applicable regulations and some known problem patents. Staff with regulatory responsibilities are sometimes trained and tested for compliance to regulations (new and existing). Products/processes found to be out of regulatory compliance are sometimes documented and rectified eventually with no disclosure to senior management. Infrequent meetings of an intellectual property committee composed of the VP R&D, patent attorney, and a corporate business director to review evaluated ideas; determine priority of patent filings, and filing strategy. Outside patents that they review that may present an infringement issue are sometimes ignored with risk accepted by management.
5-1. Laboratory notebooks and a computerized idea system are used to document and witness new ideas for future patent and new product opportunities. No formal review of patent filings or the company conducts potential infringement risk patents.

6. Public Opinion: As an Influencing Factor, Public Opinion relates to the public’s sensitivity, awareness, and perception of your company, your policies, your products, and your actions and how they relate to the changing array of local and current values, concerns, and issues. As a metric for the Front End of Innovation, you would want to know how well prepared you are to sense, react, influence, and respond to the relatively uncontrollable changes in public opinion and the public’s perceptions of your policies, procedures, products, and actions.
Anchoring Statements:
6-5. An on-going, active process to monitor, assess, communicate internally the public’s opinions of the company’s reputation as a respectable, ethical, compassionate corporate citizen in their country, their region, and their locality. The process also develops and disseminates information to proactively improve the public’s opinion of the company, its policies, and products. Company policies and procedures are adjusted in an appropriate manner relative to changes in public interests, values, and concerns when necessary. Should a problem arise, trained resources are prepared and in place to respond in such a way as to minimize damage to the company’s image and to reinforce its concern for its neighbors and its compliance with all regulations.
6-4. An on-going, active process to monitor and assess public opinions around the company’s reputation as a respectable, ethical, compassionate corporate citizen in their country, their region, and their locality. Information about public opinion is used by some individuals to influence decisions about potential new policies, procedures, products, and affiliations. The company is careful to comply with both the letter and the intent of all regulations and standards of ethical behavior.
6-3. An informal process to occasionally monitor public opinion, which does not share information widely across the company. If some company policy or activity produced a negative public response in the recent past, similar issues may be discussed relative to new products, services, processes, or business models. Information about public opinion is occasionally used by anyone when making decisions about potential new policies, procedures, products, and affiliations. Minimum compliance with all regulations is addressed when designing something new or when the regulations change.
6-2. An informal process to monitor public opinion only after an incident or issue arises. Public opinion is rarely considered when making decisions about new offerings. Minimum compliance with all regulations is addressed only when designing something new; there is no monitoring of changes in regulations.
6-1. Interest in public opinion is low and attention to it is only present after some part of the company’s business is negatively impacted by a news report or other public response. There is so much concentration on financial performance that public opinion is rarely considered. There is wide spread panic by most employees when the local news van pulls up in front of the building.

7. Degree of Potential Innovation: The “Degree of Potential Innovation” metric measures how ready a corporation is to accept and pursue Innovation.
Anchoring Statements:
7-5. Senior executives actively seek and embrace disruptive technologies as growth opportunities. The corporate portfolio of growth initiatives is continuously monitored to ensure a healthy balance of risk, reward, cash flow and growth into new markets. Corporate strategy is often developed or modified on the basis of emerging trends.
7-4. Growth opportunities are actively considered when they fit with corporate strategy. New competencies are added when required to support customer needs. Risk management is in place, and appropriate risk-taking is encouraged. Significant effort in R&D and business development exists.
7-3. Customer needs often drive new product development efforts, even when they go beyond traditional product and service offering.
7-2. Effort is spent on incremental expansion of new products and services, but with modest effort and risk taking.
7-1. Focus is on current products and services and proven technologies. Little time and patience is devoted to out-of-the-box concepts and emerging trends. Risk-taking is avoided.

8. Number & Variety of Approaches: The “Number and Variety of Approaches” metric measures the ability of a corporation to capture and understand opportunities for growth.
Anchoring Statements:
8-5. Long range forecasts are developed jointly with customers. VOC and QFD are routinely used with existing and potential new customers. Results of customer meetings are continuously communicated inside the company to feed new product planning efforts and to generate growth new product ideas. Teams are in place to examine future technology and market trends. R&D is part of an integrated business team working with a similar team at the customer.
8-4. Knowledge of customer needs is shared beyond the sales force. R&D has good exposure to customers. Regular meetings are held to capture and distribute knowledge of customer needs. R&D gets input from multiple functions at the customer.
8-3. A customer needs database is in place, and kept up to date. Customer needs are reviewed annually or semi-annually with rest of corporation as business planning occurs. R&D interfaces with only R&D at the customer.
8-2. Sales force is main contact with customers to identify needs and opportunities. Importance of customer needs is recognized but there is no support for addressing it. R&D is discouraged from customer contact.
8-1. The corporation has more of an internal focus. Sales must fit with existing assets and product catalogues offerings. Communications with customers is minimal. R&D is not trusted and prohibited from customer contact.

9. Knowledge of Unmet Needs: The “Knowledge of Unmet Needs” metric measures the ability of a corporation to find and communicate unmet needs.
Anchoring Statements:
9-5. Continuously exploring and forecasting the needs of current customers and industries, as well as potential new customers and industries. Formal knowledge management methods are routinely used to ensure that unmet needs are communicated within corporation as input to business planning.
9-4. Proactively attempting to understand unmet needs of your customer (and your customer’s customer) through multiple contact points, in order to determine both current and future unmet needs. Contacts often included R&D and advanced product-planning functions.
9-3. Frequent meetings with existing customers to assess their current unmet needs. Techniques such as Voice of customer employed to identify existing needs not yet recognized by the customer.
9-2. Receptive to customer input on problems and challenges. Maintains routine communication channels to permit customer input. Little proactive probing to learn customer’s unmet needs.
9-1. Focus is exclusively on providing current products and services to existing customers. Minimal communications with customers and no formal feedback methods are used. Customer complaints and comments are seldom examined in depth in order to understand the customer’s true unmet needs.

10. Alignment with Business Strategy: A product / process development opportunity is assessed to determine if it is worth pursuing. This element outlines but does not exhaustively address the technology and market uncertainty involved in innovation, but is intended to get enough of a grasp on a perceived opportunity to gauge whether other NCD elements should be pursued. The focus of this metric is not to evaluate the quality of the opportunity; it is to evaluate the quality of the process.
Anchoring Statements:
10-5. There is a clear and concise written process, used routinely, to assess the relationship between an opportunity and to determine the fit with the corporate and business unit strategies. – or – “typical project has a clear. … ” And the alignment is compelling enough to support immediately proceeding as proposed.
I 0-4. Alignment with business strategy is outlined but not compelling enough to make an immediate decision to proceed as proposed. Some discussion and a little further evaluation will be needed to assess the risks versus the rewards of developing this technology and/or offering before proceeding.
10-3. Alignment with business strategy is fairly clear but lacks details about a few key issues. More information is needed and much more work needs to be done to clarify the degree of alignment before making a decision to proceed.
10-2. There is a rough definition of the alignment with business strategy but there is not enough information to discuss or evaluate. Much more information is needed and significantly more work needs to be done to outline strategic alignment before a decision could possibly be made to proceed.
10-1. There is no written declaration of alignment with business strategy. No decision can be made as to the merits of this concept.

11. Quality of External Information – Technical Plan: How clearly defined are the hypotheses, the technical development path, the experimental paradigm, any rate limiting factors, and the plan to deal with any regulatory requirements and issues? Is this Concept attractive to the company from the standpoint of its contribution to and defensibility of Intellectual Property estate of the firm?
Anchoring Statements:
11-5. There is a clear and concise written definition of the Technical Path along with a compelling Business Case for the company to invest in the development of this technology and/or offering. The hypotheses are clearly stated and easily understood. Plans for dealing with regulatory and intellectual property issues are detailed and specific. It is so easy to comprehend the benefits of going down the Technical Path as proposed and the fact that those benefits far outweigh any costs or risks involved in development that the only logical decision is to agree with the plan as proposed, to resource the project per the recommendations in the document, and to immediately begin development work as outlined in the document.
11-4. The Technical Plan as written is mostly clear and concise despite issues with a few details. The hypotheses are understandable as stated but not much detail is provided. Plans for dealing with regulatory and intellectual property issues are not fairly detailed. The Technical Path appears attractive enough to be worthy of consideration for future work, but the Business Case is not compelling enough to make an immediate decision to proceed as proposed. Some discussion and a little further evaluation will be needed to assess the risks versus the rewards of developing this techno logy and/or offering before proceeding.
11-3. The Technical Path is fairly clear but may lack details about a few key issues. The hypotheses are not clearly defined and the plans for dealing with regulatory and intellectual property issues are not very detailed. If more information is needed and much more work needs to be done to clarify the Technical Path for going forward before making a decision to proceed. Significant discussion, investigation, and/or evaluation will be needed to assess the risks versus the rewards of developing this technology and/or offering.
11-2. There is only a high level, very rough description of the Technical Path. A decision cannot be made whether or not to begin development work of any kind. Much more information is needed and significantly more -work needs to be done to clarify the Technical Path before a decision could possibly be made to proceed.
II-1. There is no written definition or plan for the Technical Path. It is not clear what technology needs to be developed for the Concept. No decision can be made about the Technical Path.

12. Timeliness: The Timeliness metric characterizes how “up-to-date” the information is, which is used to assess an opportunity.
Anchoring Statements:
12-5. The information used was generated very recently (days to months, depending on the rate of technological and economic change in the opportunity area) and measures exactly the activity you are trying to assess. For example, a new release of data detailing the number of cable television subscribers on a metropolitan system (government data)
12-4. The data is reasonably current but may be aggregated in a way that requires some additional analysis or is not exactly what you are trying to measure. For example, weather data for a state may not describe precise1.y the weather conditions in a county.
12-3. The information used was generated a while ago, but is still useful for reckoning. For example, weather forecasts from the Farmers’ Almanac.
12-2. The information is old and updates are available, but could be reliably estimated. An exaI11ple of data in this category might include financial data from a publicly traded firm that is more than three quarters old, as those publicly traded firms release non-audited data more often.
12-1. The information was old and cannot be adjusted to aid reckoning. Examples of data in in is category might include full census data more than) 0 years old, any non-current legal information, and syndicated studies in which the source says a major update is available.

13. Number and Variety of Approaches for Idea Generation & Enrichment: New ideas documented as part of technical / marketing memos as well as a number and variety of new idea generation processes (brainstorm with customers, trade shows).
Anchoring Statements:
13-5. Many parallel and diverse approaches are used to generate and enrich ideas and the output is documented and shared widely. For example, an established, system for soliciting ideas exists on the company’s intranet; the database is searchable and individuals from throughout the company can “tag on” to other individual’s ideas; the system is continually improved through a cross-functional team that meets on a regular (quarterly) basis. Ideas are generated from a variety of sources (minimum of five); a cross-functional team evaluates where and how the most effective ideas are generated and make recommendations for change. Individuals or teams of experts exist that stay abreast of new techniques and train others on idea generation. New ideas are documented and shared openly and systematically as an integral part of visit reports and technical memos.
13-4. Experts exist within the company to give advice on gathering ideas. Brainstorming and other techniques are promoted, taught, and frequently applied within the company. Structured interaction between technical and sales and marketing personnel occur on a frequent basis. Regular, organized visits are made to universities, trade show and industry meetings.
13-3. More than one approach is used to generate and enrich ideas. Idea generation sessions are infrequent, but the output is documented and shared. Customer and supplier visits are documented and shared to extract ideas. Technical people are encouraged to travel with sales and marketing people to gather ideas.
13-2. Researchers are encouraged to visit trade shows and industry meetings to stimulate new ideas. Idea generation sessions are rare and there is limited sharing of ideas or output. Human resources has available methods and reference materials for seeking new ideas, but they are used infrequently.
13-1. There is no system for encouraging, soliciting, or sharing ideas. The generation is random and haphazard and results from individual initiative. There is no training for individuals seeking new ideas.

14. Effectiveness, Efficiency, and Quality of Germination and Enrichment Processes: Effectiveness of early evaluation/idea germination process; quality of ideas generated and efficiency of early evaluation/idea germination process.
Anchoring Statements:
14-5. Individuals or teams exist whose full-time job is to coach individuals on developing their ideas. Ideas are tracked from generation through implementation by a central group allowing analysis. Ideas are generated within a field (strategy) defined by management. Goals for new idea generation are included in performance management; individuals and teams receive rewards for generating ideas and greater financial rewards when their ideas are commercialized. A cross-functional team surveys individuals and teams that have generated and developed new ideas that have gone both gone into the NPPD (succeeded) or died (failed) to determine strengths and weaknesses of the process; recommendations are made to upper management for improvement.
14-4. Mentoring is provided by senior experts who have been successful in new product development. Individuals and teams that originate new ideas are recognized and rewarded. Benchmarking of idea generation processes are made with other companies. There is a formal idea tracking process at a business unit level. A high (15%) percentage of time is allowed to develop ideas. Strategies are developed that allow idea generation to be focused in fertile areas.
14-3. Time is allocated for “fleshing out” new ideas but time in reality the time is rarely available. Tracking is done at a functional level. Individuals and teams that originate new ideas are recognized after the product reaches commercialization. Ideas are aligned with the business strategy. Analysis of success/failure is done intermittently on an ad-hoc basis.
14-2. Time is not allocated for “fleshing out” new ideas. Sometimes acknowledgement or feedback is given to the person or team that originated the idea. Tracking of ideas is done intermittently. No monitoring of the quality of the ideas is made.
14-1. No help is provided to individuals or teams; only strong-willed idea-generators are successful. No tracking of ideas is done. No analysis is made of success versus failure. No acknowledgement or feedback is given to the person or team that originated the idea. Time is not allocated for developing new ideas. There is no system for encouraging or soliciting ideas.

15. Quality of Ideas: Number, Variety, Level (Platform versus One-Offering), and Blossoming of fertile platforms and sponsor ‘wow’ factor.
Anchoring Statements:
15-5. Such a large number of high-quality ideas are generated that the company has the flexibility to select only the best ideas and still fill their long and short-term portfolio needs. A targeted number of ideas evolve to be platform technologies and products that lead to many other new product offerings. A few ideas have to the potential to be disruptive and/or significantly impact the future of the company. Ideas often receive extraordinary attention from upper management; for example, eliciting phone calls and letters of congratulations. Ideas are useful and unique and result in strong patents that defend products or are marketable themselves.
15-4. New ideas originate within a variety of centers: research, sales, marketing, etc. Ideas are appropriately distributed within the four quadrants of marketing versus technical and new-to-the-company versus new-to-the-world. New product ideas are illustrated in the company’s annual reports and/or enhance the company image, and everyone generally agrees these are “real” rather than “superficial.” A big backlog of new ideas exists exceeding the number to fill the long and short-term needs of the portfolio. Portfolio of new ideas creates new platforms for the company.
15-3. Number of ideas is sufficient to challenge the new product teams and provides sufficient number of ideas to keep the pipeline filled with good ideas. A few of the ideas are out-of-the-box and stretch the marketing and/or technical capabilities of the company. New ideas are sources of potential patents.
15-2. Numerous ideas are generated but the process does not yield the number of new ideas required for your portfolio. Ideas are sound but rarely elicit upper management accolade. New ideas concentrate on a limited number of competencies. A relatively low number of people generate most of the ideas.
I5-1. Few ideas are generated; a high percentage of those get into the NPPD by default. Ideas are centered on existing business areas; only incremental improvements of existing products are envisioned. The incremental ideas generated are standalone and do not lead to other products. Ideas do not result in patents.

16. Effectiveness, Efficiency, Quality, and Speed of Processes: This metric pertains to the processes used to arrive at the final ideal selection.
Anchoring Statements:
16-5. Ideas are reviewed on a regular basis with consistent formats. There is a high quality of dialogue and clear decisions or constructive guidance are offered in all of the cases. Key parties (decision-makers) are present and represent all key stakeholders. All ideas are reviewed within three months or if necessary as required. Decisions include consideration for the rest of the business and the portfolio.
16-4. All ideas are reviewed on a regular and predetermined schedule. Clear decisions are made in all cases based upon good understanding of the value to the business and the benefit to the customer. Quality of dialogue is high. Key parties and Key stakeholders are always present. Completeness of review, consistency of criteria, and full stakeholder involvement are always evident.
16-3. Most ideas are reviewed on a regular basis with reasonably consistent formats. Clear decisions are made in half or more of the cases based upon an understanding of the value to the business and the benefit for the customer. Quality of dialogue varies by project and champion. Key parties are present but not consistently. Key stakeholders usually dealt with inside of the formal sessions. Completeness of review, consistency of criteria, and full stakeholder involvement are always evident.
16-2. Ideas are reviewed occasionally, but structure and content of the information varies and is of varying quality. Clear decisions are made only in some of the cases, but are not based on well-developed rationales. Operation is by a representative committee that does not include the key parties. Key stakeholders are dealt with outside of the formal sessions. Ideas are reviewed as desired by management or as a result of individual championship.
16-1. Ideas are not reviewed or are reviewed in inconsistent and incomplete ways. No clear decisions are made. Key stakeholders, if they are even identified, are involved only on an ad hoc and occasional basis.

17. Value of 1deas (Fit with Strategy, Goodness) : Idea Selection – as pertaining to absolute value of ideas that are approved by the NCD process – this metric characterizes the potential business impact of the idea, which ultimately is selected in terms of overall contributions.
Anchoring Statements:
17-5. Approved ideas are selected with processes that clearly balance short-term and long-term business objectives. Approved ideas typically are selected that will add significant new lines of business and/or rejuvenate/protect existing business strategies. The pipeline has a balanced mix that affords the company flexibility with a variety of options. Risks exist regarding success, but they are fully understood and are embraced by all key executives.
17-4. Approved ideas are selected with processes that begin to balance short-term and long-term business objectives. Several ideas provide the potential to add significant new lines of business and/or rejuvenate or protect existing business strategies. Expectations regarding approved ideas vary significantly in terms of the likely fit with and contribution to existing business strategies. A portfolio of ideas is selected with a healthy mix of possibilities and time frames. Its character and needs are only partially understood by key executives.
17-3. Approved ideas are selected with processes that take into account business strategies and commercial impact. However, they tend to be conservative and incremental in terms of their ability to add significant new lines of business and/or rejuvenate or protect existing business strategies. The risks involved relative to success are not appreciated and short-term results are emphasized.
17-2. Ideas are selected that are often the ‘flavor of the month’ or are the result of a single thought and are not evaluated consistently relative to the potential impact on business strategies and commercial impact. Ideas are selected due to personal preferences and biases.
17-1. Approved ideas have a distant or very uncertain relationship to the business strategies and commercial relevance.

18. Clarity of Value Proposition: How clear is the value proposition, including the consideration of alternatives? How much confidence is there in the projection of return and value over time?
Anchoring Statements:
18-5. Concept Definitions and Value Propositions are so clear and compelling that high quality decisions are made as to project selection. The assumptions, alternatives, and trade-offs are clearly documented, validated, and understood and facilitate meaningful discussion. For selected concepts, there is a clear and concise written definition of the Concept along with a Business Case for the company to invest in the development of this technology and/or offering. Note that “written” in this case does NOT necessarily mean a formal, “polished” report. It could refer to an electronic mail message, PowerPoint presentation, or even a handwritten set of notes, so long as it meets the needs of the business for clarity. It must be so easy to comprehend the benefits of this Concept and the fact that those benefits far outweigh any costs or risks involved in development that the only logical decision is to agree with the plan as proposed, to resource the project per the recommendations in the document, and to immediately begin development work as outlined in the document.
18-4. Concept Definitions are clear and concise despite issues with a few details. The Value Proposition provides sufficient information to determine the attractiveness of the concept and has been validated by objective and respected individuals, alternatives and trade-offs have been considered, but the Business Case is not enough to make an immediate decision to proceed as proposed. Some discussion and a little further evaluation will be needed to assess the risks versus the rewards of developing this technology and/or offering before proceeding.
18-3. Concept Definition is fairly clear but may lack details about a few key issues. The Business Case does not highlight or effectively communicate key issues and it may be difficult to understand some of the points that are made. The Value Proposition does not clearly indicate the attractiveness of the concept or highlight key risk factors. More information is needed and much more work needs to be done to clarify the concept, the Business Case, and/or the plan for going forward before making a decision to proceed. Significant discussion, investigation, and/or evaluation will be needed to assess the risks versus the rewards of developing this technology and/or offering.
18-2. There is a rough definition of the Concept but there is no Business Case or development plan to discuss or evaluate. A decision cannot be made whether or not to begin development work of any kind. Much more information is needed and significantly more work needs to be done to clarify the concept, the Business Case, the Value Proposition, and/or the plan for going forward before a decision could possibly be made to proceed.
18-1. There is no written definition, plan, or Business Case. It is not clear what the Concept even is. No decision can be made as to the merits of this concept.

19. Clarity and Quality of Technical Path: How clearly defined are the hypotheses, the technical development path, the experimental paradigm, any rate limiting factors, and the plan to deal with any regulatory requirements and issues? Is this Concept attractive to the company from the standpoint of its contribution to and defensibility of Intellectual Property estate of the firm?
Anchoring Statements:
19-5. Technical Paths are so clear and compelling that high quality decisions are made as to project selection. For all concepts, there is a clear and concise written* definition of the Technical Path along with go/no go milestones and deliverables and a Business Case for the company to invest in the development of this technology and/or offering. The hypotheses are clearly stated and easily understood, no important details are missing. Plans for dealing with regulatory and intellectual property issues are detailed and specific. It is so easy to comprehend the benefits of going down the Technical Path as proposed and the fact that those benefits far outweigh any costs or risks involved in development that {he only logical decision is to agree with the plan as proposed, to resource the project per the recommendations in the document and to immediately begin development work as outlined in the document.
19-4. The Technical Plan for all Concepts as written is clear and concise despite issues with a few details. The hypotheses are understandable as stated and some detail is provided. Plans for dealing with regulatory and intellectual property are well understood but implementation plans are not complete or fully detailed. Alternatives have been identified and fully explored and activity-based milestones and deliverables have been defined. The Technical Path appears attractive enough to be worthy of consideration for future work but the Business Case is not attractive enough to make an immediate decision to proceed as proposed. Some discussion and a little further evaluation will be needed to assess the risks versus the rewards of developing this technology and/or offering before proceeding.
19-3. The Technical Path for most Concepts is fairly clear but may lack details about a few key issues. The hypotheses are not clearly defined and the plans for dealing with regulatory and intellectual property issues are not very detailed and some milestones and deliverables have been defined. More information is needed and more work needs to be done to clarify the Technical Path for going forward before making a decision to proceed. Significant discussion, investigation, and/or evaluation will be needed to assess the risks versus the rewards of developing this technology and/or offering.
19-2. Most Concepts only have a high level, very rough description of the Technical Path and no alternatives have been identified. A decision cannot be made whether or not to begin development work of any kind. Significantly more information is needed and significantly more work needs to be done to clarify the Technical Path before a decision could possibly be made to proceed. No consideration has been given to competitive technologies or intellectual property.
19-1. In general, there is no written definition or plan for the Technical Path. It is not clear what technology needs to be developed for the Concept. No decision can be made about the Technical Path.

20. Output from NCD Model: The desired output from the New Concept Development Model is a dynamic portfolio of concepts whose overall purpose is to provide options for company growth and support of specific strategies. It also recognizes the importance of capturing the interest(s) and commitment of business managers in anticipation of (eventual) implementation. This element provides the only entree into the New Product, Process or Technology Development. The output is a Concept Definition, which requires a clear statement of the business case, customer needs, a definition of features and benefits, a broad understanding of the technology path, and risks, assumptions, and trade-offs. At this gate, a decision is made whether or not to launch a project and commit significant resources (time, money, and people).
Anchoring Statements:
20-5. The portfolio of projects output from the NCD efforts provides a healthy and balanced spectrum of growth options that accounts for resource availability, return on investment, risk, timing, and overall corporate strategy. The portfolio contains a robust balance of attractive business cases that provide the flexibility to grow the present businesses and/or to create new businesses. Business interest is shared by all key managers and stakeholders and alignment with business strategy is excellent.
20-4. The financial outcomes from the NPD efforts are clear and offer substantial benefit to the business and the customer when considered independent of other projects. The portfolio contains a balance of attractive business cases that range from easy fits with existing infrastructure to those requiring new investments. Business interest is shared by most key managers and stakeholders and alignment with business strategy is good.
20-3. The financial outcomes from the NCD efforts are clear and offer significant benefit to the business and the customer. There are a reasonable number of attractive business cases that range from easy fits with existing infrastructure to those requiring new investments. However, business interest varies based upon persona) interests of key managers.
20-2. The financial outcomes and benefits from the NCD efforts are clear. There are a small number of concepts with good business cases, but business interest is poor because the choices within the portfolio appear irrelevant, too meager in outcome, or too risky.
20-1. The financial outcomes and benefits from the NCD efforts are uncertain. The portfolio of choices looks risky at best to the existing businesses. Organization is not satisfied with the output of the NCD and constantly questions continued investment.

In-Process Metrics for Assessing R&D Development and Commercialization Projects

Examples of anchored scales for rating development projects on their pathway to technical and commercial success are:

Proprietary position
Competencies and skills
Complexity
Access to external technology
Manufacturing capability
Customer and market need
Market and brand recognition
Distribution channels
Customer strength
Raw material supply
Environment health and safety

When using these anchored scales the scale of 1 to 5 has been used. The answers to each question need not be absolutely precise, but should enable comparisons. Each of the attributes should also have a weighting factor. Note that the weightings used when screening potential projects in the concept stage at the fuzzy front end might be quite different from those that are for more mature projects. The anchored scale score times the weighting of the factor would give a weighted success factor score. These factor scores would be totaled to give an overall rating for various projects.

In-Process Metrics for Assessing Agile / Lean Projects
Projects

In-Process or Activity metrics are great for tracking Agile / Lean projects too. Indeed, for each team, before beginning any experiment they should set the minimum success or fail criteria. These criteria will set the benchmark for how the team will know whether their assumptions are supported by the evidence. In the end of each experiment the team should analyze lessons learned and the then make decisions about what to do next. This process of analyzing experimental evidence and decision-making is innovation accounting at its most granular level, as embedded in the scientific problem-solving methodology and the Agile / Lean create-test-learn loop methodologies.

Each metric is not equally important at every innovation sprint. Depending on the business model, innovators must focus on the metrics that matter most for their innovation stage. During the early stages confirming customer needs and validating solutions are key impact metrics. Later on in latter stages, showing traction, gaining revenues and profits are key. This is where metrics such as customer churn rates, cost of customer acquisition, and customer lifetime value become important.

Examples of impact / output metrics are:

risky assumptions identified
hypotheses developed
minimum fail criteria set
experimental results
cohort analysis
decisions made (pivot, persevere, retest)
rate of customer acquisition
percentage of customers having a great experience
is the product meeting their job needs
are customers being retained
are customers willing to pay for product
are customers paying enough to cover costs and make a profit
are customers happy enough to refer product to other people
customer lifetime value
cost per learning
time cost per learning
learning velocity
validation velocity
number of patents granted
new business models ready to scale
cost savings
innovation conversion
new market segment entered
assumptions and knowledge ratio
percent of products at each stage and or gate
percent of products a problem-solution fit
percent of products at product-market fit
percent of products at scale
number of validated business models
validation velocity
returns on product development expense
process improvement metrics

In-Process Metrics for Self-Assessing R&D Organizations

At the organization level, R&D measures for self-organizational assessment are based upon Open Culture, Value Creation Passion, Articulating Compelling Business Cases, Organizational Learning Processes, Catalyzing Breakthrough Options, External Focus, Addressing The Full Company Value Chain, Robust Decision-Making, Incentives, And Implementing In The Face Of Risk and Uncertainty. When using this assessment, anchored scales of -3 to +3 are used. The following guide offers anchoring statements at each of the extremes. Metrics for each of the Activities of the FFE are provided. The Activities are:

Anchored Scales for Rating R&D Organizations

Output or R&D Impact Metrics

The goals of the Output or Impact Metrics are to improve the quality of decision making and the “practice of R&D”. Note that there is overlap between In-Process and Output metrics. Many can be the same; it is just the scope of subject matter under evaluation that changes, e.g. a project or sub-activity within the R&D organization (In-Process), or the output of the entire R&D laboratory or group (Output).

The Output or Impact Metrics use three different approaches for investigating the effectiveness and overall output of R&D. These are:

SECTION 1. The Technology Value Pyramid (TVP) section contains metrics related to:

Value Creation
Managing a Portfolio of R&D Activities
Integration of R&D with Business
Value of Technical Assets
Practice of R&D
What Typical Stakeholders Are Interested In

SECTION 2. The Process of R&D section contains metrics associated with:

the process of R&D for both a project or a portfolio of projects
Stage Gate / Agile Sprint Model
Stage Gate / Agile Sprint Financial Investment Regimes

SECTION 3. The Critical Success Factors section contains metrics correlated with:

Critical success factors from various studies
Success factors and metrics for new venture success
Success factors and metrics for Strategic Business Unit’s success
Success factors and metrics for a project’s success
A systems approach to sustained growth

SECTION 1. The Technology Value Pyramid (TVP)

This Technology Value Pyramid (TVP) section contains a hierarchy of metrics that evaluate the fundamental elements of R&D and their relationships with business results in the short and long term. This section provides a description of each of the components of the TVP, their interrelationship with one another and implications for their use. The section continues with a list of subjects of interest, a description of the stage-gate process and the use of the Merrifield Criteria for project selection.

The Value Creation sub-section contains hard financial type metrics which directly demonstrate the value of R&D activities to the positioning, profitability and growth of the corporation and creation of shareholder value.

The Managing a Portfolio of R&D Activities sub-section contains metrics which communicate the total R&D program results and which allow optimization of the total R&D program for the corporation’s benefit.

The Integration of R&D with Business sub-section contains metrics which indicate the degree of integration, the commitment of the business to the R&D process and program, teamwork, and ability to exploit technology across the organization.

The Value of Technical Assets sub-section contains metrics that indicate the technology strength of the corporation and relate to the probability of success versus competitors for the opportunities which have been selected by the firm are contained in this category. This is an indicator of the potential for the creation of future value.

The Practice of R&D sub-section contains a wide range of metrics associated with the operation of the R&D organization. Metrics covered include decision making, project generation and selection, use of information technology, environmental policy, personnel development and intellectual property.

The What typical stakeholders Are Interested In sub-section reviews metrics which may be of interest to Stakeholders in the R&D process such as business management, finance, and the CEO.

The TVP encompasses descriptors and a potential ‘menu’ of metrics of the fundamental elements of R&D and the relationships with business results in the short and long term. Not all measurements are right for all companies, or at all times, for describing or tracking the most important aspects of R&D.

This is parallel to a comprehensive financial model of a business that is composed of many more descriptors and ‘metrics’ than are necessary for examining the critical factors of the moment that are needed to guide the decisions of senior managers and investors.

From an R&D perspective the critical factors of the moment are dependent on the situation of the company, the perspective required and the basic dynamics of the model. Let’s start from the top.

The model provides a top-down perspective that is output oriented. ‘Value Creation’ indicators are the prime drivers of overall business returns that are derived from technology-based new products/processes, predictors of business growth and (implicitly) a critical aspect of the soundness of strategic business reviews. These indicators are used to answer critical questions, such as:

Are we spending the right amount on R&D?
Are we getting good returns on our R&D?

If the ‘Value Creation’ indicators are being maintained or going up: (a) the corporation has the likely raw material to extend a technology- based or innovation-based growth program; (b) the investors have the possibility of an extended stream of positive returns from the accumulation of financial pay-offs from technology- based innovations; and (c) the R&D units enjoy the likelihood of consistent funding to reinvest in various aspects of technology application for the near term and base building for the future.

The key words here are likely or possible. ‘Value Creation’ is a necessary but not sufficient condition for growth. It is also only a measure of the moment, whether it is looking to the past or to the future. And, any downward movements will predict the difficulties the business will have in achieving solid gains against the competition. These indicators are crucial to assessing the total returns from R&D investments, whether enough is being spent on R&D, and what is the likely future value of the company from a technology perspective.

‘Value Creation’ is the result of an accumulation of effort by R&D and the business to produce new ideas and to put the best ones into practice. Due to the lag effects and to all of the factors that are involved, momentum is built into these factors over time. They will change, but not rapidly. Change is caused by the drivers of ‘Value Creation’. The drivers are strategies that transform the R&D foundations of competencies, know-how, etc. into specific projects and implementation. These are the strategies that are represented by the ‘Portfolio Assessment’ and the ‘Integration with Business’ indicators.

When ‘Value Creation’ is very positive, these strategies are most likely working well. When ‘Value Creation’ is going in the wrong direction, look first to these areas as to the cause. In fact, one could look at ‘Value Creation’ indicators simply as the double integral of these strategy indicators over time.

Given this importance to ‘Value Creation’, it is no surprise that companies have routinely focused in recent years on methodologies and activities dealing with ‘Portfolio Assessment’ and ‘Integration with Business’ issues. These are correctly perceived as the means to (the ends that) improve the future stream of results from R&D.

‘Portfolio Assessment’ indicators describe the state of the various pipelines that run through the R&D enterprise, as well as the targets that are being pursued. They provide a view of how the R&D $ are being spent in terms of the timing, risks and returns that are possible. The ‘Portfolio Assessment’ indicators are a prime place to look for answers if there are problems with ‘Value Creation’, competitive or market share problems, or if there are problems with internal satisfaction.

Some of the critical management questions affected by the dynamics of these indicators include the following:

Are we allocating the R&D budget optimally (to the elements of the portfolio)?
Are we maximizing our investment yield from R&D?

Unlike ‘Value Creation’ indicators which tend to have modest levels of momentum associated with them and provide significant underpinnings to the business returns, the ‘Portfolio Assessment’ indicators can vary quickly and have little immediate effect on the business. Their major effects are cumulative. This dynamic often leads to short term, risk adverse behaviors that over time undermine the ‘Value Creation’ indicators. Maintaining an aggressive monitoring of the ‘Portfolio Assessment’ indicators is extremely important to the long term support to ‘Value Creation’.

Similarly, when there are problems within the portfolio that are corrected, it is necessary to give the solution enough time to work.

If ‘Portfolio Assessment’ indicators show strategy by what categories of R&D and targets are being developed, the ‘Integration with Business’ indicators show strategy of how it is being done…and, consequently, with what level of quality and execution.

The ‘Integration with Business’ indicators focus on process, culture, teamwork and organization. They also touch on many of the aspects of cycle time. The issues addressed by these indicators change slowly in reality and are probably the true pacing factors that are applied by the organization to the ‘Portfolio Assessment’ indicators that, in turn, put limits on the ‘Value Creation’ that is realizable.

These indicators are often the subject of efforts aimed at TQM, cycle time reduction, or re-engineering. They are difficult to build up to high quality levels because of the various organizational pressures, agendas, incentives, etc. And, they are easy to degrade. The organizational stability of these indicators varies from weak to very strong and are the sum of dozens of behaviors.

When there are difficulties attributable to barriers, for example, that can be removed, then the indicators and the results can be changed relatively rapidly. However, when there are difficulties due to lack of cooperation, lack of contact with the market, lack of good competitive intelligence or with a lack of risk taking, then new attitudes and new behaviors are required. This takes time to build into the culture and the indicators and the results will be slow to change.

In all aspects, the dynamics of ‘Integration with Business’ indicators depend on organizational matters versus those of the ‘Portfolio Assessment” indicators, which depend on investment decisions. And, a consideration of organizational and investment matters quickly brings the factors of business and technology leadership into the model. Second and third order dynamics pushing for immediate change are easy to see as overlays to deeper cultural values.

It is also logical to see both the importance of these two sets of transformation strategy indicators to managers who want immediate change and the criticality of both sets to the impacts on sustained ‘Value Creation’.

Unfortunately, the underlying dynamics do not allow the conversion of management desires for immediate gratification (at low risk) into sustained profitable growth. There must be allowances made for the application of enough time to link all of the elements on an ongoing basis. Over time, all of the indicators will point to a consistent improvement in the transformation strategies and to the attendant output in ‘Value Creation’.

Conversely, monitoring the transformation strategy indicators of an otherwise healthy technology-based enterprise, will show the early warning signs of degradation to the degree possible, presuming the technology foundations are sound.

The foundations for the strategies represented by the ‘Portfolio Assessment’ and the ‘Integration with Business’ are built on the ‘Asset Value of Technology’ and the ‘Practice of R&D Processes’. Some of the critical questions regarding these dynamics are:

Are we becoming more or less productive with our R&D?
Are we building a strong enough future base of competencies?
Are we getting an early warning of any declining capability?

Foundation indicators have the most momentum of any category. They are very slow to change and provide the real rate limitations to growth through technology-based innovation. However, they are also very vulnerable to neglect. They need nurturing, leadership and the execution of well focused technology strategies to become strong elements of a company’s growth foundation. And, just as when weak transformation strategies lead to degradation of ‘Value Creation’, they also degrade the foundations.

Aside from weak transformation strategies, the other major dynamic that affects foundations are external technological changes.

In these cases, technological paradigm shifts occur that undermine the foundations comprised of traditional technical competencies. These bring in new competitors, topple current competitors and may, in fact, redefine the structure of an industry. Although these paradigm shifts take time to develop, R&D and the business are usually both entrenched in the traditional areas, and either don’t see the changes coming or insist on devaluing the importance of them until it is too late. The foundations which, if strong, took considerable time to build, must nonetheless be constantly extended and rebuilt to provide an ongoing set of options that are the logical growth paths for the business’s future. Otherwise, over time, they will significantly erode.

Thus, the dynamics of the foundation indicators are that they are slow to build, a rate limiting source of quality for the options to be taken up by the transformation strategies, and that they are relatively easy to degrade in spite of strong momentum. They are also fundamental to a strong competitive basis for ongoing strategy (based on technical core competencies), to the heart of R&D creativity, productivity and a significant contributor to cycle time reduction. The drivers of these areas are the R&D leadership.

The principal use of the model is for communication and control. This includes the overall R&D program, the research program, the business technical programs and the development of external relationships for which new capabilities are the goal. The metrics are not intended to aid project evaluation except as a project would contribute to the improvement of various categories and their respective indicators.

Depending on particular needs of stakeholders or of decisions to be taken, different categories and factors should be examined. In addition, time periods (unit of analysis) should be adjusted for different industries, technologies and types of R&D.

If the business depends on technology and technology-based innovation, the model has great predictive and diagnostic value. The layers of activity that compose the R&D to business linkage are identified for inspection and monitoring. Taken together, the elements of the model and the structure connecting them, represent a single entity, the R&D enterprise, which may be interpreted and examined in different ways by different groups with different purposes in mind.

Each of the primary stakeholders will tend to concentrate their attention on different parts of the model. And, while that is logical, it is important to note that all of the categories are connected with time lags by the basic dynamics. It remains for R&D and, in particular, the CTO to make sure that some awareness is made of all the categories of the menu, that each stakeholder is reminded of their interconnectedness, and that a consistency is maintained (or that corrective action is taken to achieve consistency) between the collective expectations of the stakeholders and the realities of the model represented by the Technology Value Pyramid.

Metrics Relevant to Value Creation

Financial return
New sales ratio
Cost savings ratio
R&D yield
R&D return
Projected value of the R&D pipeline
Projected sales
Projected income
Number of ways technology is exploited
Product quality and reliability
Customer evaluation
Reliability/defects
Market share
Gross margins
Product technology rating
Customer rating
Economic trade-offs
Sales protected by proprietary position
Patents only
Percent proprietary sales
Technology planning
Intellectual property management

Metrics for Managing a Portfolio of R&D Activities

Strategic Alignment
Projected value of R&D pipeline
Projected sales
Projected income
Distribution of technology investment
Number of ways technology is exploited
Market share
Gross margin
Comparative technology investment
Information Technology Use in R&D
Probability of success
Technology planning
Environmental management in R&D

Metrics Relevant to the Integration of R&D with the Business

Strategic alignment
Use of Project milestones
Customer satisfaction
Internal
Number of ways technology is exploited
Projects with business/marketing approval
Product quality and reliability
Customer evaluation
Reliability/defects
Use of cross-functional teams
Delayed Stage kills
Decision Gate processes
Technology planning

Metrics Relevant to the Value of Technical Assets

Strategic alignment
Cycle time
Market cycle time
Project management cycle time
Customer satisfaction
External
Internal
Quality of personnel
Internal customer rating
External customer rating
External recognition
Market share
Comparative manufacturing cost
Gross margin
Rating of technology features and benefits
Response time to competitors moves
Comparative technology investment
Customer rating of technical capability
Number and quality of patents
Percent useful
Value ratio
Percent retention
Patents per R&D Dollar
Sales protected by proprietary position
Patents only
Percent proprietary sales
Core technical competency
Intellectual property management

Metrics Relevant to the Practice of R&D

Gate effectiveness
Defects reported
Delayed Stage kills
Cost vs. budget
Decision Gate processes
Probability of success
Environmental management In R&D
Idea generation and creativity
People development
Intellectual property management
R&D climate
Information Technology use in R&D

Metrics Relevant to Typical Business Stakeholders

CEO and BOARD OF DIRECTORS
Strategic alignment
Financial return
Projected value of the R&D pipeline
Cycle time
Product quality and reliability
Market share
Core technical competency
Technology planning
Intellectual property management

CTO and TECHNICAL MANAGERS (Complete Listing)
Strategic alignment
Financial return
Projected value of the R&D pipeline
Use of project milestone
Cycle time
Product quality and reliability
Use of cross-functional teams
Information technology use in R&D
Gate effectiveness
Defects reported
Core technical competency
Delayed Stage Kills
Cost versus budget
Decision Gate processes
Probability of success
Technology planning
Idea generation and creativity
People development
R&D climate

CTO METRICS LISTING (with Top Ranking Scores)
Strategic Alignment 11.5
Financial Return 10
Projected Value of the R&D Pipeline 9
Distribution of Technology Investment 7.8
Use of Project Milestones 6.8
Development Cycle Time 5.5
Customer Satisfaction 5.5
Number of Ways Technology is Exploited 5
Number of Projects Having Bus./ Mkt. Approval 5
Market Share 3
Goal Clarity 2
Development Pipeline Milestones Achieved 2.3
Use of Cross-Functional Teams 1.8
Comparative Technology Investment 1.5
Project Championship 0.7
Sales Protected by Proprietary Position 0.25

CRO and FINANCIAL MANAGERS
Financial return
Number of ways technology is exploited
Comparative manufacturing cost
Information Technology use in R&D
Cost versus budget
Probability of success

SBU MANAGER
Strategic alignment
Financial return
Projected value of the R&D pipeline
Product quality and reliability
Gate effectiveness
Core technical competency
Decision Gate processes
Probability of success
Technology planning
Environmental management In R&D
Intellectual property management

MARKETING and SALES MANAGERS
Strategic alignment
Financial return
Customer satisfaction
Number of ways technology is exploited
Market share
Funding by business
Probability of success

OPERATIONS and MANUFACTURING MANAGERS
Product quality and reliability
Comparative manufacturing cost
Technology transfer to manufacturing
Defects reported
Environmental management In R&D

Metrics Relevant to Annual Reports

FINANCIAL
Total assets ($)
Total assets/employee ($)
Revenues/total assets (%)
Profits/total assets (%)
Revenues resulting from new business operations ($)
Profits resulting from new business operations ($)
Revenues/employee ($)
Customer time/employee attendance (%)
Profits/employee ($)
Lost business revenues compared to market average (%)
Revenues from new customers/total revenues (%)
Market value ($)
Return on net asset value (%)
Return on net assets resulting from new business operations ($)
Value added/employee ($)
Value added/IT-employees ($)
Investments in IT

RENEWAL & DEVELOPMENT
Competence development expense/employee ($)
Satisfied employee index (#)
Marketing expense/customer ($)
Share of training hours (%)
Share of development hours (%)
Employees view (empowerment index) (#)
R&D expense/administrative expense (%)
Training expense/employee ($)
Training expense/administrative expense (%)
Business development expense/administrative expense (%)
Share of employees below age 40 (%)
IT development expense /IT expense (%)
IT expenses on training/IT expense (%)
R&D resources/total resources (%)
Customer base (#)
Average customer age (#)
Average customer education (=)
Average customer income ($)
Average customer duration with company (months) (#)
Training investment/customer ($)
Direct communications to customer/year (#)
Non-product-related expense/customer/year ($)
New market development investment ($)
Industry development investment ($)
Value of EDI system ($)
Upgrades to EDI system ($)
Capacity of EDI system (#)
Ratio of new products (less than 2 years old) to full company catalog (%)
Ratio of new products (less than 2 years old) to product family (%)
R&D invested in basic research (%)
R&D invested in product design (%)
R&D invested in processes (%)
Investment in new product support and training ($)
Average age of company patents (#)
Patents pending (#)

CUSTOMER
Market share (%)
Number of customer (#)
Annual sales/customer ($)
Customer lost (#)
Average duration of customer relationship (#)
Average customer size ($)
Customer rating (%)
Customer visits to the company (#)
Days visiting customers (#)
Customers/employees (#)
Field salespeople (#)
Field sales management (#)
Average time from customer contact to sales response (#)
Sales closed/sales contacts (%)
Satisfied customer index (%)
IT investment/salesperson ($)
IT investment/service & support employee ($)
Support expense/customer ($)
Service expense/customer/year ($ )
Service expense/customer/contact ($)

PROCESS
Administrative expense/total revenues (%)
Cost for administrative error/management revenues (%)
Processing time, outpayments (#)
Contracts filed without error (#)
Function points/employee-month (#)
Pcs/employee (#)
Laptops/employee (#)
Administrative expense/employee ($)
IT expense/employee (#)
IT expense/administrative expense (%)
Administrative expense/gross premium (%)
IT capacity [CPU & DASD] (#)
Change in IT inventory ($)
Corporate quality goal (#)
Corporate performance/quality goal (%)
Discontinued IT inventory/IT inventory (%)
Orphan IT inventory/IT inventory (%)
IT capacity/employee (#)
IT performance/employee (#)
Human
Leadership index (#)
Motivation index (#)
Empowerment index (#)
Number of employees (#)
Employee turnover (%)
Average employee years of service with company (#)
Number of managers (#)
Number of women managers (#)
Average age of employees (#)
Share of employees less than 40 years (%)
Time in training (days/year) (#)
Number of directors (#)
Number of women directors (#)
Number of full-time or permanent employees (#)
Average age of full-time or permanent employees (=)
Average years with company of full-time or permanent employees (=)
Annual turnover of full-time or permanent employees (=)
Per capita annual cost of training, communication, and support programs for full-time permanent employees ($)
Full-time or permanent employees who spend less than 50% of work hours at a corporate facility (#)
Percentage of full-time permanent employees (%)
Per capita annual cost of training, communication, and support program ($)
Number of full-time temporary employees (#)
Average years with company of full-time temporary employees (=)
Per capita annual cost of training and support programs for full-time temporary employees ($)
Number of part-time employees or non-full-time contractors (=), average duration of contract (#)
Company managers with advanced degrees: business (%), science and engineering (%), liberal arts (%)

SECTION 2. The Process of R&D

Stage Gate Innovation Model Metrics

1. Definition of the Innovation Process
The term “innovation process” refers to the overall process in a company for conceiving and developing ideas and concepts into profitable products and processes for the company business(es). It is one of the 5 to 7 overall high- level business processes within the corporation,(e.g., Hammer) . In order to maintain, diagnose, and improve the process, both effectiveness and efficiency metrics are required to support decision-making. The relevance of using a process model to support the choice of metrics is discussed in the next section.

Note that in the context discussed herein, the assumption is made that the innovation model will be technology-based. That is, innovation is founded in the efforts of the R&D organization, and that the overall function of the process is the embodiment of technology in the products of the business of the corporation. Progress through the full innovation process will involve the efforts of all corporate resources, including marketing, engineering, manufacturing, and service and support functions.

Relevance of Innovation Process Model to Metric Choice

2. Relevance of Innovation Process Model to Metric Choice
Often stakeholders in the innovation process — R&D management, financial management, and corporate officers — need to utilize metrics which supply specific decision support information about corporate capabilities in innovation. These metrics are overall, or outcome metrics, which provide a grade or rating of the overall result of the innovation capability of the corporation or business in question. Such metrics might relate to value creation by the process, effectiveness of the overall process in terms of new product revenue or profit versus total R&D investment, or the ratio of new product revenue or profit to total company or business revenue or profit. Recommendations for stakeholder metrics can be found elsewhere in this metric advisor.

However, these metrics are often long-term and retrospective in nature, and while they serve to evaluate the overall process as a whole (and generally, but not always, on a lagging basis), they may offer little, if any, insight into the current or future state of the corporate innovation process or its many and diverse activities. To be useful for the on-going management and evolution of corporate innovation, a diverse set of metrics must be available which offer insight into current condition and future vision state of the process and many of its constituent parts. For many such metrics to be useful, they must be not simply outcome-oriented, but process-oriented, and specifically diagnostic of discrete sub processes within the overall innovation process.

Each in-process metric must be related to a process model which ties it to the portion of the process of interest, from early idea or technology exploration through proof of concept to technology development to commercialization. The following process description is a relatively simple process model and defines the various sub-processes within that model in order to provide anchors for useful metrics within the innovation process.

Stage Gate Metrics for the Innovation Process

3. Overall Description of the Stage Gate Model for the Innovation Process Used in this Metrics Guide
There are a number of possible representations of the corporate innovation process. One simple yet fairly effective representation is the so-called “stage gate model”.

The model in Stage Gate divides the overall innovation process into four stages, as shown in the “Stage Gate” figure, from concept exploration or “ideation” to the process of commercialization. This division into four sections is arbitrary. An argument can be made for a three-stage model, or a more complex model of multiple sections and sub-sections. The four-stage model is chosen primarily for convenience, and to match a number of corporate models and those defined by scholars in the area of business processes. The shape of the model recognizes several realities of modern business.

For example, the process funnel narrows as the process proceeds from exploratory research or concept exploration through proof-of-concept to technology development and commercialization, illustrating pictorially the selective filtration that occurs in the process. There are typically many more ideas and concepts that are explored than are developed into significant technology capabilities in a business, and fewer still that emerge finally into finished products. In each stage of the process, the candidates dwindle, until only the most promising are brought to full production.

Similarly, the four gates, labeled A, B, C, and D, stand for the tests or decision-making activities that are exercised between the stages in the innovation process. Economic reality imposes a limit on the total level of technology and product development that each company can support. A second reality is that as each decision gate is passed, the resources and funding required to carry a given project to the next stage increases dramatically; a good rule of thumb is a 10X expenditure increase at each gate. These corporate “facts of life” impose a stringent set of conditions on the suitability of R&D programs which approach a gate. Each gate requires a set of metrics which ensure that only those programs most suitable to meet corporate business needs pass into the next sub-process.

Decision Gates and Stage Processes Shown In the Stage Gate Model

The following sections describe the decision gates and stage processes shown in the stage gate model. Underlined terms link to appropriate metrics for the gate or process under discussion.

4. Description of the Major Sub-process Divisions in the Innovation Process Model
As noted in the introduction to the process model, the division of the innovation process model is arbitrary, but in general it seeks to approximate reality. The four divisions cover the very early technology or idea exploration phase of innovation (when possibilities are defined), a proof-of- concept phase when the mapping of ideas into the realities of the business world occurs, the development, and finally commercialization. Each of the separate phases, or sub-processes, is covered in the following sections, together with the entry gate which defines conditions for admission to that sub-process. Refer to the stage gate model in the discussions below.

4.1 Process Entry Gate (A) and Exploratory Concept Sub-process (I)
Gate A is the entry not only to Process I but to the entire innovation process. The purpose of Process I is to explore new ideas and concepts and set in motion as many promising “seed” projects as possible. The cost of research and investigation is small here. In companies where exploratory concepts consist of exploring product ideas and concepts, a single worker may have a project or even several in work simultaneously. Where true basic research is involved, it is most often at the University level, with industry participating through grants, contracts, or research agreements. In either case, there are typically many avenues being explored and no valid idea or concept is neglected.

The main consideration at Gate A is whether the idea or concept is strategically appropriate (at this stage, the alignment to corporate business goals may be ephemeral in some cases), and whether the expertise available to address the concept or idea is adequate. Metrics at Gate A should address these issues.

The purpose of Sub-process I is the validation of concepts or physical principles. Metrics for this process should simply address the validity of results and whether or not basic principles are established. Competency metrics may also be valuable to support assessments of required resources to execute projects.

4.2 Conceptualization Gate (B) and Proof-of-Concept Sub process (II)
Gate B primarily tests whether a concept is validated or some physical principle has been established. Entry to Sub-process II depends mainly on appropriate resources being able to establish proof-of-principle in a business context, and whether the concept or idea to be tested has potential application areas within the business goals of the Metrics should test these concerns. The “filtration” function at this gate is fairly strong, since although the cost of research in Sub-process II is still not great, there will be many more candidate ideas than there will be resources to address them.

In Sub-process II, the emphasis is on proof-of-principle in real business applications. Business considerations such as market window and competitive reaction begin to be important, although there will still be concern about options and possible spin-offs of the technology or concept. Project metrics for tracking milestones and execution time become more important, although time frames are still lengthy and product requirements may be vague or broad-brush. Metrics concerning resources and technical capability are also appropriate.

4.3 Technology Development Gate (C) and Product/Process Development Sub-process (III)
The emphasis at Gate C is in suitability for product development. Gate C is also a strong filter; projects that pass this gate will be the few that are highly promising for commercialization and meet all the requirements for profitable business products. Candidate technologies which pass this gate will have forecast long-term corporate benefit, and the projects entering Sub-process III meet all the strategic requirements of fit, alignment, and attractiveness for the business. Metrics for Gate C must address these strategic requirements, as well as the tactical issues of assuring that the candidate projects have successfully completed the requirements of Sub-process II.

In Sub-process III, the emphasis shifts to harder-edge issues, such as timing and execution to assure that market windows are met and product needs are satisfied. There is also emphasis on maintaining and extending technologies to keep a competitive edge in the marketplace. Milestones are important due to cycle time issues, and project funding must be managed more carefully due to budgets which are typically millions of dollars rather than the 100X lower investment that may be typical on a project in Sub-process I. Metrics for Sub-process III must reflect the concerns of timely, accurate execution and tight budget control.

4.4 Product Launch Gate (D) and Commercialization Sub-process (IV)
Gate D is the last test prior to full product launch in most cases. Where concerns — and associated metrics — at the first three gates will have been primarily strategic (fit with corporate goals and competencies, strategic timing, alignment with business need), the test at Gate D is primarily whether execution in Sub-process III was timely and efficient. Concerns for entering full commercialization are about whether all major technical hurdles are cleared, and whether commercialization costs allow for profitable entry into the marketplace. Some strategic questions must still be addressed, including market need and timing, and metrics utilized here must address both the strategic and tactical issues.

Sub-process IV is obviously market-oriented, with careful management of commercialization and product costs, timing, and execution the key issues. Since budgets of hundreds of millions of dollars may be at stake, program and resource management are paramount, and metrics appropriate to the concerns will be chosen.

4.5 Gate Efficiency
A key concern at each gate is that decisions be as efficient as possible. That is, that projects passed through each gate be good candidates to benefit the business, providing the subsequent Sub-process are navigated successfully. Since the “filtration” function of the decision process at each gate will be partially imposed by the economic constraints of a business (only so much total funding is available for the projects in each stage), even some “fit” projects will simply “miss the cut” at each gate. However, those projects which pass the fitness criteria, whether actually entering the next stage or not, represent a measure of the efficiency of the previous gate decision process. A metric (discussed in the section on stage gate metrics) may be developed in terms of the percent of successful projects in the next stage for the previous gate.

The above Stage Gate metrics can also be applied to the Agile / Lean model of sprints around the cycle. The metrics used correspond irrespective of process.

Stage and Decision Gate / Agile / Lean Metrics

METRICS ASSESSING THE EFFICIENCY OF THE STAGE I PROCESS
Cycle time
Market cycle time
Project management cycle time
Quality of personnel
Internal customer rating
External customer rating
External recognition
Published works
Milestones achieved

METRICS ASSESSING THE EFFICIENCY OF THE STAGE II PROCESS
Cycle time
Market cycle time
Project management cycle time
Quality of personnel
Internal customer rating
External customer rating
External recognition
Milestones achieved

METRICS ASSESSING THE EFFICIENCY OF THE STAGE III PROCESS
Cycle time
Market cycle time
Project management cycle time
Customer satisfaction
External
Internal
Quality of personnel
Internal customer rating
External customer rating
External recognition
Product quality and reliability
Customer evaluation
Reliability/defects
Use of cross-functional teams
Millstones achieved

METRICS ASSESSING THE EFFICIENCY OF THE STAGE IV PROCESS
Quality of personnel
Internal customer rating
External customer rating
External recognition
Product quality and reliability
Customer evaluation
Reliability/defects
Technology transfer to manufacturing
Milestones achieved
Cycle time
Market cycle time
Project management cycle time
Use of cross functional teams

METRICS USED AT DECISION GATE A
Strategic alignment
Financial return
New sales ratio
Cost savings ratio
R&D yield
R&D return
Distribution of technology investment
Core technical competency
Metrics assessing the efficiency and effectiveness of the decision making process
Delayed Stage kills
Cycle time (Decision)
Decision Gate processes
Gate effectiveness
Criteria for overall business success
Supporting R&D metrics

METRICS USED AT DECISION GATE B
Strategic alignment
Financial return
New sales ratio
Cost savings ratio
R&D yield
R&D return
Distribution of technology investment
Development pipeline milestones achieved
Percent achieved
Performance level at each
Core technical competency
Probability of success
Metrics assessing the efficiency and effectiveness of the decision making Delayed Stage Kills
Cycle time (Decision)
Decision Gate processes
Gate effectiveness
Criteria for overall business success
Supporting R&D metrics

METRICS USED AT DECISION GATE C
Strategic alignment
Financial return
New sales ratio
Cost savings ratio
R&D yield
R&D return
Customer satisfaction
Milestone achieved
Response time to competitors moves
Core technical competency
Probability of success
Metrics assessing the efficiency and effectiveness of the decision making
Delayed Stage kills
Cycle time (Decision)
Decision Gate processes
Gate effectiveness
Criteria for overall business success
Supporting R&D metrics

METRICS USED AT DECISION GATE D
Strategic alignment
Financial return
New sales ratio
Cost savings ratio
R&D yield
R&D return
Customer satisfaction
Milestone achieved
Response time to competitors moves
Probability of success
Metrics assessing the efficiency and effectiveness of the decision making
Delayed Stage kills
Cycle time (Decision)
Decision Gate processes
Gate effectiveness
Criteria for overall business success
Supporting R&D metrics

Overall Process Efficiency Metrics

Use of project milestones
Cycle time
Delayed Stage kills
Cost versus budget
Efficiency of internal technical processes
Use of cross functional teams
For overall process effectiveness see Value Creation metrics

How to select successful R&D projects based upon R&D Financial Regimes. The management funding decisions for R&D are usually guided by one or another of two prevailing viewpoints:

1. R&D As a Necessary Cost of Business – In effect, R&D is supported as an overhead expense. This is most clearly appropriate for early-stage or exploratory research efforts and for developing or maintaining technical expertise in areas judged to be essential to future competitive advantage. However, in practice there are real limits in the amount of funding which senior management feels comfortable in treating this way. See Stage 1.

2. R&D As an Investment – Underlying the treatment of R&D programs as business investments is the basic notion that scarce investment funds should be allocated to alternative company opportunities according to consistent and explicit financial criteria, whether these lie in more traditional investment domains (e.g., capital budgeting for production or distribution facilities) or in R&D. This rationale is clearly most appropriate for those technical development and engineering programs with sufficiently well-understood market and financial implications to permit meaningful quantification and analysis of important decision parameters. An example of these is when projects are in Stages III and IV.

The underlying problem faced by many research directors in the discussion of the overall research program portfolio is that with only two available funding models, all R&D which falls outside the limits management feels comfortable in treating as a necessary cost of business must be justified on the basis of ROI or similar capital budget analysis. This often requires the “force fit” of investment models to R&D situations, in which uncertainty is often directly equated with risk, and future possibilities are significantly discounted.

Putting R&D into Strategic Perspective

An essential step in dealing with the overuse of such investment criteria is to recognize that technical programs are aimed at a wide range of strategic objectives.

Most of the technical work within large corporations is clearly directed toward a well-understood business investment. The technical activity involved is usually development and engineering, and the technical community is usually comfortable with the notion that the financial approach most often suited to its evaluation is an ROI or another capital budgeting framework. At the other end of the spectrum, much of the exploratory or fundamental/basic work in industry is clearly aimed toward knowledge building. For this work, where the business impact is often poorly defined and wide ranging, the more appropriate financial approach is that of considering R&D as a cost of doing business.

However, an important segment of the technical activity, often covering applied research, exploratory development, and sometimes feasibility demonstration, is concerned with the transition between these two broad strategic objectives. The concern is with reducing technical uncertainties and building a strong technical position, to the point where the corporation feels confident it can turn its technical strength into a profitable investment.

It is here where most difficulty is experienced with the two prevailing funding models. On one hand, the expenditures are often too large for management to feel comfortable treating them as an overhead or cost of doing business. On the other hand, the potential impact of the programs is often still sufficiently uncertain as to preclude meaningful ROI measurements.

An important first step in dealing with R&D for strategic positioning is to recognize that these expenditures are not so much directed toward an investment as they are toward the creation of an option. By this it is meant that the corporation is committing relatively modest R&D expenditures now to provide the opportunity to make a profitable investment at some later date.

The second step is to recognize that at least one class of options has been analyzed in some detail, that is, stock options, and that there are parallels with R&D options which suggest important insights that overcome some of the difficulties caused by the force fit of ROI. The concept of treating R&D as an investment option has been in the literature for some time, however, it is just recently that the concept is now being accepted more generally and the literature is expanding.

R&D As a Strategic Option

R&D programs directed toward strategic positioning are in many ways parallel to the American call option which will permit the owner to purchase stock at a specified price (exercise price) at any time prior to an agreed-upon expiration date. The value of a stock option varies with stock price. In establishing the parallel between the R&D option and the stock option:

(a) The price of the call option is analogous to the cost of the R&D program.
(b) The exercise price is analogous to the cost of the future investment needed by the company to capitalize on the R&D program (Potential Value at some future time T) when the investment is made.
(c) The value of the stock for the call option is analogous in the R&D case to the returns the company will receive from the investment (Potential Value of expected returns at a time.

The value of a stock option has been developed formally as a function of most of the parameters involved in the options transactions. From the perspective of the industrial research community, there are two significant observations.

1. The value of an option varies in ways that are counterintuitive. That is, the value of the option moves in the opposite direction to the value of an investment with respect to volatility (uncertainty) and time. These findings apply generally to options as the relationships arise primarily as a result of limited downside risk, and are not overly sensitive to the specific assumptions of the stock options model.

2. The intuitive viewpoint often taken by the research director when justifying technical programs directed to strategic positioning, logic that often seems counterintuitive to the financial community, more closely parallels the relationships developed from the analysis of options than from the rules for business investment and ROI.

Downside Risk – The downside risk for an investment, whether in the stock market or directly in the business, is that the complete investment may be lost. By contrast, the downside risk for a stock option is that the option will not be exercised. The loss is thus limited to the price of the option, whatever the value of the stock.

The equivalent situation of an R&D option occurs when the corporation, for whatever reason, does not make the follow-up investment necessary to capitalize on the R&D program (exercise the R&D option). The equivalent loss is the cost of the R&D program, which in general will be a much smaller than the follow-up investment. In practice, this represents the maximum possible loss, as the results of the R&D program, if not used directly; often provide significant insights into subsequent investments.

Volatility – As volatility or general uncertainty associated with an investment increases, the value of the investment will be discounted as a result of risk aversion. Often no business investment will be made if the level of uncertainty falls above the range with which management feels comfortable.

Volatility has the reverse impact for a call option, as the downside risk is limited to the cost of the option. If the volatility of the stock price is zero, the option is worthless; increased volatility in the stock price increases the chance that it may exceed the exercise price before expiration (without increasing the downside risk).

The R&D option parallels the call option in that R&D programs which address high-impact opportunities, with a modest or low probability of success, do not imply higher risk. This distinction between options and investments is very important for R&D.

Since the downside risk is limited in this way (i.e., the distribution of possible returns truncated below zero), the upside or high potential benefits, albeit uncertain, are not offset by possible losses and are thus important elements in the selection of alternative programs. If viewed from this options perspective, the priority of earlier phase, far-reaching and more basic R&D is often improved over that produced by ROI analysis.

Time – In the investment model, the value of returns is discounted as a direct result of the time value of money. For the call option, time has the reverse effect. Increasing the time for which the option may be exercised increases the probability that stock price may exceed the exercise price during that period, and thus increases the value of the option.

The parallel situation for the R&D option is that R&D programs which offer the corporation flexibility in the timing of the subsequent investment or financial commitment, and particularly those providing the opportunity to make a series of investments over a period of time (even though the outcome of any single one may be uncertain), should be preferred to those projecting a short-range limited window of application.

Merrifield Criteria Metrics for Business Success

The statistics for R&D success are less than encouraging. Perhaps one of 20 programs that start out in the laboratory ever produces a positive cash flow, and the average time required from the idea stage to commercialization for major programs is seven to ten years. This is too long — the need may have disappeared or been satisfied in another way.

The constraint analysis that is described here, however, has produced eight out of ten successes and telescoped the time to market from seven to ten years to two to four years. This process when it is combined with the Stage Gate Process minimizes the downside risk.

Constraint analysis is basically a logic sequence or decision tree that asks three questions. Is this a good business for anyone to be in? If yes, is this a good business for us to be in? If yes, what is the best method of entry?

The first two of these questions has an expanded list of critical factors that have been selected through experience. It is important to determine, even before the first laboratory experiment is run, whether or not a new development could ever result in a commercially attractive product. (The shelves of research labs are cluttered with great mouse traps for which there have turned out to be no mice or for which the manufacturing, marketing, management, capital, and other requirements for commercialization were inadequate.)

Twelve factors are rated from 0 – 10, giving a maximum possible score of 120 : 60 for “business attractiveness” and 60 for “company fit.” Over many years of experience, those projects with scores of 80 or above were successful in eight out of ten cases. The few that did fail did so because of unexpected regulatory interventions or from increased costs for raw materials because of the oil embargo, and occasionally a major program was discontinued because a parallel effort turned up a second-generation development that was superior to the first (all of which were, indeed, surprise factors). Below 70 points the probability of success falls off rapidly to 30% or less.

BUSINESS ATTRACTIVENESS
1. Sales/Profit Potential (Max 10 Points)
a) Does this opportunity have the potential to make a significant contribution to current sales? (Max 5 points)
b) Will this opportunity provide 10% or higher after tax profits? (Max 5 points with lower scores below 10%)

GROWTH POTENTIAL (Max 10 Points)
Will annual real (inflation-corrected) growth exceed 10% per year? (10 points, with lower scores for slower growth)

COMPETITION (Max 10 Points)
a) Is the competition fragmented with small-weak competitors unlikely to react quickly? (Max 4 points)
b) Will this product or process have a long life (more than 5 years) or will it be quickly obsolete? (Max 3 points)
c) Are there strong patents, copyrights, and other types of proprietary technology not easily evaded? (Max 3 points)

DISTRIBUTION RISK (Max 10 Points)
Does this opportunity involve generic technology that can apply to four or more market segments, sufficiently differentiated from each other by manufacturing, marketing, geography, or other requirements, such that if a new technology emerges for one of these segments, the others will remain viable? (Max 10 points, with lower scores for fewer segments)

POTENTIAL FOR INDUSTRY RESTRUCTURING (Max 10 Points)
Does this opportunity involve breakthrough technology of such magnitude that it restructures an entire industry (10 points) or some segments of an industry (less than 10 points)?

POLITICAL/SOCIAL FACTORS (Max 10 Points)
Does this opportunity meet strong political or social needs (national security, safety, environmental) that will provide incentives for use of the technology (10 points), is it neutral (5 points), or does it risk penalties (liability concerns, tariff barriers, negative exchange rate differentials, tax penalties (less than 5 points)?

BUSINESS FIT
1. Capital Availability (Max 10 Points)
Time-critical access to needed capital for investment in facilities, equipment, and operations is essential to maximize an opportunity in the life cycle projected (10 points if readily available)

MANUFACTURING–PRODUCTION (Max 10 Points)
Time-critical access to at least interim production capabilities with assurance of increased cost-effective capacity, consistent with maximum growth, would rate 10 points. (If production facilities are unnecessary, score 10 also)

MARKETING–DISTRIBUTION (Max 10 Points)
Rapid market penetration, with an existing marketing-distribution capability, on a global scale would rate 10 points.

TECHNICAL COMPETENCE (Max 10 Points)
a) Technical service competence: A high level of effective support of field operations would rate 3 points
b) Incremental improvements: A strong program of continuing development work to maintain current market share and profitability would rate 4 points maximum.
c) Next generation systems: Significant investments simultaneously being made in advanced technology through strategic alliances would rate 3 points maximum.

ACCESS TO COMPONENTS (Max 10 Points)
Timely access to important raw materials or components’ is essential, preferably from more than one source. Politically inspired embargoes, fires or explosions can interrupt supplies. Score 10 points for readily available materials and components.

MANAGEMENT (The Champion) (Max 10 Points)
Strong and sustained top-level management support is essential for success, particularly for multiple-year developments. Also, a dedicated champion, determined to succeed, is required to manage each project. Weighs a maximum of 5 points for top management support, and 5 points for a champion.

Metrics that Drive New Product Success at the Strategic Business Unit Level

Profitability
Profitability Relative to Spending
Financial return
New sales ratio
Cost savings ratio
R&D Yield
R&D Return
Profit Impact
Gross Profit Margin
Meeting Sales Objectives
Strategic alignment
Meeting Profit Objectives
Profitability versus Competitors
Comparative manufacturing cost
Customer Satisfaction
Product quality and reliability
Market share
Response time to competitors moves
Efficiency of internal technical processes
Technology transfer to manufacturing
Percentage of Sales by New Projects
Technical Success Rating
Number of ways technology is exploited
Impact on Sales

Metrics that Drive the Performance of a Business Unit

DOMINANT FACTORS
A high-quality new project process
Use of project milestone
Efficiency of internal technical processes
Project championship
Number and quality of patents
Percent Useful
Value Ratio
Percent Retention
Peer evaluation
Internal
Sales protected by proprietary position
Patents Only
Gate effectiveness
Cost relative to budget
Decision Gate Processes
Probability of success
Quality of technology plan
A defined new product strategy for business unit
Strategic alignment
Goal clarity
Cost relative to budget
Quality of technology plan
Adequate resources of people and capital
Comparative Technology Investment
Adequate resources
R&D spending as a percentage of sales
Projected value of the R&D pipeline
Projected Sales
Projected Income

NEXT LEVEL FACTORS
High-quality new project teams
Development cycle time
Quality of personnel
Internal customer rating
External customer rating
External recognition
Published works
Product quality and reliability
Efficiency of internal technical processes
Customer rating of technical capability
Number of defects reported
Senior management committed to, and invovled in, new products
Goal clarity
Management support
An innovative climate and culture
Employee morale
Idea generation and creativity
R&D climate

Metrics Based Upon Business Success Criteria

Business Attractiveness
Sales profit potential
Financial return
New sales ratio
Cost savings ratio
R&D Yield
R&D Return
Projected value of the R&D pipeline
Sales
Income
Growth potential
Financial return
New sales ratio
Cost savings ratio
Projected value of the R&D pipeline
Sales
Income
Competitive position
Market share
Sales protected by proprietary position
Product synergy
Strategic alignment
Number of ways technology is exploited
Industry restructuring by breakthrough technology/processes
Political/social factors
Business Fit
Capital availability
Number of projects having business/marketing approval
Percent funded by business units
Response time to competitor moves
Marketing
Cycle time
Response time to competitor moves
Customer satisfaction
Manufacturing/production
Cycle time
Technology transfer to manufacturing
Response time to competitor moves
Technical competence/core competencies
Distribution of technology investment
Quality of personnel
Product quality and reliability
Milestones achieved
Response time to competitor moves
Team/supplier access
Response time to competitor moves
Management support
Project championship
Customer contact time

Detailed List of 50 Metrics with Definitions, Limitations, Uses, Options, Variations, References

Below are the specific metrics referenced in the above sections. For each the following is provided:

Advantages and limitations – Benefits of the metric, areas of concern when using the metric.
How to use the metric – Examples of how the metric can be applied. Parameters for measurement.
Options and variations – Different ways to apply the metric.
References – Literature references pertaining to the metric for further investigation.

FINANCIAL RETURN

1. METRIC DEFINITION
a. New Sales Ratio

The New Sales Ratio is the % of current sales originating from new products. There are two sub-definitions that are required. What is a new product? and, How old is new?

The most frequently used and simplest definition of a new product is any new SKU (inventory code: Stock Keeping Unit) that has required R&D support to implement. This avoids counting new SKUs which are only packaging changes or other modifications made easily by marketing or manufacturing. When is a new product old is different for each business and technology. In rapidly changing and evolving fields, such as electronic chips and software, new might only be one year, but certainly not more than three. Three years is more likely the norm for businesses that are a mix of fashion and formulation, such as cosmetics & toiletries. And, for more intense capital and industrial products, three to seven is a more likely range to select a number that is right for you.

b. Cost Savings Ratio

The Cost Savings Ratio is the % reduction in cost of goods or cost of operations (including depreciation charges) that are realized in a year to year comparison that have originated from technology changes that are new. Again, the same issues must be resolved as sub-definitions as required for the New Sales Ratio. In other words, what is being attributed to R&D and how long is new?

Since SKUs are not used to catalogue changes made in operations an alternative must be found that will work for your firm. A simple solution can often be created based on the capital approval process. Most companies require a specific approval for all individual capital projects. These can be coded and tracked for R&D involvement and for realized cost savings. However, it is recognized that the accounting involved in examining the impacts on cost savings may be more difficult than that for new sales. It is therefore a more common metric only in those cases which are more significantly impacted by cost savings than by new product sales. And, since new technology for operations or manufacturing has a different useful life than a new producy per se, it must be tracked for a different length of time that is industry specific. In some cases, it may be very linked to the product life in other cases, it may go on much longer. In any case, it is not likely that ‘new’ will reach beyond 7-10 years.

c. R&D Yield

R&D Yield is the contribution of R&D to current financial performance. It is a metric that is composed of definitions from New Sales Ratio and Cost Savings Ratio, plus an evaluation of gross profit from the new sales.

It is the annual combined financial benefit that is derived from the annual gross profit of new products and the annual cost savings of new processes. This is the current contribution that the company receives that is associated with its past stream of R&D investments, i.e. the part of the ‘bottom line’ that is relatively ‘new’ and derived from R&D.

d. R&D Return

R&D Return is the relative ROI measure that relates to R&D. It is composed of the R&D Yield divided by the annual investment in R&D. Hopefully, this is a large number that is proportional to the risks and variances that are part of R&D.

2. ADVANTAGES AND LIMITATIONS
The advantages of these financial metrics are that they relate directly to the financial benefits to the company, they are quantitative and they are comparable to metrics that can be used in different parts of the same firm or between firms. They capture the degree to which R&D is truly making a financial contribution to the value of the enterprise. They answer the question: What have you [R&D] done for me [the Business] lately? However, they only represent the tip of a process that takes place over a number of years and that involves other functions besides R&D. This means that the numbers reflected by these metrics are associated with activities that are in the past. These metrics are lagging indicators. They are a nice track record, but they may not be reflecting accurately a current level of effectiveness.

3. HOW TO USE THE METRIC
The metrics should be tracked at least on a once a year basis. Because of measurement and definition problems, a baseline of two years or more of historical data is needed before accurate judgments can be made about trends and ratio efficiencies.

The metrics should be examined carefully for consistency with business strategies and the results required vs. the investments in R&D. In situations where the metrics, requirements and available resources are not in balance, there will be a difficulty in executing the overall business and technology strategies. One or the other must be shifted, and variations in how R&D is conducted need to be examined.

If the Financial Return metrics are being maintained or going up: the corporation has the likely raw material to extend a technology-based or innovation-based growth program; the investors have the possibility of an extended stream of positive returns from the accumulation of financial pay-offs from technology-based innovations; and the R&D units enjoy the likelihood of consistent funding to reinvest in various aspects of technology application for the near term and base building for the future.

The key words here are likely or possible. Positive Financial Returns are a necessary but not sufficient condition for growth. It is also only a measure of the moment, whether it is looking to the past or to the future. And, any downward movements will predict the difficulties the business will have in achieving solid gains against the competition. These indicators are crucial to assessing the total returns from R&D investments, whether enough is being spent on R&D, and what is the likely future value of the company from a technology perspective.

4. OPTIONS AND VARIATIONS
There is always something new and innovative that is contributing to revenue or profit. These financial return metrics are intended to capture the new portion of these changes in the business that are related to R&D. They require definitions of what is to be considered new and for how long. These elements can then be tracked separately or together, in ratio or absolute form, by themselves as benefits or as an investment return vs R&D. The options and variations fall into place based on each company’s views of these items.

The most common variations are based on the length of time that is new. the most frequent categories are three years, five years and seven years. Another variation is to use these same metrics in a prospective, future mode.

PROJECTED VALUE OF THE R&D PIPELINE

1. METRIC DEFINITION
a. Projected Sales

Projected Sales is the calculated sum of future sales from current R&D projects. This metric may be expressed in absolute terms or a % of future sales.

Definitions must be provided for how this is to be evaluated. Normally, it is the forward side of the newness range, i.e. if new products are those introduced within five years, projected sales will be calculated for five years after commercialization. A probability of attainment is usually figured into this metric.

b. Projected Income

Projected Income is the income stream associated with the Projected Sales. Similar definitions apply. This metric may be expressed as an absolute number or as a fraction of net income.

2. ADVANTAGES AND LIMITATIONS
This metric provides an ongoing anticipation of the expected results from R&D. Because it is projected, it provides an evaluation of the benefits that are being created with today’s R&D investments.

The limitations are due to the intrinsic difficulties of obtaining estimates about the likely commercial benefits if the technology is successful.

3. HOW TO USE THE METRIC
This is perhaps the single most important and least used metrics. It is the singular indication of the future business that is to be developed as a result of successful outcomes from R&D. Are the sales impacts large enough, are there enough new products, is the timing of elements in the pipeline adequate, are the overall returns related to R&D adequate? Are these metrics on a year to year basis showing constancy, increase or decline?

This metric provides ongoing guidance to the company regarding the future gain to be expected in the business due to R&D. This should be used as a check that both the strategy and the resource allocations are correct.

If this metric is staying constant or increasing, particularly with respect to the R&D resources, then the effectiveness of R&D is being maintained or increasing.

If, alternatively, this metric is declining, then further diagnostics should be examined to understand the reasons and to take corrective action.

4. OPTIONS AND VARIATIONS
Common options are few because this is an underused metric. One is to look at absolute sales or net income over a five year horizon on the presumption that some projects will finish and be productive within the next two-three years and that others will impact a bit later, i.e. in the fourth and fifth year. A related option is not to adjust the commercial impact by any probabilities.

COMPARATIVE MANUFACTURING COST

1. METRIC DEFINITION
Benchmarked manufacturing cost data vs. competition for same type of unit cost.

2. ADVANTAGES AND LIMITATIONS
It is extremely useful and in some companies paramount to know how R&D is helping to provide an advantaged cost position to the operations. This metric reflects the quantification of that goal. Unfortunately, cost accounting and even further comparative cost accounting can be extremely difficult.

3. HOW TO USE THE METRIC
While most firms have very accurate manufacturing cost data for thremselves, the generation of accurate manufacturing cost data for competition is considerably more difficult. Therefore, when using this measurement, there should be an estimate made of the variance of the competitive estimates.

4. OPTIONS AND VARIATIONS
This metric is intended to be based on unit process comparisons. There can be many options created that are aggregates of production processes, but these simplifications can be misleading. Therefore caution is urged.

PRODUCT QUALITY AND RELIABILITY

1. METRIC DEFINITION
1.1. Customer or Consumer Evaluation.

Relative quality and reliability compared to competitive products through evaluation by customers or consumers.

1.2. Reliability/Defect Rate Assessment.

Fraction of a firm’s output, either by individual product or by sum of all products, that meets or exceeds the established quality standards.

2. ADVANTAGES AND LIMITATIONS
2.1 The advantage of this metric is that product benefits resulting from R&D activities are directly evaluated by the customer or consumer. Comparison with the competitors’ products is usually the basis for evaluation. Limitations of the metric chiefly are related to the reliability and accuracy of the survey techniques chosen as appropriate for the industry, though firms usually gain confidence in their preferred methods through repeated use and incremental improvement.

2.2 Similar to the preceding discussion, the advantage to this metric is that the direct benefit from R&D activities can be obtained through specific measurements made by the firm.

3. HOW TO USE THE METRIC
For the Customer or Consumer Evaluation metric, each firm will generally have a preferred technique for directly or indirectly obtaining data showing how well the firm’s products perform in comparison to competitive products. Data for the Reliability/Defect Rate

Assessment result from internal quality measurements. When both metrics are utilized, the impact of product quality improvements on customer satisfaction should be demonstrated. Product Quality and Reliability metrics are retrospective, showing the results of past technology or product introduction to the market.

4. OPTIONS AND VARIATIONS
This metric fits well with the trend toward greater input to R&D planning from customers and consumers and with the “Quality” protocols that have adopted by firms of all types. Though basically a retrospective measure, product needs that arise during the data collection can be used for prospective purposes.

GROSS PROFIT MARGIN

1. METRIC DEFINITION
Gross Profit as a percentage of sales, where gross profit equals net sales minus cost of goods sold (product costs plus direct manufacturing costs).

2. ADVANTAGES AND LIMITATIONS
To some extent, gross profit margin reflects value of the firm’s technology assets and the value created by R&D. However, raw material, production, and distribution costs also directly affect the firm’s gross profit margin. Each firm should attempt to understand the correlation of Gross Profit Margin to R&D effectiveness.

3. HOW TO USE THE METRIC
Value assessment should be based on change in gross profit margin from period to period. (Periods should be appropriate to an industry and may be in excess of one year.) Changes in Gross Profit Margin in relationship to changes in values of other metrics (e.g., financial return , technology transferred to manufacturing, sales protected by proprietary position) should be followed in an attempt to uncouple the contribution of R&D from other factors. This is a retrospective metric that can be used as a benchmark with the competition, if gross margin data from competitive firms are available.

MARKET SHARE

1. METRIC DEFINITION
Firm (or business unit) market share in various product categories measured as appropriate for the industry or category, expressed as a percentage of the total market.

2. ADVANTAGES AND LIMITATIONS
This metric is meant to reflect value creation for the firm and the value of the firm’s technology. Similar to Gross Profit Margin metrics, caution must be taken in the interpretation of Market Share data from the perspective of measurement of the contribution of R&D activities to the whole. There can be many confounding factors in a market share determination, such as the size and quality of the marketing effort, the competitive response, the relative state of the economy, etc.

3. HOW TO USE THE METRIC
Changes in Market Share should be assessed at least annually to determine the rate of progress or decline. The expectation is that improvement in a firm’s technologies and products will result in a greater share of the market. Market Share is a retrospective metric, showing the results of past technology or product introduction to the market. Since competitors’ market share data is usually also available, this is a metric that can be used as a benchmark with the competition.

4. OPTIONS AND VARIATIONS
As an indication of threats or opportunities, share data in markets related to a firm’s products can be followed. This “Related Market Share” metric can serve as a component of a strategy to anticipate the potential application of similar technology into a firm’s marketplace.

STRATEGIC ALIGNMENT

1. METRIC DEFINITION
This R&D metric assess the degree of alignment of an R&D project or an R&D portfolio with the strategic intents of the company or a division of the company. The strategic intents are often the corporate goals embodied in its business plan.

2. WHY IT IS USED
This metric is used to gauge the degree of relevance of the R&D program to the corporate goals. The strategic intents of a company may change more rapidly than the R&D program can respond to those changes creating various degrees of misalignment. The misalignment can be with regard to work area, long term versus short term needs, or degree of risk.

3. HOW TO USE METRIC
There are a number of ways this metric can be applied, including both prospective and retrospective views. It can be applied by R&D management, general management, or by both working in partnership. An alignment index would be assigned to each project; a linear scale of 1-5, for example, would work well. These scores may also have value when considering relative merit of individual projects in the portfolio. A composite score for the entire portfolio would then be determined. This could be a weighted average reflecting sizes of projects with regard to technical head count, project budget or some other appropriate weighting factor. Once a baseline for alignment has been established, R&D management can then decide if and how this index should migrate to greater or lesser degrees of alignment through modifications of the portfolio. Applied to individual projects, there could be a cut-off point for the alignment parameter below which projects are not supported.

Consideration must be given to the degree of alignment desired. Although the desired state in many cases is toward greater degrees of alignment, you can envision situations where that may not be the case. A research organization charged with taking the company in new directions may not want its project portfolio highly aligned with the current business plan. A more visionary business plan may capture new directions as well as current businesses, but many organizations find that some degree of decoupling is desirable.

4. OPTIONS
Both retrospective and prospective views are options of this metric. The retrospective view entails applying the metric to an existing portfolio of R&D projects to determine the degree of the portfolio with corporate goals.

The prospective view is to apply the metric to a proposed project or slate of projects. If the R&D organization is trying to increase its alignment index, then the management will be less likely to initiate projects that move the composite score in the wrong direction. Similarly, an alignment index cut-off may be instituted. Projects falling below some minimum value of alignment would not be supported.

5. References: Third Generation R&D, P. A. Roussel, K. N. Saad, T. J. Erickson, Harvard Business School Pres, Boston, MA (1991); ISBN 0-87584-252-6

Winning in High Tech Markets, Joseph G. Morone, Harvard Business School Press, Boston, MA (1993), ISBN 0-87584-325-5

DISTRIBUTION OF TECHNICAL INVESTMENT

1. METRIC DEFINITION
This metric provides a means of assessing how well an R&D program is protecting the technology investment and technical position of the company. It forces consideration of how the technical assets should be distributed, setting directions for modifying the R&D portfolio.

2. ADVANTAGES AND LIMITATIONS
The portfolio of an R&D organization may not be protecting the strategic interests of the company for any number of reasons such as skill set mismatch, slow response to changes in the company’s mission and markets, and a rapidly changing competitive environment. There can be an over-emphasis on certain business units and products. This metric causes the management to examine how well the R&D effort is protecting and expanding the technical position of the company in areas of greatest importance.

3. HOW TO USE THE METRIC
This metric is applied by first determining how the technology investment should be distributed. As an example, consider a company with six business units. The R&D portfolio can be distributed among these six business according to a number of models. Six examples of distribution models are listed below:

The revenue that each business generates.
The opportunity market share (potential market growth). [1]
The impact that technology can make in the different business units.
Competitive Impact (Base, Key, Pacing) [2]
Some combination of the above distribution models.
The profitability that each business demonstrates.
These considerations often involve the concept of the technological basis of competition [1], that is how does technology provide a sustainable competitive advantage in a particular product or market. The discussion could also consider the distribution of the competitive impact of the company’s technology investment by categorizing them as Base, Key or Pacing [1]. Base technologies are essential to the business but widely exploited by competitors. Key technologies are highly differentiating to the company’s current products. Pacing technologies are new technologies where the competitive impact is less certain but likely to be high. All three of the impact categories require protection of the competitive position, but the distribution of resources among the three categories may vary based on the company’s business plans.

Once a distribution model has been agreed upon, the portfolio is measured against that model. The distribution could be with regard to number of R&D projects in each segment of the distribution, head count devoted to each segment, or expected value of projects in each segment. Modifications are made to the portfolio to move the distribution toward the desired state.

4. OPTIONS
Both prospective and retrospective views are supported by this metric. In the retrospective view, the R&D projects are categorized according to the corporate investments or markets they are intended to protect or create, or the competitive impact that they offer. This does not have to address the entire R&D effort of the company, since the metric can be applied to any subset of the portfolio. The projects are appropriately weighted to reflect their size and cost. A distribution of the R&D efforts supporting each of the categories is determined. This current state distribution is then compared with the desired state. The degree of misappropriation can then be quantified as the fraction or percentage of the R&D effort that is improperly distributed.

The prospective view for this metric involves consideration of how a proposed project shifts the distribution of technology investment toward or away from the desired state.

5. REFERENCES
Competing for the Future, G. Hamel and C.K. Prahalad, Harvard Business School Press, Boston, MA (1994); ISBN 0-87584-416-2

Third Generation R&D, P.A. Roussel, K. N. Saad, T. J. Erickson, Harvard Business School Press, Boston, MA (1991); ISBN 0-87584- 252-6

Curtis, C. C., Non-Financial Performance Measures in New Product Development, J. of Cost Management, 8(3): 18-26. (This article addresses the distribution in terms of major projects, minor projects and extensions, and relates these to financial measurements.)

NUMBER OF WAYS TECHNOLOGY IS EXPLOITED

1. METRIC DEFINITION
This metric assess the number of ways a technical asset can bring value to the corporation.

2. ADVANTAGES AND LIMITATIONS
This metric is applied to gauge project attractiveness, or to understand the value of a technical asset already developed. It is generally agreed that a larger number of potential uses, both within the company’s current markets and in markets not yet developed, makes a technical asset more valuable. The metric is a bit arbitrary and can be misleading in cases where there are few, though very large and/or lucrative commercialization.

3. HOW TO APPLY THE METRIC
This metric is applied by taking an existing or potential technical asset, such as a project to develop a new type of lower cost, light-weight composite material, and conducting a thoughtful analysis of how many ways this asset can be exploited commercially. The count could consider:
Number of business units in the corporation that could make use of the asset
Number of markets the company serves that could be impacted by this technology
Total number of markets served by the corporation and other companies where the technology may have an impact.

Number of products that could utilize the technical asset.
Used in this way, the metric is a single numerical value. A larger number of potential uses means that the corporation is not depending on a single or small number of products to succeed in order for the technical asset to deliver value. The risks associated with the exploitation of the technical asset are spread over a larger number of potential uses. A larger number also provides greater opportunity for unforeseen benefits, like taking the company into new markets and new products.

Using the low cost, light weight composites as an example, the primary market for the company may be the automotive market, with four different auto parts that could use the strong, light weight tubes produced by the new process. In addition, there could be a market for the technology in the aeronautical industry, served by another business unit of the company. The third exploitation could be in high performance bicycle frames, a market that is new to the company.

4. OPTIONS
An option in the implementation of this metric can involve adjusting the number for relative importance of the commercialization, or keeping sub-metrics of the number of exploitations with certain value ranges. These might be segmented as:
No. of Markets for Technical Asset
No. of Product Offerings
No. of Product Offerings with (Value > $10M)
No. of Product Offerings with ($2M < Value < $10M)
No. of Product Offerings with (Value < $2M)

Another technical asset which also had eight product exploitations, but with two products in the ($2M < Value < $10M) category and six products in the (Value < $2M) category is not as attractive as the light weight composite project.

5. REFERENCES
Competing for the Future, G. Hamel and C.K. Prahalad, Harvard Business School Press, Boston, MA (1994); ISBN 0-87584-416-2

NUMBER OF PROJECTS HAVING BUSINESS/MARKETING APPROVAL

1. METRIC DEFINITION
Percent of projects in the total R&D portfolio with explicit business unit and or corporate business management sign-off.

The intent of this metric is to provide an indicator of the degree of alignment with business and corporate strategy and tactics. The metric is closely related in some corporate structures to metric 12, “percent Funding by the Business”

2. ADVANTAGES AND LIMITATIONS
Advantages: Several studies have suggested that close alignment of R&D to marketing and to business and corporate strategies increases the odds of success for new products and processes. Thus actions which drive this metric to higher values can be expected to improve the amount of R&D spent on successful projects and the predictability of the outcome from R&D efforts.

Limitations: The metric will be valuable to the extent business/marketing management and R&D management jointly develop strategy and plans. Use of the metric to drive R&D without such teamwork will likely lead to short term projects and suboptimal use of R&D resources. In those companies where R&D is corporately funded, business/marketing management may also be tempted to give approval to projects in their market segments to insure that they receive “their share” of R&D resources. Finally if the corporation uses a formal innovation process which requires business/marketing approval at some stage, the metric runs the risk of becoming a measure of compliance with use of the process or a measure of the percent of project past the approval stage.

3. HOW TO USE THE METRIC
Explicit approval may be sought at any point in the innovation process. Seeking approval early in the innovation process probably provides maximum value. One form of approval is the provision of a sales forecast from marketing management for each new product and agreement to commercialize if the product meets technical requirements in a timely manner.

The level of approval from the marketing/business management and the point where approval should be sought should be explicitly defined if the firm uses a formal innovation process. If not, the level should be commensurate with the amount of R&D resources and commercialization resources which will be required.

4. OPTIONS AND VARIATIONS
For projects having broad corporate strategic value, approval of a director of corporate planning or director of corporate business development might be an appropriate substitute for the business/marketing management approval. For corporations where out licensing of technology is a major thrust, approval of a director or vice president of licensing may be an appropriate substitution.

USE OF PROJECT MILESTONE SYSTEM

1. METRIC DEFINITION
1.1 Percent of projects in the total portfolio going through a defined project management system with defined milestones.

1.2 Percent of R&D expenditure on projects using a defined project management system with defined milestones.

2. ADVANTAGES AND LIMITATIONS
Project management systems including milestones can provide a way of reducing cycle time and providing R&D and business management with a sense of the health of projects. These systems also can improve the linkage between R&D, marketing and business management by enabling coordination with manufacturing and marketing to hit key windows. When these systems include documentation through status reports, they can be used to promote organizational learning. Driving behaviors which increase the numerical value of this metric should therefore improve both the linkage of R&D to the business and improve the effectiveness of R&D.

When used with a formal stage gate process this metric provides a measure of compliance with that system. Since companies will generally use a defined project management system and establish milestones in the later phases of innovation, this metric may also be an indicator of the distribution of projects in the innovation pipeline (see metric 8 “Distribution of Technology Investment” ).

The metric may be limited by the difficulty of counting projects outside the project management system. Further since project management systems may not be appropriate early in the innovation process, the ideal value for this metric will depend on the firm’s desired balance of early and late stage projects. For short term projects such as minor product or process variations, use of formal project management systems and this metric may create unnecessary red tape and potential delays.

3. HOW TO USE THE METRIC
3.1 As a concurrent metric, the total number of projects with defined (written) project plans including definite milestone dates can be divided by the total number of identifiable R&D projects (X100) to calculate the metric.

3.2 When used with the appropriate accounting system this metric can be calculated from the cost of projects divided by the total R&D cost. In this case the budgeted projects should be audited to determine compliance with requirements for plans and milestones.

4. OPTIONS AND VARIATIONS
The metric can be used as a concurrent metric (a snapshot of the current R&D activities) or as a retrospective measure to determine how many R&D projects used a defined process. It should be equally suitable for service and manufacturing companies.

PERCENT FUNDING BY THE BUSINESS

1. METRIC DEFINITION
Fraction (or percent) of the R&D budget or actual expenditure from business unit sources.

As used in this metric, the intent is that the funding carries with it program control. That is, an R&D “Tax” imposed at the corporate level on businesses should not be counted as a business source if the business does not have program control and the ability to vary the level of funding.

2. ADVANTAGES AND LIMITATIONS
For companies which use central or corporate laboratories, this metric can provide an indicator of linkage to the business unit strategy since presumably only programs and projects which support that strategy would be funded. The ideal value of this metric will depend on the nature of the company (whether businesses are closely aligned to each other or the company is more of a conglomerate) and the corporate strategy (growth and improvement in existing businesses versus R&D for growth outside current business). The metric could be used for comparisons within an industry and for tracking trends within a company.

Limitations: By itself, this metric cannot indicate the degree of collaboration between the business and technology communities in establishing strategy. An emphasis on driving this metric to high values in the absence of long term business strategies can lead to short term oriented R&D which could endanger the long term health of the firm. Moreover, overemphasis on business funded R&D can reduce funding of projects which support overall corporate competencies to benefit more than one business.

3. HOW TO USE THE METRIC
In most companies the accounting system should be able to provide the required numbers. R&D overhead costs should be treated in a consistent manner to enable comparisons to be made between companies. Generally these are allocated as a percentage of direct R&D costs.

4. OPTIONS AND VARIATIONS
The metric can be used prospectively as budgets are assembled or retrospectively with actual expenditure data. It should be equally applicable to service and manufacturing industries.

TECHNOLOGY TRANSFER TO MANUFACTURING

1. Metric Definition

1.1. Amount of Technology Successfully Transferred to Manufacturing.
The output of R&D is technology which must be embodied into products or services. This metric measures the amount of technology which is successfully transferred to manufacturing. It is a measure of value created by the R&D function. The form of the metric may vary:

Number or percent of projects transferred
$ of R&D expense or percent of total R&D expense related to transferred projects
Projected Value (future sales or profits) of projects transferred.

1.2. Quality of the Technology Transfer Process.
A firm may rate its technology transfer process by using a subjective scale ranging from 1 to 4. Firms operating at Level 1 are characterized by unsupported hand-offs. Firms operating at Level 4 are characterized by a development process which includes involvement by manufacturing in all phases of the project in order to ensure that the transfer will be seamless. This metric is one of many relating to how well the firm conducts the Practice of the R&D Process. It also relates to how well R&D is integrated with the business.

2. ADVANTAGES AND LIMITATIONS
The advantage of this type of metric is that it relates to value created by the R&D function without needing several years to collect data. It is a surrogate for financial return. Presumably, the more technology transferred to manufacturing, the higher will be the financial return. However, it should be clear that this is not necessarily the case. Some technologies transferred could result in financial losses.

The measurement of the quality of the transfer process can be used to diagnose problems with the process and plan for improvement.

3. HOW TO USE THE METRIC
The amount of technology transferred to manufacturing during a time period, usually one year, should be tracked over time. The expectation is that a steady state will be achieved with a flow of technology occurring at a rate which can be accommodated by manufacturing and which meets the needs of the firm for new products and services.

The projected value of the financial return from projects transferred during a time period should also be tracked over time. The expectation is that the value will increase due to the choice of better projects and more efficient management of the R&D process. Actual financial returns should be compared to the projections, although there will be a large time lag. Efforts should be made to improve the quality of the projections.

These metrics can be used retrospectively to measure the output of R&D over the past period, or prospectively to set targets for future accomplishments. The use of projected value is a prospective estimate of financial return.

4. OPTIONS AND VARIATIONS
Service companies and companies who sell technology may wish to use this metric to relate to transfer of the output of R&D in a more general sense to manufacturing, into services, for sale, for license, or any other use appropriate to the firm and which would be considered a successful outcome that creates value.

USE OF CROSS-FUNCTIONAL TEAMS

1. Metric Definition

1.1. Number of Cross-functional Teams
Current management philosophy suggests that the use of cross-functional teams will improve the effectiveness and efficiency of the R&D Process and will help R&D to be integrated with the business. The number of such teams can be counted if they are established on a formal basis.

1.2. Evaluation of the use of Cross-functional Teams.
A firm may rate its practice of using cross-functional teams by using a subjective scale ranging from 1 to 4. Firms operating at Level 1 are characterized by the existence of strong organizational boundaries, lack of cross-functional involvement in R&D projects, and no cross-functional team structure. Firms operating at Level 4 are characterized by a well developed and supported team structure which effectively places all R&D work in cross-functional teams responsible for the entire project rather than functional silos responsible for parts. This metric is one of many relating to how well the firm conducts the Practice of the R&D Process. It also relates to how well R&D is integrated with the business.

2. ADVANTAGES AND LIMITATIONS
This metric will be important if the use of cross-functional teams contributes to the effectiveness of the R&D function in the firm, given its peculiar situation. This is usually thought to be the case, but there may be cases where other factors are more dominant.

Evaluation of the use of teams can be used to diagnose problems with the organization or the R&D process and to plan for improvement.

3. HOW TO USE THE METRIC
A simple count of teams is rarely as valuable as an assessment of the how the firm uses such teams. This can be accomplished using the rating scale suggested above. Input for the evaluation should be gathered from a broad cross-section of the firm. The evaluation is important for planning improvements.

Trends across time are probably more valuable than benchmarking.

RATING OF TECHNOLOGY FEATURES AND BENEFITS

1. METRIC DEFINITION
The value of technology output from an R&D organization is closely related to the Technology Features (those attributes of the technology which are intended to enhance competitiveness) and Technology Benefits (those attributes of the technology which are recognized and valued by the market).

1.1 Product Features & Benefits

Metric 1: Competitive Technical Performance of Product (Project Metric)
Comparison of technical performance of a product in those dimensions where the customer is likely to perceive a benefit. This may be used in a prospective sense to appraise the value of a feature or in a retrospective sense to register the value of a benefit which the market has recognized.

Examples include:
The use level required to achieve a needed result in the customer application.
The yield strength of a high-performance alloy
The measured softness provided by a textile softener.
The measured UV resistance of an external architectural coating.

Metric 2: Customer rating of Products (Business Segment or Firm)
Customer rating using a scale of 1-5 of the technology benefits that is perceived in a firm’s products. This can be compared to the rating for the best competitor, usually in the form of a ratio. This is an aggregate subjective measure for a business segment or for the firm.

Metric 3: Economic Value of Products (Project, Business Segment, or Firm) This metric is the price differential per unit obtained by virtue of the technology feature minus the cost of providing the feature. The differential can be multiplied by the volume to assess the total benefit to the firm.

Metric 4: Market Share Evaluation (Business Segment or Firm)
If differential pricing does not occur, the advantages of superior product technology can appear as differential market share. In this case, the relative market share (the firm’s share divided by the largest share) can be used as a surrogate for the value of technology embodied in the products.

1.2 Process Features & Benefits:

Metric 5: Competitive Technical Performance of Process (Project Metric)
Comparison of technical performance of a product in those dimensions which are important to manufacturing cost or product performance. This may be used in a prospective sense to appraise the expected value of a new or improved process or in a retrospective sense to register the demonstrated value.

Examples include:
Manpower requirements
Efficiencies of raw material conversion
By-product or coproduct costs or values
Consistency, controllability, and other such quality parameters.
1.2.2 Metric 6: Economic Value of Processes (Project, Business Segment, or Firm)

The differential in profitability (versus the target or competitor) attributable to new or improved process technology.

Metric 7: Profitability Evaluation of Processes (Business Segment or Firm)

In the same way that Market Share is a surrogate measure of product performance in those business areas where the basis of competion is product performance, overall profitability in a business segment of the firm is a measure of process performance, in those business areas where the basis of competition is cost and/or quality.

2. ADVANTAGES AND LIMITATIONS
Product Metrics 1 through 4 attempt to assign value to technology used in products. However, differential value or market share are normally the result of many different factors. These metrics can give some indications if factors are carefully sifted, but may be misleading if the analysis is superficial. Objective measurement of product performance and customer ratings relative to competitors are the most accurate measures of product technology. But note that comparison of features that the market has not recognized as benefits may be self-serving and deceptive. Competitive rankings of measured product performance and or customer ratings may be averaged over market segments or over the firm to obtain average values.

Process Metrics 5 through 7 attempt to assign value to new or improved process technology resulting from R&D. However, economic value and profitability are normally the result of many different factors. Objective measurement of process performance is the most accurate measure of process technology.

3. How to use the Metric

Metric 1
Define the key parameters which measure features the customer is likely to perceive as benefits. Measure product performance as accurately as methods allow. Compare to the same measurements of competitive products. Rank performance versus best competitors.

Metric 2
Ask customers to rate the technology attributes of a product line relative to solving their problems. This requires a carefully constructed survey instrument.

Metric 3
This metric is valid for products which are differentiated by performance. It is not applicable to commodity products.

Metric 4
See Definition

Metric 5
Define the key parameters which measure or impact cost or quality. Measure process performance as accurately as methods allow. Compare to the same measurements of competitive processes. Rank performance versus best competitors.

Metric 6
See Definition

Metric 7
See Definition

These metrics may be used retrospectively to measure the output of R&D over the past period and prospectively to set targets for future accomplishments.

4. Options & Variations
There are many variants on these metrics. Exploring these is beyond the scope of this document.

5. REFERENCES
Ellis, L. W., and Curtis, C. C. 1995. Measuring Customer Satisfaction. Research/Technology Management, in press.

Kaplan, R.S. and Norton, D. P., 1993. Putting the Balanced Scorecard to Work, Harvard Business Review, September-October, 134-147.

RESPONSE TIME TO COMPETITORS MOVES

1. METRIC DEFINITION
This metric measures the ability of the firm to respond to new technical innovations introduced by competitors. Depending upon corporate strategy, it could be the time required to match or exceed the competitive offering.

2. ADVANTAGES AND LIMITATIONS
This metric is an indicator of technical leadership in a given field. The technical leader will not spend a significant amount of time matching competitive innovations, while a follower will be more reactive than proactive. The utilization of resources for this function as a percentage of total resources should be tracked over time. An increase in this percentage would indicate that new technology programs are not as effective as desired and technical position relative to the competition is eroding. This metric also measures the flexibility and creativity of the firm to change priorities to meet competitive challenges. A strong market intelligence function is required to identify competitive entries at an early stage in the introduction and to assess the technical merits of the offering so that appropriate responses can be made. The technical merits of competitive offerings must be critically assessed to differentiate from market repositioning of existing technology.

3. HOW TO USE THE METRIC
The time between the introduction of a competitive offering and the internal development of a comparable or superior offering can be measured and compared to product development times for similar products. A rating system of 1 to 4 can be used.

1. Organization slow to recognize significant competitive offering in the marketplace; slow to launch program to respond; unable to get their offering into the marketplace in acceptable time to be combatant. Slow to recognize impact of technical innovations.

2. Organization recognizes need for competitive offering; has trouble in launching program to develop counter-offering and gets offering to market barely in time to have impact.

3. Organization responds to competitive offering and develops counter offering to maintain relative position.

4. Organization anticipates potential for competitive offering and has counter offering into the marketplace with superior product allowing a gain in market/competitive position. Very fast to recognize impact of any hints of technical innovations.

The percentage of resources used in matching competitive moves could be measured over time. An increase in spending in this function should raise questions regarding core research and development efforts for organizations who strive for technical leadership.

4. OPTIONS AND VARIATIONS
The importance of this metric will depend upon the technology strategy of the firm. Firms pursuing technical leadership will be very interested in minimizing response time. The importance may vary across different strategic segments in the same firm.

COMPARATIVE TECHNOLOGY INVESTMENT

1. METRIC DEFINITION
This measures the current annual expenditure for R&D staff and capital compared to the best competitor and/or the industry average.

2. ADVANTAGES AND LIMITATIONS
This metric measures the rate of current activity in developing the technology of interest with the intent of predicting whether the firm is expected to gain or lose ground in the technology. It should be kept separately for the KEY and PACING technologies most critical to the strategy. Retrospectively it measures the efficiency of the investment in meeting new product and technology development goals.

This metric should be as quantitative as possible, but in some industries it may be necessary to make estimates as to the size of the development effort of the best competitor and the industry average. Analyses must be based on a comparison of similar functions. For example, some organizations include sales support as part of their report for technology expenditures. The components of the expenditures under study must be understood in making the comparison.

3. HOW TO USE THE METRIC
Information on a firm s overall technology expenditures are available in the firm s annual reports or industry publications. These can be used for comparison to internal overall investment. Information by industry is available in industry publications and from organizations such as IRI (IRI/CIMS survey).

Rationing the firm s current investment in technology versus the best competitor and industry averages provides an insight into the efficiency of the technology investment. Performance exceeding expectations in value creation goals at competitive investment rates indicates an efficient organization while sub-standard performance raises concern about the quality of the investment. Smaller firms may require an investment higher than industry norms to maintain a competitive position to offset critical mass issues.

QUALITY OF PERSONNEL

1. METRIC DEFINITION
This is a measurement of the skills and ability of the R&D staff to execute strategic programs.

1.1 Internal Customer Ratings.
Internal customers rate the quality of the R&D staff on their ability to execute programs. Measures such as percentage of mileposts met versus project plans, novelty of concepts, patentability of concepts, and competitive advantage of the technology are parameters that can be considered.

1.2 External Customer Ratings.
External customers rate the quality of the R&D staff of their ability to meet customer expectations. Problem solving, novelty of approach, responsiveness, knowledge of customer’s operations are parameters that can be considered.

1.3 External Recognition.
Publications in refereed or industry trade journals, external presentations, citations in the literature, invited lectures and patents are parameters to be considered.

2. ADVANTAGES AND LIMITATIONS
The internal and external customer ratings measure the ability of R&D to meet customer expectations and contribute to the growth of the corporation or enhance competitiveness. They are largely objective measures that can be tied to tangible value.

The value of external recognition via patents, publications and presentations is more difficult to measure objectively. The numbers of different subjects covered by public disclosures should be evaluated rather than the total number of all disclosures. It is easy to become subverted to a self servicing metric if only numbers are considered. Maintenance of technology as trade secrets must be considered in this evaluation.

3. HOW TO USE THE METRIC
Internal customer surveys can be conducted using a 1 to 4 scale for rating. A 1 represents below standard execution on a given project. The causes for this poor performance have to be determined since they could arise from inadequate skills, poor judgment, lack of responsiveness, poor planning, etc. The causes may not be related to the quality of the personnel but poor management practices. Having multiple internal customers (marketing, manufacturing, sales, etc.), conduct the evaluation is a form of 360o review. Superior performance by reaching targets ahead of schedule, lower than expected costs, developing a significant competitive advantage, etc., should be rated as a 4. In establishing this survey system, agreement should be reached on the different levels of performance. The survey should be applied to different projects with the same population of the internal customers as raters. This rating should be conducted on a regular basis and over time trends will emerge.

External customer surveys should be conducted using the same 1 to 4 scale. A 1 rating would indicate that the customer was not satisfied with the parameter being measured, while a 4 would indicate that expectations were exceeded. Key parameters should be selected beforehand and could include timeliness of response, knowledge of products, knowledge of customer s operations and knowledge of customer needs. The parameters will vary by industry. Several levels of the customer s organization should be sampled such as plant operators, first line supervisors and management. A simple postcard type of survey instrument mailed after customer contact can be used. A database can be developed over time and trends will emerge.

The subjects covered in public disclosure should be tabulated and compared to strategic technology goals. A subjective 1 to 4 rating system can be created by R&D management to determine fit with the goals. A rate of 1 corresponds to a poor fit and a need to enhance skills, while a 4 implies that all areas are being addressed.

4. OPTIONS AND VARIATION
Service and consumer product companies may find the external customer survey to be a valuable tool in assessing the effectiveness of their R&D organization. One may also measure the quality by the number and type of external awards from recognized organizations (ACS, AIChE, IRI …)

DEVELOPMENT CYCLE TIME

1. METRIC DEFINITION

1.1 Market Cycle Time
This metric measures the elapsed time from identification of a customer product need until commercial sales commence.

1.2 Project Management Cycle Time
This metric measures the elapsed time from establishment of a discrete project to address an identified customer product need until commercial sales commence.

For both 1.1 and 1.2 above, the end point can be time when manufacturing feasibility is established for those cases where no commercialization occurs. Compare to historical values and benchmark vs. competition, if possible. Group by categories of projects (e.g. major new product, minor product variation, etc.) Can also be used to track milestone attainment rate for firms using a stage gate management process.

2. Advantages and Limitations

Market Cycle Time
The advantages of this measure is that it is quantitative and can be used to measure the entire process or various parts of the process if stage gates are examined. The process can be analyzed to determine what parts are driving the overall cycle time so that improvements to the process can be made.

The limitations of this metric could include R&D’s position that it does not adequately influence the process until after a need has been more defined. Another limitation is that a strong documentation system is helpful to make the cycle time metric as accurate as possible. An additional limitation is that defining the commencement of sales as the end of the cycle does not account for post start-up issues such as efficiency, waste, % of manufactured product within specification, etc.; this could lead to focusing on shortening the cycle time at the expense of later, non-measured parts of the cycle. One must also keep in mind that for breakthrough or paradigm shifting projects,, cycle time measurements. The advantage of this metric is that it supports having the clarity of when a project is initiated based on approvals, assignment of resources, start of spending, etc. The metric can be used to measure the entire process or various parts of the process if stage gates are examined. The process can be analyzed to determine what parts are driving the overall cycle time so that improvements to the process can be made.

3. HOW TO USE THE METRIC
Both metrics should preferably be used in combination with a project reporting system that can track the project initiation date (based on approval and assignment of resources), the length of time in each stage gate of the innovation process, and date of sales commencement. For the first metric, the initiation date could be the date the customer need was determined (i.e. marketing request or date of customer research results). Cycle times for different types of projects (new products, cost savings, product improvements, etc.) should be compared to help predict and manage resource allocation. Cycle times for different divisions could also be compared (with caution) to identify practices driving lower cycle time to adapt where possible.

4. OPTIONS AND VARIATIONS
Variations could exist regarding looking at cycle times for only certain parts of the process for which R&D feels it has most control or influence. Cycle time could be extended past the commencement of sales based on what is important to the organization and R&D’s involvement, i.e. when target efficiency is achieved, target manufacturing cost, % of product within spec, etc.

5. REFERENCES
The first three references state the case for faster R&D response time, while the last three references raise cautions about going too fast:

Burkart, R. E. 1994. Reducing R&D Cycle Time, Research-Technology Management, 37(3), May – June, 27-32.

Patterson, M.L., Accelerating Innovation: Improving the Process of Product Development, New York: Van Nostrand Reinhold, 1993.

Smith, P.G., and Reinertsen, D.G. 1992. Shortening the Product Development Cycle. Research-Technology Management. 35(3), May – June, 44-49.

Curtis, C.C. 1994. Nonfinancial Performance Measures in New Product Development. Journal of Cost Management, 8(3), 18-26.

Von Braun, C.F. 1990. The acceleration trap. Sloan Management Review. 32(1), 49-58.

Ellis, L.W., and Curtis, C.C. 1995. Speedy R&D: How Beneficial? Research-Technology Management, 38(4), July – August, forthcoming. This article calls metric 1.1 idea-to-customer time, and defines other times.

CUSTOMER RATING OF TECHNICAL CAPABILITY

1. METRIC DEFINITION
This metric measures the average customer rating (internal or external) of overall technical capability of the firm (interval rating scale) in providing technical service and/or new product innovations. It can be rationed to ratings for relevant competitors for benchmarking purposes.

2. ADVANTAGES AND LIMITATIONS
The advantages of this measure is that is based on customer feedback and is therefore based on what is important to them vs. the area being measured. It also is a good measure to assess overall technical capability vs. some quantitative output or result. As with any subjective, measure, it is less objective and date-based than some other measures. It must also be clarified if the area is being measured for proving technical service OR new product innovations, as the results could be very different.

3. HOW TO USE THE METRIC
A rating system of 1 to 4 can be used for this measure. Regarding technical service, a 1 could be described as an organization having poor technical service (extremely slow response time, long problem-solving time vs. expectation, and poor/failing results); a 4 could describe an organization which provides an immediate or proactive response with extremely short turn-around time, with the problem being prevented into the future, and consistent exceptional proactive service. Regarding new product innovations, a 1 could describe an organization whose products consist of outdated, archaic systems and technology which are difficult to use, maintain, adapt, etc.; a 4 could describe an organization whose products are constantly exceeding customers’ expectations in their rate of introduction, new features, adaptability, cost, anticipating of needs, while clearly standing out from the competition.

4. OPTIONS AND VARIATIONS
Other criteria could be considered for technical capability rather than technical service or new product innovation and a similar scale could be developed. These criteria could include an overall technical assessment of design capability to quickly meet customer needs, development of products which leverage existing core capabilities or competencies (vs. requiring new technologies), ability to create products which have a significant competitive or sustainable advantage, etc.

5. REFERENCES
Ellis , L. W. and Curtis, C. C. 1995 . Measuring Customer Satisfaction. Research -Technology Management, forthcoming (hopefully in time to identify the issue & date ) .

NUMBER AND QUALITY OF PATENTS

1. METRIC DEFINITION
1.1 Percent Useful is a metric which measures the percentage of active patents from the company’s total patent estate which are incorporated into or used to defend the firm’s commercial products or processes.

1.2 Value Ratio is a metric which measures the interval rating (1 to 5) for potential strategic value times rating (1 to 5) for strength of protection divided by 25 (maximum attainable value). It yields a number between 0 and 1.

1.3 Retention Percent is a metric which measures the percent of granted patents maintained.

1.4 Cost of Invention measures the number of patents from R&D/R&D effort costs. One can also calculate this just using the number of useful patents from R&D

2. ADVANTAGES AND LIMITATIONS
1.1 The advantage of this metric is that it allows one to look at the utilization of patents and not just the number generated. By examining the percent useful, the value of the patents is much more obvious. The disadvantage of this metric is that it weights all of the useful patents equally, whereas some could be more valuable than others based on the amount of revenue they protect, etc.

1.2 The advantage of this metric is that it considers the strategic value of a given patent and the degree of protection, vs. just the potential use of the patent. The limitation of the measure is that it is more qualitative and subjective and involves multiplying, which increases the variability around the measure.

An additional limitation is that the calculation across all patents can become rather cumbersome.

1.3. The advantage of this metric is that it looks at patents which are being maintained, likely due to their potential for future utilization vs. just current utilization. A limitation is that it does not include the potential value of maintaining these patents.

1.4 The advantage of this metric is that it allows the user to assess the cost of the invention process to better determine if the cost is reasonable vs. the strategic value. A limitation is that it potentially focuses on a negative vs. the positive outcomes of the patented technologies.

3. HOW TO USE THE METRIC
3.1 This measure can be used to track over time the use of technology’s patents. This can provide signals as to whether or not patent work is being supported, is of adequate usefulness, or if the company’s core and protected technologies are deteriorating.

3.2 This measure can be used to assess the true financial and business value of patents to the organization and should be done by a cross-functional team knowledge about both the business potential and the technology. Depending on the size of the businesses in which the company invests, a 1 for strategic value would represent a business of minimal attractiveness in size (i.e. volume, revenue, or profits) and a 4 would represent a business of maximum attractiveness in size. Depending on the product life cycle for the business, a 1 for strength of protection could mean minimal “tactical” (less than 2 years) competitive advantage or easily replicated; a 4 for strength of protection could mean a transforming technology which would be expected to last 10 years or beyond or where the technology is virtually impossible to replicate by competition or the patent almost impossible to circumvent successfully.

3.3 This measure could be tracked by the legal department and updated as part of an analysis done at a set frequency to assess whether or not certain patents should be maintained. Key decision criteria should be developed i.e. likelihood of future use, strategic value of future use, etc.

3.4 This measure could be used as some set frequency to assess the cost of invention vs. the strategic value so that the organization can track this measure and relationship over time.

4. OPTIONS AND VARIATIONS
4.0 In addition to any of the above metrics, one could also consider measuring the number of times a patent is cited by others. This is assumed to indicate the value it offers to others in developing technology. If of interest only internally, then only references made via internal (vs. external) patents should be counted.

4.2 The interval rating scale for potential strategic value could be based on an even more qualitative assessment such as 1 being of minimal strategic value (in which case one would question why a patent was utilized) up to 4 being of critical strategic value (i.e. new business category, key competitive threat, high growth potential, etc.)

5. REFERENCES
Acs, Z. J. and Audretsch, D. B. 1989. Patents as a Measure of Innovative Activity, Kyklos, 42 (2)171 – 180.

Chakrabarti, A. K. , and Anyanwu, C. L. 1993. Defense R & D , Technology, and Economic Performance: A Longitudinal Analysis of the U. S. Experience, IEEE Transactions on Engineering Management , 40 (2) , 136 – 145, May. This article has many references on patents as a technology indicator.

Curtis , C. C. 1994. Nonfinancial Performance Measures in New Product Development. Journal of Cost Management, 8 (3) : 18 – 26. This article covers the relationship of patent number and patent quality with other metrics.

Griliches, Z. 1990. Patent Statistics as Economic Indicators : A Survey, Journal of Economic Literature, XXVIII, 1161 – 1707 , December.

Honig – Haftel , Sandra. 1990. The effect of reward systems on the development of patents in high technology firms . Sc. D. diss. , University of New Haven.

SALES PROTECTED BY PROPRIETARY POSITION

1. METRIC DEFINITION
Sales of products protected by patents owned by the company, and a result of the R&D effort, is a simple and meaningful research outcomes measure. The patent can cover a composition, process, or use, provided that it clearly creates monopoly or obvious market leadership position for the company. Rigorous review is required to assure that incremental improvement patents are excluded unless clearly leading to market leadership. The following formula best describes the metric;

Sales of products in year x protected by patents
Total business unit sales in year x
Foreign sales and patents may be considered if they represent a significant portion of the total. A broader metric encompassing all proprietary positions, e.g., via licensing, acquisition, joint venture agreement, trade secret, etc., could be used but it would likely be much more subjective and much less credible.

2. ADVANTAGES AND LIMITATIONS
The main advantages of this metric are:

Easy to measure
Very credible if used properly. Patents are clearly a top R&D outcome
Quantitative

The disadvantages are:

Could be a lagging indicator; may measure progress a decade or more old. May be beneficial to also include some measure of remaining years of protection, or list of expiration dates to show how current the protection is.
Some businesses or market segments may place more emphasis on trade secrets or market prowess than on patents because of short product lifetimes or other reasons.

3. HOW TO USE THE METRIC
Using the formula shown above, all of the previous year’s sales are examined to determine which are protected by patents. The Patent Department should participate and corroborate the assessment. Only those patents that clearly give a product a major technological and hence market advantage should be included, e.g., a composition patent on a me too performance drug for a minor market share, probably should not be included. International patents and sales should be included if they are significant ( over 15% of total ).

4. OPTIONS
A log of patent expiration dates should be reviewed and discussed along with the metric annually to assure that a precipitous drop in the metric is not imminent. Projections could be made 5 or so years out to illustrate continuity of the protection.

5. REFERENCES
Acs, Z.J. and Audretsch, D.B. 1989. Patents as a Measure of Innovative Activity, Kyklos, 42 (2 ), 171-180.

Chakrabarti, A.K., and Anyanwu, C.L., 1993. Defense R&D, Technology, and Economic Performance: A Longitudinal Analysis of the U.S. Experience, IEEE Transactions on Engineering Management, 40 (2 ), 136-145, May. This article has many references on patents as a technology indicator.

Curtis, C.C., 1994. Nonfinancial Performance Measures in New Product Development. Journal of Cost Management, 8 ( 3 ): 18-26. This article covers the relationship of patent number and patent quality with other metrics.

Griliches, Z., 1990. Patent Statistics as Economic Indicators: A Survey, Journal of Economic Literature, XXVIII, 1161-1707, December.

Honig-Haftel, Sandra, 1990. The Effect of Reward Systems on the Development of Patents in High Technology Firms Sc.D dissertation, University of New Haven.

PEER EVALUATION

1. METRIC DEFINITION
External peer evaluation is another method of assessing the strength of a firm’s technology. Alternately, an internal panel of top scientists could be assembled, but this group would probably have less credibility because of their stake in the outcome. The evaluation in either case would involve a comparison of the firm’s technology in a particular area compared to the current state of the known art and particularly against competition, if known.

2. ADVANTAGES AND LIMITATIONS
Advantages of the metric are;

Good diagnostic tool for future growth and progress if unbiased panel of experts can be assembled
Can be very credible if the panel members are credible to the major stakeholders ( suggests external, blue-ribbon panel )
Can lead to remedial action based on panel suggestions

Limitations/Disadvantages

May be difficult to benchmark competitive technology
Subjective, and qualitative
Confidentiality problems with outside panels
Some stakeholders distrust all technical people

3. HOW TO USE THE METRIC
Selection of the panel is extremely critical. People must be selected with credibility to the stakeholders as the predominant criteria. Technical competence in the field is essential, and the panelist must be impartial. Internal panels generally will be less believable to top management because of the stake they have in the R&D organizations success. An external panel could include outside directors, local university faculty and department chairmen, consultants, members of technical or trade organizations, etc.

A numerical rating scale should be used, e.g., 5=well above art or competition, 3=equal to , and 1= well below.

Panelists should vote independently ( perhaps secretly ) after extensive discussion with R&D participants and by themselves.

4. OPTIONS
The use of internal panels probably would have little credibility outside the technical community. However, it could be an excellent reality check within an R&D organization.

The best, and most credible results, might be obtained via paid, technical consultant teams, e.g., A.D.Little, SRI, etc. These organizations would have no incentive to benchmark low. They could also conduct blind surveys to determine competitive positions.

CUSTOMER SATISFACTION

1.0 METRIC DEFINITION
The customer satisfaction metric has two variations:

1.1 Measures of external (end-customer) satisfaction. These may be such metrics as ratings of quality of technical personnel or technical capabilities, or technology benefits within products or processes.

1.2 Internal customer satisfaction. Since the immediate customer of R&D is normally the businesses within the corporation that R&D serves, measures such as customer satisfaction in engineering, marketing, or manufacturing may be appropriate. Typical metrics might include on-time technology delivery, competitiveness or appropriateness of the technology solutions delivered, and overall satisfaction with the R&D track record of technological support.

2. ADVANTAGES AND LIMITATIONS
There are few disadvantages to good customer satisfaction metrics. In the case of external, or end-customer satisfaction metrics, one complication may be that the entire innovation cycle is under review by the end customer. A bad grade by the customer, while a valid rating of the corporate innovation process, may not be merely an indictment of the R&D operation, but a judgment of the overall product development process within the company, involving manufacturing and product engineering, market forecasts, consumer needs and attitudes, and competence of corporate management.

One the other hand, a well-thought-out customer satisfaction metric (or set of metrics) for the internal or immediate customer within the corporation — normally the corporate businesses and their various R&D-related organizations — may be the key diagnostic to indicate that R&D processes are lacking and need adjustment or redesign. The R&D organization itself is probably better served with well thought-out internal metrics than with external metrics that complicate the diagnostic process for R&D when problems are indicated.

3. HOW TO USE THE METRIC
3.1 External customer satisfaction metrics. These metrics will normally be marketing-related or implemented. One variation would be to use a marketing survey, in which various aspects of technology benefits are rated on a five -point scale

3.2 There are two dimensions for internal customer satisfaction metrics: strategic and tactical.

3.2.1 Strategic metrics deal specifically with whether the R&D function is meeting the strategic needs of the customer. The review process might involve matching technology and product roadmaps in a joint meeting, in which technology timing mismatches are resolved. Information can be exchanged; technology previews by R&D to alert the businesses to possible market-creating or market share increasing discontinuities, and the businesses to share future market window and product definitions with R&D. Various metrics can be used, including the five-point rating system mentioned above, or a metric which highlights number or percent of mismatches between product and technology roadmaps.

3.2.2 Tactical metrics deal with whether specific projects are meeting the goal or delivery requirements of the internal customer base. For technology projects in the latter stages of development (near to or entering product development), regular project reviews with the intended customer(s) is important. At quarterly or perhaps semiannual reviews, customer and R&D representatives join in a review of project progress. A useful metric in this case is a report card which each customer representative is required to fill out in the review meeting. This is a very simple questionnaire which has 3-5 survey questions on project progress, and suitability of both the project and the technological approach to satisfy the customer(s) needs. Typical questions might be:

Does this project meet your product technology needs?
Do project milestone dates meet your market window?
Have any strategy changes on your part not been addressed?
What is the overall project score (typical scale 1-5)?
As a related metric, trend analyses can be made both by project and organizationally.

4.0 OPTIONS AND VARIATIONS
One variation on the customer satisfaction process is to establish steering teams to address technology, business and market issues and provide guidance to the R&D organization on strategic issues. For external issues, the teams might consist of focus groups that meet periodically, or focus teams that convene groups which are diverse either geographically, ethnically, or with respect to age group, for example. The focus might be on functional needs that technology capabilities address. The metric would be meetings held, or issues addressed and settled. Another metric might be problems surfaced to be addressed and reported on by the R&D team.

Internal issues might be addressed by customer teams composed of manufacturing, engineering, marketing, financial, and related personnel (including even external consultants) who provide guidance and assist in forming cross-functional project teams. Appropriate metrics are meetings held, issues settled, or problems surfaced. An ongoing metric can also keep track of the % of problems addressed and resolved versus those surfaced.

DEVELOPMENT PIPELINE MILESTONES ACHIEVED

1. METRIC DEFINITION
Development Pipeline Milestones Achieved is a metric which is useful in grading the effectiveness of management and planning of each R&D project. It may also be used as an indicator of performance problems on a given project which can be used to initiate diagnostic and recovery procedures to give each project the best possible chance of success. There are two possible variations of the metric which may be used:

1.1 Percent of project milestones achieved — the percent, by project or overall by sub-organization or laboratory, of all project milestones completed on schedule or within some acceptable time window (90 days, for example) of the forecast date. Trend studies can then be established for organizational performance based on analysis of the data collected by quarter or for whatever other time period is appropriate for the particular industry group.

1.2 Performance level at each milestone — on a project basis, percent of all expected objectives met at the milestone date by which they are forecast to be completed.

2. ADVANTAGES AND LIMITATIONS
The primary advantages of this metric are for project diagnostics and as an indicator of the overall planning and management health of the organization. There are several possible limitations. First, a consistent management system must be in place to assure that the variability of the number/quality/difficulty of the milestones does not cause random fluctuations in the metric. Secondly, the calibration of the metric is important. Since both the stage of the R&D project and the industry of the business involved can greatly affect the trend analyses, the metric user must be careful to identify what the real danger signals are in terms of management and planning deficiencies, and what really constitutes an indicator of project problems for this metric to be useful.

3. HOW TO USE THE METRIC
1.1 Percent of project milestones achieved — a simple approach is to tabulate milestones met on schedule, 1-90 days late (which may constitute on schedule in some cases), 90-180 days late, and so forth. The tabulation can be done quarterly, and the trend information publicized to the organization to focus attention on performance issues.

1.2 Performance level at each milestone — This is more useful as a project diagnostic. One approach is to require project managers to use this metric to summarize performance quarterly or semiannually at an operations review and be prepared to cover diagnostic or recovery steps if a given project shows trends over two or more time periods of failing to meet an acceptable performance level (which will depend on the R&D discipline, the industry, and the stage of the innovation process).

4. OPTIONS AND VARIATIONS
One problem with using milestone achievement as a key metric is that pressure is exerted to make all the milestones. With strong pressure to meet milestone dates, the project manager may be tempted to populate the project plan with easy milestones, to assure that his milestone hit rate is good. It is important therefore to make sure that milestones are realistic and aggressive. Especially in the later stages of the innovation cycle, in product design and productization or manufacturing process development, it is important to assure that the business customer, marketing representative, etc. approve the project milestones to assure that the product development cycle meets the market window requirements of the business. In this case, a much closer tracking of milestone achievement may be necessary than in the earlier stages of the innovation cycle (for example, monthly or in some cases even more frequently).

Another problem with using the milestone approach is that early in the innovation cycle (the R part of R&D), milestones are often hard to define and even harder to forecast. With the heavy level of uncertainty in a long-term research project, the definition of a milestone may be vague, or even made in terms such as define or develop concept. In these circumstances, project managers feel especially uneasy in setting milestone dates for which they become accountable. An approach here is to realize that (1) exact dates are less important, since product or market need windows probably have not been established (and in some cases, market needs or even product existence!), and (2) many of these projects which are early in the innovation cycle will certainly fail, since in the research phase, many more ideas are explored than have a positive outcome. In this context, using milestones as a performance metric may not be useful. If used, the metric should probably be relaxed in some way. For example, instead of setting a goal of completing 100% of milestones, perhaps a goal of 50, 60, or 80% may be set. Another possibility is to declare any milestone made within 90 or 120 days of the forecast date to be on time. The specific approach taken will depend on the industry, the research area, and the stage in the innovation cycle.

CUSTOMER CONTACT TIME

1. METRIC DEFINITION
A major concern in corporate America is that R&D output be relevant to the businesses which it supports. There are several metrics which can be enforced to assure that R&D is satisfying its immediate customers in the corporation. These include Strategic Alignment, Business Funding of R&D Projects, Customer Satisfaction, and Customer Contact Time. This metric is normally a simple count of time (hours or fractional days) spent with the customer. The time may be counted for R&D management and scientists separately, as well as in a total for all customer engagement time. Note that the term customer can refer to both internal customers (representatives of the businesses that R&D supports, either engineering or perhaps marketing) as well as external customers (the end customers of the business of the company).

2. ADVANTAGES AND LIMITATIONS
The amount of time that scientists on research staff spend with customers appears to correlate well with successful innovation in many companies. Probably the more normal scenario for contact will be with internal customers. With strategic planning maturing in many corporations, businesses today are able to share relatively accurate long-term plans with R&D, which can give the R&D organization an insight into competitive requirements in the future. Correspondingly, R&D can highlight differentiating technologies which can provide future competitive advantage. A metric which enforces the regular association of R&D with internal customers ensures that this regular communication occurs. A metric which counts contact time with external or end customers may be especially desirable in certain industries.

A limitation is that simply counting contact time provides only a quantitative measure of interaction time. It does not ensure that the time spent with customers produces the desired result of coupling business and market needs into technological planning and alignment efforts.

3. HOW TO USE THE METRIC
A simple method is to count hours spent with customers in conferences or meetings. Most contact metrics will probably be with respect to the internal customer, since R&D will normally rely on business and marketing sources to translate the end-user needs into technology or functional requirements. There may be a separate count for managers spending time with counterparts in the businesses or in marketing, as well as for scientists. In some organizations, the process of converting end-customer needs to product requirements may impose special end-customer contact requirements on R&D. In this case, there might be metrics for both end-customer and immediate-customer (I. e., business) contact time.

4. OPTIONS AND VARIATIONS
An alternative is to count functions as the contact metric. For example, the R&D organization may have a steering committee (or several technology- or business-related steering committees) which meet regularly; the regularity and frequency of these meetings may be a key strategic metric. Also, R&D organizations may have operations reviews with selected customers regularly to discuss progress on key projects. Counting these meetings may be an acceptable way to judge customer contacts in a class of projects which are entering the mature (near development) stage.

PRESERVATION OF TECHNICAL OUTPUT

1. METRIC DEFINITION
The product of research is information. In the latter part of the innovation cycle (the D of R&D, also known as engineering), preservation of the knowledge gained is relatively easy, since it is embodied in the production drawings, manuals, and source code related to the products and processes of a business. Earlier in the cycle, in the research process, the preservation of information is sometimes less orderly and institutionalized. The purpose of the Preservation of Technical Output metric is to ensure the documentation of information gained in the research portion of the innovation process.

2. ADVANTAGES AND LIMITATIONS
The advantage to using a preservation metric (and associated information process) is that important information is preserved. This is normally accomplished via a documentation requirement associated with each research project (details discussed below). It is especially important to establish a reporting process, using the metric to enforce process discipline, in the long-term research area, since often knowledge gained in such efforts does not have an immediately apparent use. Documenting and preserving the knowledge can be extremely important so that retrieval at a later (possibly much later) date, in an entirely different context, is guaranteed.

The major potential limitation or disadvantage is that the value of measuring the number and/or quality of research reports for research organizations has not correlated well with financial success or value creation of the R&D organization or its associated businesses. As utilized in many R&D organizations, the practices of counting research reports, publications, conference papers, etc. has had debatable value to industry.

3. HOW TO USE THE METRIC
Preservation of Technical Output is a report metric. In many companies, it simply consists of counting the number of research reports per organization, per project, or by technical competency area. This metric can also be expressed as the % of key outcomes of projects that are captured in reports. A related metric is to require a certain number of reports for each research project, either timed (by quarter, semiannually or annually, etc.), or related to certain specific project milestones which are especially significant. Some organizations feel that a more important metric is report requests. In this view, corporate research is for the benefit of internal customers — the businesses of a company that R&D supports with technology inputs. Since report requests indicate interest on the part of the business units, requests for reports show the level of interest in each technology by the businesses which represent the R&D customer base.

The fact remains that reporting metrics do not correlate well with successful innovation (one study showed a counter correlation). Since the preservation of technical information is important, possibly the simplest approach is to require that each project document results (for example, an annual or semiannual report); the metric is simply a check-mark that the documentation is accomplished in an acceptable form.

4. OPTIONS AND VARIATIONS
To reduce the reporting overhead, a simplification is to allow any major publication or conference report which covers the reporting requirement to also satisfy the reporting metric. The report request metric can be dealt with more easily if reports are distributed in electronic form (or at least with electronic abstracts) and requests are limited to electronic requests, which can be easily tabulated.

EFFICIENCY OF INTERNAL TECHNICAL PROCESSES

1. METRIC DEFINITION
This set of metrics seeks to provide a measure of both the efficiency and effectiveness of the operation of R&D processes within the firm.

1.1 Project Assessment

1.11 The total cost of all commercially successful projects divided by the number of commercially successful projects. (Useful when tracked over time with similar projects with similar scopes)

1.12 The ratio of actual to projected costs (and timing) for all projects.

1.13 Percentage of costs devoted to commercially successful projects.

1.2 Portfolio Assessment

1.21 The total R&D budget divided by the number of projects with commercial output. Subdivide by projects of similar type (technical service, short term, long term) and used in conjunction with project value assessment.

1.22 Use the various approaches to Portfolio Assessment from Third Generation R&D (see ref. 1)

2. ADVANTAGES AND LIMITATIONS
This set of metrics needs to be adapted to the needs of each firm — considering the goals, objectives, and priorities for the firm. Assessments need to be made for individual projects (stage-gate, PACE, or similar processes), and for the collection of projects (Portfolios).

3. How to use the metric

4. OPTIONS AND VARIATIONS
Each firm will need to set the metrics relative to its specific goals and objectives. An assessment of the selection termination and management of projects can be made using the following four stages: (ref. 7)

Technical projects: Selection, termination and project management:

Level1.
a) favors short term projects
b) politically driven selection
c) no project monitoring or pre-project planning
d) little inter-functional participation in project teams
e) erratic turnover of team staffing
f) project leader roles not defined
g) no training for project leaders
h) unclear charters for project teams

Level 2.
a) mix of short and medium-term projects
b) no inter-product-line analysis
c) priorities set erratically
d) project tracking
e) some inter-functional participation but not all key functions represented.
f) formal release process for new products
g) some project team stability but conflicts over work priorities
h) project leaders given only minimum guidance or training

Level 3.
a) selection based on multiple inputs from internal and external sources
b) balance of short-, medium- and long-term projects
c) risk analysis incorporated at key phases
d) projects still schedule driven
e) inter-functional teams wherever needed
f) clear allocation of project and functional responsibilities
g) training for project leaders

Level 4.
a) clear links between selection criteria and business and product-line strategy
b) disciplined process for project termination
c) cross-functional planning and execution
d) continual improvement- postmortens, quality measures of both project process and product performance
e) projects are milestone driven
f) differentiated project management procedures for different types of projects
g) scheduling and capacity planning avoid resource contention by competing projects

5. REFERENCES
Portfolio Analysis: P.A. Roussel, K.N. Saad, and T.J. Erickson,Third Generation R&D, Harvard Business School Press, 1991

M.E. McGrath, M.T. Anthony, and A.R. Shapiro, Product Development; Success Through Product and Cycle-time Excellence, Butterworth-Heinemann, Newton MA, 1992

Assessment of work processes: G.A. Rummler and A.P. Brache, Improving Performance: How to Manage the White Space on the Organization Chart, Jossey-Bass, San Francisco 1990

C.C. Curtis, “Nonfinancial Performance Measures in New Product Development, Journal of Cost Management, 8(3):18- 16, 1994. Addresses the distribution of the portfolio in terms of major projects, minor projects and extensions.

R.N. Foster, L.H. Linden, R.I. Whiteley and A. Kantrow, “Improving the Return on Research and Development”, Research Management, 28(1): 12-17, and 28(2): 13-22, 1985.

L.W. Steele, “Selecting R&D Programs and Objectives”, Research-Technology Management, 31(2) March-April, 1988 17-36

P.S. Adler, D. William McDonald, F. MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37

EMPLOYEE MORALE

1. METRIC DEFINITION
This metric takes quantitative ratings of key aspects of employee satisfaction and morale as shown by direct employee survey. It is recognized that employees may feel good and have high morale, yet produce nothing of value for the firm — the real question is are they motivated and committed to create and innovate profitably?

2. Advantages and Limitations Extensive surveys are time consuming and expensive to conduct and must be conducted with sufficient frequency to establish base lines and understand real trends. One must understand also that technical populations tend to have certain biases in such surveys. Many employees feel “surveyed-out”.

3. How to use this metric The typical survey uses five point scales for agreement (strongly agree, agree, neither agree or disagree, disagree, strongly disagree) for importance (extremely important, very important, somewhat important, of little importance, not at all important) and for performance (very good, good, fair, poor, very poor) in answering sets of questions related to work environment, feelings about the company, ratings of the company, ratings of the organization/work location/work group, feelings about the individual’s job, and general satisfaction. Opportunities are given for comments. Such extensive surveys are most often conducted by third parties to maintain confidentiality.

Four tested questions are job satisfaction are:

3.1 If a good friend was interested in a job like yours for your firm, what would you tell the friend?

3.2 All in all, how satisfied are you with your present job?

3.3 Knowing what you know now, if you had to decide all over again, would you take your current job?

3.4 How satisfied are you with the overall employee- employer relations at your firm?

4. OPTIONS AND VARIATIONS
One firm which uses e-mail extensively, every 9-14 months conducts a broad “pulse” survey and asks employees to provide two ratings (using 0-10 point scales with descriptors) one rating the employees work climate/environment and the other rating personal feelings about the work itself. Employees spend less than 4 minutes to reply by e-mail, or to be anonymous by fax or to a voice mailbox. Employees often write additional comments that give information sought in the more extensive surveys. Confidentiality is assured and rapid feedback (within 2 weeks) of survey results to participants maintains a high level of interest and participation.

5. References.
6.1 C.J. Cranny, P. Cain-Smith, and E.F. Stone, Job Satisfaction, Lexington Books, New York, 1992

6.2 R. Katz (ed), Managing Professionals in Innovative Organizations, Harper Collins, New York, 1988

GOAL CLARITY

1. METRIC DEFINITION
This metric uses an interval rating scale assessing the extent to which project performance objectives are clearly defined and understood by all participants on the project team.

2. ADVANTAGES AND LIMITATIONS
While this metric may provide a semi-quantitative assessment there is a degree of subjectivity for an individual to assess the extent to which he/she really understands objectives and roles on the team.

3. HOW TO USE THE METRIC
A member of the team would provide to all members of the team a survey form to assess understanding of the project objectives and commitments. This can be done using some interval rating scheme. It would be wise to conduct this survey several times through the life of the project to really assess the level of understanding.

4. OPTIONS AND VARIATIONS
A four stage assessment can be constructed with the following as the highest stage for individual team members:

a) clear understanding of the expected product of team effort
b) personal belief that a team is the right way to develop/achieve the expected product
c) understanding whether the team is an implementation, recommendation and/or informational team
d) understanding the team’s operational ground rules and end-result boundaries
e) belief that the team has all the appropriate knowledge, functions, diversity, levels and locations, so that the expected product of the team effort can be achieved using the minimum number of people?
f) high personal commitment to achieving the objectives
g) clear understanding of personal role
h) acceptance of the recognized team leader
i) capability to draw on additional resources to keep the core team to a minimum

5. REFERENCES
J.R. Katzenbach and D.K. Smith,The Wisdom of Teams, Harvard Business School Press, Boston 1993

IRI Quality Director’s Network study on self-directed teams (in progress)

H.P. Dachler, and W.H. Mobley “Construct Validity of an Instrumentality-Expectancy-Task-Goal Model of Work Motivation”, Journal of Applied Psychology, 58. 397-418, 1978

PROJECT OWNERSHIP/EMPOWERMENT

1. METRIC DEFINITION
Empowerment involves management endowing the project team with the authority to make decisions on the project and to carry them out independently without constantly having to seek management permission or approval.

Project Ownership encompasses a variety of feelings and beliefs of the project team that they:

believe in the project and its goals
are committed to the project and want it to be successful
will share in the credit and reward if the project is successful
have authority and accountability for making decisions and carrying out the project
This metric has been reported to correlate positively with successful innovation from R&D. It is also likely related to employee morale on a project team.

Level 1
Projects are defined by the businesses and R&D management without the involvement of project leaders and technical people
Project team members are working on projects because they were assigned to them. They may or may not believe in the goals or be committed to achieve them* Decisions are made by R&D management and business managers; project leaders and team members are not consulted

Level 2
Some EMPOWERMENT and project ownership, but not extensive

Level 3
Extensive empowerment and project ownership, but not uniform throughout the organization

Level 4
Project leaders and technical people are involved from the beginning in defining all projects and their goals
All project teams have a high degree of belief in and commitment to their projects
Project leaders and team members have authority and accountability for making decisions and carrying out their projects; they review their decisions and progress with management and business partners on a regular basis * Project leader has financial authority and accountability for the project

2. ADVANTAGES AND LIMITATIONS
This metric is relatively easy to assess. It requires an honest and objective assessment of practices in defining and carrying out projects, but it does not require a large amount of data collection, history or benchmarking.

3. HOW TO USE THE METRIC
It is recommended that input be obtained from both managers and project teams to compare their perceptions. This metric would lend itself to measurement in an employee opinion survey. The metric is an overall assessment of practices throughout the organization that is being measured. If practices are different in different units or projects, the metric will be an average. Levels 1 and 4 are relatively straightforward. Levels 2 and 3 are interpolations between 1 and 4.

4. OPTIONS AND VARIATIONS
In general, Empowerment is a necessary factor which precedes and contributes to Project Ownership. They might be split out into separate metrics.

5. REFERENCES
Weisbord, M.R. 1990. Productive Workplaces: Organizing and Managing for Dignity, Meaning, and Community, Jossey-Bass, 121-141.

MANAGEMENT SUPPORT

1. METRIC DEFINITION
Management Support is critical to successful innovation. It is in essence showing the project teams that they are respected and trusted and providing for them all the tools they need to carry out the projects successfully.

Management Support is manifested in many ways, including:

Empowerment of project leaders and project teams
Providing appropriate input, guidance and direction on larger issues and allowing the team to attend to the details
Providing resources for the project, including people, equipment, consultants, attending technical conferences, customer contacts, etc.
Personal attention, such as attending project meetings and visiting labs
Verbal affirmation of the teams, their approaches, and their progress
Peer recognition, company and departmental awards
Defense and support of the teams in response to questions and criticism
Allowing project teams to fail without negative consequences
Low ratings in this area may point to a breakdown in relations and credibility between R&D management and the project team and, perhaps, between R&D management and business management.

Level 1
Scenario A – Micromanaging* Managers generally do not trust technical people to carry out their projects and are very critical of technical people
Managers define and direct the projects
Managers are running the show, in meetings and in the labs
Technical people are punished for project failure in performance appraisals

Scenario B – Neglect
Managers pay no attention to the projects and project teams
Managers trust the technical people implicitly and leave them to themselves
Managers rarely attend project meetings or visit the labs
Projects are understaffed
Project teams are making do with inadequate equipment and resources
Technical people are rarely rewarded in peer recognition or company awards.

Level 2
Some management support, but not extensive

Level 3
Extensive management support, but not meeting all the goals and/or not uniform throughout the organization

Level 4
Managers respect technical people and trust them to carry out their projects
Managers participate with technical people in defining projects
Managers contribute ideas, but give broad guidance and direction in carrying out the projects

Managers regularly attend project review meetings and frequently visit the labs
Project teams are given the resources they need to successfully complete their projects
Management acts as an advocate for the project team, facilitating interactions with other functions

Technical people are regularly recognized and rewarded in peer recognition and company award programs
Project failures are acknowledged, analyzed constructively, and used as a learning tool to see what could have been done better, but team members are not punished

2. ADVANTAGES AND LIMITATIONS
This metric might be perceived differently by managers and employees. Although it might be evaluated simply by an honest and objective assessment of management practices, resource allocation, relationships between managers and technical people, and how project successes and failures are dealt with, it would be more meaningful to get input from project teams as well. Assessment will require little history or benchmarking.

The metric is an overall assessment of practices throughout the organization that is being measured. If practices are different in different units or projects, the metric will be an average. Levels 1 and 4 are relatively straightforward. Levels 2 and 3 are interpolations between 1 and 4.

4. OPTIONS AND VARIATIONS
There are several aspects to Management Support, such as empowerment, resources, and recognition. They may be measured altogether or split out into separate metrics.

5. REFERENCES
Farris, G.F. and Ellis, L.W. 1990. Managing Major Change in R&D, Research-Technology Management, 33 (1), Jan-Feb.

PROJECT CHAMPIONSHIP

1. METRIC DEFINITION
An effective Project Champion is a person who believes deeply, but objectively, in the project and the need for its pursuit to commercialization. This is a person who takes ownership of the project and has the authority and resources to ensure that:

a. adequate market information is gathered up-front to justify the project and define appropriate goals
b. the project team is supported and motivated
c. if the project is successful in meeting the goals, the resulting product will be taken to market.

Ideally there should be a champion associated with the sponsoring business, such as a division manager, business manager, or product manager, as well as a champion in R&D. The Project Champion can lend an air of enthusiasm, optimism, and urgency that tends to permeate the project team.

Level 1
There are no effective champions for projects* R&D people are responsible for gathering market information and setting project goals
Business people are not involved throughout the project* R&D people usually are responsible for field trials and taking products to market

Level 2
The champion for most projects is within R&D
Some business participation in projects, but not extensive

Level 3
Extensive business participation in projects. Some business champions of projects. * Not uniform throughout the organization

Level 4
All projects have an effective champion who is associated with the sponsoring business unit as well as a champion in R&D

Business people actively participate in obtaining upfront information to define and justify the project and establish its goals* Business people support and help motivate the project teams* Business people take a leadership role in arranging field trials and commercializing the product

2. ADVANTAGES AND LIMITATIONS
It is critical that the champion believe strongly in the project, however, the champion also must be analytical in assessing the project and its progress, so as not to drive the project beyond when it appropriately should be terminated.This metric is relatively easy to assess. It requires an honest and objective assessment of the role of business partners in projects, but it does not require a large amount of data collection, history or benchmarking.

3. HOW TO USE THE METRIC
The metric is an overall assessment of practices throughout the organization that is being measured. If practices are different in different units or projects, the metric will be an average. Levels 1 and 4 are relatively straightforward. Levels 2 and 3 are interpolations between 1 and 4.

4. OPTIONS AND VARIATIONS
Business involvement typically varies throughout the stages of a project, such as project definition, technology development, product development, and product commercialization. This metric might spawn additional metrics for the different stages of each project.

5. REFERENCES
Farris, G.F. and Ellis, L.W. 1990. Managing Major Change in R&D, Research – Technology Management, 33 (1), Jan-Feb.

Kanter, R.M. 1983. The Change Masters. New York: Simon and Shuster.

INFORMATION TECHNOLOGY USE IN R&D

1. METRIC DEFINITION
This set of metrics measures the extent of use and the ways in which information technology is used within R&D.

1.1 Extent of IT Use in expenditures in the R&D budget.
IT expenditures are a measure of the enhancement of R&D staff effectiveness by the use of information technology. The thrust of the metric is also how to justify this IT spending to higher management, and how to determine an optimum amount. This may be a non-linear effect – while too little use may not match the capability of your competition, too much use may involve R&D staff in activities less optimal than the use of their time directly on R&D. The form of the metric is usually to measure the amount, or ratio of the total R&D expenses of the firm, spent on information technology hardware and software.

1.2 Ways in which IT is Employed – Impact of IT on R&D
The intent of this metric is to measure how much impact information technology (IT) has on the innovation chain of your company. It is usually measured on a scale of 1 to 4. Level 1 is when IT is only used to keep track of staff times and costs of R&D activities. In level 2, IT is also used to provide tools for technical computing and/or hardware and software for the company’s products and services. In level 3, IT also enhances the effectiveness of innovation management by substituting for human effort in R&D. Level 4 IT also lets you rethink how R&D is done by enabling doing things that can not be done any other way. Where unit scores vary considerably from one unit to another, managers should consider using a median score from all the sub-units.

1.3 Extent of IT use in Managing R&D
The intent of this metric is to measure how far along you are at using IT in managing the innovation process. Level 1 is when IT is only used by R&D groups on an individual basis. In level 2, IT is integrated between the IT function and a few groups in the R&D activity. In level 3, IT is fully integrated in the process of managing technology within the R&D department. Level 4 is when IT is fully integrated in the process of managing technology with the all department managers from marketing through R&D through manufacturing/operations.

1.3 1.4 Other possible metrics that might be developed
Breadth of usage by application category vs. benchmarks; Quality of usage by application category vs. benchmarks; Percent of R&D staff using IT at benchmark levels.

2. ADVANTAGES AND LIMITATIONS
The advantage of this metric is that it enables the justification of IT in R&D both at financial budget time, and in benchmarking IT use and impact in R&D against competition and strategic goals. Thus, the stakeholders most interested would be financial officers and managers, and strategic planners.
One limitation is that these are relatively untried metrics, with minimal research support as to their effectiveness. Thus, it is not yet clear how many R&D groups track IT costs. Another limitation is that it may cost more to follow than the benefit derived from it. Information technology is evolving and developing so rapidly that its dimensions are not fully appreciated and some of the measures may be difficult to make.

3. HOW TO USE THE METRIC
The amount of IT expenditure in R&D should be tracked over time, usually the annual budget cycle. The ratings might need to be done with input from IT professionals. The rating can be compared to other measures of R&D effectiveness to see if the change in IT expenses has enhanced effectiveness. If information is available on competitors, this too can be tracked and used to justify future IT budgets.
The ratings of extent of impact and use should also be tracked over a time period for comparison with competition and strategic needs.

4. OPTIONS AND VARIATIONS
Service companies and companies with a high IT content in their operations may wish to use alternatively any of the software effectiveness metrics such as the Capability Maturity Model developed by the Software Engineering Institute at Carnegie-Mellon.

5. REFERENCES
There is an IT project of the R-o-R Committee started in 1995. This is a new area with little published references. The most general reference is U. S. National Research Council, Information Technology in the Service Sector, Washington, DC: National Academy Press, 1994.

The SEI Capability Maturity Model is reviewed, and compared with Total Quality Management (TQM) and other evaluation techniques, in H. Saiedian & R. Kuzara, “SEI Capability Maturity Model’s Impact on Contractors,” Computer (IEEE), 28(1), Jan., 1995, pp. 16-26. This source has 12 additional references.

GATE EFFECTIVENESS

1. METRIC DEFINITION
In a stage-gate or milestone R&D management process, each project is subject to a design review at periodic stage gates or milestones.

1.1 Quantitative Yield
This metric is designed to measure how effective the design review process is by measuring in terms of successful yield at each gate or milestone from project initiation to project commercialization. The assumption is that the higher the yield, the more effective is the stage-gate or milestone design review process. However, a very high number on this metric indicates that the gate is not working very effectively, i.e., there is no need for a filtering process.

The metric is most accurately used by the percent of the value of R&D projects passing each gate or milestone which meet the criteria for passing the next gate or milestone. The simpler metric of percent of the number of projects may also be used, but runs the risk that it might be excessively influenced by small, easy to understand projects and misrepresent the true picture.

1.2 Income Contributors
This metric is the percent that pass through the gate that become significant income contributors. This is a more retrospective number than metric 1.2, but better reflects what the stage-gate process intends.

1.3 Costly Failures
Another metric is to document the reasons costly failures passed through the gates and were not screened out. This would identify retrospectively what criteria were missing.

2. ADVANTAGES AND LIMITATIONS
The advantage of these metrics is to calibrate the stage- gate, milestone, and design review process, thus enabling diagnosis of the R&D management process. While metric 1.1 is a somewhat retrospective metric, the delay time is only one gate or milestone period long, and thus it is a more prompt feedback metric than overall success or failure in the market place indicated by metrics 1.2 and 1.3. Too low a yield at any gate or milestone indicates that too many projects are getting through for further expenditure which do not deserve to have been supported since they did not pass the subsequent gate or milestone. The limitation is that this is a subjective metric, and needs some historical data within the firm on which to base a benchmark percentage.

3. HOW TO USE THE METRIC
The only way to use this metric is to keep records for a period of time, and use them to evaluate trends. This is somewhat of an administrative burden, justified by its use in continuous improvement of the R&D management process.

4. OPTIONS AND VARIATIONS
For the final gate or milestone before commercialization, metric 1.1 and 1.2 become the same as Metric 28: Percent or value of R&D which is commercially successful.

5. REFERENCES
Van Remoortere, F., and Cotterman, R. 1993. Project Tracking System Serves as Research Management Tool, ResearchTechnology Management, 36 (2), March-April, 32- 37. This reference covers milestones in general.

Ellis, L. W. 1984. The Financial Side of Industrial Research Management. New York: Wiley. This reference, on page 105, covers the histogram approach to measuring milestone, gate and project completion.

Patterson, M. L., Accelerating Innovation: Improving the Process of Product Development, New York: Van Nostrand Reinhold, 1993. This reference introduces the measurement at each originally forecast milestone date of the percent actually completed of the originally estimated work.

NUMBER OF DEFECTS REPORTED

1. METRIC DEFINITION
This metric is intended to measure the lack of quality in the finished product or service. As such it is a retrospective metric, summing up all things that did not go right in the R&D process. This metric may take any one of the following forms: Number of defects reported at any stage by downstream operations; “bugs” found in computer programs; defects as shown by the number of change orders issued to manufacturing or operations; number of end customer complaints.

2. ADVANTAGES AND LIMITATIONS
The advantage of this metric is that it tabulates the ultimate cause of customer dissatisfaction. It is important for diagnosis to identify which defects were caused by design in R&D, and which were caused by downstream operations and not attributable to R&D such as faulty scale-up, manufacturing flaws, packaging problems, etc. This has the advantage over metric 4 because it records the “bugs” identified along the technology transfer process.

Its primary limitation is that it comes rather late, after the internal or external customer has found the problem that was missed in the R&D process. Just as for any inspection system of quality, it is better to do it right the first time, because quality cannot be inspected in after the project has left R&D.

3. HOW TO USE THE METRIC
Like other quality metrics, this is best kept as a running average over a time period – short enough to provide as prompt a feedback as it can, but sufficiently long to smooth out the ups and downs.

4. Options and Variations.
See also Product Quality & Reliability metric : .

5. REFERENCES
Lutz, Robert A. 1994. Implementing Technological Change with Cross-Functional Teams, ResearchTechnology Management, 37(2), March-April, 14-18.

CORE TECHNICAL COMPETENCY

1. METRIC DEFINITION
The understanding, definition, and maintenance of strategic Core Technical Competencies is a key to both current and future R&D performance. The strength of Core Technical Competencies may be characterized using the categories described in the Third Generation R&D Tutorial.

1.1. Scoring of Core Technical Competencies

The metric is defined by first establishing the core competencies of the the firm in the business segment(s) of interest and then rating the specific competitive position in each Core Technical Competency: Score Level Definition 5 Dominant Sets the pace and direction of technological development and recognized for such in industry to express independent technical actions and set new directions3FavorableAble to sustain technological competitiveness in general and/or leadership in technical niches Unable to set independent course. Continually in catch-up mode1WeakUnable to sustain quality of technical output versus competitors. Short-term firefighting.

1.2. Aggregate Rating of Core Technical Competencies.

To obtain an aggregate for the firm or business segment(s), these scores may be averaged over all the competencies defined.

2. ADVANTAGES AND LIMITATIONS
The first advantage of such a metric is that it is considered at all. Use of the metric forces agreement within the firm on Core Technical Competencies and the competitive position in each. Understanding and acceptance can drive programs to maintain competencies or to remedy deficiencies. The process of applying the metric provides a healthy dialogue among the commercial and technical functions.

3. HOW TO USE THE METRIC
To rate the position of the firm with respect to each Core Technical Competency and to develop an aggregate rating of the firm over several competencies, it is first necessary to discern those technical competencies which are essential or most important to the success of a firm in selected markets or arenas of business. The prime references on core competencies from HBR and other current references in the field are recommended. There must be agreement between technology and commercial functions on these Core Technical Competencies, whether or not the firm has an advantaged position in any of them.

4. Options & Variations
For some industries and competencies, it may be possible to define an objective metric having to do with competitive technical performance, such as lead time relative to competitors on major innovations; or position along the curve of improving product performance relative to competitors ( e.g. 20% better).

5. REFERENCES
Arthur D. Little, Inc., Third Generation R&D, IRI Tutorial, April 29, 1994.

Prahalad, C. K. and Gary Hamel, Core Competence of the Corporation, Harvard Business Review, May-June 1990.

DELAYED STAGE KILLS

1. METRIC DEFINITION
This metric is central to the basic notions of R&D yield, productivity, effectiveness and risk/performance abilities. It is also multi-functional in character.

This metric is defined as the failure of projects to move beyond a given stage; particularly in reference to the later stages of the product/process development activity. Since the purpose of R&D is to produce knowledge that reduces risk, then it follows that later stages should be represented by projects that have been risk-reduced. If this is so, there should be more successes and fewer failures at later stages. Conformance to this goal is captured by this common metric as expressed in its negative form.

The most frequently used form of this metric is the number of projects that are terminated either in commercial introduction or in the last phase of development.

However, this can also be generalized by examining the actual failures (or rejections) at a given stage and comparing this to a predicted level. An example might be that at a certain stage you expect that only 10% of the projects should fail, but you observe that 30% are failing at this point. This indicates a later stage kill ratio that is three times higher than expected. There is debate as to whether this metric should be measured only at the stage in question or should be made on a cumulative basis that integrates all future failures that occur beyond a particular stage. This may be more appropriate to some organizations than to others. The important concept is to establish a useful measure of a result to be avoided, i.e. making decisions to terminate later than they could have been made. Overall, these metrics are critical in describing the kind of risk can be attempted and executed well for a given R&D portfolio.

2. ADVANTAGES AND LIMITATIONS
The advantages of these late(r) stage kill metrics is the focus they provide on the cost of ineffective processes. No one wants to spend time and money in significant amounts at later stages just to see it all thrown away because of an unanticipated failure. This is particularly true if it the cause of the failure should have been uncovered by the R&D process at an earlier stage.

These metrics capture the degree to which R&D is truly its risk reduction role for the corporation and thereby increasing its contribution to the value of the enterprise. They answer the question: Have you [R&D] helped me [the Business] avoid problems and used my resources wisely? Because of their value in this regard, they are useful metrics for targeting future R&D performance. However, they only represent part of a process that involves other functions besides R&D. And, given that they focus on the quality of a ‘process’ and ‘processes’ take time to change, the numbers reflected by these metrics are associated with activities that are in the past. These metrics are lagging indicators. They are a nice track record, but they may not be reflecting accurately a current level of effectiveness.

The metrics should be examined carefully for consistency with business strategies and the results required vs. the investments in R&D. In situations where the metrics indicate a high late(r) stage kill, there will be a difficulty in taking on high risk programs. This means that this metric should provide a guide to the level of risk that can be undertaken in executing the overall business and technology strategies. One or the other must be shifted, and variations in how R&D is conducted and managed need to be examined.

4. OPTIONS AND VARIATIONS
The most common option is to apply these metrics to all stages and not just the last one. Originally, and most typically referred to as ‘late stage kills’, these metrics can be generalized to apply to all stages. However, they lose the potency and impact of having to ‘write-off R&D’ at a time period when this should not be the case. Another variation is to apply these metrics to future prospective.

COST RELATIVE TO BUDGET

1. METRIC DEFINITION
The cost effectiveness of R&D is measured by comparing the value of technology output to the cost of producing that output. Technology output is generally measured with respect to objectives, such as short-range or annual targets or long-range or strategic targets. Costs are generally measured with respect to budgets, such as annual budgets or longer-range or cumulative project budgets.

1.1 Metric -1: Annual Budget Performance

Absolute and relative variance of actual total annual project cost compared to the agreed-on annual project budget, where both are accounted on the same basis. This metric may be defined for an individual project or class of projects, for all projects from a specific R&D unit, for all projects supporting a business segment, or for the total R&D in the firm.

1.2 Metric -2: Cumulative Project Budget

Absolute and relative variance of total cumulative project cost compared to the agreed on budget for an individual project. This is especially important in the early stages of a multi-year project as an indicator of future trends; it is important toward the end of the project as a measure of total cost for comparison with total benefit.

2. ADVANTAGES AND LIMITATIONS
These metrics are important to management in a firm and have the advantage that they are numerical and appear to be objective. The accuracy of the metrics is limited by the quality of the accounting processes in the firm and by the quality of the effort to assign costs accurately, including those costs which come from shared or purchased resources. Since the standard of measurement is the project budget, the quality of this metric also depends on forecasting a budget which is justified by the merits of the project and provides appropriate, affordable resourcing to project R&D. As knowledge increases and the project progresses, it may be necessary to adjust the previous forecast budget in order to make the performance versus budget a meaningful measure of effectiveness.

3. HOW TO USE THE METRIC
The first key to use of these metrics lies in good analysis and understanding of how manpower, support, and other costs are incurred in R&D for the project, collection of projects, or R&D unit of interest. The second key lies in establishment of project budgets which are meaningful and which reflect accurately the expectation of costs to be incurred. The third key lies in accurate assignment of costs on an ongoing basis. Then costs can be compared to budgets at appropriate times and the metrics can be computed. The fourth key to use is the avoidance of misuse: the purpose of R&D is to create value, not to demonstrate that project budgets can be forecast precisely.

4. Options & Variations
These metrics can be defined for an individual project, for a classification of projects (such as all exploratory projects), for the projects supporting a business unit, or for the total R&D effort of the firm.

5. REFERENCES
Ellis, L. W. 1984. The Financial Side of Industrial Research Management. New York: Wiley.

Ellis, L. W. 1984. Viewing R and D Budgets Financially. Research Management, XXVII (3), 35-40.

DECISION GATE PROCESSES

1. METRIC DEFINITION
This metric is intended to assess the decision process itself at decision points (gates) of the stage gate innovation process. For the generalized case of a work process, such a metric is important to the management and improvement of the process. There are two important metrics: (1) an objective measurement of the degree to which a formal decision process is applied; and (2) a subjective rating of the effectiveness of the decision process.

1.1 Use of Decision Process

The use of a decision process is characterized by computing the proportion of programs (either budget % or number %) passing through the stage gate process which have been subjected to a formal decision process at the last gate encountered.

1.2 Quality of Decision Process

The quality of the decision process is rated subjectively (on a scale of 1-4) with respect to the primary dimensions of Fit, Attractiveness, and Capability. 4Parameters which define Fit (Attractiveness, Capability) are precisely defined and were the objective basis for decision at the last gate3Parameters which define Fit (Attractiveness, Capability) are well defined and were given primary consideration in the decision at the last gate.2Parameters which define Fit (Attractiveness, Capability) are poorly defined and received limited consideration in the decision at the last gate.1Parameters which define Fit (Attractiveness, Capability) are neither defined nor given significant consideration in the decision at the last gate.

2. ADVANTAGES AND LIMITATIONS
These metrics are subjective measures of what is intended to be an objective decision process. They are intended to drive the use of objective decision processes at the gates of a stage-gate innovation process. It should be recognized that the best R&D decisions may not be obtained by deterministic processes. It may be that (3) in the scale above is actually the optimum, when combined with excellence in subjective technical and commercial judgement.

3. HOW TO USE THE METRIC
This metric is relevant to a stage-gate innovation process in which go, no-go, turn-back, abandon decisions are made at each gate. Metric 1.1 measures the use of the process and Metric 1.2 rates the quality of the decision process. Metric 1.1 may be computed by clearly identifying the formal decision process and by computing the proportion of projects subjected to it. Metric 1.2 requires a subjective evaluation by knowledgeable people.

4. Options & Variations
The best measure of the effectiveness of the decision process is the quality of subsequent results (e.g. the proportion of projects surviving the subsequent gate) but that is retrospective. The current metric is intended to be prospective and focuses on the nature of the decision process.

5. REFERENCES
Van Remoortere, F., and Cotterman, R. 1993. Project Tracking System Serves as Research Management Tool, Research¥Technology Management, 36 (2), March-April, 32-37.

PROBABILITY OF SUCCESS

1. METRIC DEFINITION
This is obviously a prospective measurement. It is a good measure of the wealth and depth of the new product pipeline. It can also be used as a key criteria for passage through gates in the stage gate process.

The probability of success on an R&D program ( project ) depends on the likelihood that technical problems can be resolved and that the market opportunity can be realized. The product of the two factors is an educated guesstimate of the probability of commercial success. The probability of success times the projected peak sales is the projected, probability corrected sales estimate.

The sum of all major projected, probability corrected sales is a good prospective measure of the richness of the R&D pipeline. Major changes in the total,particularly on the downside, should be thoroughly investigated for cause and remedial action taken if warranted.

The probability of success of a program can be used as a key criteria for passage through one or all of the stage gates. The probability should remain constant or increase as the development proceeds.

2. ADVANTAGES AND LIMITATIONS
Advantages of the measurement are:

Among the best prognosticators of R&D’s future success
Invaluable in analysis of portfolio balance
Measured over time, provides a good measure of the organizations predictive ability
Can provide warning alarm for problems and fears not yet articulated

DISADVANTAGES/LIMITATIONS

Qualitative and very subjective
Subject to gamesmanship to push favorite projects
Requires solid market knowledge and forecasting ability
Subject to usual technical overconfidence
Early market forecasts and become “engraved in stone”

3. HOW TO USE THIS METRIC
An informal panel of program or project members and other lab experts should review the program and especially the technical issues and obstacles impeding it’s progress. Each panel member should rate the likelihood of project success ( including meeting all product attributes ) on a scale of 1 to 10 where 10= very simple to complete and 1= insurmountable obstacles exist. Marketing personnel should provide regular updates on the probability of commercial success using the same numerical system. The product of the two factors is the overall success probability. Significant changes, particularly drops, should be investigated for underlying causes. The sum of the probabilities times projected peak sales across the R&D portfolio is an excellent prospective of the firms new product prospects.

4. OPTIONS
The projected, probability corrected sales forecast spread over 5-10 years is a valuable measure of the R&D new product pipeline. Gaps are easy to spot and remedial adjustments in the portfolio can be made. Probability trends on a program can be used as an indication of enthusiasm or lack of it. Calculations can be refined using net present value calculations, but the rough nature of the estimates usually negates the effort.

5. REFERENCES
Merrifield, D. Bruce, 1981, Selecting Projects for Commercial Success, Research Management, 24 (6), 13-18. This is an early source article for the technical probability of commercial success.

Ellis, L.W., 1984. The Financial Side of Industrial Research Management,, New York, Wiley. Chapter 6 discusses the probability distribution of outcomes.

TECHNOLOGY PLANNING

1. METRIC DEFINITION
This metric employs a four stage/interval scale for an assessment of key elements in the technology planning process and in the technology plan itself.

Technology Planning/Plans: Self Assessment

The following will be put in matrix format: criteria vs. levels of performance 1-4:

Planning Processes:

Involvement:
business planning and technology planning are either non-existent or remain superficial and are not used as a basis for action.
Business planning and technology planning are carried out as separate processes with little or no involvement of technology in business planning.
Technology function is asked for input into business plans and business plans provide basis for technology plans; technology is often in a responsive mode.
Technology function is an integral part of business planning significantly impacting the plans and technology planning is an important part of the business planning process. Senior professionals have opportunity to be involved and provide input.

Timing:
technology planning is done sporadically and is not built upon prvious plans.
technology planning is done periodically responding to organizational needs (demands for planning or for budget preparation)
technology planning is done regularly on an annual basis and results reviewed regularly. Plans are iterative — building on last year’s plan.
technology plans are “evergreen” and are used regularly to guide action and decision making and are modified periodically based on results and external events.

Internal Communications:
technology plans are not communicated to the R&D technology organization (non-existent, or incompletely formulated)
technology plans and related business plans are communicated to and understood by less than half of the technical organization. Some familiarity by the members of the business team.
technology plans and related business plans are communicated to and understood by more than half of the technical organization; members of the organization understand how their programs are supportive.
major parts of the technical organization are integrally involved in the planning process; communication/understanding/doing are are all interrelated.

External Communications:
Little if any communications of needs and opportunities outside the immediate organization.
What cooperative efforts and discussions take place with those outside the immediate organization are not done within the framework of the technology plan or critical needs of the business.
members of the technical organization meet sporadically with technical resources outside their organization to discuss needs spelled out in their plan.
members of the technical organization under the leadership of the technology director interact regularly with others outside the organization (corp. other R&D units, Universities, other external research sources) to look for opportunities to address the issues in the plan. Continuing relations/cooperative efforts are built upon past successes. (addition: where R&D is separate from the business/operating unit — may want to have gate keeper/formal liaison).

Learning/Improving (Quality)
No learning takes place based on results of the previous plan and projects.
Some lessons are simply acknowledged; some actions taken in response
The planning process and/or results of individual programs/projects are reviewed in a semi-formal manner; some actions taken within the organization to to improve the process and/or execution of future projects.
The planning process and results are reviewed and improvements/responses to learnings are carried throughout the organization.

Elements of the Plan Itself

Business Issues
technology programs are not explicitly related to key business needs/strategies
technology programs are related to business needs in only vague terms
it is clear how results from technology programs contribute to business success
it is clear how results from technology programs not only contribute to business success but how some results can provide a real change in the game.

Nature Of The Markets, Customers
little if any reference to markets and customer needs
a few customer needs are mentioned
the “voice of the customer” is heard in the plan; market structure/environment are well understood
customer needs are spelled out but future needs in the marketplace are anticipated

Competitors
little if any reference to competitors
some competitors and their position in the market are recognized
Competitive intelligence is reflected in the plan with an understanding of most competitive threats, in-kind and not-in kind
. technical program strategies recognize the thrusts of competitors — including their technological strengths and weaknesses (addition: include recognition of potential end runs from non-traditional competitors)

Human Resources
people’s skills/competencies are not mentioned in the plan — other than number of budget people for individual projects
people needs are addressed — but not in connection with skills and competencies
skills and competencies needed are mentioned but with no analysis.
the skills and competencies needed to carry out the technology plan are addressed in the plan — both near and longer term. Needs to fill gaps are discussed, and a plan to fill gaps

Contingencies/Alternative Plans
no alternatives to the planned programs are presented
any alternatives or contingency plans are not realistic and offer no real choices
limited cut and dried alternatives are presented
alternatives and contingencies offer options in the case of changes in resources or major changes in economic or market conditions.

Other Elements Of The Plan
none of the following are addressed to any extent: capital productivity costs product quality opportunities for growth in the business environmental challenges maintenance/development of (core) competencies opportunities for breakthroughs
some of these elements are addressed
most of these are addressed where they are important to the business
None are overlooked and impact on technology programs are seen.

Core Competency Considerations
Core competencies (technology) are not considered part of the planning process.
Core competencies are discussed but are not central to the planning process.
The planning process defines core competencies required and the plan to develop and maintain.
Core competencies that are in place are understood and are an important consideration in the planning process. Competencies that are required are clearly defined with a clear plan of action to acquire the competencies and to prune those not needed.

Time Frame of Plan
essentially one year
1-3 years
3-7 years
7-10+ years (may also be framed around the planning cycle)

Content/Documentation of the Plan
plan covers the relevant details and consists of prose and data. No effort made to use visual planning charts.
plan covers the relevant details with back up prose and data. Visual planning charts are included in the plan.
plan covers the relevant details with back up prose and data. Executive summary defines clearly the most important issues.
Plan covers the relevant details with back up prose and data. Plan is summarized in less than 15 slides using visual planning charts which easily and clearly define needs and actions.

Patents/Intellectual Property
no mention of value and impact of patent estate
brief mention of key patents currently held
patent estate is presented along with key patents of competitors.
value of present patents is understood and analyzed along with plan to build estate in key areas; key patents of competitors are also included.

Other Areas To Consider In The Planning Process:
a) training and skills needed to be developed or improved to do the planning
b) Continuous improvement of the planning process (applying TQM to this process)
c) basis for alignment/resource allocation for all parts of the organization.
d) assuring that plan delivers results — accountability; relation to performance management.

Other Areas To Be Considered As Elements Of The Plan (some may be in other criteria):
a) explicit connection between technology development and sustainable competitive advantage.
b) balancing the portfolio and aligning with other business initiatives
c) identifying and nuturing critical technical capabilities.
d) meeting needs of both external and internal customers.
e) understanding evolution of markets and changes in technology needs.
f) understanding relation of technology and product life cycles. (S curves and technolical limits)
g) metrics for measuring implementation of the plan and success of the planning process.
h) time-sequencing of steps for implementation (technology road maps?)

2. ADVANTAGES AND LIMITATIONS
This assessment matrix can provide some common ground for assessing the quality of a process and the result of the planning process. Since a common matrix is being used by a number of firms it provides a way to make some external comparisons. The assessment factors/elements however should be modified to reflect the process and plans for the individual firm.

3. HOW TO USE THE METRIC
Those doing the planning and developing plans should use the matrix as a self assessment tool and then focus on areas for improvement.

4. References
“Integrating Technology and Business Planning in IRI Companies” 73 minute video, self assessment matrix, Slides used by speakers in the video, written summary, and bibliography; available from Industrial Research Institute, 1550 M St. NW Suite 1100, Washington D.C. 20005-1708

P.S. Adler, D William McDonald, F MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37

ENVIRONMENTAL MANAGEMENT IN R&D

1. METRIC DEFINITION
This metric uses a stage/interval self assessment/ appraisal framework to determine the quality of environmental management in the R&D function.

“Reactive”

a) policies and strategies: environmental policy not developed or well-communicated; environmental problems and opportunities given limited attention by management and staff.
b) scope of activities: focus solely on regulatory compliance by the R&D facility; no environment-related R&D or employee awareness activities
c) level of participation: limited to a few (safety and environmental group)
d) Management processes and systems: administrative procedures in place only for compliance management
“Participative”

a) policy developed and publicized; management encourages participation and contribution
b) good citizen initiatives underway; awareness and training programs in place
c) entire staff involved in one way or another in citizen initiatives; limited involvement of management
d) Total Quality of Continuous Improvement Teams deployed
“Active”

a) strong ownership of policy by staff; management ensures environmental planning and review for R&D projects
b) projects subject to environmental planning and analysis
c) entire staff involved in environmental planning and review of R&D activities
d) environmental check-lists used for project screening; stage-gate system used to guide developments
” Innovative-Leader””

a) policy encourages active search for environmental innovation and influences corporate direction; strong commitment of corporation to R&D initiatives
b) large portion of portfolio aimed at cleaner technologies
c) all units and functions in the corporation feel challenged to contribute or support innovation initiatives
d) corporate, business, and R&D plans are aligned on environmental strategies and priorities; senior management provides oversight with periodic reviews”

2. ADVANTAGES AND LIMITATIONS
Advantages and limitations are similar to other “quality transformation grids” or assessment matrices. “

3. HOW TO USE THIS METRIC
The matrix is to be used to find areas for improvement and to assess the job being done in comparison with others.

INFORMATION USE IN R&D

1. METRIC DEFINITION
This set of metrics measures the extent of use and the ways in which information technology is used within R&D.

1.1 Extent of Information Technology Use in expenditures in R&D budget
IT expenditures are a measure of the enhancement of R&D staff effectiveness by the use of Information Technology. The trust of this metric is also how to justify this IT spending to higher management and how to determine an optimum amount. The form of the metric is usually to measure the amount or ratio of the amount spent of IT hardware, software, and effort to total R&D expense.

1.2 Ways in which Information Technology is employed- Impact of IT on R&D
Four stages may be developed:

IT is only used to keep track of staff times and activities
IT is also used to provide tools for technical computing and/or hardware and software for the company’s products and services
IT also enhances the effectiveness of innovation management by substituting for human effort in R&D
IT enables the rethinking of how R&D is done and doing things that cannot be done in any other way; IT may also allow entirely new output from R&D.

1.3 An alternative to 1.2
A 9 box grid may be created:
Columns are Processes Impacted: Technical Computing; Mgt. of Technology; Integration of R&D with rest of Company
Rows are Type of IT impact: Tools to improve efficiency / effectiveness of R&D; Substitutes for old or existing ways of doing R&D; Tools that enable R&D to do things not possible before.

1.4 Extent of IT use in Managing R&D
The intent of this metric is to measure how far along an organization is at using IT in managing the innovation process. Four stages can be seen:

IT is only used by R&D groups on an individual basis
IT is integrated between the IT function and a few groups in the R&D activity
IT is fully integrated in the process of managing technology within R&D
IT is fully integrated in the process of managing technology across the firm.

1.5 Other possible metrics that might be developed
Breadth of usage by application category vs. benchmarks
Quality of usage by application category vs. benchmarks
Percent of R&D staff using IT at benchmark levels
2. ADVANTAGES AND LIMITATIONS
The advantage of this metric is that it enables the justification of IT in R&D both at financial budget time, and in benchmarking IT use and impact in R&D against competition and strategic goals.

A limitation is that these are relatively untried metrics, with minimal research support as to their effectiveness.

Information Technology is evolving and developing so rapidly that its dimensions are not fully appreciated and some of the measures may be difficult to make.

3. HOW TO USE THE METRIC
The amount of IT expenditure should be tracked over time, usually at the annual budget cycle. This can be compared to other measures of R&D effectiveness to see if the change in IT expenses has enhanced effectiveness.

The ratings of extent of impact and use should also be tracked over a time period for comparison with competition and strategic needs.

4. Options and Variations.
Service companies and companies with high IT content in their operations may wish to use alternately any of the software effectiveness metrics such as the Cpability Maturity Model developed by the Software Engineering Institute at Carnegie- Mellon.

5. REFERENCES
The most general reference is U.S. National Research Council, Information Technology in the Service Sector, National Academy Press, Washington D.C. 1994

The SEI Capability Maturity Model is reviewed and compared with Total Quality Management and other evaluation techniques: H. Saiedian and R. Kuzara, “SEI Capability Maturity Model’s Impact on Contractors,” Computer (IEEE), 28(1), Jan.,1995 , 16-26; 12 references

IDEA GENERATION AND CREATIVITY

1. METRIC DESCRIPTION
This metric uses an interval rating scale assessing the state of creativity and innovation within an organization. Four stages of development/performance can be given:

a. managerial control systems and organization discourage individual and organizational creativity
b. Ideas are encouraged and creativity is values in assessing performance of individuals, but no formal idea management tools are used. Creativity training is not generally available or promoted. Risk taking is discouraged directly or indirectly. Innovation takes place within well known and understood arenas.
c. Champions for new ideas are sought and supported. Creativity skills are taught and formal mechanisms are used to obtain new ideas from employees. Idea awards are presented to individuals and teams. Higher risk innovation for the creation of new opportunities is valued, but the technology management for existing businesses is typically a preferred route for advancement and status.
d. Innovative and entrepreneurial behavior among employees is encouraged and rewarded. Model behavior is highly visible. Funding programs are available to test new ideas outside the mainstream. The organization demands that scouting time be used to generate new approaches and ideas. Organizational status is higher for creation of new opportunity than for maintaining existing business. External sources are integrated into new idea processes. Idea banks and support systems are commonly used.

2. ADVANTAGES AND LIMITATIONS
Assessing idea generation and creativity within an organization is not straightforward and most likely other measures will need to be developed.

3. HOW TO USE THE METRIC
Members of the organization use this self assessment tool and then seek areas for improvement.

4. OPTIONS AND VARIATIONS
Organizations may also want to study the extent to which “creativity tools” are used: Brainstorming, brain writing, lateral thinking (random input, escaping from thought patterns, building on provocations), metaphoric thinking, forced visual associations, guided imagery, six hats thinking, criteria setting

5. REFERENCES
G.F. Farris and L.W. Ellis, “Managing for Change in R&D”, Research-Technology Management, 33(1), Jan.-Feb. 1990. 6.2 various works by DiBono

PEOPLE DEVELOPMENT

1. METRIC DEFINITION
1.1 This metric uses an interval rating scale for assessing the recruitment, development, evaluation, and rewarding of R&D personnel : (ref. 1)

Level 1
a) recruit only for specific openings
b) recruit only from local area
c) no training or development programs
d) rewards based on who you know; favors most senior people
e) no support to first-line supervision in evaluation process

Level 2
a) recruit locally and regionally
b) training programs unrelated to strategy needs
c) limited communication on reward system
d) no significant difference in monetary rewards for different performance levels e) no meaningful rewards other than salary increases

Level 3
a) active nationwide college recruiting
b) career development programs
c) training at all levels
d) three-year personnel plan
e) realistic appraisals
f) appraisal training
g) written evaluations, annual reviews
h) significant differences in rewards for top performers

Level 4
a) recruiting based on skill mix, competency analysis ,and long term staff development planning
b) all management levels involved in selection
c) effective dual ladders
d) recruitment and development recognize need for global technology management
e) planned balance of roles
f) incentives for entrepreneurial behavior
g) interfunctional and international career opportunities
h) mix of individual and team rewards
i) personnel development accomplishments a key factor in evaluations of managers.
1.2 Measure of the inventory of ideas in the pool

A listing is maintained of ideas coming from various sources within the firm. The organization develops some standards on number of ideas contributed in the last three months and for the year. A measure of the number of backlog ideas is also developed.

1.3 Measure of the amount of effort devoted to idea generation

An assessment is made of the typical % of time devoted to idea generation by members of the R&D staff.

2. ADVANTAGES AND LIMITATIONS
The advantages and limitations are the same for the use of any “quality transformation grid.

3. HOW TO USE THE METRIC
The grid descriptions should be tailored to the particular organization; then members of the organization make the assessment to find areas where improvements are needed.

4. References
P.S. Adler, D William McDonald, F MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37

C.J. Cranny, P. Cain-Smith, and E.F. Stone, Job Satisfaction, Lexington Books, New York, 1992

R. Katz (ed), Managing Professionals in Innovative Organizations, Harper Collins, New York, 1988

L.W. Ellis and S. Honig-Haftel, “Reward Strategies for R&D”, Research-Technology Management, 35(2) March- April, 1992 16-20.

INTELLECTUAL PROPERTY MANAGEMENT

1. METRIC DEFINITION
This metric uses a four stage interval rating scale to assess the state or stage of intellectual property management within an organization: (ref. 1)

Level 1
a) ignored

Level 2
a) rewards for patents
b) intellectural property issues left to legal

Level 3
a) selective patenting based on evaluation of pros and cons of disclosure
b) in-licensing if needed and out-licensing if asked
c) trade secrets defended in court

Level 4
a) intellectual property opportunities are part of business strategy, project selection, and project management criteria
b) in-licensing to maintain focus, speed external comparison, and learning opportunities
c) technical personnel rotate through intellectual property assignment
d) out-licensing based on business and technical assessments
e) comprehensive trade-secret policies

2. ADVANTAGES AND LIMITATIONS
The advantages and limitations are the same for the use of any “quality transformation grid.

3. HOW TO USE THE METRIC
The grid descriptions should be tailored to the particular organization; then members of the organization make the assessment to find areas where improvements are needed.

4. References
Adler, D William McDonald, F MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37

Griliches “Patent Statistics as Economic Indicators: A Survey” Journal of Economic Literature, XXVIII, December 1990, 1161-1707

Narin, “Technology Indicators in Strategic Planning”, Science and Public Policy, December 1992, 369

R&D CLIMATE

1. METRIC DEFINITION
This metric provides a way to score the overall climate in an R&D organization — especially to see the extent to which it is characterized by “Third Generation R&D”: If the index is put into four stages similar to other metrics we can outline stage 1 and stage 4 (ref. 1)

Stage 1
No explicit strategic framework for technology management
R&D a second class citizen
R&D isolated
R&D strategy is not linked to division strategy
Management is not tolerant of failure
Business planning does not include R&D
R&D funding of programs varies with earnings forecasts
Priorities are set according to near-term business conditions
R&D performance measured according to product successes
R&D has poorly defined projects
Progress is evaluated only when things go wrong
“Fires” constantly cause unplanned changes to programs
Manufacturing becomes involved only during late stages
Marketing input only at instigation of project
Funding is a function only of internal needs/requirements
Sales and Marketing planning is the province of GM and marketing managers
All projects are managed by individuals
R&D projects are evaluated by R&D on an individual basis
Projects are never terminated

Stage 4
Corporate-wide strategic framework for technology management
R&D is respected and a partnership exists between it and all other areas of the organization
R&D is fully integrated
R&D and business strategy are integrated corporate-wide
Management is tolerant of failure
Division and corporate business includes R&D participation
R&D funding varies with technological maturity and competitive impact
Priorities are set according to contribution to division or corporate strategic objectives
R&D performance is measured according to division or corporate business objectives and technological expectations
All R&D projects are well defined
Progress is evaluated regularly and whenever events warrant
“Fires” are dealt with without interfering with planned R&D programs
Manufacturing included in project planning
Marketing participates in project reviews
External competitive environment also considered in funding decisions
R&D staff have input to sales projections and marketing plans
Large complex projects are “managed” by multifunctional teams
R&D projects are evaluated as part of a portfolio taking into account risk/reward, time horizon etc.
Projects are continually weeded out

2. ADVANTAGES AND LIMITATIONS
This metric allows for a quantitative comparison between R&D organizations. Some aspects may indeed by subjective.

3. HOW TO USE THE METRIC
The scoring is probably best done by a number of different individuals within the organization.

4. OPTIONS AND VARIATIONS
This scoring system has been developed by AD Little; an assessment scheme for R&D decision making has also been developed by Strategic Decisions Group and could be used instead.

5. REFERENCES
Roussel, K.N. Saad, and T.J. Erickson,Third Generation R&D, Harvard Business School Press, 1991

Output Metrics for Assessing R&D Technology Assets

The logic for utilizing this point of view is derived from the argument that the ability to visualize and measure intellectual capital is a prerequisite for effective technology management. Intellectual capital (e.g., competencies, technologies, brands) and the ability to use it explain to a large extent the success and market value of technology companies. Most of these companies recognize that technology assets are a critical part of their intellectual capital and that their challenge is to generate more value from these assets by more efficient acquisition, protection, and utilization.

In the “General Categorization of Technology Assets” figure the general areas of measurement are numbered from 1 to 7. In all there are 31 sub-measures that backup assessment of a company’s technology assets by this model.

Looking at this model in more conventional terms the measures related to acquisition of technology are normally considered in-process metrics for R&D and in-licensing activities. The metrics associated with the different forms of technology stock are often considered output measures for R&D, partnership and licensing efforts. The metrics associated with the commercialization of technology section are often considered outcome or governance metrics. These are typically internal exploitation by selling a product with an advantaged position, licensing-out for revenues, or use of the technology assets in partnerships and other cooperative work that will bring the company future value.

The “Measuring Seven Classes of Technology Assets” figure presents metrics identified for each class of technology related asset. In the first column, each metric is represented as an absolute target of measurement. In the same column alternative ratios are suggested which can be used to measure the target against some comparison point. Absolute indicators are suitable for internal use wears indicator ratio still better job of enabling a comparison between companies.

For the areas listed, metrics in Commercialization of Technology generally received the highest priority from surveyed companies. Depending upon the company’s internal strengths and priorities, Internal Exploitation is typically ranked above the other two forms of exploitation. However, for very specific companies who grow by acquisition, cooperative exploitation also receives a high priority.

Technology acquisition was ranked the second highest priority by surveyed companies. This metric received a higher priority for production oriented companies than from product focused companies. Of the different forms of technology acquisition measuring the efficiency of Internal R&D and R&D Cooperation were considered the most important.
The remaining items on the list vary in importance depending upon the market area and technology maturity of the environment in which the company is operating.

Outcome / Governance Metrics and Inconsistent Correlations

At the highest level of abstraction lies Outcome or Governance metrics. These measure the results of R&D that shareholders can see and experience. From these stakeholders’ perspective questions like “what is the appropriate amount of money for to spend on R&D?” and “how do we know our investment in R&D is required for shareholder returns?” To see how the whole R&D portfolio is doing the following are examples of Outcome or Governance metrics:
1. innovation contribution to revenue and profits
2. innovation contribution to cost savings
3. revenue from new products developed in the last three years as a percent of total revenue
4. profits from new products developed in the last three years as a percent of total profits
5. innovation conversion rate per stage
6. new product market share
7. new segment market share
8. instant increase shelf space
9. increased share of wallet
10. increased distribution footprint
11. number of patents granted
12. number of registered trademarks
13. customer satisfaction (net promoter score, usability testing)

High-Level Innovation Success Criteria by Stage

These are individual metrics aimed at evaluating the success of innovation programs. The more generalized questions to be asked and answered are shown in the “High-Level Innovation Success Criteria by Stage” Figure. Notice that these questions are more qualitative statements than either the in-process or output metrics. Answers to these governance questions are typically done on a scale of 1 to 5. Visualizing the answers to these questions is done by adding up the total score for project, either with or without weighting the individual question answers. A better methodology however has been to use star or spider diagrams to visualize the results of a project’s governance status. Looking at the answers to the questions over a large number of projects using spider diagrams outperforms strictly numerical scales by a wide margin when it comes to an Investment Board’s making a fast high-quality and sticky business decision.

Advocates of R&D spending point out that using metrics to measure the performance of R&D is valuable. This is because the correlation between R&D performance in corporate performance is measured by stock price or shareholder value is positive at the extremes. Studies that use Standard & Poor’s compustat database of financial data often segment the top and bottom performing S&P 500 companies into two groups of top and bottom 50. They then determine the R&D investment by each group. The average return of the top 50 companies is usually over 30% versus only 2 to 4% for the bottom 50 group. Top-performing companies are considerably more research intensive, investing 8.2% of sales versus 4% for the bottom 50. Although blindly investing in R&D is no guarantee of business success, it does appear the long-term commitment R&D and stop performance do go hand-in-hand.

This positive correlation between R&D spending in corporate performance is also true of European based companies. The UK Department of trade and industry creates an R&D scorecard survey which shows strong evidence for positive correlation between R&D intensity and sales growth. In the past they found the turnover expands six times faster in companies with higher than average percentage of sales from new products stunning competitors with a lower than average for Sen. of sales from new products. It is also argue the comparisons between industrial sectors also shows a correlation between R&D intensity and sales growth. The pharmaceuticals, aerospace, IT hardware and software industries spend more on R&D and are expanding faster than the chemicals, engineering and food industries. Additionally when business downturns occur companies that have cut R&D investment have often found that their products and services comparable less well against competitors went up terms come in they find it more difficult to protect market share and value-added.

Critics of R&D spending point out that two thirds of the top 20 R&D spenders have less than average price to earnings ratios. If one looks more closely at R&D intensity versus outright investment in R&D, it’s also true that one half of the top R&D to sales spenders have less than average P/E ratios. Because of these returns critics can forcefully argue that some of the R&D spending should be redirected toward customer centric innovation versus traditional R&D. This criticism has impart brought about the transition from the fundamental stage-gate managed R&D to more agile or lean innovation systems.

Furthering the critics viewpoint are studies which show the relationship between stockholder value and R&D spending have mixed results for the computer / high tech industries. On the one hand, they identify a positive, short-term relationship between stockholder share price movements and firm announcements of plans to increase R&D expenditures, and show that industries with higher R&D intensities grow faster than those with low R&D intensities. However, it is also observed in the computer industry that R&D intensity has a statistically significant negative relationship to future stockholder returns over both one and five year time periods. Such work suggests that computer firms are overspending on R&D at the expense of their stockholders.

To see the effect of various industries, and differing R&D Game Types, the “Share Price / Payoff” figure shows the potential change in share price corresponding to a one unit change in investor perceptions of nonfinancial performance; the scale is from 0 to 10 units, which is less than half a standard deviation, a moderate change. Note that only in the pharmaceutical industry, a Safety Journey R&D game type, is investment in new product development a high payoff investment.

Visual Display of Metrics for Projects’ Progress and R&D Dashboard

Sometimes a picture is worth 1000 words. Pictures can also convey a lot more emotion, eliciting accolades or empathy.

As shown in the three figures, “Visual Display of Technical Progress”, “Visual Display of Market Progress”, and “Visual Display of Fun”, graphically displaying how projects are doing and how the people on the projects are feeling can be concisely communicated. Oftentimes in R&D organizations where motivation is important for project success, pictorial views of the projects are an improved communication tool over monthly reports.

Studies have shown that it is important to show not just technical progress but also whether or not progress is being made in the marketplace. Both have to be happening for commercialization success.

The other metric that studies have shown important is to acknowledge the natural cycle of excitement and despair that goes with any project. The cycle goes from initial elation at the start of a project, then into the pit of despair, and finally out the other side as a project goes commercial.

Another way to more quantitatively graph a project’s progress is to plot specific elements of sales and marketing success on a vertical scale in specific elements of technology manufacturing success on the horizontal scale. These milestones come from the hurdles present in Stage-Gate or Agile/Lean Project Review questionnaires. An example is shown in the “Project’s Progress” figure. Ideally a project goes up the 45° line but often either marketing or technical progress goes faster slower than expected. This visual display of information shows a manner in which projects are succeeding are struggling. It is especially useful when used to create a dashboard of such images for all projects in an R&D or innovation organization’s portfolio.

At the organizational level the “R&D Organization Dashboard” figure shows a sketch of a dashboard used to track the performance of a Fortune 500 Corporate R&D laboratory. The indicators across the display show in pictorial form anchored scales that the executive team and CTO have previously agreed upon. The advantages such a display are that the dozen metrics that the executive team finds important in their allocation of operating and capital budgets to the R&D group are displayed in one spot. As a corporation’s environment changes over quarters and years, the executive team can easily see if the corporate R&D organization is being managed in a way that matches the environmental changes. From a presentation standpoint it was easier for the CTO to have one graphic display that could be used to ground the executive team at the start and close of any presentation.

The key to understanding information is to display information in a comprehensible form. A checklist of possible display options is captured in the “Periodic Table of Visualization Methods” figure. Understanding what an audience can comprehend most quickly and what image will remain in their mind over time (sticky) should drive selection of the most appropriate display.

Appendix 1: Competitive Intelligence Metrics

In order to excel as a technical organization it’s imperative to have good business competitive intelligence to understand both the business and technology environments. Without this understanding the best projects for commercialization will not be picked and project teams cannot be resourced appropriately to deal with competitive efforts.

The “Cultural Adaptation Model Dimensions” figure provides anchored scales from levels 1 to 5 for assessing competitive intelligence efforts. There are five categories in which competitive intelligence is assessed. These are use, awareness, needs, collection and dissemination, and champions. Organizations operating at level III or below are strongly disadvantaged in their innovation efforts.

The “Business Competitive Intelligence Questionnaire” below provides almost 100 detailed Yes/No questions for assessing competitive intelligence efforts in a comprehensive manner. Companies with strong competitive intelligence programs answer yes to all these questions. These questions can be used both as an overall scorecard metric (output metric) and from an “In-Process” perspective, used to identify gaps in performance that need to be addressed.

Intelligence practices currently in place in your organization. Answer Yes (1) or No (0). Sum all answers to get a company score:

Our Company regularly prepares profiles of our competitors.
We use secondary sources of information (public literature, reports. etc. to learn about key competitors.
Our company has developed legal and ethical guidelines for the conduct of CI activities.
Our company is concerned about the companies with which we directly and indirectly compete.
Our company produces intelligence reports and assessments on the competitors and/or emerging technologies that we believe are most important.
Our company continuously and systematically monitors our technologies globally to determine whether new competitors or technology substitutes are emerging.
We monitor and assess the activities and plans of organizations and groups (such as regulatory agencies or NGOs) whose view of our company could affect us.
Our company focuses its intelligence efforts on competitors that management has identified as important.
Our company produces assessments that address several possible outcomes of our competitors’ actions and that identify the threats and opportunities those outcomes present for our company, new products, etc.
Our employees regularly report information about our competitors to appropriate managers.
Our Company maintains a network of human contacts outside the company that we call on to answer senior management’s questions in a timely and credible fashion.
Our company analyzes our competitors’ plans and strategies to predict and anticipate their actions.
We use formal competitor analytical models such as SWOT and resource gap analysis.
We use formal psychological models such as competitor management profiling.
Our employees understand clearly that our proprietary information and intellectual property should not be disclosed and what to do if they become aware of potential inappropriate disclosure or access.
Our company recognizes CI as a legitimate and necessary activity for business.
Our Company collects and uses patent and scientific literature to assess R&D programs and/or emerging technologies.
Senior company management supports intelligence activities.
Our CI activities include counter-intelligence aimed at assessing the success of CI efforts directed against us.
Our company has incentives to encourage employees to report their competitive observations and information.
We proactively communicate the company’s intelligence needs to employees.
We have convenient ways for employees to report observations & information.
We interview our executives regularly to understand their intelligence requirements.
Senior managers use CI regularly in their planning and decision making
In our communications activities (media relations; marketing. advertising; etc.) we consider the potential CI benefits that competitors could gain from our disclosures
Senior executives’ stated intelligence requirements are used to focus our intelligence efforts and resources.
Our company uses advanced analytical techniques (e.g., on-line data screening, photography/imaging of competitor technology) to analyze our competitors’ and assess their future business implications
Our company develops profiles of emerging technologies to better understand their characteristics, potential applications and market advantages
Our company has a variety of methods for collecting current intel, such as organized methods to exploit conferences
Our staff distributes intelligence findings only to those who are authorized to see them
We use information management techniques, such as data-mining, data-warehousing, OLAP or “business intelligence” software, to understand our customers.
Company personnel (e.g. scientists, marketing staff, etc.) are our most important source of information.
Our contacts outside the organization are our most important source of information.
We use the corporate Intranet as a means of storing and accessing competitive information that our employees need in their day-to-day work.
Secondary information sources (e.g. publications. web pages) are our most important sources for information.
Our staff distributes intelligence findings throughout the company to anyone who is interested in them.
Our corporate Intranet is specifically designed to facilitate and support our CI activities.
We recognize the potential CI value of information held within our company.
We maintain a comprehensive map or inventory of internal information and knowledge.
Key decision-makers are regularly surveyed / interviewed to verify that the intelligence products produced for them, satisfy their needs and provide value
Kev decision-makers are interviewed on a regular basis to determine best methods to deliver CI findings to them
Competitive intelligence is used primarily in sales/marketing
Our company has developed guidelines for employees on what type of information should not be disclosed
As long as the collection method is legal. we will use it
All information collected is checked for accuracy and validated by at least one other source
Our company asks all employees what their view is of industry trends
We coach our employees every time they go to trade shows, exhibitions, conventions, and so forth about what type of information they should look for
We know the mindset of the CEO’s and other key executives of our top customers – how they view the industry, the degree of risk they are willing to take, the priority of their business goals. etc.
Our company lets all new employees know what information to look for
Every employee is given counterintelligence training
Results from exit interviews and job interviews are used in our intelligence system (information is shared)
We regularly ask our employees, as to what they view as future opportunities for our company
There is a central coordination point for receiving competitive intelligence information
We make intelligence training available to all our employees
Our employees are generally aware of any legal and ethical guidelines for the conduct of CI
Our employees have been told what information is considered confidential/sensitive
Our orientation session for new employees includes a briefing on CI policies
We are concerned to understand the plans and intentions of not only our key competitors but also of key allies and partners. such as suppliers, distributors, investors and collaborators.
We coach our employees when they go to trade shows conventions and so forth about what they should not talk about
The results from our intelligence process influence our corporate strategy and direction
Employees understand that sharing information is important to the success of the company.
We record centrally/monitor all requests from people outside our organization for information
Our company produces forecasts of key government policy changes that can affect our industry
For a typical intelligence project we spend at least 50% of our time collecting information
Our company provides feedback to employees on how the information they provided was used
We have a standard template for the presentation of intelligence findings
We regularly poll our employees, as to what they view as future threats to our company’s health
We have dedicated staff and resources for the organization of competitive intelligence information
Our web site has been examined to ensure it does not reveal information that would compromise us
Most employees understand exactly what intelligence is
We evaluate our intelligence results
We believe that competitive intelligence can be used to create a competitive advantage
We know the mindset of the CEO’s and other key executives of our top competitors – how they view the industry, the degree of risk they are willing to take, the priority of their business goals, etc.
We try to collect all available information on our competitors
Our intelligence staff regularly takes intelligence seminars/training programs
We have a formal knowledge management system
Our corporate culture encourages information sharing
There is a secure storage and retrieval system for competitive intelligence information gathered
Our company has a policy that identifies the security level of information
Our company maintains a central record of all known reliable sources of information
We know our competitors costs, sales and margins
Competitive intelligence is used primarily in sales/marketing
We conduct intelligence projects only when asked to – we are demand driven
We conduct intelligence project regardless of whether we have been asked to do it
Our employees have received formal training on how to collect information (searching or interviewing course)
Our company regularly scans help wanted ads to detect any possible hiring trends by our competitors
Our employees are aware of the competitive intelligence methods used by our competitors
We have a long range competitive intelligence plan
We have a competitive intelligence unit with at least one full time resource
Our competitive intelligence unit reports directly to the President or a senior Vice President
Intelligence projects start out with a hypothesis (an idea of what management thinks we will find)
After collecting information whether it is from a person or from a documented source (e.g. the internet) we make a note about the quality/value of the source
We get more than 50% of our information from sources like the internet (this includes newspapers, libraries, databases, consultants reports, government reports. etc.)
We use time-lining techniques to forecast when a competitor will be entering our market
We have conducted an internal knowledge audit
We conduct internal knowledge audits on a regular basis (at least once every two years)
We have more than five intelligence objectives (things that we really want to know)
Before doing a telephone or in person interview our employees spend more time preparing for the interview than they actually spend on the phone/in person

Appendix 2: Intellectual Property Metrics

Metrics related to intellectual property have been covered lightly in some of the previous chapters on intellectual property. This section specifically addresses intellectual property metrics is related to R&D or business performance. Key high-level intellectual property metrics relating to R&D performance are:

R&D $ / Patent
% Patents Utilized (by category of Existing Products, future products, licensing, or no business use).
Key and Emerging Technologies With IP Dominance
Key and Emerging Technologies with IP Subservience

To determine whether a company’s patent strategy matches its business and R&D strategies, the performance of the company’s patent assets can be assessed using the following evaluation indicators:

Number of reported inventions / R&D employee
Number of reported inventors / number of reported inventions
Average age of patents
Total number of patents
Total number of patented inventions
Percentage of patented inventions commercially used by the company
Percentage of patented inventions involved in disputes
Percentage of total sales protected by patents
Percentage of new to market products and services protected by patents
Sales protected by patents / R&D expenditure

There are also many ways to get a single indicator of the value of a Corporation’s overall patent portfolio. One common patent strength indicator used in some commercial software programs has the following factors, and their weighting, on a five-point scale.

Number of patents (5.0)
Number of patents with large number of forward citations (4.0)
Average age (2.5)
Average number of claims per patent (2.0)
Average number of pages of specification and drawings (1.5)
Number of patents litigated (4.0)
Number of patents reissued or re-examined (3.0)
Number of patents licensed (4.0)

A more complete set of indicators that show the value of a corporation’s overall patent portfolio was provided by the Patent Scorecard. This scorecard was organized at a company level, including all US patents held by each company across multiple industry sectors. This commercially available scorecard allowed comparison of company positions in various industry segments. The attributes measured were:

Technology strength. Technology Strength is the basis of the rankings and provides an overall assessment of a company’s intellectual property and innovation strength.
Current Impact Index. The Current Impact Index showcases the broader significance of a company’s patents by examining how often its US patents are used as the basis for other innovation in the current year
Science Linkage. Science Linkage reflects the core science referenced in a company’s US patents. A high figure indicates a company is closer to the cutting age than its competitors with lower values
Technology Cycle Time. The Technology Cycle Time indicates a firm’s speed in turning proprietary research and innovation into intellectual property.
Patent Count. The Patent Count equals the number of US patents awarded, excluding design and other special case inventions.

For corporations who derive a significant portion of their revenue and profits from patent related activities, the metrics used to evaluate performance or dictated by the licensing objectives.

TO GENERATE ROYALTY INCOME:
Estimated EVA of the licensing function.
Annual royalty income and number of transactions.
Royalty income as a percentage of R&D expenditures.
Ratio royalty income to IP protection costs.
Ratio royalty income to patents licensed.
TO PRIME NEW MARKETS:
Estimated EVA.
Number of new market center through licensing out.
DEVELOP INDUSTRY STANDARDS:
Estimated EVA.
Number of technologies licensed by at least X percent of the market.
TRADITIONAL PROTECTION ACTIVITIES:
Number of patents available through cross-licenses and licensing-in.
>Ratio of issued patents to annual R&D expense
Ratio of issued patents to total sales
Aging of the patent portfolio
FREEDOM TO OPERATE:
Number of patents relative to competitors.(Overall, annually, by major product line)
Number of lawsuits in which you been sued for infringement by one or more of your competitors
Number of patent secured through cross licenses
Number of lawsuits in which you are the defendant relative to the number of lawsuits in which your competitors are the defendant
ASSESSING THE IP PORTFOLIO:
Number of invention disclosure submitted
Number of patent applications filed.
Number of patents received annually total portfolio size (patents and trademarks)
Ratio of patents granted to applications filed.
Average prosecution costs per patent granted.
Annual maintenance fees
Cycle times for various activities (i.e., invention disclosure to patent filing

Board of Director Metrics: Although the form of questions, there are metrics which a company’s Board can assess the value of IP to shareholders. These high-level questions Board Directors should be asking about Intellectual Property follow. Each metric has proposed anchored scales for ease in evaluation.

GROUP A: OVERALL POSITIONING
this first set of questions is designed to develop an overall high-level sense of the role that IP plays in your industry, the company’s relative position with respect to its IP portfolio, and it’s IP management level of sophistication.
Question 1: How sophisticated is our industry with respect to invention and innovation?
This question relates to the industry your enterprise competes in. Where are the majority of competitors with respect to their level of sophistication with respect to invention and innovation?
A. Most companies use technology invented by others: competitiveness depends upon adopting and using the right technology, not inventing or adapting it. Examples are banking, professional services, and retailing.
B. Invention and innovation takes place at low to moderate level in our industry: most competitors have some technology adaptation capability, but it is possible to be successful without it. Examples are commodity manufacturing.
C. All competitors have a basic innovation capability: it is a critical success factor in our industry, and it is impossible to be competitive without it. Examples are auto-parts and other high-end manufacturing.
D. Competitive position in our industry is directly related to the sophistication of innovation and invention capability. Examples are chemicals, IT, communications.
Question 2. How does our invention and innovation capability compare to our competitors?
How is the enterprise positioned against their competitors with respect to invention and innovation capability? Or would you locate your enterprise along the following continuum:
A. Our invention and innovation capability on balance legs compared to most of our competitors.
B. Our invention innovation capabilities are generally on a par with the majority of our competitors.
C. Our invention and innovation capabilities are superior to the majority of our competitors.
D. We are considered among the top companies in our industry with respect or invention and innovation capability.
Question 3. How large are IP portfolios held by the leaders in our industry?
This question focuses on the average size of IP portfolios held by the leading companies within an industry. Is measured by reference to IP families. The relevant number for IP portfolio management purposes is the number of discrete patents, trademarks, or other items which may be filed and registered in multiple jurisdictions. Of course there are many characteristics of an IP portfolio that are important, not the least of all being the quality of the IP in the portfolio. But as an additional indirect measure of importance of IP in an industry, it is useful to consider the size of the IP portfolio held by leading companies in an industry as follows:
A. Small to none: average portfolio is less than 50 IP families.
B. Moderate: average IP portfolio is less than 500 IP families.
C. Significant: average IP portfolio is less than 5000 IP families.
D. Large: average IP portfolio was greater than 5000 IP families.
Question 4. How does our IP portfolio size compare with our competitors?
Where would you locate your enterprise along the following continuum?
A. Our IP portfolio is smaller than those of most of our competitors.
B. Our IP portfolio is about the same size as most of our competitors.
C. Our IP portfolio is larger than that held by most of our competitors.
D. Our IP portfolio is acknowledged to be the largest in our industry.
Question 5. How sophisticated is our industry with respect to IP management practices?
This question focuses on the level sophistication of our industry with respect to IP management practices. While this can be asked as a series of complex questions, as a form of shorthand in this context, it is suggested that a level of sophistication with respect to IP management practices can be determined by distinguishing among managing IP as a legal, business, or strategic asset. Given this context how would you locate your industry among the following continuum?
A. Most competitors are not active in acquiring or managing IP
B. Most competitors are focused on building up an IP portfolio and/or focused on managing IP is illegal asset.
C. Most competitors are focused on managing IP is a business asset.
D. Most competitors are focused on managing IP is a strategic asset.
For definitions: (I) companies that manage IP is illegal asset can have IP attorneys in charge of the portfolio, tend not to have significant revenues from IP or technology licensing, and tend to file for new and renew IP is a matter of course. (II) companies that manage IP as a business asset tend to have created an IP management function at the corporate and/or business unit level, are selective about what IPRs are created and renewed, tend to have an active program for creating revenues from IP licensing, actively exploit their IP rights. (III) companies that manage IP at a strategic asset are good at managing IP as a business asset, and take active steps to protect the company’s ability to innovate in the future, use IPRs to create competitive advantage through a variety of approaches, such as selective litigation, using IP to protect standards, and use IP rights to channel industry activities along desirable paths
Question 6. How does our IPM level of sophistication compare to our competitors?
Where would you locate your enterprise along the following continuum?
A. Our IPM practices in general are weaker and lag those of most of our competitors.
B. Our IPM practices are on a par with most of our competitors.
C. Our IPM practices on average are more sophisticated than most of our competitors.
D. Our IPM practices are acknowledged to be the leading edge for our industry.

GROUP B: ASSESSING IP MANAGEMENT STRATEGIC ALIGNMENT AND POSITIONING
Question 7. Importance of IP and IP management to our business strategy
What is your judgment about the role that IP and IP management have in the company’s business strategy? Where would you locate your enterprise along the following continuum?
A. IP is not an important factor in our business strategy, and our level of IP management capabilities is not a competitive factor for us.
B. IP is of moderate importance compared to other competitive factors, and maintaining a competitive level of IP management capability is a factor in our strategy.
C. IP is an important factor for us and IP management is a significant factor in our competitive positioning.
D. IP plays an important and essential role in our strategy, and the level of our IP management capabilities is one of the most important critical success factors for us.
Question 8. How are IP and IP management positioned in our latest business strategy document?
One of the clues to the perceived importance of IP in IP management with respect to the company strategy is the amount of detail devoted to it in the latest strategy document. How would you locate your enterprise along the following continuum?
A. IP and IP management is not explicitly mentioned in our latest business strategy document.
B. IP and IP management is mentioned in passing in our latest business strategy document, but not a lot of attention is given to it.
C. Our business strategy document explicitly describes in detail the role of IP in IP management in our strategy.
D. Our latest business strategy document includes a detailed analysis of our IP portfolio and IP management capabilities, and includes specific targets, actions, timelines, and responsibilities for future enhancements.
Question 9. Do you have an IP function and if so what is its role.
How would you locate your enterprise along the following continuum?
A. We do not have a defined or formalized function with the responsibility for managing intellectual property, knowledge, or intellectual capital.
B. Our IP function is largely perceived as an administrative function that focuses on securing and maintaining protection of our intellectual property rights arising from current inventions.
C. Our IP function collaborates actively with business managers to support business unit strategy and realize optimum value from IP rights, as well as securing and maintaining those rights.
D. Our IP function makes a major ongoing role to strategy formulation, and is highly proactive in securing IP rights that are highly significant to the future evolution of our key technologies and lay ground work for future inventions.
Question 10. Speed of adoption of IP management best practices
Another way to get a fix on the positioning of IP management function is its approach to adopting IPM best practices as they emerge over time. Or would you locate your enterprise along the following continuum?
A. We adopt new IP management practices after they have diffused widely in our industry.
B. We monitor the IP management practices of the leaders in our industry, and adopt similar practices once a clear direction has emerged.
C. We monitor IP management developments broadly, and are early adopters of new approaches that help enhance our capabilities.
D. We are recognized as an organization that creates new practices that are eventually adopted by others as best practices.

GROUP C: ASSESSING OUR IP MANAGEMENT PROCESSES AND CAPABILITIES IN GREATER DETAIL
Question 11: IP generation and protection
IP generation and protection includes the processes and capabilities that are involved in identifying innovations that should be protected, and deciding on the best method of protection. Where would you locate your enterprise along the following continuum?
A. Our processes and practices for managing and screening disclosures, and making decisions about patenting or other protection approaches are largely rudimentary three or nonexistent.
B. We have reasonable processes and practices for managing and screening disclosures, and making decisions about patenting or other protection approaches, and they come into play late in the innovation process.
C. Our processes and practices for managing and screening disclosures, and making decisions about patenting or other protection approaches are employed early in the innovation process, and are supported by advanced IT enabled analytical tools.
D. Our IT enabled processes and practices for managing and screening disclosures, and making decisions about patenting or other protection approaches are extensively automated in ways that are regarded as leading-edge within our industry and set the pattern for others to follow.
Question 12. IP portfolio management
IP portfolio management includes processes and capabilities that are involved in decisions about what IP rights to renew, monitoring the performance of the portfolio, and tracking its evolution. Where would you locate your enterprise along the following continuum?
A. Our portfolio management practices and capabilities are less developed than the leaders in our industry.
B. Our IP database focuses on patent management and helps is managed patent renewal costs and identify licensing opportunities.
C. Our IP database is comprehensive and is supported by advanced decision support tools that help us to decide when, what, and how to protect all forms of IP.
D. Our analytical and performance measurement systems enable us to demonstrate the systematic evolution of our IP portfolio, which is been demonstrated to be an important source of competitive advantage.
Question 13. IP value extraction commercialization
IP value extraction commercialization includes the processes and capabilities that are involved in extracting value from the portfolio. Where would you locate your enterprise along the following continuum?
A. We do not extract significant value from our IP portfolio except through protection afforded to strategically important technologies.
B. Value extracted from the IP portfolio is mainly through patent licensing, and generates Limited, incremental revenues.
C. We use various advanced analytic and decision support tools to identify value extraction opportunities, and generate results which are significant to the Corporation’s overall performance..
D. Our value extraction capabilities and systems are recognized as a critical competitive advantage for the enterprise.
Question 14. Strategic leverage from IPM
Strategic leverage from IPM focuses on the processes and capabilities that can help build a strategic advantage for the enterprise. Where would you locate your enterprise along the following continuum?
A. We have limited competitive intelligence about IP portfolios, strategies, and capabilities of the leading companies in the sectors within which we operate.
B We periodically undertake initiatives designed to enhance our knowledge of competitors IP portfolios, strategies, and capabilities.
C. Our advanced and IT enabled systems provide us with the intelligence needed to undertake strategic patenting, strategic disclosure, and take other types of preemptive actions designed to enhance our competitive position.
D. Our advanced IT enabled systems provide us with the intelligence needed to lead standards and other types of collective initiatives with the potential transformer industry.
Question 15 IP management performance measurement
IP management performance measurement focuses on the processes and capabilities attract the overall performance of the IP management function on an ongoing basis. Where would you locate your enterprise along the following continuum?
A. We have no specific processes to track the performance of the IP management function.
B. The IP management function is tracked using operational metrics to track patent filings and related activities and standard financial metrics.
C. We have detailed drill down capabilities for IP management that allows us to link goals of individual organizational units to the broader performance targets for the enterprise as a whole.
D. We use a variety of specialized analytics that allow us to comprehensively track value creation from IP and IP management and that link performance to the overall strategic performance objectives of the enterprise.

NEW FRONTIERS
The questions in this section explore the company’s stance with respect to leading-edge practices and new frontiers in and beyond IP management.
Question 16: Intellectual Capital Management
This question explores whether we are only focused on IP management, or have the adopted practices for management of other types of intellectual capital. Where would you locate your enterprise along the following continuum?
A. We do not actively manage other forms of intellectual capital other than IP.
B. We recently have begun developing some specific processes for managing other forms of intellectual capital.
C. We have well-developed processes for managing specific types of intellectual capital beyond IP.
D. Managing intellectual capital is virtually identical with managing the enterprise. Our key processes take IC into account, and all employees are part of the IC management process.
Question 17: New Business Models
Where would you locate your enterprise along the following continuum?
A. Our business model is exclusively traditional, and we expect it to stay that way.
B. Our business model is primarily traditional, but we earn significant revenues from IC related activities.
C. Almost all business units in our enterprise are traditional; we are actively developing some which are primarily earning revenues from IC activities.
D. We can foresee a time when the majority of our revenues are derived from intellectual capital related activities.
Question 18:
All businesses participate in a value chain or network. Increasingly, leading businesses are recognized for their role for managing IC beyond the boundaries of the organization, in relation to other participants in the value chain or network. Where would you locate your enterprise along the following continuum?
A. Our intellectual property and intellectual capital management activities take place entirely within the boundaries of our organization.
B. We actively manage ongoing relationships with universities and research organizations that have knowledge and/or technologies that can support our innovation activities.
C. We actively participate in upstream and/or downstream co-creation activities and have processes in place to manage the intellectual property and intellectual capital issues across organizational boundaries.
D. Our business strategy includes active management of intellectual property and intellectual capital processes throughout the value network within which we operate.

Appendix 3: Business Level Metrics

There’s a great deal that is been written on metrics for business. The intent of this section is not to review such material. Instead this section has some unique collections of metrics it provides insight into managing innovation and intellectual property.

In the “Business Logistics Model” figure performance indicators for various industries are shown. Of interest from a technology management standpoint is the mention of “suggestions per employee” in the Assembling Cell B4. Otherwise new product development is pretty much absent from the chart. Thus for most industries, short-term performance indicators are not driven by innovation. For intellectual property Duplicating Cell F2 about “brand awareness”, but again there is no mention of patents being a primary driver of business performance. Again this underscores that the lack of innovation or proper intellectual property protection is only a cause for very poor business performance, but the presence of innovation and intellectual property is rarely a key performance indicator on the plus side.

Somewhat in contrast to the above view, the “Business Valuation Multipliers” figure shows at that higher multipliers are affected by intellectual property positions. The highest valuations are usually retailers and manufacturers who get the maximum benefit from significant capital investment, being in a stable industry, large growth opportunities, and intellectual property.

In the “Patent Effectiveness Scores” figure the benefit manufacturers get from specifically patents to prevent duplication by others is shown in the first column labeled “Yale”. The range of the scale is from 1 to 7. In the second column labeled CMU, the percentage of product innovations for which patents are considered effective in protecting competitive advantage is listed.

Contrasting these three various viewpoints it is clear that intellectual property and innovation are important attributes of the successful businesses, but their contribution is mostly in preventing surprises and loss of current business position. They are not key performance indicators of executive short-term performance of most interest to stockholders.

Appendix 4: Selected Benchmarking of R&D Metrics

The following benchmarking data is provided to show the variability in metrics that exist between R&D organizations in different business areas, and with differing strategies to be labeled as a “high-performance brand” versus a “value brand”.

Input Metrics Benchmarks:

Type of R&D / Innovation Funding Distributions

R&D / Innovation Funding Sources / Allocations

Sustainability Goals Are Set for R&D / Innovation Projects

Accuracy of Sales Forecasts Used for Selecting R&D / Innovation Projects

Sources of Difference Between Forecast and Actual Final Sales Used for Selecting R&D / Innovation Projects

In-Process Metrics Benchmarks:

Use of Employee Time Tracking on Specific Activities

R&D / Innovation Output Metrics Benchmarks:

Duration of an Average R&D/Innovation Project

Metrics Used to Track Past R&D / Innovation Performance

Metrics Used to Forecast Future R&D / Innovation Performance

R&D / Innovation Business Outcomes Metrics Benchmarks:

Expected ROI from Innovation Investments

Expected Top Line Revenue Growth from Innovation Investments

Starting Time Used for Tracking Percentage of New Product Sales

Reference and Selected Bibliographic Information

“Wissen ist Zukunft”, by Jutta Rump and Gaby Wilms, Ministerium fur Wirtschaft, Verkehr, Landwirtschaft und Weinbau, Dezember 2005.
“New Metrics for a New Age” by Michale S. Malone, Forbes ASAP, April 7, 1997
“Reengineering the Corporation: a Manifesto for Business Revolution”, by Michael Hammer and James Champy, Harper Collins Publishers, 1993.
“How to Select Successful R&D Projects”, by D. Bruce Merrifield, Management Review, December 1979.
“Japanese Corporate Ventures: Success Curve,” by Takeru Ohe, Shuji Honjo, and D. Bruce Merrifield, Journal of Business Venturing 7, 1992.
“Winning Businesses in Product Development: The Critical Success Factors”, by R.G. Cooper and E.J. Kleinschmidt.
“A anchored scales for determining technical and commercial probabilities of success” by the IRI Research on Research Committee’s Subcommittee on R&D Portfolio Evaluation, committee report, November 1998.
“Value Creation Through Value Innovation” by the IRI Research on Research Committee’s Subcommittee on Valuation, committee report, March 2001.
“Identify, Measure, Visualize your Technology Assets”, by Pekka Utunen, Research Technology Management, June 2003.
“A Balanced Scorecard framework for R&D”, by Teresa Garcı ́a-Valderrama, Eva Mulero-Mendigorri and Daniel Revuelta-Bordoy, European Journal of Innovation Management, April 2008.
“The 2017 EU Industrial R&D Investment Scoreboard”, by European Commission – Joint Research Centre, Publications Office of the European Union, 2017.
“Metrics That Matter to R&D”, by Wayne Mackey, Workshop by The Management Roundtable, 2008.
“Commonly Used Metrics and Indicators”, by the Knowledge Roundtable Group, White Paper by The Management Roundtable, 2006
“Indicators of Technology Excellence”, by CHI Research Inc., Manual, 1994.
“Cultural Adoption Model”, Michelle Settecase, Competitive Intelligence Magazine, October 2003.
“Total Shareholder Return Calculation”, by Bill James, White Paper from Procter & Gamble, 1999.
“Decreasing Returns to Shareholders from R&D Spending in the Computer Industry”, by Halvard Nystrom, Engineering Management Journal, Sept. 2001.
“Turning Technology R&D into Market Cap Growth”, by Ian McMillan, Presentation to Industrial Research Institute Annual Meeting, May 2006.
“Survey – R&D Scorecard”, by Clive Cookson, Article by Clive Cookson, Sept 2001.
“Non-Financial Factors Affecting Share Price”, Figure by Ernst & Young, Strategy & Leadership, April 1998.
“FASB141 and 142”, White Paper by Deloitte & Touche, 2002.
“Developing a Model for Measuring the Impact of Intellectual Capital”, by Presentation by ICM Group, 1997.
“20 Questions Directors Should Ask About IP Strategy”, by Chris Bart, The Canadian Institute of Chartered Accountants, 2003.
“Solving the Measurement Puzzle”, by Marcos Ampuero, Jesse Goranson, and Jason Scott, unknown journal, circa 2000.
“Using EVA to Measure Performance and Assess Strategy”, by Al Erhbar, Strategy & Leadership, June 1999.
“Intangible Assets”, by Baruch Lev, Brooking Institute Press, June 2001.
“New Measures for the New Economy”, by Charles Leadbeater, A discussion paper for the Institute of Chartered Accountants in England and Wales, 2005.
“A Value Network Approach for Modeling and Measuring Intangibles” by Verna Allee, White Paper, Nov. 2002.
“Relating IP to R&D Performance”, Presentation by PricewaterhouseCoopers, 1999.
“Building Shareholder Value Through Effective Patent Asset Management “, by Lex VanWijk, lesNouvelles, Dec. 2005.
“Licensing”, Presentation by Fairfield Resources, 1998.
“The patent scorecard“, white paper and data base provided by IPIQ Global, 2015
“Metrics”, Presentation by the IPC Group, 1998
“IP Roadmap to Value”, by Don Drinkwater, lesNouvelles, March 2003.
“20 Questions Directors should ask about intellectual property”, Draft for Discussion by Canadian Institute for Chartered Accountants, 2008.
“How do you account for R&D spend?”, by IRI Community Forum, Innovation Research Interchange, Nov. 2018.
“How are you improving your team’s technical capability?”, by IRI Community Forum, Innovation Research Interchange, Mar. 2019.
“How do you allocate R&D costs?”, by IRI Community Forum, Innovation Research Interchange, Feb. 2018.
“Tell us about your sustainability metrics”, by IRI Community Forum, Innovation Research Interchange, Mar. 2019.
“What metrics are used for your innovation process?”, by IRI Community Forum, Innovation Research Interchange, Dec. 2017.
“How are new product sales calculated?”, by IRI Community Forum, Innovation Research Interchange, Mar. 2018.
“How do you track R&D project execution performance?”, by IRI Community Forum, Innovation Research Interchange, Aug. 2017.
“Real Numbers: Management Accounting In a Lean Organization”, by Jean Cunningham and Orest Fiume, Managing Times Press, 2003.
“Competing Against Time”, by George Stalk and Thomas Hout, The Free Press, 1990.
“Vital Signs”, by Steven Hronec, Arthur Anderson and Company, 1993.
“The metrics of Science and Technology”, by Eliezer Geisler, Quorum Books, 2000.
“Updating IP Reporting Practices”, by Patrick Sullivan and Rupert Mayer, Intellectual Asset Management, Oct. 2011.
“Are the Wrong Metrics Driving Your Strategy?”, by Michael Sisk, Harvard Management Update, Mar. 2007.
“Designing a Best-In-Class Innovation Scorecard”, by Amanda Eager, Research Technology Management, Feb. 2010.