Measuring the Performance of Knowledge Management Projects based on Balanced Scorecard in an Engineering and Manufacturing Environment

Measuring the Performance of Knowledge Management Projects based on Balanced Scorecard in an Engineering and Manufacturing Environment

Mbassegué, Patrick Gérard

École Polytechnique de Montréal, Montréal, Québec, Canada

Gardoni, Mickael

École de Technologie Supérieure (ETS), Montréal, Québec, Canada

Abstract – In any organization, knowledge production and its enhancement now represents one of the pillars of value creation. But to get there, organizations must learn to manage their knowledge and especially to measure it. The challenges are important because knowledge is difficult to measure, given the fact that it is intangible by nature and largely resides within the workforce. This paper highlights the issues and challenges regarding the evaluation of knowledge management initiatives in an organizational context, and explores the adaptation of the balanced scorecard model through two distinct case studies. This exploratory study shows that it is possible to measure organizational knowledge, but it remains difficult to identify all of its benefits.

INTRODUCTION

During the past 25 years, knowledge management has been hailed as a key solution to build a competitive advantage and enhance business performance. Engineering management helps organization to deal with continuous improvement, value creation and the optimization of processes. Engineering management and Knowledge Management (KM) both contribute to organisational performance.

Various benefits for an organization are attached to KM projects at different levels: individual, group, financial, quality, productivity, etc. (Forstenlechner and al., 2009 and 2007; Lettice and al., 2006; Marr and al. 2004; Carrion and al. 2004; Davenport and Probst, 2001; Alvesson, 2000). For managers, this interest is supported by various examples carried out by a certain number of organizations with convincing results concerning improvement of the productivity, customer satisfaction and product and service innovation (Scarso and al., 2011; Hagi, 2004). Many organizations have carried out KM initiatives, but not so many can claim or identify specific results. In spite of these proclaimed benefits and advantages, it is important to establish the nature and value of the impacts of these KM initiatives in various organisational contexts. Many researchers and managers are therefore seeking methods, metrics and frameworks to assess properly the impacts of KM projects in organizations.

Difficulties in assessing Knowledge Management initiatives KM projects are structured around 3 types of resources: human resources, processes and infrastructures. Their combination will generate the expected positive impacts. It is therefore necessary to raise the question of what is the object of assessment: is it human resources, processes or infrastructures? In addition, knowledge is intangible (Akhavan and al., 2013; Malhotra, 2005), which makes difficult to count on conventional methods, such as financial or accounting ones, to measure the impact of KM projects (Skyrme and al., 1998). Financial resources are necessary to put in place KM projects, so managers are looking for Return On Investment (ROI). Measurement is necessary to justify all these investments, but also to continue to improve organization at different levels, namely business processes, customer satisfaction, innovation, etc. It becomes difficult to establish the link between investment in KM and organisational performance. With these multiple variables in mind, which framework managers can use to take to account the complexity of measurement of KM projects?

These multiple variables that shape these initiatives induce a problem of mismatch between methods and measurement models available and the nature of knowledge; in short, based on which framework can the results of KM initiatives be adequately assessed?

Research Methodology

The methodological approach selected is based on a literature study divided in two parts. In the first one, the emphasis is put on the identification of different frameworks for measuring KM projects. It consists of briefly presenting their characteristics, in order to show what aspects are taken into consideration. In the second part, the emphasis is placed on a specific framework, the Balanced Scorecard approach. Following the literature, two case studies are proposed to test what the application of the Balanced Scorecard. Indeed, a case study approach is favored in a process of exploratory research (Sushil, 2000). Also, since knowledge measurement implies both socio-technical and organizational aspects, it is important to place emphasis on issues of ownership, socialization, internalization, etc. A case study also enables

an inductive approach that helps identify the variables that affect the research object; here the measurement of knowledge management. The case studies take place in two distinct organizational contexts that illustrate two different types of KM initiatives. To prepare them, the data collection was elaborated on completed projects through interviews with project participants. The results summarized in this paper are based on verbatim transcripts of these interviews.

Literature review

Part 1: Measurement frameworks

Literature about KM addresses the measurement issue in a differentiated way. These differences are mostly due to the profile, experience and disciplinary field to which belong those who express solutions to it (Hanley and Malafsky, 2004). Thus, all proposals aiming for the measurement of KM, within an organization, can be grouped into three main approaches. A first one in which the focus is on metrics, a second one in which the focus is on methodological aspects and a third one that prioritizes measurement models. In the first approach, various authors propose "metrics" that enable to establish the level of knowledge within an organization. Those metrics enable to find a characteristic or a condition of the organization. No processing measure is seen between an initial state and a final time. In this respect, it favors a static vision.

In the methodological approach, it is critical to consider steps that can lead to the assessment of knowledge (Hanley and Malafsky, 2004; Chamorro and al 2003.) This is in fact a more relevant way to establish a measure, but in practice cannot be done without setting first objectives and goals. Consequently, the approach and the measurement of knowledge cannot be taken as a simple process of calculation of metrics", hence the need for a systematic approach. Although they are relevant, methodological approaches only apply to the conceptual framework of the measurement approach. These steps do not enable to establish the necessary links between project activities and benefits. As to the measurement models, several are distinguished. They can be classified as economic, systemic or strategic in nature. On this basis, various syntheses and categorizations are established. Table 1 below shows the parameters on all approaches.

Table 1: Perspectives on KM assessment

Although they are all relevant, these categorizations remain difficult to operationalize. They certainly indicate what needs to be done, however, no precise indication is provided to determine from which measurement the calculation of metrics is done, or of what is the nature the data to be entered. Given the nature of KM, identifying a measurement model requires meeting a group of criteria. Indeed, the following aspects should be taken into consideration to measure the impact of knowledge management: a) – Include both qualitative and quantitative metrics; b) – Take into consideration both financial and non-financial dimensions ;

c) Specifically localize on what organizational level is the measuring of the impact and effects focused ; d) – Recall the purpose and objectives pursued; e) – Distinguish relative flows and stocks dimensions; f) – Set changes in the practices of organization members involved in the project ;

g) – Reveal improved or optimized results obtained following the project.

Part 2: The Balanced Scorecard: a suitable model for Knowledge Management assessment in an engineering and manufacturing environment

In the literature, various authors suggest that the Balanced Scorecard approach (Kaplan and Norton, 1996) meets several of the aforementionned criteria. Indeed, the Balanced Scorecard (BSC) is a framework which takes into account several dimensions. The emphasis is placed on 4 perspectives, namely: customer, finance, internal processes, learning and growth. Based on this architecture, Kaplan and Norton (1996) stress that the financial results are only a consequence of the other elements. This complementarity between these 4 perspectives makes it possible to ensure a multi-factor approach to performance measurement. Moreover, the BSC integrates categories that are both internal and external to the organizational system under study. Finally for each dimension, it is possible to identify both qualitative and quantitative metrics. For measurements relating to customer satisfaction are rather of qualitative type whereas with regard to financial results, metrics are typically of a quantitative nature. For managers and researchers, the BSC offers many advantages in terms of measurement of the performance. It is also applicable at different organizations levels, namely: within business units, projects or the organization as a whole. The fact that it is non-prescriptive, the Balanced Scorecard can be adapted to various industries and sectors (private, public, service oriented, manufacturing, etc.). With this in mind, it becomes important to specify under which conditions the BSC framework will be applicable in a context of KM performance assessment.

Chen and al. (2005) adapted the BSC for KM purposes. Starting from the work of various authors (Nonaka and Takeuchi, 1997; Alavi, 1997; Liebowitz, 1999; Chen and Wu, 2001), Chen and al. (2005) established that the process of KM can be declined in 4 core activities, namely: 1- creation, 2- conversion, 3- circulation, and 4 – completion. These processes are used as substitutes for the four initial

ones proposed in the original Norton and Kaplan model. Conceptually, the proposal made by Chen and al. (2005) summarised in Table 2 adapts the BSC so that it can answer the specific needs of KM performance measurement.

Table 2 : The Balanced Scorecard model adapted by Chen and al. (2005)

Another adaptation of the BSC for KM performance assessment has been proposed by Wu (2005). Here, a more qualitative and integrated approach is adopted by associating on the one hand aspects relating to the organization (human capital, customer capital, organisational capital) and on the other hand operational dimensions of the BSC (finance, process, learning, etc). This combination makes it possible to distinguish elements relating to KM as a stock (all aspects concerning organizational capital) from the dynamic aspects relating to the transformation from stock into flow. Table 3 summarises the adaptation developed by Wu (2005), which proves to be very relevant in a non-commercial organisational context, because it is not necessary to have financial or quantitative data to obtain results.

Table 3 : The Balanced Scorecard adapted by Wu (2005)

Forstenlechner and al. (2009) also make use of the BSC to measure the results of an initiative of KM in a law firm. They underline the benefit searched by the firm, namely: 1)

– to promote the training between lawyers; 2) – to identify

the best practices; 3) – to diffuse knowledge within the organization; 4) – to increase innovation; 5) – and finally to controlcosts. Starting from these objectives, four specific dimensions related to the firm are substituted for those of the initial method, namely: lawyers, organisational culture, process interns and finances as summarised in Table 4.

Table 4. Balanced Scorecard adapted by Forstenlechner and al. (2009)

On the basis of metrics and these dimensions, the data- gathering made it possible for the authors to establish causality relationships between the 4 levels. Thus, dimensions related to culture and organization, by means of standardization processes, affect the quality and the transfer of knowledge, indicators connected to internal processes. In the same way, commitment of lawyers and innovation influence the development of knowledge; this affects the transfer of the knowledge. Thus, the 4 indicators of dimension culture and organization affect the three connected to internal processes. The latter influence the indicators relating to lawyers, namely the use of knowledge made available and their effectiveness to work: what contributes to their satisfaction. Consequently, by the means of the indicators and collected information, dimensions culture and organization, internal process and lawyers are interrelated. Finally the 3 indicators relating to the lawyers who are the use of knowledge made available, the effectiveness and satisfaction have an impact on the performance, the productivity and the costs: indicators connected to financial dimension. It thus prevails a series of interrelationships connecting dimensions the ones to the others by the means of the indicators selected. Consequently, the authors prove that the KM initiative made it possible to reach the targets. The approach thus consists in establishing a link between causes and their consequences. These causality relations are established based on the indicators. As a whole, various authors propose approaches relating to the evaluation of the measurement of the benefit related to the initiatives of knowledge management. In the following pages two case studies are presented in order to illustrate the advantages and limitations of both these adaptations of the BSC model to measure the performance of KM projects.

Case study 1: Knowledge management in an engineering research unit

This case relates to an industrial research Chair in which various researchers took part. One of their area of activity focuses on solutions of optimization and streamlining of business processes. It should also be noted that a research Chair is a unique place to pursue the practice of KM, because all the activities and procedures that characterizes KM can be studied: i.e. creation, codification, generation, transfer, and application. As part of their mandate, a group of three researchers got to work on problematic of identification of market segment on a three-year period, so that the industrial partner could develop a competitive advantage. The three researchers worked on this mandate at different periods, coming across different obstacles, challenges and issues.

Based on the progress of this initiative that has an informal approach to KM, the Chair wanted to know, from the actual benefits in terms of knowledge development, transfer and creation. The proposed evaluation did not cover all employees, only the coordinator and researchers who participated in the period of 3 years project only. The BSC is the model chosen for this case study, for the reasons already given in this text. Wus adaptation (2005) is the one chosen, partly because a data collection by interview is privileged, and also because the actual impact is emphasized: what constitutes the objectives pursued by the chair. To evaluate this impact, the analytical framework proposed by Wu (2005) was chosen and the results are presented in Table 5 below.

Table 5: KM benefits of case study 1 based on Wus model

Table 5 suggests the benefits that the project participants experienced. Starting from the mission of the organization, we find that in any context of problem solving produces knowledge in many ways, from the theoretical framework. They accumulate and create a pool of solutions that would be useful to all those who gravitate around the Chair. As stressed by the coordinator, "the fees being paid are to me the best indicator of outcome." Incidentally on the basis of those results, the following conclusions can be drawn: in terms of human capital, profits are present at all levels (financial, customer, process and learning). In passing, human resources involvement which is mobilized in a KM initiative is estimated to help generate positive spillovers. Concerning the organizational capital, valuing what the Chair already earns is a cornerstone for its development. Indeed, for one participant, being confronted to real organizational issues has allowed me to develop knowledge in related fields to mine, including performance analysis, in order to better calibrate the solutions developed." Thus, in both financial and internal process levels, the practice of KM helps generate more business volume for the Chair, which coincides with the position already expressed by various other authors for this purpose ( Andreeva et al, 2012; Carlucci et al, 2006; Massey et al, 2001).

At the customer capital level profits also exist, and one of the reasons for this relates to the mission of the Chair, which is to produce applied knowledge for the industry. On this basis, focusing on specialized knowledge to meet customer expectations contributes to the development of internal skills. This capacity is fueled by the accumulated knowledge the Chair already has. This first evaluation of a non- formalized initiative, based on this adapted SC , can be made at several levels: a) – the applied research field, mostly in engineering and more particularly in its management, would do well to formalize procedures for sharing and using knowledge; b) – such an institutionalization of the procedure of knowledge production would result inter alia into developing an ability to find new contexts for the application of knowledge and developed solutions within the Chair; c) coming into contact with the real needs of customers also contributes to innovation and creativity through adapted solutions This enables the organization to reach its goals and to demonstrate, to some degree, the relevance of a systematization for KM, even for an entity such as a Chair that does not work in the private sector.

Nevertheless, this profit-measuring approach is limited to the participants impressions. It is necessary to evaluate various cases in different contexts of the Chair, not only to obtain a variety of cases as advocated in qualitative research, but also to obtain an overcrowding wealth of situations (Hlady-Rispail, 2002 Sushil, 2000).

Case study 2: formal knowledge management approach in the manufacturing sector

The company named Bois-Franc for confidentiality reasons specializes in the manufacture of custom natural wood furniture for the Canadian and U.S. markets.. Even though their products are sold in luxury or high-end market

segments, sales are suffering. To improve its competitive position, the company decided to develop a knowledge transfer initiative through external consultancy. The mandate for the consultants was to develop a set of solutions, one of which would focus on a new approach to market segmentation strategy, in order to precisely determine the localities or regional markets in which Bois- Franc should devote its efforts and its resources to generate add value. This formal KM initiative represents a transfer of knowledge between two entities, one internal and the other external. The impact is measured in the receiving companys developed knowledge. Given the formal characterization of the project, the adaptation of Chen et al.s Balanced Scorecard (2005) is used. On one hand enables to take into account the transfer aspects and those related to organizational performance on the other hand. The collected and processed information is cosigned in Table 6.

Table 6 : KM benefits based on Chen model

Based on Chen et al.s adapted Balanced Scorecard model (2005), Table 6 shows a good amount of results. 1 – Designing activities have effectively enabled a transfer of the developed solution within the company Bois-Franc. This transfer was made thanks to socialization and externalization, as advocated by Nonaka and Takeuchi (1997). 2 – Movement activities also generated tangible profits, because the 3 employees indicated (market development agent, computer technician and managers) have appropriated the new segmentation solutions, thus enabling Bois-Franc to improve its knowledge of the market in one hand and to give access to knowledge in the other hand. Conversion activities show embryonic results, based on indicatorsthat were selected and comments that were gathered; it is difficult to establish the impact for certain. It is at the completion level, which corresponds to the achievement of the final objectives, that profits are the most persuasive. Indeed, the statement about the improvement of organizational performance results is very clear; as emphasized by participants, they are often related to the knowledge transfers initiative.

CONCLUSION

The BSC represents a viable option to establish the impact of KM initiatives on organization. The flexible and adaptable character of the Balanced Scorecard enables it to be used in different contexts (manufacturing company, Research unit) and for different kinds of initiatives. Moreover, it can be used as much for qualitative as for quantitative analyses.

Nevertheless, many dimensions are not taken into consideration. For example, the timescale is not taken into account. The effects observed 3, 6, 12 or 24 months after a KM initiative are different from one period to another: this makes this mode of operationalization essentially static, while KM is a dynamic phenomenon. Furthermore, the behavioral aspects that emphasize ownership that individuals adopt to integrate new ways of doing, like knowledge sharing, its distribution or its combination are not taken into consideration, but it is through ownership that the short, medium and long term effects can be appreciated. Incidentally, implementing a KM initiative reflects a will and a strategy of organizational change. In this context, it is therefore appropriate to stress a few ideas for future consideration, including:

• Whom the question of profits should be asked for?

• Establishing the situation before initiating, as well as establishing the differences between the situation before and after.

• What should lead to integrate an audit of the status of knowledge management before any measurement approach.

Moreover, it would be relevant from the BSC model to determine a complementary framework partner who would enable an understanding of the impact, while still preserving the multi-criteria aspect.

Also, it seems that in an input-output logic, the emphasis is on more on the outputs and less on the outcomes. This dimension appears essential to us, because they are the benefits, effects and impacts generated by the knowledge management initiatives looked for. It is therefore necessary that a measurement framework to develop could take this aspect into consideration. We believe that efforts related to future research should pay a special attention to it.

REFERENCES

1. Akhavan, P., Shirazi, H., Sabzaligol, A. and Pezeshkan, A. (2013), A Framework fo organizational Knowledge Assessment by Combining BSC and EQFM: A case of Besat Industry Complex, Iran in The IUP Journal of Knowledge Management, vol. XI, 2, p. 7-18.

2. Alavi, M., Leidner, D. (2001). Knowledge Management and Knowledge Management Systems : Conceptual Foundations and Research Issues, MIS Quarterly, vol. 25, no 1, mars, 107-136.

3. Andreeva, T., and Kianto, A. (2012), Does Knowledge Management Really Matter? Linking Knowledge Management Practices, Competitiveness and Economic Performance, Journal of Knowledge Management Economics and Information Technology, 6, octobre, 354-377

4. Anvarie, A., Alipourian, G.-A., Moghimi, R. and Baktash, L. (2011), Analysis of Knowledge Management Within Five Key Areas, Journal of Knowledge Management, 16, 4, 617- 636

5. Bontis, N., Ragonetti, N.C., Jacobsen, K. and Roos, G. (1999), The Knowledge Toolbox: A review of The Tools Available to measure and Manage intangible Resources

6. Management in European Management Journal, vol. 17, 4, p. 391-402.

7. Bose, R. (2004), Knowledge Management Metrics in Industrial Management & Data Systems Management, vol. 104, 6, 457-468.

8. Canadian Management Accountants (CMA; 1999), Focus group Draft: Measuring Knowledge Assets in Holsapple,C.W. (eds) Handbook of Knowledge Management.

   CMA, Mississauga

9. Carlucci, D., and Schiuma, G (2006), Knowledge Asset Value Spiral: Linking Knowledge Assets to Companys Performance, Knowledge and Process Management, 13, 1, 35-46

10. Chamorro Del Rey, F.M., Roy, R., Van Wegen and Steele, A. (2003), A Framework to Create Key Performance Indicators for Knowledge Management Solutions in Journal of Knowledge Management, vol 7, no. 2, p. 42-62

11. Chen, M.-Y. and Chen, A.-P. (2005), Integrating option Model and Knowledge Management Performance Measures: An Empirical Study in Journal of Information Science, vol. 31, 5, p. 381-393.

12. Chua, A. Y. K. and Goh, D. H. (2008), Untying The knot of Knowledge Management Measurement: A Study of Six Public Service Agencies in Journal of Information Science, vol. 34, 3, p. 259-274.

13. Davenport, T., and Probst, G. (2001), Knowledge Management Case Book, Verlag, Wiley & Sons, Germany

14. Forstenlechner, I., Lettice, F. and M. Bourne, (2009), Knowledge Pays: Evidence From A Law Firm in Journal of Knowledge Management, vol. 13, 1, 56-68.

15. Forstenlechner, I., Lettice, F., Bourne, M. and Webb, C. (2007), Turning Knowledge In to Value in Professional Service Firms in Performance Measurement and Metrics, vol. 8, 3, 146-156.

16. Hanley, S. and Malafsky, G. (2004), A Guide for Measuring the value of KM Investments in Holsapple,C.W. (eds) Handbook of knowledge management. Knowledge Matters.

    Vol.1, Chapter 49, p. 369-390, Spring Verlag, Berlin

17. Hlady-Rispal, M. (2002), La méthode des cas. Application à la recherche en gestion. De Boeck Université, Bruxelles

18. Hubbard, D. W. (2007), How to Measure Anything. Finding the Value of Intangibles in Business, John Wiley & Sons, New Jersey

19. Kaplan, R. S. and Norton, D. P. (1996), The Balance Scorecard, Harvard Business School Press, Boston

20. Lettice, F., Roth, N. and Forstenlechner, I. (2006), Measuring Knowledge in The new Product Development Process in International Journal of Productivity and Performance Measurement, vol. 55, no. 3/4, p. 217-241.

21. Liebowitz, J. (2000), Knowledge Management Handbook,

    CRC Press, New York

22. Malhotra, Y. (2005), Integrating Knowledge Management Technologies in Organizational Business Processes: getting Real Time Enterprises to Deliver Real business performance in Journal of Knowledge Management, vol. 9, no. 1, p. 7-28.

23. Marr, B., Schiuma, G. and Neely, A. (2004), The Dynamics of Value Creation: Mapping your Intellectual Performance Driver in Journal of Intellectual Capital, vol. 5, no. 2, p. 312-325.

24. Massey, A.P., Montoya, M.M., Holcom, K. (2001). Re- engineering The Customer Relationship: Leveraging Knowledge Assets at IBM, Decision Support System, vol. 32, no 2, December, 155-170.

25. Nonaka, I., and Takeuchi, H. (1997), La Connaissance Créatrice: La Dynamique de lEntreprise Apprenante, De Boeck, Bruxelles.

26. Roos, J., Roos, G., Dagronetti, N. and Edvinsson, L. (1998), Intellectual Capital: Navigating in the New Business Landscape, New York University Press, New York

27. Scarso, E., Bolisani, E. and Padova, A. (2011), The Complex Issue of Measuring KM Performance: Lessons from The Practice in Holsapple,C.W. (eds) Handbook of knowledge management. Knowledge Matters. Vol.1, Chapter 49, p. 369- 390, Spring Verlag, Berlin

28. Skyrme, D.J. and Amidon, D.M. (1998), New measures of Success in Lournal of business Strategy, January/February, 20-24

29. Sveiby KE (2007) Methods for Measuring Intangibles Assets in Vallejo-Alonso, B., Rodriguez-Castellanos, A. et Arregui-Ayastuy, G., Identifying, Measuring and Valuing Knowledge Based Intangible Assets- New Perspectives. Business Science Reference, New York.

30. Wu, A. (2005), The Integration Between Balanced Scorecard andIntellectual Capital in Journal of Intellectual Capital, vol. 6, no. 2, p. 267-284