Evaluation of the Implementation of the Electronic Information System from the Perspective of Users: a Case Study at the Federal Institute of Education, Science and Technology of Tocantins

Evaluation of the Implementation of the Electronic Information System from the Perspective of Users: a Case Study at the Federal Institute of Education, Science and Technology of Tocantins

Brenno Barbosa De Alencar Fernandes

Degree in Internet Systems Federal Institute of Education,

Science and Technology of Tocantins Palmas TO, Brazil

Jadson Vieira De Oliveira Federal Institute of Education, Science and Technology of Tocantins Palmas TO, Brazil

Daniel Cândido De Oliveira Federal Institute of Education, Science and Technology of Tocantins Palmas TO, Brazil

AbstractThis research aims to investigate the implementation process of the Sistema Eletr̫nico de Informa̵̤es (Electronic Information System) РSEI, at the Instituto Federal de Educa̤̣o, Ci̻ncia e Tecnologia (Federal Institute of Education, Science and Technology of Tocantins) РIFTO, focusing on the IFTO public servants. For this, a search was made in the specialized literature, in which success factors related to user satisfaction in the system implementation were extracted. Then, to confirm the state of the art, a descriptive case study was developed. The data were collected through a judgment matrix and later treated in order to validate the presented proposal.

KeywordsUsers;Satisfaction;Sei

1. INTRODUCTION

   Information technology has contributed to the improvement of the relationship between government, citizens and representative groups of society [4] [5]. In the last years the public power has been adopting these resources as a form of improvement in the provision of services to the citizens.

   One of the systems that has been widely adopted in organs of different spheres is the Electronic Information System (SEI), which aims to implement the electronic process in place of existing physical processes, seeking greater agility, speed, transparency and efficiency in services provided [17]. The SEI was adopted within the scope of the IFTO in order to comply with the Presidential Decree No. 8,539, of October 8, 2015, which provides for the use of electronic means to carry out the administrative process [7].

   The SEI was developed by the Federal Regional Court of the 4th Region (TRF4), and disseminated to other federation bodies, free from charge, and an official letter was sent to the Ministry of Planning, Development and Management (MPDG), formally showing intention to obtain the right to use the system. Subsequently, the Agreement of Technical Cooperation is signed, which formalizes the assignment of the use of the system, and alongside with this term, the Plan of Work must also be signed, where the initial planning for the implementation of the system in each body or entity is defined [15].

   In the process of implementing an Information System (SI) it is not possible to control all present factors in the implementation, critical factors are crucial for the success or failure in an information system implementation [18] [19] [24] [25] [29] [30]. Users satisfaction is shown as a matter of prime importance, once the perception over the benefits provided by the system might improve its performance and productivity [1] [3] [6] [8] [9] [18] [28].

   This work aims to be a useful tool in the aid of decision making and IT governance. For such purpose, it has IFTO as focus and it presents, in this context, an evaluation of the impacts in implementing SEI in IFTOs scope under the users perspective. In specific, this work aims to present a theoretical survey that substantiates the research, as well as presenting users opinion, from which the collected data referring to the system implementation and using experience were verified through a judgment matrix and a proper statistic treatment. Therefore, at the end of this work it shall be possible to compare the state of the art and the state of the practice, relating critical factors yet to be developed in order to assure its continue usage with efficiency.

2. METHODOLOGY

   Performing a descriptive research aims to expose the characteristics of a certain population or phenomenon, and in order to reach this goal it is necessary that the investigator acquires several data about the object of study [14] [27], allowing reality to be analyzed from the participants perspective providing a global , coherent and thorough view to the investigator [12] [23]. This study focuses on analyzing a certain situation, which must have relevance and be representative, picturing reality without the researchers interference.

   By performing the theoretical survey the key words users satisfaction and information systems were used, and from the works found, the ones having a theme related to the project were selected. From these, it was possible to extract methods that contained efficiency proof in the analysis to be produced.

   Based on the work of several researchers, [26] defined four dimensions: (a) task productivity, (b) innovation, (c) customer satisfaction and finally (d) managerial control, which combined, help describing the impact of an application on the individuals of an organization [10]. From this assumption, this research focuses on the dimension of users satisfaction.

   In order to allow the evaluation of the impact of the SEI implementation, a questionnaire was applied to the public servants with closed questions, and the answers were organized according to the Likert scale, which includes statements on a scale between "totally agree" and "totally disagree" [10] [21]. In addition, according to [20], we used statistical methods to perform a more sophisticated analysis of the data.

   The questions had as goal to verify the effectiveness of the implementation of the SEI in IFTO from the users perspective, involving several questionings observed in the theoretical cut- offs, which were grouped in four critical factors that have influence on the users satisfaction, being able to be observed in Table I.

   TABLE I. CRITICAL FACTORS ON THE PROCESS OF IMPLEMENTING AN INFORMATION SYSTEM BASED ON USERS SATISFACTION

   Critical Factors	Authors
   Business processes	[6] [8] [9] [18] [28]
   Reliability of information	[1] [2] [6] [11] [17]
   Training and education	[9] [28]
   Monitoring, control and support	[3] [18] [28]

   Fig. 2. Age range.

   Fig. 1 presents the conceptual model for validation of our hypotheses to be verified throughout the work. Critical factors have a greater or lesser effect on users satisfaction.

   Fig. 1. Conceptual model.

   TABLE II. QUESTIONNAIRE RELIABILITY ACCORDING TO CRONBACH ALPHA

   Cronbach Alpha	Reliability
   Bigger than 0,9	Excellent
   Between 0,8 and 0,9	Good
   Between 0,7 and 0,8	Acceptable
   Between 0,6 and 0,7	Questionable
   Between 0,5 and 0,6	Poor
   Smaller than 0,5	Unacceptable
   Source: [13]

   a. Source [13]

3. RESULTS

   The research had the participation of 347 public servants categorized by age range, academic formation and assignment time in the institution as shown in fig. 2, 3 and 4.

   The data were collected through an electronic questionnaire, madeavailable to the institution's public servants. IFTO has a functional population of 1221 public servants, of which 354 participated. In order for the answers to be consistent and relevant to this work, some control questions were inserted. From these, 347 were validated for statistical treatment. The achieved proportion of validated respondents, relative to the total population, provides the survey a 95% confidence level, with a sample error margin of 4.46% [22].

   In order to determine a level of confidence of each critical factor and the correlation of the items we used the Cronbach's alpha value [16]. Table II presents the classification levels of the Cronbach Alpha.

   Fig. 3. Academic formation.

   Fig. 4. Assignment time.

   The statistical treatment was applied to reduce the subjectivity of the answers, so that the results of the research would be able to reflect the reality of the whole. We considered satisfied users those who had an average of each critical factor between 5 and 7, based on the Likert scale. In order to evaluate the influence of the critical factors, according to table 1, in the users satisfaction, follow the analysis below.

   Looking at Table III we see that the user's perception regarding the critical factor business processes presents a weighted average of 5.15. Since the item cost reduction by archiving and storage presents higher average of 5.78 and lower standard deviation. This critical factor reached 66.92% satisfaction of the participating users.

   TABLE III. CRITICAL FACTOR BUSINESS PROCESSES

   Items	Average of items	Standard deviation	Average of the critical factor	Alpha value
   P1 – Was there a decrease in labor costs?	4,60	1,90	5,15	0,857
   P2 – Was the time for carrying out the activities reduced by using the system?	4,86	1,85
   P3 – Did IFTO reduce the costs of filing and storing documents?	5,78	1,60
   P4 – Did SEI make your service more productive?	4,98	1,85
   P6 – Is SEI useful in relation to your work needs?	5,51	1,71

   Table IV presents the critical factor "Reliability of information", which obtained approximately 77% approval from its users. The item P5 obtained an average of 5.99, being the highest among all the items surveyed.

   TABLE IV. CRITICAL FACTOR RELIABILITY OF INFORMATION

   Items	Average of items	Standard deviation	Average of the critical factor	Alpha value
   P7 – Whenever necessary do you have access to the information in SEI?	5,57	1,66	5,49	0,803
   P5 – Is the information provided by SEI reliable?	5,99	1,37
   P8 – Do you consider the system safe?	5,35	1,67
   P9 – Does SEI respond quickly to your requests?	5,04	1,77

   A control question was used to separate the answers from those who participated in the trainings offered to use the system, totaling 57.7% of the interviewees. It can be seen from Table V that P10 received the average 4.24, this was the lowest average of all the research, and it may be related to the short time available for training.

   TABLE V. CRITICAL FACTOR TRAINING AND EDUCATION

   Items	Average of items	Standard deviation	Average of the critical factor	Alpha value
   P10 – Was training for SEI satisfactory?	4,24	1,71	4,82	0,862
   P11 Does the training environment (http://treinamento.sei.ifto.ed u.br) aid in learning to use the system?	4,95	1,66
   P12 – Was the material available for study satisfactory?	4,97	1,56
   P13 Has the guidance page on the use of SEI, (http://sei. ifto.edu.br/site-sei/), assisted in the use of the system?	5,11	1,59

   Based on a control question, as well as with the critical factor "Training and education", the data of those who used support were analyzed in some way, being that they number approximately 65% of the respondents. Based on these data, Table VI was created, analyzing the high degree of satisfaction with the offered support, with the average being 5.68.

   TABLE VI. CRITICAL FACTOR MONITORING, CONTROL AND SUPPORT

   Items	Average of items	Standard deviation	Average of the critical factor	Alpha value
   P14 – Have your demands always been attended?	5,81	1,43	5,68	0,910
   P15 – Were the demands attended quickly?	5,55	1,56

   About 35% of the participants did not need support in any way, of these 57% feel satisfied with the system.

   A. Critical factors level of importance

   Through the results it is possible to demonstrate, through Table VII, the relative importance of the critical factors. Given that these data can be used to assist in decision making and IT governance, in order to guarantee efficient continued use of the system.

   TABLE VII. CRITICAL FACTOR MONITORING, CONTROL AND SUPPORT

   Critical factor	Position	Average	Standard deviation
   Monitoring, control and support	1º	5,68	1,49
   Reliability of information	2º	5,49	1,62
   Business processes	3º	5,15	1,78
   Training and education	4º	4,82	1,63

4. DISCUSSION

This paper aims to evaluate the influence of critical factors on the users satisfaction dimension in helping decision making and IT governance. For this, a survey of the systems evaluation practices regarding the users satisfaction was carried out in the light of the specialized literature. The study sought to ascertain if the state of the art reflected in the state of the practice through an electronic questionnaire. For that, an exploratory research was carried out with direct users of the SEI – IFTO public servants – that would be able to respond on the effects, based on their experiences and technical knowledge.

The data collected through the electronic questionnaire validated the presented proposal. Critical factors were evaluated from the Likert scale and impact weights were assigned to them. From this, a positive response was obtained regarding the hypothesis tested by this study, and it was verified that the practices focused on users satisfaction cause a greater or lesser effect on the prospecting of technology in the value chain of systems implantation, where practices were considered to have a moderate effect, tending to be high, on the dependent variables, and even some of them presenting a minor impact, no practice had a below-moderae impact.

It was possible to perceive a high degree of satisfaction of the users with the system in question. But in order to be able to attest this affirmation, new researches must be carried out periodically, later through a comparative between the surveys we can affirm whether the implantation was successful. However, it is important to emphasize that the research was limited to the implementation of SEI in IFTO, which makes it possible and recommended to expand the study to other experiences, as well as extend to other specialists with technical and scientific knowledge on the subject addressed here.

REFERENCES

1. Akkermans, H. and Helden, K. (2002). Vicious and virtuous cycles in erp implementations: a case of interrelations between critical success factors. European Journal of Information Systems.

2. Albernaz, C. M. and Freitas, A. L. P. (2010). Um modelo para avaliação da qualidade de serviços de suporte de tecnologia da informação. Anais do XXX ENEGEP.

3. Albertin, L. A. (2001). Valor estratégico dos projetos de tecnologia da informação. Revista de Administração de Empresas, 41(3):4250.

4. Anjos, G. M. A. A. and Ezequiel, V. d. C. (2011). Cidadania Virtual: O Espetáculo do Governo Eletrônico. Estud. sociol, 16:5976.

5. Balbe, R. d. S. (2010). Uso de tecnologias de informação e comunicação na gestão pública: exemplos no governo federall. Revista do Serviço Público, 61:20.

6. Bergamaschi, S. (2001). Fatores críticos de sucesso para implementação de sistemas de gestão empresarial. Atlas, São Paulo.

7. Brasil (2015). Decreto N o 8.539, de 8 de outubro de 2015. Dispo- nível em http://www.planalto.gov.br/ccivil_03/Ato2015- 2018/2015/Decreto/D8539.htm. Accessed in: 2017-07-10.

8. Catersels, R., Helms, R., and Batenburg, R. (2010). Exploring the gap

   between the practical and theoretical world of erp implementations: results of global survey. Proceedings of the International Conference on Research and Practical Issues of Enterprise Information Systems (CONFENIS).

9. Esteves, J., Casanovas, J., and Pastor, J. (2003). Using the partial least squares (pls) method to establish critical success factors interrelations in erp imple- mentation projects. Proceedings of the Americas Conference on Information Systems (AMCIS).

10. Ferreira, A. P. D. and Ferreira, L. (2008). Medindo a Satisfação dos usuários de um Sistema Integrado de Gestão. SEGeT Simpósio de Excelência em Gestão e Tecnologia, pages 1 11.

11. Fitzsimmons, J. A. and Fitzsimmons, M. J. (2000). Administração de Serviços: operações, estratégia e tecnologia de informação. Bookman, Porto Alegre, 2 edition.

12. Fonseca, J. J. S. (2002). Metodologia da pesquisa científica. UEC, Fortaleza.

13. George, D. and Mallery, P. (2002). SPSS for Windows step by step: A simple guide and reference, 11.0 update. 4th Boston: Allyn & Bacon.

14. Gil, A. C. (1999). Métodos e técnicas de pesquisa social. Atlas, São Paulo, 5 edition.

15. Governo-Federal (2017). Processo Eletrônico Nacional – Sistema Eletrônico de Informação – PEN – SEI. Disponível em https://softwarepublico.gov.br/social/sei. Accessed in: 2017-07-10.

16. Hora, H. R. M., Monteiro, G. T. R., and Arica, J. (2010). Confiabilidade em questionários para qualidade: Um estudo com o coeficiente alfa de cronbach. Produto & Produção, 1(2):85103.

17. Klein, M. M. (2012). Processo eletrônico na Justiça Federal da quarta região: verificação do resultado de sua implementação.

18. Laudon, K. C. and Laudon, J. P. (2007). Sistemas de informação gerenciais. Prentice Hall, São Paulo, 7 edition.

19. Oliveira, M. M. d., Ponchio, M. C., Neto, M. S., and Pizzinatto, N. K. (2009). Análise dos fatores de resistência na implantação de sistemas de informação na manufatura de eletrônicos. JISTEM Journal of Information Systems and Technology Management, 6:507524.

20. Roesch, S. M. A. (2012). Projetos de estágio e de pesquisa em administração: guia para estágios, trabalhos de conclusão, dissertações e estudos de caso. Atlas, Atlas, 3 edition.

21. Salvalaggio, A. P. d. S. (2015). Avaliação do Sistema Erp Através da Satisfação e Produtividade dos Usuários Finais da Empresa Vitasons Centro de Apoio Auditivo. pages 1 79.

22. Santos, G. E. d. O. (2017). Cálculo amostral: calculadora on-line. Disponível em http://www.calculoamostral.vai.la. Accessed in: 2017-07-10.

23. Severino, A. J. (2007). Metodologia do trabalho científico. Cortez, São Paulo, 23 edition.

24. Somers, T. M. and Nelson, K. (2001). The impact of critical suc- cess factors across the stages of enterprise resource planning implementations. Proceedings of the Hawaii International Conference on System Sciences.

25. Sun, Y. A., Yadzani, A., and Overend, D. J. (2005). Achievemente asses- sment for enterprise resource planning (erp) system implementations based on critical success factors (csfs). International Journal of Production Economics, 98:189203.

26. Torkzadeh, G. and Doll, W. (1999). The development of a tool for measuring the perceived impact of information technology on work. Omega, 27(3):327339.

27. Triviños, A. N. S. (1987). Introdução à pesquisa em ciências sociais: a pesquisa qualitativa em educação. Atlas, São Paulo.

28. Umble, E., Haft, R., and Umble, M. (2003). Enterprise resource planning: Implementation procedures and critical success factors. European Journal of Operational Research, 146(2):241257.

29. Valente, N. T. Z. and Riccio, E. L. (2004). Implementação de ERP em Pequenas e Médias Empresas: Estudo de Caso em Empresa do Setor da Cons- trução Civil.

30. Xue, Y., Liang, H., Boulton, W. R., and Snyder, C. A. (2005). ERP imple- mentation failures in China: Case studies with implications for ERP vendors. International Journal of Production Economics.