Information Service Quality Evaluation for Analytics Service Assessment and Comparison

Information Service Quality Evaluation for Analytics Service Assessment and Comparison

Dr . M . Suresh Babu,

Professor & Head, Department of Computer Applications, Madanapalle Institute of Technology & Science,

Mr . D . Venkata Siva Reddy,

Head, Department of Computer Science, Besant Theosophical College, Madanapalle 517325, AP, India.

Dr . D . Asha Devi, Professor, Department of ECE,

Madanapalle Institute of Technology & Science, Madanapalle,India.

Abstract – Various information services are now becoming customizable in order to satisfy users information needs on the Web or in a mobile environment. Therefore, we now need a new approach for evaluating the information service quality from the users perspective. This paper proposes a new evaluation model for assessing information service quality of high value-added services such as business or technology intelligence services. It compares two analytics services, Google Analytics and InSciTe Adaptive, using the proposed evaluation model. The proposed information service quality model contains two evaluation factors based on usefulness and reliability as well as 22 detailed evaluation criteria. The result of the service quality evaluation shows that the evaluation model is useful for accessing the information service quality of various analytics services and for finding the main characteristics between different types of services in the practical domains of business and research.

Keywords: Information Service Quality, Service Evaluation, Business Intelligence, Service Usefulness, Service Reliability, Google Analytics, InSciTe Adpative

1. INTRODUCTION

   The Web 2.0 technology that promotes interoperability, information sharing, and participation has acknowledged the Internet to have an important impact on our lives. We attempt to obtain various types of information required in everyday life through the Internet by browsing professional information, shopping online, listening to Internet broadcasts, exchanging information through social communities, and searching for real-time information such as weather and traffic. However, the users information needs are not entirely met because of the excessive amount of information provided in response to the users inquiry as well as irrelevant information services. Furthermore, problems such as disorientation and embedded digression are raised.

   Many Web service technologies have been developed to solve these kinds of problems, and with the emergence of a semantic web service based on the Web 3.0 technology,

   information services on the Web are becoming customizable to satisfy the users information needs. Through semantic processing, not only does serving of facts resolve the users inquiry but also the creation and serving of information with a high added value satisfies the users information needs. It can be said that the environment of Web services is changing to one where information service systems replace the users role of deciding what and how to find. Consequently, there is a need for a method to appropriately evaluate the quality of an information service; however, regulations and methods for evaluating the quality of an information service are almost nonexistent.

   The existing ISO (International Standard Organization) regulations either only partially deal with the quality of an information service or do not sufficiently reflect the trend in the development of information service technologies. For example, ISO 9126:1-3 defines the characteristics of software quality. Nonetheless, there is a problem in applying this to the quality characteristics of an information service because it defines quality characteristics from an engineering perspective in relation to software functions. Further, even though ISO 25000 is a consolidated model for software quality evaluation, it can only be considered as a quality evaluation model for developing software because it combines the software evaluation model ISO 9126 and the software evaluation procedure model ISO 14598 .

   Research on software service quality assurance applying ISO 9000 or ISO 14001 as well has not been appropriate to evaluate the information service quality from the users perspective.

   One reason for this can be obtained from the characteristics of information service systems. Nowadays most information services are provided through the Internet. From the users perspective both a systems quality as a Web software and the quality of information provided by the system determine the Quality of Service (QoS). Therefore, the quality of information service is dependent on both the quality of its function as a software (functionality, usability, reliability, etc.) and the quality of the information provided (accuracy, reliability, and added value).

   Moreover, the quality of an information service is strongly related to the subjective information needs of the user and their satisfaction. Therefore, the quality evaluation of an information service should be conducted from the users perspective. In this research such an evaluation method is introduced.

   Finally, we introduce two major services in the areas of business intelligence and technology intelligence, Google Analytics and InSciTe Adaptive. We assess these two analytics services based on the proposed evaluation model. The assessment task aims to find some meaningful characteristics between the two kinds of analytics services by considering differences among system functions, service goals, operating ways, service experience environments, and so on.

   managed by large-scale databases. As examined above, this kind of technological advancement requires a new quality model for information service to be defined.

   We define the quality of an information service as the usefulness of information system service models that can satisfy the users information needs and the reliability of the served information. Here, the quality model of information service is defined based on the usefulness of the service model of information systems. The usefulness is defined from the quality in use perspective, using the definitions from ISO 9241-11 and ISO/IEC 9126:4 as a guideline. The reliability of the served information is based on the definitions of reliability from ISO 15489 [13] and Vedder et al.

2. BACKGROUNDS

   The World Wide Web Consortium (W3C) abstractly deals with the general properties of Web services, mentioning resource discovery, reliability, precision/recall, availability, latency, authenticity, access control, attributes, and meta- information as parameters of the quality of a Web service. Saroja and Sujatha define the concept of information service quality from the perspective of total quality management (TQM). They mainly consider the information systems competence to be related to the accuracy of its services, adequacy of need-based services, and timeliness of the services, and they determine the quality of the service system in terms of adequacy that is closely related to the users perspective.

   Caruana defines service loyalty in the field of marketing using the relationship between service quality and customer satisfaction. He reports that customer satisfaction acts as an intermediary for service quality. Thus, owing to interaction between service quality and customer satisfaction, service quality is not independent of customer satisfaction. Customer satisfaction is determined by the difference between the users expectations and the real service product. The users pre-existing knowledge and experience affect his or her satisfaction. Vedder et al.investigae the reliability of information on the Internet and analyze it from a philosophical point of view. They generically classify information reliability into content criteria and pedigree criteria. Content criteria are related to the reliability of the information content itself, and pedigree criteria are related to the reliability of the information source. Content criteria are further composed of evidence criteria and logical/subject matter criteria of the relevant information and they do not guarantee the reliability of the information. Therefore, Vedder et al. deduce that the information source for the pedigree criteria should be supported in order to guarantee information reliability.

   2.1. Information Service Quality Model

   Current information systems aim at going beyond servicing information about simple facts, by creating and servicing information with a high added value through the so-called semantic processing, which uses relationship reconstruction, statistical processing, knowledge-based inference, and pattern classification of facts that are

   Figure 1. Basic Concept for Quality Model of Information Service Service model :

   This is a planned model developed to carry out operations that can satisfy the users needs; it can also be understood as a systemized functional unit of information service systems. The service model (or servicing model) generally describes an approach whereby labor unions aim to satisfy members demands for resolving grievances and securing benefits through methods other than direct grassroots oriented pressure on employers. It is often contrasted with the organizing model and with a rank and file organization.

   Information Service Usefulness:

   This is an evaluation criterion to assess time and cost efficiency, the degree to which the service functions correspond with the users operation goals, and the degree to which the service model that reflects the users needs matches with user expectations. This criterion assesses whether a software product allows users to suitably solve real problems. Usefulness implies that a software product has practical utility, which partially reflects how closely the product supports the users own task model. Usefulness therefore depends on the features and functionality offered by the software product

   .

   Information Reliability :

   This is an evaluation criterion to assess the degree to which the contents of the served information match the users information needs, as well as the source reliability.

   Business intelligence (BI) services such as Google Analytics and Micro strategy technology intelligent (TI)

   service such as InSciTe provide information through various service models. In this process, the usefulness of the service model is dependent on the functional quality of the relevant service and the information quality provided by the service model. Evaluation factors for service usefulness can be summarized as in the table1

   Table 1. Evaluation factors for Service Usefulness

   	Functional Quality Factor	Information Quality Factor
   Efficiency	Is it possible to efficiently carry our the desired operation	Is it possible to provide the information in the format desired by the users ?
   Effectiveness	Does the operation effectively provide the desired operation results ?	Does accurate information that the user wants exist, and can it be served?
   Satisfaction	Is performing the operation easy, and is it satisfactory?	Can the provided information be individualized?

   The second evaluation factor of the model of information service quality is information reliability. The information reliability evaluation factor consists of evaluation criteria and metrics that include factors such as accuracy, fullness, and argumentation as in the table2.

   	Reliability Factor
   Accuracy	To what extent can the accuracy of the served information be trusted?
   Fullness	Is the served information sufficient?
   Argumentation	Is it possible to confirm the source and validity of the served information?

3. Analytics Services

   Recently, core techniques such as semantic web, text mining, and data mining have undergone significant developments. These techniques have begun to be applied to technology intelligence and business intelligence for companies who wish to analyze business opportunities such as technologies emerging in the near future.

   Technology intelligence (TI) is an activity that enables companies to identify technical opportunities and threats that could affect the future growth and survival of their business, whereas business intelligence (BI) can be defined as the ability for an organization to take all its capabilities and convert them into knowledge. BI mainly aims to support better business decision-making and can also be called a decision support system.

   1. Google Analytics

      Wikipedia describes that Google Analytics is a service offered by Google that generates detailed statistics about the visits to a website. The product is aimed at marketers as opposed to webmasters and technologists from which the industry of web analytics originally grew. It is the most

      widely used website statistics service. Some major features are extracted from its official website as below.

      Content Analytics

      Content reports help you understand which parts of your website are performing well, which pages are most popular so you can create a better experience for your customers.

      Figure 2 : (official web site) Google analytics Social Analytics

      The web is a social place and Google Analytics measures success of your social media programs. You can analyze how visitors interact with sharing features on your site and engage with your content across social platforms

      Mobile Analytics

      Google Analytics helps you measure the impact of mobile on your business. Additionally, if you build mobile apps Google Analytics offers Software Development Kits for iOS and Android so you can measure how people use your app.

      Conversion Analytics

      Find out how many customers you're attracting, how much you're selling and how users are engaging with your site with Google Analytics' range of analysis features.

      Advertising Analytics

      Make the most of your advertising by learning how well your social, mobile, search and display ads are working. Link your website activity to your marketing campaigns to get the complete picture and improve your advertising performance.

      Figure 3 : Real time reporting

   2. InSciTe Adaptive

      With regard to the trends in TI, Korea Institute of Science and Technology Information (KISTI) has been developing an advanced analytics service called InSciTe Adaptive that is based on text mining and semantic web technologies. InSciTe Adaptive includes three major parts: user adaptive features, technology focusing analysis, and agent focusing analysis.

      1. User Adaptive Features

         InSciTe Adaptive includes three user adaptive features: user modeling, insight-making function and auto-generated report. The user modeling is the first part of the InSciTe Adaptive service and consists of four levels of questions. By analyzing the answers of these questions, the system classifies a user into a group and suggests a suitable starting service. The insight-making function is applied to each service in InSciTe Adaptive. The system analyzes a users service usage pattern and suggests related technology or other services, including a user-customized final report, which may also be useful to the user.

         User Modeling

         The user modeling process consist of five steps: (a) key category selection, (b) constitution element selection, (c) constitution function decision, (d) service decision, and (e) user group decision. In the first and second steps, the system requests the user to manually select options to determine the users intention. For the precise recognition of the users purpose, the system supports the users optimized selection based on mapping information constitution elements in the second step and constitution functions in the third step.

         Insight-Making Function

         A decision support system can only implicitly support business or organizational decision-making activities, not explicitly give any conclusions or insights from data analysis. For the next generation of TI system, we have been developing a real-time insight-making system.

         Auto-Generated Report

         InSciTe Adaptive gives a user an auto-generated report by tracking the users behavior pattern. The service flow customized by the users intention affects the structure of the automatic report in real time and makes a more professional report using the insight function.

         Figure 4 : User Adaptive and adaptive features and Auto generated report

      2. Technology Focusing Analysis

         InSciTe Adaptive includes three technology focusing analysis services: technology trends, core elementary technology, and convergence technology. Each service focuses on technology analysis and the relationships between related technologies.

         Technology Trend

         The technology trends service analyzes technology emergence status, development speed and forecasts when a technology will achieve maturity. The technology trends service is based on three analysis models: technology life cycle discovery (TLCD) model, technology maturity forecast (TMF) model, and emerging technology discovery (ETD) model. The TLCD model determines the emerging phase of a specific technology and the TMF model forecasts the speed of development of a specific

         technology. The ETD model decides whether a specific technology is an emerging technology.

         Element Technology

         The core elementary technology service analyzes various elementary technologies of a specific technology. Elementary technologies are extracted separately using papers, patents, and web resources. Because companies usually focus on applying patents while universities focus on publishing papers, searching for elementary technologies using papers and patents creates completely different results.

         Each elementary technology has a relative share ratio and this service evaluates the degree of emergence for each elementary technology.

         Convergence Technology

         The convergence technology service discovers candidate technologies for convergence, for instance Augmented Reality + Car = Smart Car. Convergence technologies are extracted based on the degree of shared element technologies. If one technology shares many of the same element technologies with another technology, they can be converged in the future. The convergence technology service is a typical forecasting service that analyzes the current status of each technology and predicts a blueprint of the future. In addition, the service evaluates each candidate convergence technology based on the convergence proportion between candidate technologies and search keywords given by users.

      3. Agent Focusing Analysis

         InSciTe Adaptive includes two agent focusing analysis services: agent level and agent partner. In this service, an agent refer to a company, university, or institution. Each service focuses on agent analysis and the relationship between agents and technologies.

         Agent Level

         The agent level service provides the relative rank of organizations and nations for a given technology. The service analyzes two dimensions, commercial and academic calculated by the amount of patents and papers, respectively. In addition, the service evaluates agents based on an overall ratio, which summarizes both dimensions. The service also provides a list of technologies of the selected organization or nation. Users of the service understand which agent concentrates commercially or academically on a given technology.

         Agent Partner

         The agent partner service provides collaborating or competing organizations. Organizations can have both types of relationships because they develop many kinds of technologies and products. If there are two organizations and one produces LCD and tablet PC while the other

         produces tablet PC, then this organization can collaborate in terms of LCD for tablet PC, but they can compete in terms of tablet PC. This service analyzes the collaborating and competing aspects between organizations for technologies and products. The services users know an

         organizations relationships and the technologies and products that form these relationships.

         Figure 5 : Technology Trend and Agent level

4. SERVICE ASSESSMENT AND COMPARISON

   The information service quality model defined for the evaluation of the quality of information systems includes the metrics summarized in Table3. Consequently, the info service quality model contains service usefulness and reliability evaluation factors. Service usefulness is further classified into functional quality and information quality. Functional quality is related to the quality of the usage of model functions, and information quality is a quality factor. For the service usefulness evaluation factor, efficiency, effectiveness, and satisfaction are considered as the quality- in-use properties from the users perspective. Service reliability includes factors such as accuracy, fullness, and argumentation; these are also information quality properties from the users perspective and they rely on the users pre-existing knowledge and experience. As given in Table 3, the usefulness evaluation factor contains eleven and six evaluation criteria for functional and information quality evaluations, respectively. The service reliability evaluation factor contains four evaluation criteria.

   Table 3 : Evaluation model for information service quality

   Eval Element		Eval Factor	Eval. Criteria
   Service Usefulness	Functional quality	Efficiency
   		Effectiveness
   		Satisfaction
   	Information quality	Efficiency
   		Effectiveness
   		Satisfaction
   Service reliability		Accuracy
   		Fullness
   		Argumentation

   • Minimal action

   • Operability

   • Feedback

   • Navigability

   • Time behavior

   • Consistency

   • Accurateness

   • Completeness

   • Attractiveness

   • Flexibility

   • User guidance

   • Information type

   • Report type

   • Information accuracy

   • Information completeness

   • Information navigability

   • Information individualization

   • Timeliness

   • Sufficiency

   • Proof

   • Confirmation

   For the tasks of service evaluation and comparison, six experts in charge of strategic planning in the government, research & development (R&D), and business domains assessed and compared two analytics services by testing and answering 21 questions, shown in Table 4.

   Table 4 : Questions list for the Evaluation Tasks by Experts

   Criteria	Questions
   Functional Quality of Service Usefulnss	Minimal action: Is it possible to achieve the goal by performing the minimum number of actions? Operability: Are adequate operational functions (e.g., zoom in, zoom out, and move to center) provided? Feedback: Is an adequate system reaction provided in response to the users action? Navigability: Is it formulated for the user to easily use the desired service? Time behavior: Is the time taken to carry out the functions and the loading time adequate? Consistency: Are the interface components consistent and is the meaning sufficiently implied to the user?
   	Accurateness: Are the operation results reliable? Completeness: Do any errors occur while carrying out the operation, and can the operation be carried out completely?
   	Attractiveness: Is the environment sufficiently attractive when carrying out the operation? Flexibility: Can the user change the interface according to his or her individual preferences? User guidance: Can a user easily find help when curious about the usage of a feature or function, and is the help content adequately useful?
   Information Quality of Service Usefulness	Information type: Is it easy to recognize the desired information by using visualization that suits the informations characteristics? Report type: Is the report or summary information provided in the format requested by the user?
   	Information accuracy: Does the served information match the users information purpose and needs? Information completeness: Is there any inconsistency in the information provided and that required. Can the information provided be intuitively understood? Information navigability: Is it possible to search the information easily and quickly, and is it possible to obtain an overall understanding of the information?

   	Information individualization: Does it support individualization of served information according to the users preferences?
   Service reliability	How accurate is the served information at the time when the transaction is executed according to the users request?
   	How completely and sufficiently does the served information match the users request
   	Does it serve evidence documents or basic information? Can the user verify the source of the served information?

   The result of the evaluation reveals the strong points of each service (marked in bold), as shown in Table 5. The Avg. scores for InSciTe Adaptive and Google Analytics are the mean values from the absolute evaluation of six experts who are in practical fields, such as CEOs of companies or researchers in charge of strategic planning at government or public institutions. Absolute Evaluation means that the experts assessed the two target services at a high level, i.e., from the viewpoint of the evaluation standard model, the Information Service Quality Model, as proposed in Section 2.

   Table 5 : Service evaluation result by experts

   Consequently, the two analytics services have the same level of information service quality in that the Total Score for each service is almost equivalent. The results are similar because while Google Analytics is generally strong in terms of functional quality criteria, InSciTe Adaptive gets a higher score for informational quality criteria. Because Google Analytics is a type of business intelligence service to help businesses survive and grow, the approaches are mainly based on statistical methods and quantitative measurement. Moreover, it has the strong advantage of a complete system including a user interface, visualization toolkits, and well designed functions developed with a large amount of the companys capital investment. In contrast, InSciTe Adaptive aims to capture the emerging technologies in the R&D strategic planning field. Because the semantic web, text mining, and information extraction technologies of InSciTe Adaptive can produce higher value-added information from raw materials such as scholarly papers, patents, and web articles, experts hold the information quality generated from the serial process of the automatic intelligent system in higher regard. Finally, all of the services show a similar level of reliability in terms of the service reliability criteria (see Figure 6).

   Figure 6 : Comparison of characteristics between Analytics services.

5. CONCLUSION

   In this paper, we have proposed an evaluation model for information service quality from the users perspective and compared two major analytics services by assessing and comparing their service quality. This evaluation model focuses on applying analytics services in the areas of business intelligence and technology intelligence. The information service quality model contains service evaluation factors to measure usefulness and reliability, while the final evaluation metric has 22 evaluation criteria

   to cover the full range of information service quality assessment.

   The result of the evaluation and comparison shows the characteristics of each service from the users viewpoint. Google Analytics, as a business intelligence service, generally is strong in terms of functional quality, whereas InSciTe Adaptive, as a technology intelligence service, shows a higher score in informational quality. The reasons for this are that Google Analytics is mainly based on statistical methods and quantitative measurement, and has a strong advantage of an industrially developed system, while InSciTe Adaptive captures emerging technologies by using high-level technologies such as semantic web, text mining, and information extraction, producing higher value added information from raw data.

   This evaluation model is useful for assessing and comparing the information service quality of the various analytics systems in practical business and research domains. We intend to evaluate other practical systems in the areas of business and technology intelligence by using this evaluation model and to find the characteristics of each service from the users viewpoint.

6. REFERENCES

1. Jeong, D.H, Hwang, M., Kim, J., Song, S.K., Jung, H., Peters, C., Pietras, N., Kim, D.W.: Information Service Quality Evaluation Model from the Users Perspective, The 2nd International Semantic Technology (JIST) Conference 2012, Nara, Japan, 2012.

2. ISO 9126:1-3 : Software Quality Model.

3. ISO 25000 : Software Quality Requirement and Evaluation.

4. ISO 14598 : Information Technology – Software Product Evaluation.

5. ISO 9000 : Quality Management.

6. ISO 14001 : Environmental Management.

7. Quality of Service (W3C), Home page at, http://www.w3.org/Architecture/qos.html, viewed September 25 2012.

8. Saroja, G., Sujatha, G.: Application of total quality management to library and information services in Indian open universities, http://www.col.org/forum/pcfpapers/saroja.pdf, viewed July 15 2012.

9. Caruana, A.: Service loyalty: The effects of service quality and the mediating role of customer satisfaction. in: European journal of marketing, Vol.36, 2002.

10. Vedder, A., Wachbroit, R.: Reliability of information on the internet: some distinctions. in: Ethics and Information Technology, 2004.

11. ISO 9241-11 : Ergonomic Requirements for office work.

12. ISO/IEC 9126:4 : Software Engineering Product Quality Quality in use metrics.

13. ISO 15489 : Records Management.

14. Google Analytics, Home page at, http://www.google.com/analytics/, viewed September 25 2012.

15. MicroStrategy, Home page at, http://www.microstrategy.com/, viewed August 25 2012.

16. Jeong, D.H., Kim, J., Hwang, M., Lee, S., Jung, H.: User- centerd Mobile Technology Intelligence System. Proceedings on Advanced Computer Science and Technology (AST) 2012, Beijing, China, 2012.

17. CUBIST: Your Business Intelligence, Project home page at, http://www.cubist-project.eu/, viewed August 25 2012.

18. IARPA: Be The Future, FUSE project home page at, http://www.iarpa.gov/solicitations_fuse.html, viewed August 25 2012.

19. Technology Intelligence, Wikipedia at, http://en.wikipedia.org/wiki/Technology_intelligence, viewed August 25 2012.