View towards Synthetic Monitoring using HTTP Archive

View towards Synthetic Monitoring using HTTP Archive

Mr. Siddhesh Vaidya

1. Information Technology Automation Test Engineer

   Apmosys Technologies Pvt. Ltd.

   Navi Mumbai, Maharashtra

   Mr. Prabhat Padhy MS (BITS Pilani) VP, Technology

   Apmosys Technologies Pvt. Ltd.

   Navi Mumbai, Maharashtra

   Abstract To give customers a top-quality digital experience, its important to make sure that your applications are always working properly. In that way, synthetic monitoring can help to confirm applications that are performing as intended, and if theyre not, it will help you quickly figure out whats going on. Although synthetic monitoring have become a crucial part of application performance monitoring and it also can be analyzed using various categories such as Availability Monitoring, Web Performance Monitoring and Transaction Monitoring.

   Keywords Synthetic Monitoring, Performance Monitoring, Availability Monitoring, Transaction Monitoring.

   1. INTRODUCTION

      Monitoring is the way to test and verify that end-users can interact with an application as expected. To fulfil needs of users, the monitoring system should be given charge to analyse, diagnose the application. Synthetic Monitoring is an motivative and active approach to test website or a web service by working on visitor requests to test for the availability, performance, and functions. Synthetic Monitoring is a term applied to different types of websites and also on different server monitoring solutions. Synthetic monitors work like bots (an autonomous program on the internet or another network that can interact with systems or users) that connect to websites, web services, APIs, and servers to verify availability, function, and performance using a network of checkpoints external to the websites own servers from different parts of the networks.[2]

      Synthetic Monitoring is also known as Synthetic Testing which monitors the performance of the application and emulates the path the user might take when engaging with an application and uses the scripts to generate simulated user behaviors for different scenarios, geographic locations, device types and other variables. After collecting and analyzing valuable performance data, a synthetic monitoring solution can be the following:

      • Give your crucial insight into how well your application is performing.

      • Automatically keeling tabs on application uptime and tell you how your application responds to typical user behaviour.

      • Zero in on specific business transactions For eg: By alerting you to issues users might experience while attempting to complete a purchase or fill out a web form.

   2. CATEGORIES OF SYNTHETIC MONITORING Synthetic Monitoring can be categorized as follows:

      1. Availability Monitoring

      2. Web Performance Monitoring

      3. Transaction Monitoring

      1. Availability Monitoring:

         Availability Monitoring is also known as uptime monitoring which is the utmost introductory form of monitoring and at its root means that the site or service can be accessible, but some definitions expands that definition to include checking that the website or service functions properly as well. In simple terms, an availability monitor sends a Get or a ping and checks the response for a successful code. More Advance Availability monitors may check for particular content, response time, attempt to validate a user, perform an API call, or verify a DNS or SSL certificate entry.

      2. Web Performance Monitoring:

         Web Performance Monitoring takes Availability Monitoring to the next level and may check the page load speed, show performance on an element-by-element basis, and gives report on frontend and backend response times. Generally, Web Performance Monitoring uses a native browser to load the returned content for Real Browser Monitoring and Full- Page Checks. Loading the returned content into an actual browser window can capture crimes that does not appear initially in the first response. Issues captured by Web Performance Monitoring include checking for content, crimes and slow responses caused by third-party content, and slow responding CDNs and databases. Web Performance Monitoring also includes multi-browser monitoring and mobile monitoring or can also be called as Mobile (Hand- held) device monitoring.

      3. Transaction Monitoring:

      Transaction Monitoring takes Synthetic Monitoring to more higher level than the Web Performance Monitoring. The checkpoint terminal runs a script that can login to a service, complete forms, purchase products, and respond to system prompts. Transaction Monitoring tests the happy paths to ensure that visitors (users) can complete their tasks.

   3. STEPS INVOLED IN PERFORMING SYNTHETIC MONITORING

      The steps involved in executing synthetic monitoring are:

      • The monitoring system selects a checkpoint to do the check and sends the instruction to the check point.

      • The checkpoint initiates the contact, checks the response and proceeds grounded on the type of examining the monitor requires,

      • The checkpoint reports its results and findings back to the monitoring system.

      • The system records the information for reporting, if the check turned in an error, the service immediately requests a new test from another checkpoint. If the checkpoint reports the similar error (defect), the system declares the error is confirmed.

      • The systems sends out an alert (signal) for the confirmed error based on the escalation settings and duty schedules.

        Depending on the type of test, this process may occur as frequently as every minute or up to once an hour. Most brands check for availability every 1 minute, performance every 5 minutes and transactions every 15 minutes.

   4. WHY THERE IS NEED TO USE SYNTHETIC MONITORING?

      Any company or brand that has content or services through a network or the web needs Synthetic Monitoring to guard productivity, revenue, and reputation. When availability and performance are mission critical, a brand cant watch for end-users to notify them of an issue. By the time end users begin to complain, it's too late. The brand has taken on damage because of the issues, and until the matter is resolved, the brands reputation and revenue suffer.

      • Faster Problem Resolution:

        Content and supporting technology change continuously and the slightest change to the infrastructure can bring any brand to its knees if not everything goes as planned. Ungood performance and outages appear quickly when using Synthetic Monitoring. Instant access to detailed reports about a blunder (sometimes with screenshots), allow teams to retort quickly. Often teams can find the foundation cause and implement a fix before the highest users become aware.

        party vendors in command of their products availability and performance.

      • SLAs and Synthetic Monitoring:

        Besides performance and function, companies use Synthetic Monitoring to validate SLAs (Service Level Agreement). Detailed reports allow companies the facility to point the precise availability percentage for any given period. Some companies use the reports to produce proof to their customers that they need met the obligations of their SLA, other companies use the reports to trace the supply of internet sites and services on which they rely.

   5. WHY DO YOU NEED SYNTHETIC MONITORING?

      The need is for following purposes:

      • Fix performance issues bfore they affect your end users.

      • Benchmark your websites.

      • Test your scaling capabilities.

      • Test new product features before launching.

      • Monitor third party APIs and critical business processes.

      • Ensure SLA (Service Level Agreement) compliance.

      • Reduce MTTR (Mean Time to Recovery)

   6. LIST OF TOP SYNTHETIC MONITORING TOOLS[1]

      Following are the tools used for synthetic monitoring:

      1. Datadog:

         Datadog is best for API and browser tests that will give proactive and end-to-end visibility.

         • Alerting:

           Alerts should happen before end users experience problems with a site or service. Although some passive

      2. AlertBot:

         Fig 1: View of DataDog Tool

         approaches to monitoring include alerting, Synthetic Monitorings proactive approach works well for alerting a team to issues early. Synthetic Monitoring can check and verify many details and identify situations before they become problems such as reduced performance from a database server.

         • Tracking third-party content:

         Third party content comes in many alternative forms, e.g., advertising, payment management systems, CDNs (Content Delivery Networks), and analytic solutions. The performance of those third-party vendors affects the performance of the host site. Using an independent monitoring service allows brands to carry their third-

         AlertBot is among the top players in the synthetic monitoring industry and is ideal for organizations of all sizes.

         Fig 2: View of Alert Bot Tool

      3. SolarWinds Pingdom:

      Best for ultimate visibility and enhanced troubleshooting through a combined solution of synthetic monitoring and RUM.

      Fig 5: View of Uptrends

      vi. SmartBear AlertSite:

      est for monitoring of websites, web apps, APIs from all over the globe & in private networks.

      Fig 6: View of SmartBear AlertSite

      iv. Sematext:

      Fig 3: View of SolarWinds Pingdom

      vii. Dynatrace:

      Best for digital experience monitoring.

      Best for IT system monitoring tools for DevOps.

      v. Uptrends:

      Fig 4: View of Sematext

      1. AppDynamics:

         Fig 7: View of Dynatrace

         Best for web performance and website monitoring.

         Best for intelligent cloud agents for monitoring business transactions & performance.

         Fig 11: View of eG Innovation

      2. Site 24*7:

      Fig 8: View of AppDynamics

   7. NINE REASONS TO USE SYNTHETIC MONITORING

      Here are some of the top reasons IT shops rely on synthetic monitoring.

      Best for DevOps and IT Operations teams to monitor the performance.

      1. Monitor application availability 24 x 7, even during off hours.

      2. Proactive notification of application availability issues.

      3. Identify reachability problems for remote sites.

      4. Understand the impact of third-party services have on customer-facing applications.

      5. Monitor performance and availability of SaaS applications.

      6. Test business-to-business web services that use SOAP, REST, or other web services technologies.

      7. Monitor the availability of critical databases queries.

      8. Measure service-level agreements (SLAs).

      9. Baseline and analyze performance trends across topographies.

      1. New Relic:

         Fig 9: View of Site 24*7

   8. BENEFITS OF SYNTHETIC MONITORING[3]

      1. Find and fix issues before they impact your end users:

         Synthetic monitoring helps you emulate user interactions

         Best for providing the real-time view of your operational data in one place.

         Fig 10: View of New Relic

         1. eG Innovations:

         Best for end-user experience monitoring.

         and run them as tests from global monitoring locations or from behind your firewall. Synthetic monitoring proactively watches over your APIs, websites, web, mobile and SaaS applications, even during the low- traffic periods, and alerts your operations team in case of performance degradation or availability issues. So you get enough bandwidth to spot the matter, engage subject material experts, find the foundation cause and fix issues before they impact the tip users.

      2. Baselining and Benchmarking:

      Synthetic monitoring gives you the ability to monitor your APIs and applications at frequency and location(s) of your choice, always. Over time, this monitoring data can be used for baselining your applications performance, identifying areas of improvement, and developing performance improvement strategies. You can also use synthetic monitoring to benchmark your applications availability and performance with your historic self or against competitors.

      	3.	Prepare for the peak traffic season or a new market:
      		Synthetic monitoring gives you a novel ability to observe
      		the realm of your website or application which doesnt
      		have real user traffic yet. Imagine a brand-new marketing

      campaign driving traffic to a replacement area of the application. Synthetic monitoring allows you to proactively simulate traffic there to area and facilitate you ensure availability and performance. The other use case is when you are launching your services in a new geography. Synthetic monitoring enables you to test your applications performance from that geography and address performance issues, if any, before your real end users encounter them.

      1. Monitor complex transactions and business processes:

         Only checking availability and uptime of your APIs and operations isn't sufficient when you are seeking to deliver a high-quality operation performance. Synthetic monitoring allows you to emulate business processes or user transactions such as logging in, searching, filling form, adding items to cart and check out etc. from different topographies, and monitor their performance (whether efficient or not). You can also compare performance statistics between topographies and steps (sequences) in the transactions and formulate your performance improvement plans.

      2. Measure and Adhere to SLAs:

         Service level agreements are critical to modern business. Regardless of what side of the SLA you're, measuring and adhering to the agreed upon level of service is helpful for both client and vendor parties. For vendors, synthetic monitoring helps understand the provision and performance limitations of the appliance better. Armed with this data, vendors can prepare realistic service level objectives and avoid unforeseen penalties.

      3. Hold your third-party vendors accountable:

         Modern applications depend upon multiple 3rd party components for functionality and data. The foremost common 3rd party integrations are CDNs, payment processing solutions, site search and suggestions plugins, business intelligence and analytics solutions etc. Synthetic monitoring enables the consumers of such 3rd party services to watch service level objectives, performance degradations and unavailability incidents to carry the vendors accountable.

      4. Test from the end users perspective:

      Modern applications depend upon multiple 3rd party components for functionality and data. The foremost common 3rd party integrations are CDNs, payment processing solutions, site search and suggestions plugins, business intelligence and analytics solutions etc. Synthetic monitoring enables the consumers of such 3rd party services to watch service level objectives, performance degradations and unavailability incidents to carry the vendors accountable.

   9. LIMITATIONS ON SYNTHETIC MONITORING Synthetic testing is a useful approach to Digital Experience Monitoring, but it has limitations. Synthetic onitoring is just thatsynthetic. The scripts on which it relies only emulate

      user experience. They do not measure actual end user experience. So synthetic testing cant help IT understand what users actually see when they interact with an application.

      In addition, synthetic monitoring cant see how device resources and health affect application performance. Is the application truly slow? Or is the problem that the users device lacks sufficient memory or CPU?

      Last, synthetic testing scripts have no insight into the characteristics of the end user who is experiencing poor application performance. Their identity. Role. Department. Location.

   10. SIX REASONS FOR WHY NOT TO USE SYNTHETIC MONITORING

       While synthetic monitoring helps IT identify and resolve general applications performance issues, it does not help companies address some key IT and business use cases.

       1. Resolve end user complaints: When service desk staff receive an end user complaint, synthetic monitoring products would tell them nothing about what the end user was doing or experiencing.

       2. Troubleshoot device problems: Desktop services teams need insight into the performance and health metrics of the laptops, PCs, tablets, and mobile devices used by their workforce. Synthetic testing products lack this visibility.

       3. Determine how device performance affects application performance: Under-resourced devices negatively affect application performance. Without insight into device performance, application owners cant isolate the cause of application complaints.

       4. Analyze performance for every business-critical app: Companies rely on dozens of business-critical applications. Creating and maintaining scripts for synthetic testing is time-consuming, even for one app.

       5. Validate the impact of change across the estate: When IT makes changes to devices, applications, or infrastructure, they must validate the impact of those changes on actual end user experience to determine the impact. Comparing metrics from synthetic testing wont provide the full picture.

       6. Improve workforce productivity: To measure the impact of IT on workforce productivity, IT and business execs need to assess every application in use in the enterprise. Not just one.

   11. WHAT ARE THE THINGS THAT SYNTHETIC MONITORING WILL ANSWER?

       • Is my website up?

       • How fast is my site at this moment?

       • Are transactions working?

       • Are third party components still operating?

       • If theres a slowdown or failure: Where is it?

   12. MONITORING DATA USING HTTP ARCHIVE (HAR)

       The HTTP Archive format, or also called as HAR, is a JSON

       i.e. JavaScript Object Notation formatted archive file format for logging of a web browser's interaction with a site. The common extension for these files is .har.[4]

       The specification for the HTTP Archive (HAR) format defines an archival format for HTTP transactions that can be used by a web browser to export detailed performance data about web pages it loads. The specification for this format is produced by the Web Performance Working Group of the World Wide Web Consortium (W3C). This document was never published by the W3C Web Performance Working Group and has been abandoned.

       The data can be monitored by automating the URL and then capturing the complete details and viewing it in HTTP archive viewer tool which is leaded by the extension.

       Following are the details of the HTTP Archive:

       Fig 12: Log Details after executing the code

       After executing the script, the data for the loaded page or URL is getting fetched in the file format called as HAR. Its representation is depicted in Fig. 12.

       Fig 13: Inspection of code after loading log into HAR viewer

       The above Fig. 13 shows representation of HAR content inspected in HTTP Archive loader with different time frames required to load the element in milliseconds.

       Fig 3: Detail of link after clicking on it

   13. CONCLUSION

       With Synthetic Monitoring, a brand can proactively monitor web and network assets and receive alerts when things don't work as expected. Because of the flexibility offered from Synthetic Monitoring, a brand should consider Synthetic Monitoring their first line of defense to guard availability, performance, and function.

   14. REFERENCES

[1] https://sematext.com/blog/synthetic-monitoring-tools/

[2] https://www.uptrends.com/what-is/synthetic-monitoring

[3] https://smartbear.com/learn/topic-area/benefits-of-synthetic- monitoring/

[4] https://toolbox.googleapps.com/apps/har_analyzer/

AUTHORS:

1. Mr. Siddhesh Vaidya

   M. E. Information Technology Automation Test Engineer Apmosys Technologies Pvt. Ltd.

2. Mr. Prabhat Kumar Padhy MS (BITS Pilani)

VP, Technology

Apmosys Technologies Pvt. Ltd.