Telco 2.0™ Research

Summary: Customer experience is fast becoming the new mantra for operators across the globe.  But while improvements are being made in broadband speeds, customer experience and so forth, app latency and errors have largely gone ignored. This is despite the fact that both are important drivers of customer experience and of app developer profitability. Find out the relationship between latency and app errors and our analysis of those operators that are performing well and those that are ‘lagging’ behind. (November 2015, Executive Briefing Service)

 

    

Below is an extract from this 18 page Telco 2.0 Report that can be downloaded in full in PDF format by members of the Executive Briefing Service here. To find out more about how to join or access this report please see here or call +44 (0) 207 247 5003.

Executive Summary

• STL Partners’ analysis shows that while latency and app errors are only weakly correlated across the whole of Europe, once outlying operators (SFR, Wind and those in Germany) are removed, there is a strong positive correlation between the two: as latency increases so do app errors.

• Intuitively, this makes sense: apps ‘time out’ waiting for responses causing errors and crashes.

• Latency and app errors both negatively affect customer experience – customers are more likely to abandon apps as responsiveness and error rates increase: 
  
  • 48% of users would uninstall or stop using an app if it regularly ran slowly.
  • 53% of users would uninstall or stop using an app if it regularly crashed, stopped responding or had errors.

• Historically, customers have tended to hold the app developer responsible for errors (55% of users blame the app for problems and only 22% the mobile operator) but mobile operators have a significant impact on how quickly an app runs and how likely it is to experience an error and, as understanding of the operators’ role grows, users may well use this as a criterion when selecting their mobile service provider.

• Performance among Europe’s operators for app latency and errors varies widely:
• • The worst-performing operator in Europe (3 Italy) experiences over three times the amount of requests with poor latency compared to the best-performer (Bouygues Telecom).
  • The worst-performing operator in Europe (O2 Germany) results in over twice the number of app errors than the best-performer (Bouygues Telecom again).

• Improving customer experience is rapidly becoming a mantra of operators globally and for several players (in Europe at least) improving latency performance and reducing app errors caused by latency and other factors should be a key priority. For without improvement, poor performing operators will find themselves at a disadvantage and may struggle to retain existing customers and recruit new ones. 

Introduction

Key objectives

Network latency is a key driver of user experience. In applications as diverse as e-commerce, VoIP, gaming, video or audio content delivery, search, online advertising, financial services, and the Internet of Things, increased latency has a direct and negative impact on customers. With higher latency, customers fail to complete tasks, leave applications, or experience application errors. This, in turn, results poorer core business KPIs for the application provider – lower ratings, fewer subscribers, or reduced advertising fees.

As we showed in a recent report titled Mobile app latency in Europe: French operators lead; Italian & Spanish lag, with the modern Internet dominated by flows of small packets on fast networks, latency accounts for the biggest share of total load times and tends to determine the actual data transfer rates users see. And, as web and mobile applications increasingly consist of large numbers of requests to independent ‘microservices’, jitter – the variation in latency – becomes a more significant threat to the consumer experience. Furthermore, we benchmarked major European mobile network operators (MNOs) on average latency and the rate of unacceptably high-latency events (over 500ms).

In this second report on latency, which again uses data provided by app analytics specialist Crittercism, we look at the rate of app errors – evidently, something that could not impact user experience more directly – and its correlation with both latency, and the rate of unacceptable high-latency events. We explore how often apps fail across the same set of MNOs, test if latency is a driver of app errors, and then conclude whether or not our theory that it is a real driver of consumer experience is correct.

Source data and methodology

Our partner, Crittercism, collects a wide variety of analytics data from thousands of mobile apps used by hundreds of millions of people around the world in their every-day lives and work. To date, the primary purpose of the data has been to help app developers make better apps. We are now working with Crittercism to produce further insights from the data to serve the global community of mobile operators.

This data-set includes the average network latency experienced at the application layer, the percentage of network requests above 500ms round-trip time, the 5th and 95th percentiles, and the rate of application errors. All of these data points are useful in trying to understand the overall experience of customers using their mobile apps, and in particular the delays and problems they’ve experienced such as long screen wait times and applications failing to work.

We showed in the previous report how the longest round-trip delays or ‘app-lags’ (i.e. those over 500ms) are the most important KPI to look at when trying to understand customer experience. This is firstly because people really notice individual delays of this length. For people used to high speed broadband, it’s like going back to narrowband internet – it seems incredibly slow!

Importantly though, in modern apps, the distribution of delays is even more significant, as each app or web page typically makes multiple requests over the internet before it can load fully – and each of these requests will suffer some form of delay or latency.

A detailed explanation of this and of the collection methodology is available in the first report.

 

The Impact of latency on app errors

First glance: a positive correlation overall, but a weak one

The following chart shows the error rate per 10,000 app requests, plotted against the percentage of requests over 500ms round-trip time, by carrier. Each dot represents a week’s performance and we’ve looked at 12 weeks of data from 20 operators, from the week of 03/08/15 to the week beginning 19/10/2015. The hypothesis being that the more requests with unacceptable latency there are, the more app errors, because apps ‘time-out’ or key requests are not fulfilled in time causing an app error or, worse, a crash.

Figure 1: Latency and errors for the top 20 European MNOs over the last 12-weeks appear correlated, but there are some important outliers

Source: STL Partners, Crittercism

At first glance, there appears to be only a weak positive relationship between latency and error rates but there does seem to be a natural grouping found between the two hand-drawn dotted lines on the chart with the weeks above the upper boundary (potentially) being outliers, in which at least one other factor is driving application errors up.

The lower boundary seems to represent the underlying rate of app-errors that occur when there are no latency issues (between 20 and 50 errors per ten thousand plus an increasing error rate as higher latency kicks in. For example, when 10% of requests experience latency above 500ms, the minimum error rate is around 30 per 10,000 requests, rising to 50 at the 35% mark.

 

To access the rest of this 18 page Telco 2.0 Report in full, including...