After months of editing and collaboration work on E.812, which also included contributions from SpeedChecker, ITU approved the recommendation.
You can download the full text here.
Are you looking at a crowdsourcing solution that follows ITU standards?
SpeedChecker is an ITU associate member of Study Group 12. The solutions we build follow ITU recommendations. See our crowdsourcing solution presentation.
Between 19th and 24th August 2018 over 2 million visitors arrived in Mecca for Hajj. This annual pilgrimage to the holiest city for Muslims is associated with the Prophet Mohammed who is said to have lead his followers there before consecrating it to Allah. It is considered a religious duty for all adult worshippers who are able to undertake this pilgrimage at least once in their lives. This number of visitors more than doubles the usual 1.5 million population of Mecca causing almost unimaginable challenges to the city’s infrastructure. In this article we discuss just one of these challenges : mobile Internet speed and access. It is hard to imagine how the infrastructure could cope with the huge increase in demand.
During Hajj in 2017 mobile data demand nearly doubled compared to 2016. Although an increase of 60-70% was anticipated the 100% jump was a surprise. This was attributed to the increase in popularity of YouTube and Snapchat. Despite the increased demand, 99% of calls were successful and 23,000 Terabytes of data were consumed. According to the UN Sustainable Development Goals report published in ITU News from September 2017 this was thanks to the deployment of 3700 ICT specialists and 13,000 2G, 3G and 4G mobile base stations in all Hajj cities. The report does not specify which Telcos were involved. Source: ITU News.
King Salman bin Abdulaziz and the Crown Prince Mohammed bin Salman issued a directive “to do everything possible to make it easy for pilgrims to perform the rituals of Hajj”. The initiative’s objective is to allow pilgrims to communicate with their families and enable them to access the digital services available in the Smart Hajj initiative, so that they can enhance their experience and allow them to take advantage of enhanced communication services, as per a release issued by the authorities.
In particular, a number of packages provided by some of the main mobile operators offered their customers 1 Gb for 48 hours. Source: https://www.tahawultech.com
The challenge of providing adequate mobile services during a large event is not simply trying to maintain the current service levels. It is also about balancing the needs of the visitors with key service areas that are essential during the event. Consideration must be given to protecting the critical infrastructure of the region to enable it to respond to serious incidents. One way this can be achieved is to ensure there is resilience and redundancy built in to the infrastructure. Consultation with interested parties is essential to ensure that the steps agreed will meet the essential needs of all concerned. A thorough risk and threat assessment will identify where the effort is required.
It is a balance between being good hosts to the visitors and ensuring a continuity of services for the locals. Short term measures and agreements will be a great help in achieving this balance and the generous provision of 1Gb over 48 hours in Mecca is one such example. This may be the headline initiative but it is clear that much more has been done in many other areas to ensure a successful Hajj.
Mecca has an excellent 4G network covered by a number of major operators. Building on the improvements made for Hajj 2017 this has allowed them to improve the average download speed by 83% between 2017 and 2018. They will continue to improve as they roll out 5G and it is expected that this will be further improved as part of Saudi Vision 2030.
Saudi Telecom Company (STC) has been at the forefront of this with investment in FDD and TDD LTE spectrum assets. The rewards of this investment can be seen in our results which show STC outperform the other providers in our research.
Zain have also been investing in technologies that allow them to extract the best out of their infrastructure. They are also preparing for 5G rollout.
Mobily has partnered with Ericsson to deliver 4×4 MIMO and as with STC and Zain they are preparing a 5G rollout.
As the Telcos continue to improve installed and available capacity so the Internet speeds can be expected to increase.
Ahead of Hajj, Speedchecker started data collection to gather as many data points in Mecca as possible before / during / after the event. The crowd sourced data samples were collected in the field using mobile phones carried by the pilgrims to Mecca. Measurements were run on mobile networks of the top providers using Android and iOS devices. The measurements were made towards local CDN PoP based in Riyadh. The recorded results are a good proxy for the internet quality users were experiencing in Mecca on their mobile devices. During the 15 days the data collection took place, Speedchecker received over 100,000 data samples and the included stats and analysis are based on this dataset.
The results show that not only did Mecca cope with the extra 2 million visitors they exceeded all expectations. It would be reasonable to expect that speeds would decline by up to 50% during Hajj when compared to the week before or the days after. However, the speed test result reveal that the steps taken in Mecca allowed visitors and locals to enjoy an increase in speed that was continued throughout the following days. Our analysis stops after the 26th August.
The chart shown below shows the median (middle value) of Mobily Mobile, Zain Mobile, STC Mobile and STC Fixed broadband. We only have STC data from 21st August (Hajj started on 19th) and we have separated the STC Mobile tests from the STC Fixed Broadband tests. There is an unexplained drop in speed for STC mobile on the 23rd August. We have included the STC Fixed Broadband to show that the problem only affected STC Mobile customers. Despite this 50% drop from STC the overall trend during Hajj was a gradual increase in download speed.
STC mobile download speeds are more than 50% faster than either Mobily or Zain and this shows that investment in infrastructure yields positive results and benefits to the end user.
The following graph compares how the average daily median speeds of each of the providers changed before, during and after Hajj. The average shows a remarkable increase throughout Hajj and into the following days. Zain’s speeds after Hajj are faster than those from before while Mobily has returned to before Hajj speeds.
Whatever improvements and changes were made to the Telco infrastructure during Mecca the results of the download speeds show that it was a huge success.
Using mobile device GPS data we were able to map internet speeds in Mecca to a high geographic precision. Collected data were normalized and color-coded so that the fastest areas are in red and slowest in dark blue. The outskirts of Mecca which are not colored are out of scope for this study.
Mobily
As can be observed the fastest areas for Mobily are not in the center which can be attributed to increased demand from higher concentration of people.
Zain
The Zain speed map is slightly darker and corresponds with slightly slower internet speeds than Mobily. Yet the centre is faring quite well in comparison with Mobily.
STC
The STC internet speed map looks comparatively much better than Mobily and Zain and proves that internet speeds are well distributed across whole of Mecca.
The internet quality around Great Mosque is better illustrated using more detailed heat map where you can see individual measurements (which are also color coded like on previous maps). The area around the mosque has very good speeds also for Zain, which indicates Zain did not underestimate the capacity needed in the center.
| Mobily | Zain |
 |
 |
The 2 million pilgrims arriving in Mecca in 2018 provided a huge challenge to ensure that the quality of service that visitors and locals expect can be delivered and maintained. We have seen how the demand doubled between 2016 and 2017 and this increase was sure to continue in 2018.
The Saudi Arabia Initiative and the efforts and investments of the major mobile operators has ensured that the quality of the service has not only be sustained but improved. This improvement has continued at least for the few days after Hajj (we have no data beyond this). We don’t know how much of the improvement will be permanent but, with a similar commitment in 2019, we can be confident that Hajj will continue to be a Telco success.
Looking further forward we can see that the Saudi Vision 2030 has ambitious plans that should sustain this for the foreseeable future.
Speedchecker, a private company running large-scale software-based monitoring networks and DD-WRT, the most popular open-source router firmware, announce a partnership which will aim to build the world’s largest hardware probe monitoring network.
Under the terms of the partnership DD-WRT started including the Speedchecker Probe client within the DD-WRT firmware. DD-WRT users can opt-in to the Speedchecker network and get new features for their routers in exchange for providing bandwidth for Internet measurements.
As Christian Scheele from the DD-WRT development team said:
“We are pleased to be part of this partnership to not only help fund the DD-WRT development but also be part of the project which enables Internet research be conducted on a large scale across many countries that are currently not represented in existing measurement networks”.
Since the soft-launch earlier this year over 2000 users of DD-WRT have already opted-in to the network, enabling Speedchecker to cover over 80 countries for its Internet measurements. Speedchecker offers access to its network to clients such as Microsoft and Oracle, as well as researchers in organizations such as LACNIC which publish Internet topology research.
CEO of Speedchecker Ltd, Janusz Jezowicz noted:
Historically, companies always had to make a choice of either running measurements from software probes with its wider coverage but lower accuracy, or rely on hardware probes which had limited coverage. With this partnership we are able to provide global coverage for hardware probes with low costs due to end-users running the tests on their own routers and not expensive custom hardware.
After a few months of hard work we are pleased to announce a new version of our Probe API. We decided to completely rewrite the API specification to apply all the things we have learned over the last few years without breaking API access for our existing user base. We don’t plan to sunset the old API version yet, but new clients are not able to sign up for the old version.
The new version is so much better; we have made following improvements:
Easy to use
Our API is well documented including a Quickstart guide which will get you up to speed quickly so you can start running your measurements.
Reliable
We have learned a lot of lessons over the years about how to make the API more scalable . We are pleased to say the new API already supports millions of measurements running every day!
Multi-platform
As part of new API release we are offering access to our Android probes to all of our users. API users can leverage increased coverage by testing on all available platforms, or specifically target mobile probes using Platform source targeting. We will soon have an announcement about hardware probes which will be supported in the same way, without the need for code changes.
Great level of support
Being a small company, we have always taken personal care of each client and made sure our support team provides expert advice on internet performance measurements to assist clients in fulfilling their goals.
Transparent pricing
Our API access starts from 49 EUR per month.
Please check our pricing here.
On top of those improvements mentioned above the API features also got an upgrade. Based on feedback we got from our users we have improved the API methods to include:
Improved probe targeting
Our new API offers many more options on how to select probes for measurements. Our users can select probes by location (e.g. City / Country / Lat and Long coordinates), network (Network name, ASN or IP prefix) and more.
More information about probes
Using ProbeInfo properties API users can specify what information about the probe is useful for them to return with the measurement results. We have added new properties such as DNS Resolver IP, Screen size (useful for page load tests), User connection type and more.
Extended tests and new metrics
Our API supports existing measurements such as Ping, DNS, Traceroute, HTTP, Webpage load. We have also added a new measurement type – video streaming test.
Further changes to the API endpoints:
Ping
We added an option to run TCP ping.
DNS DIG
DIG command now responds with full DNS query information .
HTTP
We now have available metrics such as TTFB, TotalLatency, DownloadedBytes, TCP connect time.
HTTP GET measurement can also return full HTTP Headers and Body. This can be very useful for many scenarios such as finding out which CDN POPs are being accessed, CDN cache HIT/MISS analysis, keyword monitoring in the HTTP response. The possibilities are endless!
Webpage load
We offer all the web performance metrics you would expect and we have added a couple more: such as the number of requests the page has loaded as well as full HAR file. HAR file is very useful in getting a complete picture of the pageload performance and allows you to construct a waterfall model which we use in our CloudPerf product.
We hope all the improvements we have made will encourage you to sign up for our 7-day FREE trial.
Every startup is rightly afraid of a new competition, especially if it comes from Internet giants like Google. The stories of how Google enters the market and dominates it in a few years are not new (such as mobile OS and Android, or recently browser wars with Chrome) . In some cases Google gets a slap on the wrist or occasional $2.7 billion fine . Nevertheless, the situation is not likely to improve and Google’s dominance in search will grow into other areas, if Google decides to compete there.
This blog post hopes to give an insight into the impact of prioritizing Google funded initiative over existing players in the market, using the real numbers from specifically our small business (which I am not sure is still right to call a startup after 10 years) 😊
But before I do so, let me give you a super quick overview of what my company – Speedchecker does – we provide easy to use and accurate speed test of your internet connection. After almost 10 years we have done over 300 million tests and provided speed test technology for many other companies.
The story begins about a year ago when Google launched their own speed test featured directly in search results in USA and followed by other English speaking markets. We knew that the UK launch would happen eventually but we did not know when.
Luckily for us Google picked an open-data solution for running their speed test , noble M-Lab. M-Lab was founded by internet visionaries such as Vint Cerf and is funded by consortium of companies including Google. This choice enabled us to analyze the rollout and provided real numbers for this blog post.
M-Lab speed test data is available to download for everyone through Google Cloud (of course). By analyzing volumes of data each day, we could produce following chart:
(Number of speed tests from UK in M-Lab dataset on random days in May, June and July 2017)
As we can see, Google started the rollout on the 15th of May. We can also observe that Google did not do an immediate rollout across all the UK users but over the course of several days, the feature was introduced to more and more users in the search results.
Here is how the search results look like in the UK for one of our main keywords:
As we can see Google speed test occupies significant space on the 1st page and pushes all results below.
Here is the chart plotting user visits from Google (and Bing for comparison) before and after the Google speed test release. We can observe the drop in visitors begins after Google launches the speed test.
To better illustrate that the drop is because of ranking change and not seasonal factors, here is zoomed in data from Bing which does not show any meaningful change before/after 15th of May when Google launched.
Looking at the average drops we can estimate the loss of about 5000 visits per day from 25000 Google visits. Overall that is about 20% traffic loss from being moved from position 1 to 2.
Comparing to industry standard data e.g. by RankScience, 20% drop is quite a good result, it could be worse.
From M-Lab dataset we can also extract quite interesting insights as it contains user IP address as well. If we cross-reference user IPs seen in M-Lab data with our internal data, we can see about 5% of users use both services. We can only speculate whether it’s a good result or not, definitely for the user it is useful to get information from 2 different sources and decide what is more relevant.
From our perspective we are quite happy the Google threat is not as serious as we originally thought. Loosing 10% of our overall traffic (and 20% of Google’s) will have impact on our bottom line but we will survive. Luckily, we provide other features that user’s appreciate such as mobile apps, storing results, mapping, comparisons and more. This I believe contributed heavily to such a small drop. I have no doubt many users will favor convenience of 1 click to get result in search results directly than going to 3rd party site such as ours. Unfortunately, there is nothing we can do to compete with that and stay in business at the same time.
With Google favoring their own speedtest, M-Lab datasets are growing at a rate of almost 1 million results per day and will achieve to serve as many customers in less than a year – something we have achieved in the last 10 years. That is the power of Google search dominance.
Added support for targeting probes with state level accuracy
We have added optional input StateCode parameter to those methods:
StartDIGTestByCountry
StartHttpGetTestByCountry
StartPageLoadTestByCountry
StartTracertTestByCountry
StartPingTestByCountry
Currently only probes located in the USA can be targeted on state level. The API response will also contain StateCode (e.g. VA) and also State (e.g. Virginia) if you request CountryCode=US.
Added new limits on the number of probes you can request test from To avoid abuse we have added limits to how many probes you can request results at the same time in 1 API call. This is controlled by the probeLimit parameter which is required.
Here are the maximum values allowed for different tests.
PING – 100
DIG – 100
TRACEROUTE – 100
PAGELOAD – 20
HTTPGET – 20
If you require those limits to be lifted, please contact us.
Added web reputation services for HTTP tests
To avoid abuse and protect the probes from accessing risky content, we have added white and black lists that will be checked for all HTTP tests (e.g. HTTP GET or Page load). Following errors can happen>
StatusCode: 221
StatusDescription: Destination blocked Message: “This URL destination has been marked as risky by our web classification engine. We are unable to test this URL. ”
StatusCode: 222
StatusDescription: Destination not classified Message: “This URL destination has not been classified by our web classification engine. We are unable to test this URL at this time. Please contact support to whitelist this URL”
We have deployed few new improvements to the API.
Changes to default timeout and commandTimeout
We have changed default timeout and commandTimeout parameters to maximize the number of returned results and minimize the timeouts on probes (of course you can set any time out as you want by specifying those parameters)
Ping timeout 8000
commandtimeout = 1000
DIGtimeout = 4000
commandtimeout = 1000
Traceroute timeout 52000
commandtimout 1000
HTTP GET timeout 8000
commandtimeout = timeout – 3000
Pageload timeout 45000
commandtimeout = timeout – 3000
Global error handler. Supported errors in the HTTP Header
To better understand returned results from the API we are now returning more information about the possible API problesms right in the HTTP Header.
Some of the new Status Codes:
200 – OK
500 – Unexpected Error from the API side
520 – Errors related with the empty results
If HTTP Status Code is <> from 200, then we also respond with [Message] element in the response body that will provide more user friendly message what the error is about.
Changes in the data output model
We are returning only objects which are currently in use. Objects as PING / HTTPGET etc are hidden for DIG. And the same rules exists for the others type of tests
e.g based on results for DIG OLD RESPONSE:
"StartDIGTestByCountryResult": [ { "ASN": { "AsnID": "AS12741", "AsnName": "Netia SA" }, "CDNEdgeNode": null, "Country": { "CountryCode": "PL", "CountryFlag": "http:\/\/speedcheckerapi.blob.core.windows.net\/bsc-img-country-logos\/pl.png", "CountryName": "Poland" }, "DIGDns": [ { "AdditionalInformation": null, "Destination": "www.broadbandspeedchecker.co.uk", "QueryTime": "13", "Status": "NoError" } ], "DateTimeStamp": "\/Date(1435158434263+0000)\/", "HTTPGet": null, "ID": 31658310, "Location": { "Latitude": 50.320525, "Longitude": 19.132328 }, "LoginDate": null, "Network": { "LogoURL": "https:\/\/speedcheckerapi.blob.core.windows.net\/bsc-img-providers\/logo_0039_ef7487217cc7_60.jpg", "NetworkID": "226", "NetworkName": "Netia SA" }, "PAGELoad": null, "Ping": null, "PingTime": null, "TRACERoute": null } ]
NEW RESPONSE:
"StartDIGTestByCountryResult": [ { "ASN": { "AsnID": "AS197480", "AsnName": "SerczerNET Malgorzata Nienaltowska" }, "Country": { "CountryCode": "PL", "CountryFlag": "http:\/\/speedcheckerapi.blob.core.windows.net\/bsc-img-country-logos\/pl.png", "CountryName": "Poland" }, "DIGDns": [ { "Status": "OK", "Destination": "www.broadbandspeedchecker.co.uk", "QueryTime": "1" } ], "DateTimeStamp": "\/Date(1435150848253+0000)\/", "ID": 17854019, "Location": { "Latitude": 53.29718, "Longitude": 23.28092 }, "Network": { "LogoURL": null, "NetworkID": "3362", "NetworkName": "SerczerNET Malgorzata Nienaltowska" } } ]
New properties available for probe results
For all probe results we added new property: Status
Status: OK, Timeout , TtlExpired (+ potentially other statuses added in the future)
For Ping and Traceroute we added also PingTime array, which can be useful to calculate standard deviations and other statistical analysis of the individual pings that were performed.
Ping Response "Ping": [ { "Status": "OK", "Destination": "www.interia.pl", "Hostname": "www.interia.pl", "IP": "217.74.65.27", "PingTime": 12, "PingTimeArray": [ "13", "11", "12" ] } ]
Traceroute response
"TRACERoute": [ { "Destination": "www.interia.pl", "HostName": "www.interia.pl", "IP": "217.74.65.27", "Tracert": [ { "HostName": "", "IP": "10.27.1.1", "PingTimeArray": [ "1", "0", "0" ], "Ping1": "1", "Ping2": "0", "Ping3": "0", "Status": "OK" }, { "HostName": "", "IP": "178.217.192.254", "PingTimeArray": [ "42", "31", "33" ], "Ping1": "42", "Ping2": "31", "Ping3": "33", "Status": "OK" }, { "HostName": "interia.tpix.pl", "IP": "195.149.232.12", "PingTimeArray": [ "12", "14", "11" ], "Ping1": "12", "Ping2": "14", "Ping3": "11", "Status": "OK" }, { "HostName": "", "IP": "217.74.64.187", "PingTimeArray": [ "11", "11", "11" ], "Ping1": "11", "Ping2": "11", "Ping3": "11", "Status": "OK" }, { "HostName": "www.interia.pl", "IP": "217.74.65.27", "PingTimeArray": [ "12", "11", "13" ]", "Ping1":"12", "Ping2":"11", "Ping3":"13", "Status":"OK"}]}]}
We have deployed few new improvements to the API.
Support for new parameters
PING Test methods
ttl – Max allowed hops for packet (default=128, max=255)
bufferSize – Size of the Packet data (default=32, max=65500)
fragment – Fragmentation of sending packets. (default=1)
resolve – IP addresses will be resolved to domain names for each hop (default=0)
ipv4only – Force using IPv4. If no IPv4 IP address is returned will return error (default=0)
ipv6only – Force using IPv6. If no IPv6 IP address is returned will return error (default=0)
DIGDNS Test methods
commandTimeout – DNS query timeout in milliseconds (default=1000ms)
retries – Total number of retries (default=0)
TRACEROUTE Test methods
maxFailedHops – Stop the command execution after maximum errors in a row (e.g. stop after 5 ping timeouts, default=0)
ttlStart – First Hop from which the trace route should start (default=1) bufferSize – Size of the Packet data (default=32, max=65500)
fragment – Fragmentation of sending packets (default=1)
resolve – IP addresses will be resolved to domain names for each hop (default=0)
ipv4only – Force using IPv4. If no IPv4 IP address is returned will return error (default=0)
ipv6only – Force using IPv6. If no IPv6 IP address is returned will return error (default=0)
PAGELOAD Test methods
commandTimeout – Maximum allowed time for pageload in milliseconds
HTTPGET Test methods
commandTimeout – Maximum allowed time to send HTTP GET request and receive the response in milliseconds
maxBytes – Max bytes to download from response stream
2. Changes in how timeouts are handled by the API
Each method has 2 main time out parameters:
timeout – Amount of time in milliseconds in which API responds with result
commandTimeout – timeout for the actual test command. For most of the tests its obvious that commandTimeout correlates with total timeout of the API method. However, in case of Ping and Traceroute its not the case. Ping methods by default execute 3 ping commands. Traceroute executes even more and it depends on many parameters such as TTL, count etc.