Secrets behind web rankings

Posted on Aug 21, 2016

Contributed by Jiaxu Luo. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between July 5th to September 23rd, 2016.This post is based on his second class project - Shiny Application (due on the 6th week of the program)

The shiny app could be found

 I. Introduction

When Tim Berners-Lee, who could have applied for a patent as the inventor of the World Wide Web, appeared at the opening ceremony of the London Olympics in 2012 and tweeted“This is for everyone", it struck me that our lives would not be as easy and efficient had the trailblazers of the Internet not been so generous. We rely on web services everyday and especially ever since the Web 2.0, when end users massively generate contents of their own. As an integral part of our lives, the Internet was developed for everyone and so, by definition,  should  transcend national boundaries.

However, the accessibility and popularity of individual websites are intertwined with issues of business, politics and geography. Could website ranking data by country reveal some of the relationships among those complicated issues. The answer is Yes!  This is demonstrated by an analysis of website popularity, and easily evidenced in the shiny app that accompanies this blog (

II. Web scraping

For the purpose of the discovery, I wrote a web scraper with python using the package "Beautiful Soup"and scraped Alexa Alexa is a wholly owned subsidiary of, providing commercial web traffic data and analytics.The sites in the top sites lists are ordered by their 1 month alexa traffic.The 1 month rank is calculated using a combination of average daily visitors and pageviews over the past month. The site with the highest combination of visitors and pageviews is ranked #1. These ranking are updated every two days.

The data extracted for the further analysis includes

  • Name and rankings of global top websites: 500 in total
  • Nationality of the providers of each of the top websites: 500 in total
  • Country names that have statistics: 127 in total
  • Name and rankings of top websites for each of the countries: 127*500 in total

III. Analysis

One surprise, when investigating the global 500 websites was the very high ranking of some of the Chinese websites. As a Chinese person, my impression has been that website services provided by Baidu, QQ and Taobao are not as popular around the globe as Google and Wikipedia. One way of understanding this is as an artifact of the way Alexa accounts for popularity, by numbers of visitors and pageviews. China, with its large population, is perhaps over represented. So, ranking, based on traffic alone may not be the best criteria for assessing the popularity of a website.


Luckily, Alexa provides data for the rankings of top websites for 127 countries (All the big players on the Internet are included).  From this data I could derive a new ranking metric based on the number of countries in which a website is ranked in the top 500.  This metric was then used as a measure of website popularity.

For example, under the new standard, "" still ranks 1st because it ranks in the top 500 in all countries.  In contrast, Chinese web service providers like "" rank 247.  It is ranked in the top 500 in only 23 countries.  However,the shortcoming of this metric is that by just considering the number of countries alone, it introduces bias.   Therefore, I derived one more metric as a trade-off  between popularity rankings based on overall traffic and those based on the spread among countries.  This metric is named Bias (Bias=rankings of popularity based on overall traffic -rankings of popularity based on spread among countries).

The following histogram shows the distribution of “Bias” scores by count.  Interestingly it approximates to a normal distribution. I categorized the global top 500 websites into 3 groups:  Negative biased websites("Bias"<-200), Neutral websites(-200<"Bias"<200 ) and Positive biased websites("Bias" >200).

Websites with positive bias (i.e, positively skewed)  are those which might not have high overall traffic, but appear in the top 500 list in many countries.   In contrast, a negative bias (i.e., negatively skewed) indicates websites which have very high overall traffic but are popular only with a limited number of countries.  Websites where the bias is closer to zero are those where there is synchronicity between overall traffic, and spread throughout many countries.  This includes the googles of the world, which are widely and heavily used throughout the world, as well as small “mom and pop” websites that have both low overall traffic and low distribution.

I categorized those global top 500 websites into 3 groups - Negative biased websites("Bias"<-200), Neutral websites(-200<"Bias"<200 ), Positive biased websites("Bias" >200), and investigated the rankings among countries for these three groups respectively.

The hypothesis here are:

  • The countries that favor a specific website show a geographical pattern
  • The inconsistency of rankings of a specific website among countries may reflect tactics of business and politics.

Neutral websites (Mainly international-market based)

In the shiny app, the color gradient from red to green represents the increasing popularity. The reasons for some of the colorless  region  are either that the data is not available for those countries or the website is not among the top 500 websites list. Looking at just the higher ranked websites within the Neutral group, the reason for low Bias seems apparent.  Many of these websites (e.g. "", "", ""), provide general services that are used widely by a majority of people and these companies target the global market. As a result, the rankings by the criteria of both overall traffic and breadth go hand in hand.


Something quite interesting here is that media providers, such as "twitter","Facebook" and "YouTube", are having trouble expanding their businesses in China and Iran. This is depicted by the sharp contrast in color for China and Iran, as compared with other countries. As far as I know, the Iranian government excludes those media services that might present a challenge to prevailing religious beliefs.  The Chinese government does this in order to direct people to use products of its own.

The next graphic depicts the popularity of Amazon’s AWS service across the globe.  Interestingly, the American website is even more popular in Nigeria than in USA.  (The relative shadings are easier to observe using the shiny app itself, which allows one to zoom in and out.)


Positive biased websites (Mainly regional-market based)

Most positively biased websites share a common feature.  These are the websites that focus on a small segment of the market, but nevertheless have global reach.  For example, the Nigeria website "" focuses on the latest world soccer news & standings; an Indian website "" focuses on discovering great art, design, architecture, and photography.



Above is a map that reflects the difference in rankings of the Nigeria website " It's interesting to see that the web service in some other countries is more welcomed than in it's own country. The warmer colors indicates the relatively lower rankings while the colder colors indicates the higher. The regions where there is no color is due to the fact that the website is not in the top 500 websites list in those area. In the perspective of marketing, I would suggest those companies to do more  in customer acquisition in the countries in warmer colors , like India,since there’s still much space to improve.

For the  countries in colder colors,which indicates a sign of relative maturity in the operation of business,I would suggest the company pay more attention to customer retention.

Negative biased websites (Mainly local-market based)

For the last group of websites, it's surprising to note that, except for some websites from United States subsidiary companies, like “, nearly all the websites are from China, Japan and South Korea. It's obvious, by exploring the app, that these websites are predominantly local and are hosted by national businesses. It's not surprising that many Chinese websites are on this list, simply due to the size of the Chinese population. It is surprising, however, to see local business websites from Japan and South Korea on this list, as they do not have the scale advantage that China has.  My hypothesis here is that people from these countries are so loyal to their own web services that page views can reach a very high level.

Below is a famous Chinese website called “Tmall” and the business of the company associated with it around the neighboring countries. It’s evident that this is a website currently focusing on local business.


Scatter plot as a summary

Below is a scatter plot visualizing the rankings of websites by two different criteria. The colors represent different countries. Nine countries with the most observations are included in the plot.  The shiny app provides a way to filter by country. Setting China as an example, most Chinese companies focus their business nationally. This can be seen from the high density of points in the top left corner. Alibaba, as one of the outliers, is represented by a dot in the far right corner of the plot.  Alibaba appears to be employing a different, more global, business model.



IV. Conclusion

  • A majority of  websites belong to neutral websites; the top neutral websites typically provide general services around the world.  (International-market based)
  • Positively biased websites are typically those websites that target very specific market segments, but which may nevertheless have global reach. (Regional-market based)
  • Negatively biased websites are mostly from China, Japan, and South Korea, where businesses operate largely on a national level or around the neighborhood. (Local-market based)
  • For top websites, this  app of provides data about the global market share of business operations and could be used as a tool to facilitate business decision making.

About Author

Jiaxu Luo

Jiaxu Luo is a recent MS graduate from Columbia University. As a child born in a business family and a student with diverse academic background in Material Science and Chemical Engineering, he grasps the importance of asking questions...
View all posts by Jiaxu Luo >

Leave a Comment

Secrets behind web rankings – Browse your web December 16, 2016
[…] – Shiny Application (due on the 6th week of the program). The original article can be found here. The original article can be found […]

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI