High-Load Websites and Applications built with Python / Django

Our company has been developing large web applications using Python / Django for quite a long time. For ease of a websites comparison, we give some statistics for each project. In general, there are many websites in the world, which are written in Python / Django, so we will cover the most interesting of them.

Below there is a small table of well-known projects (based on the data at https://builtwith.com/):

Pinterest (social network)

Technologies that have been used:

Python as a programming language and Django has been chosen as a framework. Hosting provider — Amazon. MySQL is used as the main database management system, memcached does caching of objects and the collections of objects is done by Redis. Solr is used as a search platform, and Hadoop — for a realization of search and context mechanisms, analysis of data.

A little bit of statistics:

According to the latest data, the staff includes over 140 people. About 11 million unique visitors per week in December, 2011. 500 virtual machines in EC2, 410 terabytes of user data or 80 million different objects are stored in Amazon S3. Pinterest has about 70 million users according to a French news agency called Semiocast. In total, Pinterest becomes the fourth most popular social network in the United States after Facebook, Twitter, and LinkedIn.

(https://en.wikipedia.org/wiki/Pinterest)

(the original material can be found here: https://www.businessinsider.com/how-we-scaled-pinterest-2013-4?op=1, http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html)

Disqus (service)

Technologies that have been used:

Python as a programming language and Django has been chosen as a framework. Operating System — Linux. MySQL is used as the main database management system, memcached does caching of objects as in the case with Pinterest. HAProxy is responsible for the load balancing, and Slony — for data replication.

A little bit of statistics:

33 people works at the company. The number of registered users is growing and currently is 50 million people. The site receives about 17,000 requests per second. More than 144 million unique visits per month only from the territory of the United States. Around 750,000 websites and blogs.

(source https://www.insight-it.ru/masshtabiruemost/arkhitektura-disqus/, the original material can be found here: http://highscalability.com/blog/2010/10/26/scaling-disqus-to-75-million-comments-and-17000-rps.html, https://en.wikipedia.org/wiki/Disqus)

Instagram (photo and video application)

Technologies that have been used:

Ubuntu Linux 11.04 has been selected as the primary operating system. Python is used as a programming language, and Django is used as a framework. PostgreSQL has been selected as the main database management system, caching of objects is done by memcached, Redis acts as an additional data storage. HAProxy is responsible for the load balancing. The project uses the infrastructure of Amazon, particularly EC2, ELB, Route, S3 and CloudFront. Solr has been chosen as a search platform. Gearman is used to work with tasks.

A little bit of statistics:

It all started with one small and not powerful server and two developers. On the first day, about 25,000 users were registered. Today, Instagram is used by over 200 million people, and there are about 7 million users daily. About 20 billion photos have been already published, and 60 million photos are published on a daily basis. Facebook acquired the service for 1 billion dollars in April, 2012.

(source https://expandedramblings.com/index.php/important-instagram-stats/, https://en.wikipedia.org/wiki/Instagram)

Reddit (news website)

Technologies that have been used:

Python as a programming language and as a framework, Pylons was selected. PostgreSQL is used as the main database management system, memcached does caching of objects. RabbitMQ is used for autonomic data processing. HAProxy is responsible for the load balancing. Amazon CloudSearch is selected as a search platform.

(source https://en.wikipedia.org/wiki/Reddit)

A little bit of statistics:

Approximately 112 million unique visits per month. 5,46 billion page views per month. 2,89 million people use Reddit. And all of this is maintain by the team of 28 people.

(source https://expandedramblings.com/index.php/reddit-stats/)

Dropbox (cloud storage)

Technologies that have been used:

Dropbox server and client application were written in Python. The client uses development tools GUI, such as wxWidgets and Cocoa and other important Python libraries such as Twisted, ctypes and pywin32. Dropbox depends on librsync library, which is written in C. The information about the files is contained in the storage, which was implemented on the basis of MySQL. Amazon S3 is used to store the files.

(source https://en.wikipedia.org/wiki/Dropbox_(service))

A little bit of statistics:

The staff number is 110 people. More than 50 million registered users. Every 3 minutes more than a million files are stored using the service. 500 million files are saved daily.

(source https://www.dropbox.com/static/docs/DropboxFactSheet.pdf)

Pitchfork (music e-zine)

Technologies that have been used:

Python as a programming language and as a framework, Django was selected. Hosting provider — Amazon. MySQL is used as the main database management system. In addition, PostgreSQL is used. Redis does caching of objects. ElasticSearch and Solr are used as a search platform. Also, Amazon services (EC2, RDS, SES) are used.

(source https://www.siteclass.com/www/pitchfork.com)

A little bit of statistics:

The project has more than 14.5 million visits per month (580,000 per day), 5.5 million of which are unique (410,000 per day). Page views are 38.5 million per month (1.6 million per day). The number of registered users is 4.3 million in March-April, 2014.

(source https://www.quantcast.com/pitchfork.com)

Lanyrd (portal)

Technologies that have been used:

Python as a programming language and Django was selected as a framework. PostgreSQL is chosen as the main database management system, caching of objects is done memcached. Redis is used in combination with Celery to store intermediate results of tasks, which performs by Celery asynchronously. HAProxy is responsible for load balancing. The project uses the infrastructure of Amazon, particularly S3. Solr is used as a search platform.

(source https://www.slideshare.net/InfoQ/inside-lanyrds-architecture)

A little bit of statistics:

The development team consists of only 6 people. That’s what Andrew Godwin said in his presentation. About 900,000 unique visitors per month in March 2014. The maximum value was in October 2013 and reached over a million visitors (https://www.trafficestimate.com/lanyrd.com). Daily views of pages are about 55,000.

(source https://www.slideshare.net/InfoQ/inside-lanyrds-architecture)

Mozilla (software)

Technologies that have been used:

The Mozilla company uses various programming languages including Python, which is used widely: from writing built scripts for the company website, Webmaker, and other components, as well as a synchronization server — minimalistic WSGI application using Paste to deploy on the hosting server and Sqlite3 as a database.

A little bit of statistics:

Today, the Mozilla projects are used by more than 500 million people. This is a very high indicator, considering the competition. Over a thousand developers are engaged in the Mozilla projects.

(source https://expandedramblings.com/index.php/internet-browser-stats/)

Yelp (portal with social network elements)

Technologies that have been used:

Ubuntu Linux has been chosen as the primary operating system. Python as a programming language and as a framework, Django was selected. MySQL is used as the main database management system. Also, Yelp uses Amazon services, particularly, S3 to store logs and photos, and EMR. Solr/Lucene are used as a search platform. HAProxy and LVS are responsible for the load balancing.

(source https://engineeringblog.yelp.com/, https://aws.amazon.com/solutions/case-studies/yelp/)

A little bit of statistics:

The website traffic is about 200 million people per month, about 120 million of which are unique visits.(https://www.quantcast.com/yelp.com). Since the project launch, over 53 million user reviews have been written worldwide.

(source https://expandedramblings.com/index.php/yelp-statistics/)

Foursquare (social network with the function of geolocation)

Technologies that have been used:

The project is developed with several technologies including Python, which is used to automate operational tasks and other processes. CentOS Linux is used as the primary operating system. HAProxy is responsible for the load balancing and API-requests. MongoDB is selected as the main database management system, caching of objects is done by memcached. Part of the information, namely user photos, is stored in Amazon S3. Hadoop deals with the analysis. Solr and Elasticsearch were chosen as a search platform. Library s2 from Google in combination with PostGIS are used to search for geo-index. Kestrel is responsible for handling asynchronous tasks.

(source https://foursquare.com/about)

A little bit of statistics:

The staff includes 140 people. About 40 thousand developers are engaged in the project development and maintenance in accordance with the information reported in 2013. About 45 million users. The total number of check ins are 5 billion worldwide, and about 3 million are added on a daily basis.

Rdio (music service)

Technologies that have been used:

Rdio uses several programming languages. Particularly, a part of the back end is written in Python and Django. MongoDB and MySQL are used to store information. Redis was chosen as an alternative of memcached.

A little bit of statistics:

The catalog includes more than 20 million songs. There are around 200,000 page views daily. About 300 thousand people use the services during a month in the United States.

Google (search engine)

Technologies that have been used:

Google uses a variety of programming languages. The creator of Python was working at Google for a while, so it’s easy to guess why this technology along with others was used for the project. Part of YouTube, the search engine as well as many other components are written in Python. Briefly, LevelDB is used as the main database management system. The closure is also used to work with JavaScript. It’s kind of a set of tools created by developers of Google.

A little bit of statistics:

The statistics of Google are known for everyone but let’s take a look at some indicators.12,477 billion inquiries are done monthly. Google takes about 67% of the search market in the United States. There are 191 million unique visits per month. The company employs 53 891 people.

(source https://expandedramblings.com/index.php/by-the-numbers-a-gigantic-list-of-google-stats-and-facts/)

Quora (social service ‘Questions and Answers)

Technologies that have been used:

Python as a programming language. Hosting provider — Amazon. MySQL is selected as the main database management system, caching of objects is done by memcached. HAProxy is responsible for load balancing.

A little bit of statistics:

There are 72 employees on the team. The number of unique visits is over a million according to the information reported in February, 2014. Monthly, 1,126,000 people use Quora (as of 2013).

(source https://www.quora.com/How-much-traffic-does-Quora-get, https://techcrunch.com/2013/11/12/quora-confirms-its-favoring-search-ads-for-eventual-monetization-launches-author-stats-tool/)

Conclusion

Based on the provided above information, a lot of well-known projects are built with Python / Django, and that’s fully justified. Python is a very interesting modern programming language, which is rapidly gaining popularity nowadays. Today, we observe a high demand for programming services using Python / Django because of its quality! The demand for projects effects the demand for Python experts.

If you would like to develop a high-load project, we highly recommend you to consider Python / Django as a technical platform. And a small tip for colleagues. If you would like to become a developer or learn a new programming language, Python is definitely one of the best options.

P.S. To get our new articles right away or not to miss new publications, subscribe to us on Facebook, Twitter, and LinkedIn