The founder of Daybees was frustrated by the difficulty of finding interesting and comprehensive event information. He envisioned an application aggregating every event, in every category and location - not just mainstream ticketed events happening in major cities. It wasn’t just about finding events but also being able to make use of them through the Daybees calendar. Adding, editing and sharing events, buying tickets, getting maps, avoiding clashes and finding out what else is on around another event: Daybees was about building events around one’s life and sharing the calendar with work or family.
Our job was to build a web, Android and iOS apps running on a common infrastructure and covering 90% of events happening in the UK. On top of it there would be functionality that helps users find relevant event information and organize it in a useful way.
We were faced with many technical challenges related to building such an ambitious system with the following key areas requiring an appropriate solution:
- Data collection - we had to find data sources and a way of collecting the information,
processing it and moving it to Daybees databases,
- Search engine - a large collection of events required an advanced and engine that would make
events searchable in a fast and user friendly way,
- Infrastructure - the complexity of the system and a multi-platform architecture required an
efficient, reliable and scalable server environment.
The project took off with a comprehensive research about client’s needs, market and competitive landscape, suitable solutions and approaches and the existing application prototype.
After initial planning the team made the following strategic decisions:
- Programming language & framework - we chose Python and Django as a scalable web
framework that ships with easy-installable packages for creating APIs and a customized CMS.
- Data collection - we decided to rely on crawlers as the primary source of event data.
That meant writing a bespoke crawling script for each website and building a common crawler engine to run those scripts, process incoming data and report on results and system status.
- Search engine - we chose Elasticsearch for the job as it provided us with indexing
options that perfectly fit our requirements.
- API - we’ve decided to introduce an interface for web & mobile applications - that
means an abstraction layer over database and search engine which decouples them from web and mobile applications.
The next step was to gather two teams responsible for different areas of the system:
- Design and development team composed of a CTO, designer and five developers - focused on
building web and mobile apps with all their features,
- Crawler team composed of four crawler engineers and a moderator - focused on building and
maintaining a high-quality database of event data.
Both teams were working closely together for over a year to build, maintain and grow Daybees. After the launch three apps were running on the following infrastructure:
- 4 web application servers (behind a load balancer so that more servers can be added easily once there is an increase in the number of concurrent requests and connections),
- 3 database servers (master + slave + load balancing server)
- 4 Elasticsearch servers (as a scalable cluster for search queries and indexing data)
- 1 queue server (for long running tasks)
- 1 crawler server (where multi-threaded crawling and data postprocessing was happening)