Build the UK’s largest event search engine

Daybees is a UK startup that embarked on a mission to build the world’s largest event database and search engine. At its launch in 2014, it offered information about more than 30 million events from over 10 thousand sources. It was also featured in BBC, Wired, The New York Times, and other media.

Daybees is a specialized search engine that aggregates millions of events happening across the UK, with innovative technology and an exceptional user experience.

Business

The founder of Daybees was frustrated with the poor availability of event information on the web. He envisioned an application aggregating all events in every category and location— not just the mainstream ticketed events happening in major cities.

It wasn’t just about finding events but also incorporating them into an individual schedule. The Daybees Calendar aimed to help users build a calendar of events they could share with family, friends, and colleagues. This tool would let users add, edit, and share events. Daybees would also let users buy tickets, get maps, avoid scheduling conflicts, and find nearby attractions.

Bringing such an innovation in event search to life would require an expert team and pioneering technology. Sunscrapers was happy to provide support.

Technology

Daybees’ visionary mission set multiple challenges for the technical team. First, how would we build and grow our event database to achieve maximum event coverage? We considered solutions such as Internet crawling, third-party feeds, and user-generated content.

Second, we had to process all the data aggregated from numerous sources to provide users with high-quality information. That included duplicate event detection, event merging, and handling repetitive events in various combinations.

Next, we had to serve all event data on a fast and intuitive web and mobile interface to ensure that Daybees users could easily find events they were interested in.

Finally, we required a solid infrastructure with enough power to run our applications and serve the data processing work happening in the background.

Through collaboration and initial planning, the Sunscrapers team made the following strategic decisions:

• Programming language & framework - we chose Python and Django as a scalable web framework that ships easy-installable packages for creating APIs and a customized CMS.
• Data collection and processing - we decided to rely on crawlers as the primary source of event data. Sunscrapers created a custom crawling script for each website and built a common crawler engine to run these scripts, process incoming data, and report on the results and system status.
• Search engine - we chose Elasticsearch because it provided us with indexing options that fit our requirements perfectly.
• API - we decided to introduce an interface for web & mobile applications as an abstraction layer over the database and search engine, which would decouple them from web and mobile applications.

The next step was gathering two teams responsible for different areas of the system:

1. Design and development team composed of a CTO, a designer, and five developers, all focused on building the web and mobile apps with all their features.
2. Crawler team consisted of four expert crawler engineers and a moderator, focused on building and maintaining a high-quality database of event data.

Both teams worked closely together for over a year to build, maintain, and grow Daybees. After the launch, three apps were running on the following infrastructure:

• 4 web application servers - behind a load balancer to ensure that more servers could be added easily in case of an increase of concurrent requests and connections
• 3 database servers - master + slave + load balancing server
• 4 Elasticsearch servers - as a scalable cluster for search queries and indexing data
• 1 queue server - for long running tasks
• 1 crawler server - where multithreaded crawling and data postprocessing was happening

Sunscrapers helped bring this elaborate, multi-faceted project to life and helped create the world’s largest event search engine from scratch.