Sunscrapers

Daybees

Helping a UK startup build the world’s largest event search engine.

Daybees is a search engine that aggregates millions of events happening across the UK.

Daybees is a UK startup that embarked on a mission to build the world’s largest event database and search engine. At its launch in 2014 it offered information about more than 30 million events from over 10 thousand sources. It was also featured in BBC, Wired, The New York Times, and other media.

Location
London / UK
Industry
Internet
Platforms
Web, iOS, Android

Challenge.

Business

The founder of Daybees was frustrated with the availability of event information on the web. He envisioned an application aggregating all events, in every category and location - not just the mainstream ticketed events happening in major cities. It wasn’t just about finding events, but also incorporating them to an individual schedule with the help of the Daybees calendar. Adding, editing and sharing events, buying tickets, getting maps, avoiding clashes and finding out what else is happening around another event: Daybees aimed to help users build a calendar of events they could share with family, friends, and colleagues.

Technology

Daybees’ visionary mission set multiple challenges for the technical team. The first one was deciding how we would build and grow our event database to achieve maximum event coverage. We considered solutions such as Internet crawling, third party feeds, and user-generated content. Second, we had to process all the data aggregated from numerous sources to provide users with high-quality information. That included duplicate event detection, event merging and handling repetitive events which came in various combinations. Next, we had to serve all event data on a fast and intuitive web and mobile interface to ensure that Daybees users could find events they were interested in easily. Finally, we required a solid infrastructure with enough power to run our applications and serve the data processing work happening in the background.

Solution.

After initial planning, the team made the following strategic decisions:

  • Programming language & framework - we chose Python and Django as a scalable web framework that ships easy- installable packages for creating APIs and a customized CMS.
  • Data collection and processing - we decided to rely on crawlers as the primary source of event data. That required writing a bespoke crawling script for each website and building a common crawler engine to run these scripts, process incoming data, and report on the results and system status.
  • Search engine - we chose Elasticsearch for the job because it provided us with indexing options that fit our requirements perfectly.
  • API - we decided to introduce an interface for web & mobile applications in the form of an abstraction layer over the database and search engine which would decouple them from web and mobile applications.

The next step was gathering two teams responsible for different areas of the system:

  • Design and development team composed of a CTO, a designer, and five developers - all focused on building the web and mobile apps with all their features,
  • Crawler team composed of four crawler engineers and a moderator - focused on building and maintaining a high- quality database of event data.

Both teams worked closely together for over a year to build, maintain, and grow Daybees. After the launch, three apps were running on the following infrastructure:

4 web application servers (behind a load balancer to ensure that more servers could be added easily in case of an increase of concurrent requests and connections),

3 database servers (master + slave + load balancing server),

4 Elasticsearch servers (as a scalable cluster for search queries and indexing data),

1 queue server (for long running tasks),

1 crawler server (where multi-threaded crawling and data postprocessing was happening).

services
  • Product design
  • Full-stack web and mobile development
  • API development
  • Search engine
  • Crawlers and aggregators
technologies
  • Python / Django
  • iOS / Swift
  • Android

Results.

#5

most popular free app in the UK AppStore (Jan ‘14)

$1,000,000

raised in funding

30,000,000+

events in the database

11,113 URLs

crawled by 2,085 crawlers

10

people in the dedicated Daybees team

680,749

crawled event locations

Let’s talk about your project.

Reach out to us

and receive a bespoke estimation in 2 business days

OFFICE

Sunscrapers Sp. z o.o.
ul. Pokorna 2/947
(entrance 9)
00-199 Warsaw

AWARDS


Clutch award

OFFICE

Sunscrapers Sp. z o.o.
ul. Pokorna 2/947
(entrance 9)
00-199 Warsaw

We are using cookies to provide statistics that help us give you the best experience of our site. You can find out more or switch them off if you prefer. However, by continuing to use the site without changing settings, you are agreeing to our use of cookies.

Give us some details.