How to build a recommender system: it's all about rocket science - Part 1

How it all started: one goal, one mission

30 minutes was the time it took for the excitement over our first ever live feature to be stopped cold. An urgent hotfix was required, developed and deployed but, in the end, everything went off in quite a spectacular crash and burn. How naive of us. A bit like Space X, it was with a less than glorious start that Inspire, Farfetch’s very own recommendations engine, saw daylight for the first time.

Through the veins of history

Building a recommendation system (or engine) from scratch is, to say the very least, complex. The apparent simplicity of a feature that is as old as human interactions themselves -- the first and still most used recommender is the good old word of mouth -- hides the nature of what in fact, is: an information filtering system, with all the expected bells and whistles that have been known for decades but only now are easily accessible, like machine learning, deep learning and other cool buzzwords. Before we move any further, let us assure you that yes, just like Facebook, Google, Netflix and other tech-players out there we are, too, making use of such approaches on our internal recommendations engine.

The strategic move by Farfetch to have its own engine as part of its suite of products and services available through the platform for our clients was in itself challenging enough. For reference, Amazon’s recommendations engine is driving more than 35% in revenue and Netflix’s is responsible for 75% of what the users watch. So it’s only logical that Farfetch, being at the forefront of the intersection between luxury fashion and technology, would invest in its recommendation engine as well. 

However, at the time of our first deploy a little over 2 years ago, Farfetch was using a third party for recommendations, dubbed the #1 personalisation engine in the industry (recommendations and personalisation are often used interchangeably). We were using the external solution to accelerate our evolution, to understand what our users were looking for, and at the same time to benchmark against, so we knew where the stakes were. As you can imagine, any automated algorithmic approach (some would call it "intelligent”) is a moving target. Coupled with the sheer amount of traffic Farfetch has on the website alone meant we would not get away with one approach -- it was necessary for us to divide and conquer. In this process, we iterated more than 20 times on the main touchpoints of the website, all the while dramatically improving the performance of the engine. Having the best algorithm in the world is worthless if you have no way of delivering it. So, to fulfil our vision of setting the standard in luxury recommendations, we were, in fact, fighting on multiple fronts with data science at the core and engineering at the forefront. 

Pressure, pushing down on us

Just like building a house, our team needed to build the foundations of the engine to deliver recommendations in a platform-ready manner -- we had to build the software. So we proceeded to build the services, the ETLs and the sand-boxes for conducting experiments and to drive innovation and business forward.

Like many e-commerce platforms, we started to collect and leverage various implicit interactions by tracking our users’ journeys on all available channels, as well as the explicit preferences they sometimes set on their preference centre. From this, we created Precog (go ahead and check the film Minority Report, we’ll wait). It was our first implementation of the fundamental and state-of-the-art recommender systems based on collaborative approaches. You can think of it as a word of mouth where it is the algorithms’ job to find the best person to recommend you an item based on both your navigation histories. Precog was also housing the content-based recommenders fueled by the breadth of information provided by the production teams at Farfetch. These two pillars are the cornerstone of our recommenders and serve as a tool for learning and benchmarking any innovation and experiment we can think of. The flexibility and robustness of such a piece of software allows the team to create new algorithms, or improve existing ones, just like using LEGO building blocks. In this way, we are able to deliver ideas faster, without losing the granularity and the low-level required for having confidence in what we are delivering to our customers.

Having the basics in place has also helped us keep our feet on the ground and be realistic about the progress of our innovations. As scientists, we tend to wander off a tiny bit and we end up building a rocket to go to the grocery store, just because, why not? But through experimentation we were taught that the simple solution will often beat that neat and complex model. It also taught us that we need to continuously push the boundaries to deliver a tailored and luxury-focused experience. How? Keep reading - Part 2.