Analytics Tracking - a Mobile Perspective
By
Diogo BalseiroView All Posts
Passionate about iOS, books and gaming. Now busier changing diapers. Kenzo enthusiast, on behalf of the missus.
Top Stories
1 - Analytics Tracking - a Mobile Perspective2 - MOBCAST: Engineering and Product - A Tale of Growth
Analytics tracking is often perceived as the ugly duckling in software development. After the glistening UI and the intricate foundation systems, tracking work comes as an afterthought and is usually met with a loud sigh by the prototypical developer. Curiously, it later becomes an essential and vital part of the product, and without it, bad decisions are made.
Farfetch has many front-end applications available, ranging from desktop and mobile web to iOS and Android. As the business grew, more and more applications were created to tackle different challenges, with each having its own tracking system, often resulting in disparate codebases, vendors and data models.
These implementations eventually reached their capacity, becoming slow, inefficient and difficult to expand. Not only that, but some also relied upon external vendors that were costly.
Farfetch sought out a new solution and Omnitracking was born.
Omnitracking is an internal multi-channel tracking solution used by our front-end applications, allowing the tracking of page views, page actions and system actions.
Being an internal service, Omnitracking brings a number of advantages at multiple levels, for instance:
- From an analytical standpoint, it delivers a single source of truth for all application data, enabling ETL processing and easing analysis.
- From an operational standpoint, it provides near real-time processing of user behaviour, allowing for interesting opportunities such as better user journey personalization.
- From an engineering standpoint, it grants the opportunity to deprecate old and costly systems, while creating new ones that scale better for our current and future needs.
The project itself is very complex and too broad to be tackled in a single article, so let’s focus on the perspective of mobile development, particularly iOS.
iOS Omnitracking SDK
Given that Farfetch has many front-end iOS applications, the iOS solution needed to be generic and agnostic, allowing cross-compatibility. This meant that it couldn’t have any business logic for a specific app lest we risk adding noise to the tracking of other apps.
From this perspective, we’ve created an iOS SDK. This article will describe it by focusing on three separate areas:
- The data model contract
- The client dispatcher
- Data validation and reconciliation
The Data Model Contract
The data model refers to a specific set of rules to which all Omnitracking events must adhere, providing consistency across channels. This means that these events become largely application-agnostic and consequently, their analysis and handling can be reused for multiple apps. Changing the contract requires approval from a committee, which prevents unexpected changes, validates the need and format of new fields and ensures the contract’s overall health.
The contract has a few rules and the following affect the iOS SDK the most:
- Mandates the use of specific fields that provide the minimum required information needed for an event
- Provides a curated list of optional fields that can be used to customise the event further
- Any field that is sent and isn’t in the contract is later ignored and flagged
- Limits the values allowed in each field
- Allowing only a specific value set
- Allowing any value, but restricting its format
How is the Data Model Contract represented in the SDK?
Illustrating with an example:
struct OmnitrackingWishlistPageViewExample: OmnitrackingPageView {// Mandatorylet viewType = OmnitrackingEventParameters.ViewType.wishlistlet uuid: FOTOmnitrackingUniqueIdlet uniqueViewId: FOTOmnitrackingUniqueIdlet clientTimestamp: Date// Optionallet exitInteraction: String?let skuItemList: String}
You can see how the contract manifests itself:
- A view type that identifies the screen the user saw. The value is a specific string, shared by all platforms. In this case, that value represents the wishlist;
- A universally unique identifier that identifies this particular event itself;
- A unique view id that identifies the wishlist screen in the user’s current session
- A timestamp that dates when the user saw the wishlist screen;
- An exit interaction that specifies what led the user away from the wishlist screen;
- An item list containing the products that were shown in the wishlist.
You can see some mandatory and optional fields already. This event is later hydrated with global fields shared by all events. These custom dimensions can include non-personally identifiable information about the user, device, session, app, etc.
Where are these data models stored?
The contract is probably the most important piece of Omnitracking. After all, it’s the foundation that supports its entire premise. As such, it’s tempting to want full control of the data models and to store them inside the SDK, to ensure their compliance with the contract.
If we need to track a wishlist Page, for example, we can create a wishlist page view, like the one above, and store it in the SDK, using code review as a validation anchor. However, that approach poses a problem: what happens if apps need to differentiate some attributes in their wishlist page views?
Those page views might end up looking almost identical, but not exactly the same. To handle this edge case, we decided that each app would have their own models, provided they follow the contract.
Since we use CocoaPods to manage our dependencies, we contemplated taking advantage of subspecs and having multiple specs of the contract, where different apps could store their models inside the SDK, properly segregated by app. This seemed appealing at first, but the added risk of allowing developers to see the models of one another and inadvertently making changes despite code review didn't justify the benefits.
Eventually, we followed the opposite approach: each app keeps its models away from the SDK, leading to more privacy and flexibility in their creation and maintenance.
To aid the creation of the models and ensure their compatibility with the contract, a few protocols were created in the SDK that mandate the required fields of Omnitracking. Also, we keep a local representation of the contract’s optional fields, auto-generating protocols for each one. This allows each app to internally compose their models using these protocols, therefore ensuring they use the proper field names and types, decreasing the likelihood of introducing errors.
Here’s an example of that composition:
typealias OmnitrackingMessageCentrePageViewProtocol = OmnitrackingPageView & OmnitrackingFieldKeysProtocolMessageQuantity & OmnitrackingFieldKeysProtocolHasNewMessages & OmnitrackingFieldKeysProtocolPromocodeOptionalstruct OmnitrackingMessageCentrePageViewExample: OmnitrackingMessageCentrePageViewProtocol {// Mandatorylet viewType = OmnitrackingFieldValues.ViewType.myNotificationsListlet uniqueViewId: OmnitrackingUniqueIdlet uuid: OmnitrackingUniqueIdlet clientTimestamp: Date// Optionallet exitInteraction: String?let messageQuantity: Intlet hasNewMessages: Boollet promocode: String?
}
Is this protocol composition required?
While the SDK mandates the use of the protocols that represent the required fields, it doesn't force the apps to use the optional protocol composition. It’s important to avoid being overly restrictive in the SDK usage lest it becomes a strenuous chore.
To compensate, the SDK validates each event it receives and warns the developer if the contract wasn't followed. It does this by comparing it against a mocked event composed of all protocols and inspecting the differences.
End result
The implementation of the contract in the SDK allows for better data quality and helps prevent mistakes. In the past, we’ve had bad experiences due to not having a fixed set of rules and constraints, always leading to incorrect data. While the contract is not a silver bullet and data quality will never be perfect, it’s an additional layer of security that we’ve come to appreciate. That appreciation has led to an extra effort from all developers in keeping the iOS implementation of the contract up to date and I’m sure we’ll be looking for other ways to improve it in the future.
The client dispatcher
The client dispatcher refers to the code base that provides a layer of abstraction between the server API and the iOS apps. It provides out of the box functionality and is usually divided in three distinct areas:
- Event flush strategy
- Event persistency
- Logging and debugging
Event flush strategy
Much like other Analytics SDKs, the Omnitracking SDK provides several options for how the events should be dispatched to the server API. This is important because each app might have different requirements and might want to be more or less aggressive regarding that event dispatching.
Dispatching events requires an active internet connection, which taxes the system since networking is one of the most battery-intensive operations a phone can do. This is especially true for cellular connections. As such, apps should generally want to be good neighbours and avoid taxing the system unnecessarily.
The SDK dispatcher abides by this preference by providing a few event flush strategies and configurations, that range from very aggressive to very relaxed:
public struct BatchDispatchStrategy: OptionSet {public static let boot = BatchDispatchStrategy(rawValue: 1 << 0)public static let willEnterForeground = BatchDispatchStrategy(rawValue: 1 << 1)public static let didEnterBackground = BatchDispatchStrategy(rawValue: 1 << 2)public static let reachabilityPings = BatchDispatchStrategy(rawValue: 1 << 3)public static let initialEvents = BatchDispatchStrategy(rawValue: 1 << 4)}public struct FOTBatchConfig {/// Options for flushing the current stored unsent eventspublic let strategy: FOTBatchDispatchStrategy/// Maximum amount of events the SDK will hold on topublic let maxQueueSize: Int/// Amount of events until new events trigger flushingpublic let overflowQueueSize: Int/// The target for how many events should be sent in a single network request payloadpublic let deliverySize: Int/// Interval for automatic flushingpublic let flushInterval: TimeInterval}
By using the configurations above, each app can define the behaviour they’re looking for: they can force the dispatcher to flush the events as soon as they enter the dispatcher; they can also let the dispatcher hold on to those events for a long time to preserve battery; or something in between.
Event Persistency
Given how the dispatcher can hold on to the events internally for a certain amount of time and an iOS app can crash unexpectedly, or lose internet connection, it’s paramount that these events are persisted locally in the phone. That way, if indeed that app is terminated before sending the events, the dispatcher can attempt to send them later.
As such, event persistency should be the very first thing that happens when a new event reaches the dispatcher. That process must be fast but not overly demanding to the system, since it will likely happen very frequently. Swift Codable works well when the volume of data to save is small, otherwise SQLite is a good option.
Logging and Debugging
The dispatcher provides a debug UI that can be used to quickly check what events were created and whether they were already sent via API.
Its invocation is simple:
func showMonitorViewController(presentingNavigationController: UINavigationController?)func dismissMonitorViewController(currentNavigationController: UINavigationController?)
Generally, we just use the device shake motion to toggle the debug UI. It’s a great time saver for quality assurance and validation, with the additional benefit of providing some quality of life actions, such as sorting and email exporting.
Data Validation and Reconciliation
The reliability of the data is the most important metric in this whole SDK. It does not matter how quick, pretty and scalable it is if it doesn't provide reliable event creation and delivery. Combining our experience creating other systems and the experience of creating this SDK, here’s a summary of some of the most important points to focus on when creating an Analytics SDK.
Regarding the SDK itself:
Event creation and flushing should have bulk support
Not having bulk support adds unnecessary strain on the system because it requires more singular operations to achieve the same end result, limiting opportunities for optimisation.
Event creation and event flushing must not block each other
Event creation can happen at any time, concurrently and from multiple threads. Therefore, it should be serialised in a Dispatch Queue, or similar. Likewise, event flushing can also be triggered concurrently, due to different configurations and strategies.
While both might manipulate the same internal state, it’s important that they don't block each other’s operations as much as possible. Semaphores and dispatch queues are good mechanisms to achieve this.
Be conservative with removing events mid-delivery
An event should only be removed from the internal state after its delivery to the API is guaranteed. Premature event removal isn’t ideal because the app can crash during event delivery, resulting in its permanent loss.
Also, a record log of the event’s journey throughout the SDK should be kept, at least for the current session. Not only does it help with hydrating future events, but it also provides visibility over how the SDK is functioning.
Do not discard incomplete events unless absolutely necessary
While the contract’s protocols force certain fields to be present in the events, not all required fields come from the apps. The SDK will provide a fair share of required global fields too. So, it’s possible that some of these required fields aren't available when processing the event and its delivery is in question. The SDK should always have default values when appropriate and try to deliver the events regardless, merely flagging the problem to be fixed later. The events should not be discarded though, it's better to have incomplete events than not having events at all.
Flag duplicate events
It's always possible for the SDK or the app to generate duplicate events by mistake, so it's important to flag these events as they progress through the SDK. Always deliver them regardless, for the same rationale as the previous point.
Have a background task
When an iOS app goes to the background, the system provides a courtesy period where the app can still execute freely. However, this period is volatile and can’t be taken for granted. Since going to the background might trigger event creation and event flushing, it’s useful to have a background task to try and prolong that courtesy period and allow those operations to finish. A mechanism that works well is simply having a background task that pools the internal state of the SDK and completes only after all operations are done.
Optimize the SDK as much as possible
This one is obvious. Benchmark the SDK and be aware of what operations are taking more time, especially handling corner cases. The app can crash at any time, so more time spent working equals bigger risk of losing events to unforeseen scenarios.
Regarding data validation:
Always compete against a control solution
It’s incredibly important to have an external Analytics SDK competing against your own, serving as a baseline. That way you always know if your SDK is behaving correctly on corner cases. You must always be able to match events sent by the baseline SDK with events sent by your own.
Start validating your data as early as possible
Together with Data teams, start validating the data your SDK sends as soon as you can. It's often hard to pinpoint the root cause of bad data, both in and outside of the SDK. Fixing these problems can also take time because you’ll frequently need to ship code to production to guarantee its resolution. Therefore, starting the data validation early is critical.
Looking forward
This project was a great challenge from start to finish, given how it involved different teams across different time zones, all pitching in. The collaboration was a great success, not only in terms of the finished product but also in inter-team communication and discussion. I believe that Farfetch’s values led to that. #BeBrilliant #TodosJuntos
The Omnitracking iOS SDK is already bringing value to quite a few apps in Farfetch and that’s only the beginning. The fact that the Omnitracking Contract evolves over time means that this SDK will evolve over time too, so this collaboration across many teams will continue in the future and I’m looking forward to it.
Penned by Diogo Balseiro, on the behalf of the Mobile Domain.
