Why your CEP should be built on top of your Data Warehouse

Arun Thulasidharan

Aug 30, 2022

3 mins read

CEPs or Customer Engagement Platforms have remained the backbone of all kinds of businesses varying across different verticals and growth stages for decades now. Customer engagement processes have also evolved drastically, with marketers/growth teams trying new approaches and channels to lure their customers. CEPs have stood the test of time amid this ever-changing marketing landscape and have constantly added features to keep the growth/marketing teams up to date with the latest trends.

So all is well with the world of CEPs? Well, not really! In this blog, we intend to share the learnings from the interviews we conducted with 100+ customer engagement teams. In the following sections, we will discuss in detail the major pain points they face with their CEPs and the solution we propose to solve their problems out of the box.

Exorbitant Data Volume-Based Pricing

Scared of your finance teams/business execs eyeballing you before you decide to start a campaign or sync a new event to your CEP?

If this sounds familiar, do not worry. You are not alone! Cost is, without any doubt, the prominent headache growth marketers across the globe face with the existing CEPs. Almost all the CEPs like Braze, Iterable and CustomerIO are priced in terms of the data points consumed, which can get exorbitantly expensive in this era of digital transactions. In such scenarios, your customer engagement teams are often forced to make tradeoffs in campaign efficiency to keep their marketing budgets at bay.

But why do the current tools charge in terms of data storage?

One of the primary reasons for this pricing mode is that it is challenging to build a product that scales to hundreds of billions of data points across thousands of customers. With such a product, probably 80% of the engineering cost, which comprises the developer cost, infra cost, maintenance, and support cost, will be spent on solving the scale at hand rather than building features that provide real business value to the end user.

Even from a business point of view, it makes sense for the CEPs to charge in terms of data storage since more storage means more value. But we believe that the current way of pricing exponentially in terms of data storage is not giving a fair ROI back to the marketer.

Restrictive Data Look-back Periods

Almost all the major CEPs in the market provide a data look-back period not exceeding 180 days by default. Imagine you having to pay an unreasonably high amount of money even to make sure that your most loyal customers are not treated as first-timers. You would want to remember the health package they took six months back or the products they bought from you in the last Christmas Sales.

This restriction can also be traced back to the limitation of CEPs in storing and processing massive amounts of data points across thousands of customers.

Unavailability of Quality Data

There is no doubt that the CEPs that you use currently are extremely powerful in the functionality they provide. But all of that amounts to nothing if you do not have access to the right set of product usage data in your CEP. Currently, the customer data/events are fed to the CEPs using SDKs on your website or your transactional backends. Unfortunately, this data is incomplete and does not contain a 360-degree view of your customer. Many critical customer interactions happen outside your core application in third-party sales or support tools that are often missed by these SDKs.

Your cloud data warehouse is the only source of data in your entire product-tech stack that contains a comprehensive view of your customer’s product usage. Marketing/Growth teams have recently started to realize the value of the customer data in their cloud data warehouses. As a result, they have been using Reverse ETL to sync their customer data from their data warehouse back to their CEPs. While Reverse ETL solves this problem to an extent, the downside of this approach is that it adds an unwanted dependency on your data teams to create and maintain those complex data pipelines.

Tedious Grind of Getting Started

Imagine you having evaluated and purchased a Customer Engagement tool. How much time does it take to start your first campaign? Also, if you are migrating from an existing tool to another, how much time does it take to migrate all your campaigns to the new tool and stop using the old one? Finally, are you tired of asking your tech team about the status of the developer tickets you raised?

Just because you bought a CEP does not mean you can start engaging customers right away. You will need developer time from your tech team to integrate the users/events/custom objects that you need to use in your campaigns. That’s when the next problem arises. It’s almost always the case the tech resources in a company are scarce, and their main priority is to build the core product.

So in all possibilities, the tickets you raised for your integrations will not be picked up for at least a few months in the best case and for more than a year in the worst. You are potentially losing a lot of money in opportunity costs during this time. Now, this is not even a one-time effort. The same cycle must be repeated when you decide to change your CEP, which is a fairly common scenario when your business grows.

Data Warehouse As the Backend

The solution to building a more flexible, cost-effective, and robust CEP is to build it natively on top of your cloud data warehouse. Your data team has already done all the hard work by bringing all the user/events data to your data warehouse for analysis. They have also created critical semantic models relevant to your business using a modeling tool like DBT. All that would be left for you to do is to create dynamic segments directly on top of those models and run campaigns/customer journeys using those segments.

Now let’s see how the newly proposed solution solves all our problems.

Since the CEP is built natively on your cloud data warehouse, it does not have to store your users/events/custom objects’ data. This drastically reduces the cost of a warehouse-native CEP by almost 80% compared to a traditional CEP.

Again, since all the events’ data are stored in your data warehouse, there is no restriction on data retention/look-backs from the CEP side. Storage in a cloud data warehouse is exceptionally cheap, and you should be able to retain all your events for as long as you need.

There is no need to add a dependency on the data teams to bring the data from the warehouse back to your CEP using a Reverse ETL solution. You can use the segment builder directly on top of the product usage data lying in the warehouse to create your segments.

All the data you need to run your campaigns are already lying in your data warehouse. You can start engaging your customers in a matter of few hours instead of months. Also, since your cloud data warehouse is the single source of data for all your campaigns, you do not have to worry about getting the developer time to migrate the data, even if you decide to change the CEP at some point.