An AI Crystal Ball? How We Predict Future Outcomes Using a Temporal Fusion Transformer Model

Our data science and analytics teams handle and apply lots of data for insightful decision-making. Last year, I presented the data science team with a challenge: use historical data to predict a key business driver for each of the next 8 periods. We wanted to have a data-driven preview of what we might see in the in each of the next eight periods so that we could anticipate the actual outcome and make better decisions with a an eight-period.

The data science team went to work researching ways we could do this and tested a few different methodologies. We have lots of input data from our own and public sources to feed any model we wanted to test, which worked well for us. With that said, we had low expectations about finding a predictive model that produced anything reliable.

Testing different algorithms is always our approach. For the semi-technical readers, before settling on Temporal Fusion Transformer (TFT), the algorithms we tested included ARIMA, VAR, GARCH, ARCH models (univariate), Prophet, NHits, and Nbeats. TFT is an attention-based deep learning neural network algorithm. Using a mix of inputs, it produces a forecast over multiple periods in a future time horizon that you can determine. You can predict days, weeks, months quarters (really any interval is possible) into the future. Your choice.

The picture below shows the concept of how TFT works.

Source: Bryan Lim, Sercan Ö. Arık, Nicolas Loeff, Tomas Pfister. “Temporal Fusion Transformers for interpretable multi-horizon time series forecasting.” International Journal of Forecasting. Volume 37, Issue 4, October–December 2021, Pages 1748-1764.

A continuous improvement process best describes how we developed and continue to refine the model. It’s a never ending process of improvement, as a true crystal ball is never achieved.

These are the four stages of development we went through after choosing TFT as our algorithm:

• Stage 1a: Selecting all logical observed inputs and test how they drive the model. We started with over 100 and the final model only used 20. Go to Stage 1b as needed.
• Stage 1b: Refining the time intervals of the observed inputs. Since the inputs might come in varying time intervals (daily, weekly, monthly and quarterly, etc…), we needed to find methods to standardize them. You should choose an interval that matches the decision-making forecast you are producing, if you can. Go back to Stage 1a as needed.
• Stage 2: Model iteration and improvement. Complete back testing. Examine early predictions. Go back to Stages 1a and 1b as needed. At this point, you have probably settled on one or two of the most promising algorithms.
• Stage 3: Begin using in production and comparing predictions to the future periods as they unfold. Learn and refine by going back to any previous stage as needed.
• Stage 4. Continuous improvement loop. Write long-term road map. Test new inputs as they are presented. Continuous scrutiny of the predictions against what actually happens – learn and make changes by going back to any previous stage as needed.

Note that at any of the stages of development, you can use a TFT encoder decoder to measure the importance of different inputs in the algorithm to learn which ones have the most impact on your prediction.

Below are the results of our model. The orange line is the actual result of the key driver and the blue line is the prediction of the key driver that was made 8 periods ago. The area to the right without the orange line is the next 8-period forecast. So, at Period 19, we can use the blue line forecast to take action based on what Periods 20-27 tell us. When we reach Period 20, we evaluate the updated forecast and we make a decisions accordingly for the future periods. This way, we have a rolling 8-period prediction/decision cycle.

As you can see the model has been refined to a level that it makes useful predictions and handles volatility of the prediction with some reliability. Right now, we don’t use this to predict the future down to the exact number, which would be ideal, but we use it for a directional understanding of where things are headed so we can make better decisions at the current decision point.

Key Consideration For SaaS (or Any Recurring Revenue) Financial Models

In SaaS, decoding revenue dynamics is pivotal for pushing the business forward. Let's talk about the elements of financial modeling tailored for SaaS companies:

1. Revenue Insights:

MRR (Monthly Recurring Revenue): This quantifies the predictable monthly revenue, offering immediate insights into short-term revenue trends. In my experience, I build monthly forecasts and report on the business against the forecast monthly. Having a predictable MRR with less than 1% variance to the rolling 90-day forecast is achievable and ideal. (Of course, early in the business these variances could be higher.)

ARR (Annual Recurring Revenue): An annualized view of MRR, guiding long-term planning and providing a comprehensive overview of revenue trajectory. Often ARR is used to give investors a sense of how much revenue the business has on the books that will repeat for the following year. This gives comfort to investors who see this as a baseline of revenue helping fund the company. Personally, I think contracted backlog is a more interesting way to look at this same element of SaaS, but I will cover that another day.

Churn Rate (aka cancellation rate): Measuring customer subscription cancellation, influencing MRR and ARR. Managing and reducing churn is crucial for cost-effective customer retention. Churn has two important modeling conventions that you should consider: first, does your cancellation rate change with the age of the client or contract. This is heavily influenced by the contract duration, but if you have no duration, this is an important factor to consider. Second, when modeling, it is often easier to model client counts as retention, which is (1-churn%). Always be sure that you are applying this correctly as there is a difference between churn-to-date and churn since the last period.

2. Cost Projections:

COGS (Cost of Goods Sold): Direct costs related to delivering the software service, impacting gross margin and signaling operational efficiency. Accurate forecasting is vital for profitability projections. Cloud services and direct IT support of software delivery and up-time fit into this bucket.

Operating Expenses: Day-to-day operational costs affecting operating margin and overall profitability. Monitoring ensures business efficiency and agility. This includes more typical overheads like rent, sales and marketing costs, R&D and management.

3. Customer-Centric Metrics:

CAC (Customer Acquisition Cost): Evaluating the average cost to acquire a customer. Discrepancies between CAC and customer LTV (Lifetime Value) indicate marketing or sales process inefficiencies. CAC should include all sales and marketing costs, including sales overhead for things like a CRM software, pre-sales scheduling and sales management. If you leave these items out, you are really looking at marketing acquisition cost. It’s useful in some cases to do this, but CAC, especially when you are running dynamic LTV analysis.

Retention Rate: Depending on how you want to use this, it could be a very granular financial model component. Otherwise, it can simply be the percentage of retained ARR over a specified period. The latter example again is an important metric to help convince investors you have a stable source of revenues.

LTV (Lifetime Value) aka LCV (Lifetime Customer Value): One basic approach is to calculate this as total gross profit from a customer throughout the lifecycle of the client. Personally, I like to be very granular with this and I use specific components of the above for the analysis: -CAC, +churn adjusted revenues, -churn adjusted COGS = LTV Contribution and -allocated overhead = Net LTV. In addition to the final LTV values, I look at the following ratios: LTV Contribution to CAC and LTV Net to CAC. Note: Churn adjusted revenues and expenses are very useful when you have client with changing cancellation rates over time. Pro tip: You can also look at this by subscription cohorts if you sign up a lot of contracts each month.

4. Financial Health Analysis:

Cash Flow: Tracking cash movement for informed management of working capital and expense management.

Break-Even Analysis: Predicting profitability by determining the sales volume needed to cover costs. Essential for strategic pricing and sales strategies. In this area, it’s useful to look at the count, value and consistency of new contract additions in the forecast to determine when the business becomes profitable.

Understanding these components offers a complete view of the current and future operation, empowering leaders to make informed decisions aligned with growth objectives. The interplay of revenue insights, cost projections, customer-centric metrics, and financial health analysis forms the bedrock for a robust SaaS financial model.

Unraveling the Complexities of Consumer Balances Sheets Post-Covid

As discussions go round and round regarding the potential impact of consumer finances on an impending recession or a more gradual "soft landing," recent media narratives have brought attention to a remarkable surge in the savings rate following the COVID-19 pandemic.

While the savings rate serves as a useful metric, providing insights into where consumers are directing their excess cash, it falls short in offering a nuanced understanding of the actual cash buildup. Recently disclosed data has ushered in a paradigm shift, challenging preconceived notions about consumer savings. The revelation suggests that consumers have saved more cash than previously estimated, with reported savings rates reaching 15.4% in 2020 and 11.4% in 2021—figures that are undeniably significant.

Prompted by this new data report, my focus has been directed towards the Federal Reserve's reporting on Currency and Checkable Deposits, a comprehensive analysis tracking the ebb and flow of cash in checking accounts. This alternative data perspective paints a more optimistic picture of the consumer's financial standing, revealing that the average US consumer is holding multiples of cash compared to pre-COVID levels. This newfound understanding underscores the robustness and sustainability of consumer balances, fueled by higher average savings rates.

Let's explore how Currency and Checkable Deposits have evolved for different segments of the population, particularly the Bottom 50% and Top 50% of households.

Cash and checkable deposits for the bottom 50% of households by wealth. Federal Reserve, 2023.

Cash and checkable deposits for the top 50% of households by wealth. Federal Reserve, 2023.

The charts vividly illustrate the substantial growth in cash availability, with the Bottom 50% experiencing a 2.5x increase from January 2020 to August 2023, and the Top 50% boasting over 3.5x more cash during the same period. Intriguingly, both groups have begun consuming their accumulated cash, a trend that commenced in June 2022 for the bottom 50% and October 2022 for the top 50%.

These data points collectively paint a comprehensive picture, indicating that, on average, US consumer balance sheets remain resilient and robust. The runway for these balance sheets to return to pre-COVID cash levels appears long. However, it's crucial to acknowledge that these are averages across all households, and the sustainability of cash levels varies by the decile of wealth.

In dissecting this narrative further, it becomes evident that the erosion of cash is not uniform and is likely to follow a bottom-up trajectory. This divergence is observable as the bottom 50% experiences negative growth since June 2022, preceding the top 50% that exhibits similar trends from October 2022. The implication is that less affluent households might feel the impacts of a potential recession earlier than their wealthier counterparts.

As we navigate the complex terrain of economic forecasting, it's essential to recognize that while consumer balance sheets play a role, they may not emerge as the primary driver of an economic slowdown. Other economic factors, such as hiring trends, wage dynamics relative to inflation, and industrial output, are anticipated to wield a more substantial influence in shaping the trajectory of the economic slowdown.

Reference: https://shorturl.at/uJLO1