The beginner product manager’s guide to simple post release metrics
Note: Any data/numbers mentioned in this article are just taken as an illustration.
Product managers often use the word — ‘measure’. They seldom explain what it means.
Now, ‘measure’ is something that can be done in 3 stages — initiation (before development), development and post release.
All three mentioned stages call for different ways of measuring and gauging successes and failures. But, in this write-up we’ll only talk about the Post Release metrics that are easy and important to learn for any PM new to the role.
If we think about the key things that a firm might want to achieve from a product, post release (let us consider an e-commerce product for ease of understanding), we’ll land up with the following goals:
Goal # 1. User Growth- Increase the number of users
Goal # 2. Buyer Experience and Promotion- Positive experiences and promotion of the product
Goal # 3. Monetization- Through increased transactions or other monetization strategies
To meet the mentioned goals, one needs to measure success/failures on different parameters for 3 key functions i.e. Business, Technology and Product at a regular interval to devise and revise strategies.
We can list out some simple metrics for these functions and will later trace them back to our mentioned goals.
Now, knowing that there’s a possible overlap in the required data across our focus areas, let us try and list out the output metrics and their frequency (daily/monthly/quarterly) across the focus areas.
(Again, assuming that we’re looking at the best suited ones for each)
While the mentioned metrics are the simpler ones, there are many other important metrics for the product like customer effort score (something that needs a dedicated chapter on), access and logins to new screens, unique users, data collected, hits/access on old screens, number of full workflow transitions, number of deadlocks, growth (YoY, MoM, QoQ) etc. , technology-focused metrics like application health, load time, errors (File not Found i.e. 404, Unauthorized i.e. 401, Internal server errors i.e. 500, Forbidden i.e. 403, Bad requests i.e. 400, Service unavailable i.e. 503), Failures (Login, Access) etc. and those for business — conversion rate, etc.
Now, let’s try to understand what each of these mean and how we obtain them.
BUSINESS/REVENUE METRICS
- Cost Per Acquisition
From a business perspective, you need to be mindful about how much the adoption exercise costs in terms of marketing, sales, offers, campaigns etc. Also, we need to keep in mind the customer churn alongside, in order to ensure that the changes aren’t negative on the loyal user base. To measure the CPA of a campaign, total your costs for that campaign and the value lost in churn and divide it by the conversions or acquisitions that the campaign produced.
CPA= (Cost of Campaign + Old CPA *Churn#)/Conversions
The above can be used to determine
- The Cost incurred in the process of user acquisition and the parallel churn. To reduce the costs, we can look at additional metrics like Churn User Attributes and NPS and decide how we can reduce on the overall CPA.
- Also, this metric can act as an input for determining the overall profit in the end.
2. Life Time Value (LTV) and Average Revenue Per User (ARPU)
The purpose in the end is eventually going to tie back to the value proposition through every exercise.
Hence, it’s important to understand the value obtained from the customers acquired. Lifetime value formulas will help you assess whether you’re paying too much for it or getting a good deal on your customers, given the value they bring to your company. The takeaway here is that LTV should be greater than CPA.
You can also calculate the current LTV your app has achieved since launch, by simply dividing the total revenue of your app by the number of users you’ve had. That’s called ARPU or Average Revenue per User.
LTV= Average Value of Conversion * # of Users * average customer lifetime
ARPU= LTV/#of Users
In this case (assuming we’re starting from launch and the values are cumulative thereby also receiving the ARPU),
The above can be used to determine:
- The value obtained from the users individually, and overall, across a lifetime.
- How much the Life Time Value is rising with respect to the user base.
- This also needs to compare with the competitive firms to ensure that the trends are at par or ahead of the market. Any negative variations might require us to evaluate the market, feedback and product.
3. Return of Investment (ROI)
Return on Investment (ROI) is a performance measure, used to evaluate the efficiency of an investment or compare the efficiency of a number of different investments. ROI measures the amount of return on an investment, relative to the investment’s cost.
ROI% (for the quarter) = (LTV- Cost Incurred)*100/ Cost Incurred
The above can be used to determine
1.The rate of growth, as opposed to the investment.
2. This can be compared with competitive firms and benchmarking.
4. Click Though Rate
The click-through rate indicates success of your content/ad targeting. By calculating how many people actually end up clicking on the ad after seeing it, it reveals the strength (or weakness) and the quality of your ad copy, imagery, positioning, and keywords. Improving the click-through rate is one of the fastest ways to increase conversions and ultimately generate more sales. To figure out whether you’re doing a good job at paid search and display advertising or with your email marketing campaigns, try comparing your CTR to your industry averages.
Click-through rate (CTR) is a performance metric expressed in percentages that measures the amount of times an ad or an email is clicked versus the amount of times it’s been viewed (impressions).
CTR = (# Clicks / #Impressions) * 100
The above can be used to determine
- How many users find the impression interesting enough.
2. Changes required in the algorithm or personalization engine.
5. Monetization
This allows you to assess the revenue generated by the company through the users.
In an app (paid app), the monetization can be done through two major channels i.e. ads or transactions. For subscription, we need to include the subscription fee based on the region (Asia/ EMEA WHEM assuming $20 and $50 per annum and $5 and $10 per month respectively) and the subscription type (annual or monthly).
Monetization (Monthly) = Ad Monetization + Value Earned through transactions + $20 *Asia annual subscriptions + $50 * EMEA or WHEM annual subscriptions + $5 *Asia monthly subscriptions + $10 * EMEA or WHEM monthly subscriptions
The above can be used to
- Determine the rate at which the firm is making money.
2. Evaluate the actual money made from various channels of revenue.
3. Check for high/low monetization and work on parts that need attention.
GENERAL PRODUCT METRICS
7. Retention Rate
This metric will help you understand the rate at which your users are retained. And as a result, you’ll be able to also see how many customers you’re losing.
Retention Rate = #Repeating Users Today/ #Active Users from Launch (Comparison) Time
The above can be used to determine
- If the users are having a good experience on the application and would like to return.
- Whether the growth in the user base is a positive sign or a result of a voluntary and high cost campaign.
- If there is a dip in the numbers which can be compared with competitive firms (e.g. Snapdeal, Facebook etc.) to gauge whether the situation is due to market conditions or our own drawbacks.
8. Churn Rate
Using this metric, you can calculate by subtracting your retention rate by 1. You’ll find what percentage of customers are choosing to leave your app, or are “churning.”
Churn Rate = 1- Retention Rate
The above can be used to determine
- The percentage of users who could be potential NPAs (Non-Performing Assets) for the app.
2. Further metrics with user categorization (on region, operating system, age etc.) and market analysis may help with any anomaly detection around whom and where we are experiencing the churn, thereby, assisting to find the reasons and working on it if required.
9. Daily Active Sessions (DAU)
Daily sessions per DAU gives you an idea of how often your customers make use of your app within a single day. This can help you determine whether your customers are returning to your app as often as you’d like them to.
Daily Active Sessions/DAU = Day’s Active Sessions / #Active Users
The above can be used to determine
- The frequency of usage of the application, thus, helping determine the experience.
These metrics are better assessed at a monthly level.
10. Transactions through the app
For a product and business manager, conversion is just half job done and a successful transaction is full job done. It’s important for the business to know the number of transactions happening in order to understand what percent of users logging in are actually returning value for the app through transactions.
The above can be used to determine
- The increase in the number of transactions, hence, gauging the tangible value add through purchases.
2. The percentage of sessions ending in transaction (these are great for the inclusion of Add to cart events, Navigation time etc. to understand why the sessions don’t end up in transactions).
11. Stickiness
This lets you know how often your users keep coming back to the app.
Stickiness= Daily Active Users (DAU) / Monthly Active Users (MAU)
The above can be used to determine
- The interest of a user and how repeatedly they use the app.
2. The likelihood of increasing the stickiness and collaborative attempts to convert maximum logins into transactions.
12. Direct and Indirect Net Promoter Score (NPS)
The Net Promoter Score is calculated based on responses to a single question -How likely is it that you would recommend our company/product/service to a friend or colleague? The scoring for this answer is most often based on a 0 to 10 scale.
We can determine this using 3 common metrics Direct input from user, App store rating, Feedback sentiment analysis (assuming that the data set is collected for a sufficient number of users from multiple channels like Twitter, Instagram, subjective ratings etc.)
Let’s establish the general sentiment through each value for the channels below
- Direct input from user
2. App Store Rating (combined average of iOS and Android)
3. Feedback Sentiment
The above can be used to determine
- The likelihood of users promoting the application.
2. The trends around the general sentiment for the application.
3. This can be broken down and categorized further on the basis of Operating System, Region etc. Further, a competitive analysis can help understand the differences and the reason for any sentiment.
13. Brand Awareness and Social Media followers
This can help the business and product determine the kind of growth seen in the social media/ public visibility.
This can be broken down into 2 buckets i.e. Brand Awareness (by a multichannel survey to know if users have heard of the brand) and assessing social media followers (Facebook/Twitter etc.). This can also be compared with the competitors to analyze the market situation.
In this case, we can analyze the same for a given region (for instance, India)
The above can be used to determine
- The position of the company in terms of market visibility.
2. Any variations from the expectations which can help decide the marketing/campaign strategy.
14. Payment Selection Methods
This shall help the Product Manager to understand the methods of payment selection over a period of time, to move in a direction to get traction towards the favorable channels for the organization.
These can be generated monthly to understand the trends and work if required.
The above can be used to determine
- If cash/card on delivery is being used frequently. This adds to the logistics management, the other channels that can be worked upon to make the experience seamless.
E-Wallets, for instance, are quick channels in general. If their usage is less, the reasons can be gauged from feedbacks, failures etc. Additional competitive analysis can also help understand the market trends.
ENGINEERING METRICS
16. App Crashes
This gives the number of app cashes per month for the engineering to evaluate the logs and work on fixing the key causes. This scan be further broken down into the Operating System level details.
The above can be used to determine
- How many crashes are really happening per session.
2. The trends along with the user journey logs and anomaly detection metrics which help replicate the faulty scenarios.
17. Checkout Failures
This gives the number of checkout failures per month for Engineering and Product to evaluate the logs and work on fixing the key causes or working with the transaction partners to mitigate any issues. This can be further broken down into the Operating System level details.
The above can be used to determine
- If there’s any glitch in the application, by assessing the appropriate logs and corresponding patterns.
2. If there’s any pattern in a transaction channel that can be worked upon internally or with the partner application.
18. Concurrent Users and Peak Load
This can help the Engineering department determine the performance and server needs as per the peak load experienced.
The above can be used to determine
- The server and load related requirements for infrastructure.
2. Expected load from scalability can help determine the future loads.
METRICS — GOAL RECONCILIATION (TRACEABILITY)
Goal # 1. User Growth: Increase the number of users
To know the growth in the app, we can make use of the metrics like daily/monthly active users, transactions, social media followers, brand awareness, site traffic, Net Promoter Score and gauge the level at which the user base / awareness is increasing or can be expected to increase.
Goal # 2. Buyer Experience and Promotion: Positive Experience and Promotion of the Product
Net Promoter Score can help you determine the general sentiment of the user base for your app and that if they are likely to recommend it to the others to use. A direct survey has been conducted in this case. But, app store rating, app store feedback sentiment, retention rate and churn rate could also help understand the Net Promoter Score collectively. Whereas, Customer Effort Score could hint at a projected churn or scope for improvement.
Goal # 3. Monetization: Through increased transactions or other monetization strategies
Retention Rate and Churn Rate inform you about how many users like returning to the app. We have the data around stickiness (to know if people love the app and like to return). App crashes and checkout failure will help us understand the smoothness of the experience. Also, the overall NPS will make us aware of the opinion that the user base has. All these metrics are directly helping us determine the buyer experience.
For determining the reach, Conversion Rate, Impressions and Click Through Rate are good metrics to determine how many notifications/ads are published and how many are viewed. It also helps to determine how successful the product targeting is.
Regarding monetization, overall monetization metrics will help us know the money spent on campaigning for adoption. Metrics like Life Time Value, Average Revenue per User, Return of Investment give clarity on monetization directly through user transaction. Also, brand awareness and social media follower metrics might help understand the market visibility and take appropriate decisions for scaling the same.
Revenue and transaction metrics shall help understand the tangible value obtained from the adoption.
The nature, type and frequency of the metrics may change on the basis of business and the person. But, the gist continues to remain the same — MEASURE!
