z

Model performance metrics

• accuracy
• precision
• recall

Reduce variance

Sometimes the model can overfit the training data and hence underperforms with real data. To reduce complexity, we use regularization.

• Add a penalty factor on complexity i.e. number of features and values of weights.

• Lasso: Force coefficients to zero if not relevant. Ends up performing feature selection.

• Ridge: Reduces coefficients of irrelevant features but does not drop them to zero.

Derive new features

Predict numeric values

Linear Regression builds relationship between inputs (features) and outputs (targets). It is a simple yet effective model to start with to get a benchmark.

• Using training data, the goal is to minimize the loss.

• You can measure loss function using SSE

Polynomial Regression: Linear Regression can also be used to model a non-linear relationship. To achieve this, we transform an input feature by a non-linear transformation function and create a new feature that can be used as input in the model.

Chinese proverb: One chopstick is fragile, but a group of chopsticks is robust.

A big challenge with ML models is overfitting to training data. And hence, we can use ensemble models — average multiple models into a meta model, which in turn reduces variance.

Identify unusual cases

Drive Top of the Funnel from various Channels.

Best practice: Use UTM parameters appropriately so you can attribute traffic to each channel.

When retention fails, you churn, i.e., lose customers.

Why it matters: In any business you want to make sure that you are adding customers faster than you are losing them and at the right Customer Acquisition Cost.

• Sometimes churn is good because your product isn’t for everyone! It is better to let them go than to keep supporting them or serving their needs.

Types: E-commerce businesses will measure transaction churn. In contrast, Subscription businesses will measure two forms of churn:

• Customer Churn in a specific period (say a month) is the number of customers lost divided by the number of customers you had at the start of the period. Use this if you have only one Pricing Tier. However, if you’ve more than one Pricing Tier, then you should calculate revenue churn or visualize churn by customer cohorts.
• Revenue Churn in a specific period (say a month) is the total revenue on the first day of the period minus the revenue from the same customers at the end of the period minus any value from Upsell. We need to subtract the upsells because they hide loss from churned customers.

Challenges: Churn is a delayed backward looking and you cannot get a live number.

Examples. Everyone has faced challenges with churn:

• The Wall Street Journal worries about aging subscribers.
• The Economist focuses a lot on retention.

Any new, and better way of doing things is technology.

Why it matters: Technology allows you to do more with less.

How: Technology has never been an automatic feature of history. We have to build it.

Footnotes:

• Zero to One

Cost Per Acquisition

Once someone is on your Internet Products, always push them to consume one more piece of content. Pages Per Session captures how interested are audiences in exploring your website beyond the landing page.

You can do this by capturing the audience’s intent of coming to your platform and serving them basis that intent using:

• Information Architecture buckets all your content so audiences can discover it easily. Additionally, a good Information Architecture helps you improve SEO authority.
• Optimize distribution using Personalization.
• Search interfaces give audiences exactly what they want.

To understand their intent you can rely on the piece of content they are consuming right now or the organic keywords they were searching for prior to landing on your platform.

Cost Per Mille: Businesses book a fixed number of impressions within a date range. Once the order is accepted, publishers must honor the impressions while maintaining reach and view-ability% within acceptable limits.

Net Promoter Score (NPS) is a simple metric to measure customer satisfaction that users can fill up quickly on a regular basis and it doesn’t require any analysis to digest. However, NPS is backward looking. For forward looking, ask Qualitative Feedback.

How to ask

How likely are you to recommend our company/product/service to your friends and colleagues?

The answer takes the form of a score, from 0 to 10, with 0 being not at all and 10 being extremely likely.

How to analyze

You then group your customers into three groups based on their response:

• Promoters (9-10): Customers who love your product and will recommend it to others.
• Passives (7-8): Customers who are ambivalent.
• Detractors (0-6): Customers who are unhappy and may advise against working with you.

Net Promoter Score = (% who are Promoters) – (% who are Detractors)

Most people have a ratings bias, i.e., they avoid giving low ratings and settle for something in the middle. The NPS scale keeps that bias in mind.

Benchmarks

Look for blogs talking about NPS to get an idea about benchmarks.

Advertisements sold via an intermediary ad networks like Google Ads, Taboola, etc.

If you find a heavily discounted product, then buy it and resell it quickly in a secondary market at a higher price.

Sessions Per User communicates how frequently users return to your product. At a macro level, this idea is also communicated via DAU/MAU. To improve it, use following levers:

• Off platform capture intent by perfecting SEO.
• Attract unidentified audiences to your platform using Brand Marketing and Social Media.
• Launch scheduled Habit-forming Products like Newsletters and Podcasts.

Once they land on your platform, use Audience Engagement and Onboarding to get them to:

• Identify themselves, i.e., login, sign up to Newsletters, etc.
• Download and use your Mobile Apps
• Try out Habit-forming content

Eventually, with Identified Audience, run event-triggered and personalized communication using Newsletters and Emails.

Hacks

• If you take a screenshot of a Tweet, then Twitter shows you this pop-up. Growth hacks to increase Sessions per user

The time you spend courting the media so you can get Referral Traffic.

A customer lifecycle is a form of segmentation or funnels that maps customers based on how proficient they are at finding and using your product and features.

Why it matters:

• Build awareness and tracking of how segments of customers move across levels of expertise.
• This helps you take decisions on design, features, communication, and pricing tiers to drop churn.

Subscription Lifecycle

Pre-membership: What ought to happen before users subscribe? Typically, this would involve getting users to register (login), take a Free Trials, etc.

Early-days: In this situation use Onboarding to get users to try out most of the features of the product so that users get the most of their investment. You can further break this phase down into two stages:

• Newbie: These are people who just started using your product for the first time.
• Student: These are people who are learning your product but haven’t yet mastered it.

During subscription: Use Habit-forming to expand frequency of usage. Create unique flows for users who are disengaged.

Pre-renewal: Either Upsell, cross-sell, or right-size the user so that the right plan can be targeted to the user.

Post-renewal: After Churn what is your Win-back journey.

What caused the conversion?

Establishing exact causality is very hard. For example, unknown to us, a conversion event might have occurred after the lead read one of our Candid Communication blog posts, seen a Digital Advertisements, received our Newsletters as a forward, or saw a post in their Social media feeds.

• There is also no way to perform A/B Testing to discover such causality because all of these actions are happening outside our platform.
• Additionally, conversion rate varies wildly based on the acquisition Channels.

So, who gets the credit? There are few models that can help estimate attribution:

• Distributed: Divide the credit across all Channels that the lead interfaced with.
• Last Touch: Assign 100% of the credit to the last Channels the lead interfaced with.
• Decaying: Distribute the credit across all the Channels but given highest weightage to the last channel.