Can We Measure What Matters?

Every November, I challenge myself by selecting and reading books that I would typically avoid—whether it’s a genre I avoid or an author or topic that I disagree with. This practice has been enlightening, albeit occasionally frustrating, especially when faced with a book that is more difficult than I expected. (I strongly encourage you to make this an annual tradition, even if it’s as simple as reading one article or opinion piece.)

While searching for this year’s books, I posed a question to myself: “What topic would I rather NOT write about in our Team Letter? What should we all be thinking about more right now?”

The answer: the significance of metrics in marketing and how we measure our work.

This letter addresses my stance against letting metrics dictate our team’s decisions, juxtaposed with the need to evaluate and quantify our efforts. If that’s not confusing enough, keep reading.

First, the danger of metrics and why I typically avoid metrics for our team:

The Cobra Effect:

In colonial India, British officials in Delhi offered a cash reward for each dead cobra, aiming to reduce their population. However, locals began breeding cobras for profit, ultimately leading to a surge in the cobra population when the program was terminated.

Lesson: Metrics can trigger unintended consequences due to misaligned incentives.

Soviet Nail Factory Metrics:

In the Soviet Union, nail factories prioritized producing a vast quantity of small nails to meet a production quota. Altering the metric to nail weight inadvertently resulted in the production of heavy, unusable nails.

Lesson: Focusing solely on a specific metric may undermine the intended goal, emphasizing the importance of capturing what truly matters.

A/B Tests:

Lars Lofgren is an exceptional marketer. He was early at KISSmetrics and later helped grow IWT (Ramit Sethi’s company). He’s one of the smartest marketers I know; one time we were having coffee in NYC and he told me about his approach to A/B tests.

At the time, I was at Sticker Mule where we ran “big and meaningful” A/B tests. In short, he said three things: 1) the control stands, 2) you have to wait longer than you think to pick a winner and 3) you need 99% statistical significance. Soon after that conversation he published this article: My 7 Rules for A/B Testing That Triple Conversion Rates (if you’re on the Growth team, this is required reading!). In short, Lars showed me that metrics need time and without it, the data often tells a different, inaccurate story.

Lesson: Without the right guardrails, you can inadvertently pick the wrong choice while thinking you were right. Data can be misleading if not understood in the proper context or proper timeline.

Second, metrics have a place (or learning to prove myself wrong):

Let’s use this quote as a thought experiment: “If you can’t measure it, you can’t improve it.”

We have to ask ourselves:

• What do we want to improve?
• What matters most in our work?
• What is our team’s contribution to Apptegy?

Now, let’s work that backward (based on the previous team letter):

1. Our team’s objective or contribution to Apptegy is to “make sales easier” and “hold a conversation with the entire K-12 market.”

2. What matters most in our work is to “get and keep attention, establish trust and credibility, make the complex simple and accessible, and, most importantly, persuade someone to take action.”

3. Now, what do we want to improve?

That’s where I want each team and person to spend some time thinking and writing. The last few letters have been intended for you to read and think. This one is more typical work.

Your task is to:

• Block off 35 minutes to ask yourself what do you and your team want to improve next year.
• Come up with at least 5 and no more than 10.
• Check your list against what you think is most important in your work: your unique contribution, objectives, and what matters most.

As you do, don’t forget about our contribution to Apptegy and what matters most in our work. Start by listing out all the things you want to improve (don’t worry about measuring it, yet). Then, as a team, I want us to all talk about what we came up with to discuss how (or if) we can measure those over the course of the next year.

I’m doing the same and my hope is that I can prove myself wrong and find a way to measure what matters most and make meaningful improvements over the coming months and years.

We’ll meet on Tuesday, November 12th to discuss what each person and team creates.

I’ll end on this quote, from the great marketer Seth Godin, “Just because something is easy to measure doesn’t mean it’s important.”

However, the inverse isn’t always true: just because something is important doesn’t mean you cannot measure it. Keep that in mind as you think and write out your areas to improve.

---

P.S. I haven’t selected my books yet so if you think of something I wouldn’t like or a viewpoint I should consider but am unlikely to read on my own, I’m all ears.

P.P.S. If you haven’t read it, read Alchemy by Rory Sutherland or watch his TED Talk again or watch this other (very similar) TED talk.