數據驅動的侷限：衡量的是昨天，不是明天

Andrew Chen 電子報內容(補上AI翻譯後的英中對照) We are endlessly surrounded by data but also lack the ability to make sense of it all. It’s enormously frustrating but we commonly encounter discussions with statements like the following: 我們無休無止地被數據包圍，卻缺乏理解這一切的能力。這是非常令人沮喪的，但我們經常會遇到類似以下陳述的討論： “Oh the metrics went down? That’s just seasonality. Next quarter will be better” 「哦，指標下降了？這只是季節性。下個季度會更好” “Yes, the metrics are up! That’s because of this amazing new thing we just shipped” 「是的，指標上升了！那是因為我們剛剛發貨了這個令人驚奇的新東西” “Yes, the A/B test showed this metric went down but this other went up, so it’s actually okay” “是的，A/B 測試顯示這個指標下降了，但另一個指標上升了，所以實際上沒問題” “We have to wait on the A/B test to see how the feature is doing, the data is still collecting” “我們必須等待 A/B 測試，看看該​​功能的表現如何，數據仍在收集中” “I signed up for a 5% increase last quarter on this key metric, and we hit that goal. Why didn’t revenue go up? No idea” 「上個季度我簽署了將這項關鍵指標提高 5% 的協議，我們實現了這一目標。為什麼收入沒有增加？不知道” “People love this feature, just look at the data, we just need to onboard them better!” “人們喜歡這個功能，只要看看數據，我們只需要更好地引導他們！” “Our new product feature is totally working, but the current metrics don’t reflect it since there’s a time lag. Wait until next month” 「我們的新產品功能完全有效，但由於存在時間滯後，目前的指標並未反映出來。等到下個月” “The clickthrough rate on this email was 0.15% and the other was 0.14%. That’s why 「這封電子郵件的點擊率為 0.15%，另一封電子郵件的點擊率為 0.14%。這就是為什麼 Etc etc etc 等等等等等等 Many of you are now smiling because you’ve been there! Both on the receiving end, as well as having these kinds of words come out of your mouth :) I know I certainly have. 你們中的許多人現在都在微笑，因為你們曾經去過那裡！無論是在接收端，還是從你嘴裡說出這些話:)我知道我當然有。 Why is it that we have so much data about how people use our products, and we are able to speculate as to what might make them stickier, or more valuable — yet truth-seeking is so difficult? We get all the aforementioned excuses (and more) because of the “fog of war” as we pick the product. This is doubly critical to figure out as many of our jobs now regularly involve reviewing data, then making decisions. This happens while reviewing dashboards and graphs (at product meetings, reviewing decks, board meetings, etc). Or writing about metrics you’re plan to increase as part of a quarterly review. We are in this fog constantly. 為什麼我們擁有如此多的關於人們如何使用我們產品的數據，並且我們能夠推測什麼可能使它們更具黏性或更有價值，但尋求真相卻如此困難？由於我們在選擇產品時的“戰爭迷霧”，我們得到了所有上述藉口（以及更多）。弄清楚這一點至關重要，因為我們現在的許多工作經常涉及審查數據，然後做出決策。這種情況發生在審查儀表板和圖表時（在產品會議、審查簡報、董事會會議等）。或寫下您計劃在季度審核中增加的指標。我們一直處於這種迷霧之中。 The root cause(s) for all our problems 我們所有問題的根本原因 In our quest to be data-driven, we hit many core problems based on how we collect/analyze/decision the data. Here are some of the variations: 在我們尋求數據驅動的過程中，我們根據如何收集/分析/決策數據遇到了許多核心問題。以下是一些變化： Measuring yesterday when you should be thinking about tomorrow. Data comes from the market you have today, not who you want to have. Products often resonate with a small and loyal audience, but they are often niche, early adopter types. If you make decisions using data from this audience, you may not be able to make the right decisions that get you to a broader market. Your initial nerdy audience might ask for nerdy/configurable features, or expert features, whereas a broader market wants “easy to use.” Your data often doesn’t make this argument, and instead you end up chasing a local maximum. 當你應該思考明天時，卻衡量昨天。數據來自您今天擁有的市場，而不是您想要擁有的人。產品通常會引起一小部分忠實受眾的共鳴，但它們通常是小眾、早期採用者類型。如果您使用這些受眾的數據做出決策，您可能無法做出正確的決策來進入更廣泛的市場。您最初的書呆子受眾可能會要求書呆子/可配置功能或專家功能，而更廣泛的市場則需要「易於使用」。你的數據通常不會提出這個論點，相反，你最終會追逐局部最大值。 Analysis paralysis. The cost of acquiring the data and its analysis can become cost prohibitive. Of course you’d like to have data for every decision. But if you need smart data analysts to pull the data, analyze it, check for corner cases, segment and analyze it, and then call a meeting to discuss (then analyze more, then discuss again), then it’ll just take too long. So it’s often better to just analyze the hell out of big things and otherwise ignore the small stuff, even though a truly data-driven team would bristle at this approach. 分析癱瘓。獲取數據及其分析的成本可能會變得過高。當然，您希望獲得每個決策的數據。但如果你需要聰明的數據分析師來提取數據、分析數據、檢查極端情況、對其進行細分和分析，然後召開會議進行討論（然後進行更多分析，然後再次討論），那麼就會花費太長時間。因此，通常最好只分析大事情，而忽略小事，儘管真正的數據驅動團隊會對這種方法感到憤怒。 Intuition is fast, A/B tests are slow. Related to the last point, sometimes when you become an expert about the customer, and have great intuition, you just know how to do the right thing. And now just the right thing that can move the needle in the short-term, but also the thing that will create better, long-term value. You can make a bunch of these decisions quickly, and just ship the product, and not try to A/B test every single one. Sometimes a startup’s market is moving so fast that this is actually the right approach. 直覺很快，A/B 測試很慢。與最後一點相關的是，有時當您成為客戶方面的專家並且具有很強的直覺時，您就知道如何做正確的事情。現在，正確的事情不僅可以在短期內起到推動作用，而且還可以創造更好的長期價值。您可以快速做出一系列決策，然後交付產品，而不是嘗試對每個決策進行 A/B 測試。有時，新創公司的市場變化如此之快，這實際上是正確的方法。 The tyranny of the majority. When you deal with millions of datapoints of usage, you necessarily have to aggregate people together into segments. You look at averages, and compare segments. Yet sometimes these groupings are wrong — what about the VIPs? What about the high value users? If you make decisions that appeal to a broad audience, you might attract low-value international users at the expense of high-value US users — is more always better? It takes a while to unentangle which segments you actually care about, particularly if they are the minority of your users. 多數人的暴政。當您處理數百萬個使用資料點時，您必須將人員聚合到不同的部分。您查看平均值並比較各個部分。但有時這些分組是錯誤的──VIP 又如何呢？那麼高價值用戶呢？如果您所做的決策吸引了廣泛的受眾，您可能會以犧牲高價值的美國用戶為代價來吸引低價值的國際用戶——越多越好嗎？需要一段時間才能理清您真正關心的細分市場，特別是如果他們是您的用戶中的少數。 A/B tests are good at measuring big things quickly, but not other things. The biggest tool that a data-driven team can use is to use A/B tests. Yet to reach statistical significance quickly, and to iterate on a regular basis (weeks not months), you end up comparing big metrics movements that happen quickly. Signup rate is easy to measure, as is clickthrough rate. But what about 90 day retention rate for the minority of your users that are the most high-value? That’s hard. And so often people just ignore them. A/B 測驗擅長快速測量大事物，但不適用於其他事物。數據驅動團隊可以使用的最大工具是使用 A/B 測試。然而，為了快速達到統計顯著性並定期迭代（幾週而不是幾個月），您最終會比較快速發生的大型指標變化。註冊率和點擊率都很容易衡量。但是，少數最有價值的用戶的 90 天保留率又如何呢？這很難。人們常常忽略它們。 There’s a lot of noise. Factors like seasonality adds a lot of noise. So do big ad campaigns, or when competitors launch new products, or even if the month you’re in has an extra weekend versus another. It’s often hard to compare metrics cleanly, so unless there’s a big effect, you often have to just grit your teeth and make a guesstimate. 有很多噪音。季節性等因素會增加許多雜訊。大型廣告活動，或當競爭對手推出新產品時，或即使你所在的月份比其他月份有一個額外的周末，也是如此。通常很難清楚地比較指標，因此除非有很大的影響，否則您通常只能咬緊牙關進行猜測。 Historical data may not predict future trends. This is particularly true with emerging technologies. If you take a product idea, and add AI to it, will that be better or worse? Will that increase conversion or retention or put other key metrics into decline? What if you take an idea and add a web3-first design? Sometimes, technology is just discontinuous and it’s impossible to know what will happen next. Sometimes you just have to go for it. I live with this problem every day in my venture capital job, where you’d like to simplify investing as a “you should always invest when a product has X attribute.” But the problem is, the successes are so few (a few dozen a year at most, out of tens of thousands) that you can’t really build a high-conviction predictive model 歷史數據可能無法預測未來趨勢。對於新興技術尤其如此。如果你採用一個產品創意，並將人工智慧添加到其中，效果會更好還是更糟？這會增加轉換率或留任率，還是導致其他關鍵指標下降？如果您提出想法並添加 web3-first 設計會怎麼樣？有時，科技是不連續的，不可能知道接下來會發生什麼事。有時你只需要努力去做。我在創投工作中每天都會面臨這個問題，你希望將投資簡化為「當產品具有 X 屬性時，你應該始終進行投資」。但問題是，成功的案例太少了（每年最多只有幾十個，數以萬計），以至於你無法真正建立一個高可信度的預測模型 Correlation and causation. There’s a common path of analysis where a product team tries to figure out how their most engaged users became that in the first place. They quickly find that they do some setup step more often than others — like setting their profile photo or inviting a friend or something. The effect looks strong, and then an A/B test is run forcing everyone to take this step. The result is… nothing. It turns out high-intent users do X and then become high-intent. It’s correlation not causation. Shrug. This appears all the time, in many places, and just wastes everyone’s time. 相關性和因果關係。有一個常見的分析路徑，產品團隊首先試圖弄清楚他們最活躍的用戶是如何變成這樣的。他們很快就發現自己比其他人更頻繁地執行某些設定步驟 - 例如設定個人資料照片或邀請朋友或其他什麼。效果看起來很強大，然後運行 A/B 測試，迫使每個人都採取這一步驟。結果是……什麼也沒有。事實證明，高意圖的用戶做了 X，然後變得高意圖。這是相關性而不是因果性。聳聳肩。這種情況在很多地方一直出現，只是浪費每個人的時間。 Some things are important, but hard to measure. How do you measure the effect of a new product feature on long-term brand? You probably can’t. What about how psyched your employees are about a new feature set? Or what investors think? Or maybe you have an n=1 super strategic and important partner you want to impress with some killer new tech. How does that show in an A/B test? Some of these factors are very important in the success of a new product, but it’s hard to get them to show up properly in a graph. Sometimes you just have to make the right call, if these things are important to you. 有些事情很重要，但很難衡量。您如何衡量新產品功能對長期品牌的影響？你可能不能。您的員工對新功能集有多感興趣？或者投資人怎麼看？或者，也許您有一個 n=1 的超級策略且重要的合作夥伴，您想用一些殺手級的新技術來打動您。這在 A/B 測試中如何體現？其中一些因素對於新產品的成功非常重要，但很難讓它們在圖表中正確顯示。有時，如果這些事情對您很重要，您只需做出正確的決定。 I could come up with dozens more of these. The ultimate point I want to make is that there are a lot of gotchas, and as the webcomics mention, a lot of these data-driven problems will crop up, there are too many dashboards, and too many conflicting opinions. And eventually the “highest paid person” (the HIPPO) just makes the decision, which is often where we started. 我還可以想出幾十個這樣的。我想說的最終一點是，存在著許多陷阱，正如網路漫畫提到的那樣，會突然出現很多數據驅動的問題，儀表板太多，觀點相互矛盾。最終，「收入最高的人」（HIPPO）做出決定，這通常是我們開始的地方。 For the more data-driven among you, this might seem horrible. But let me also make the argument that this is great, and actually works! 對於你們當中更注重數據驅動的人來說，這可能看起來很可怕。但我也想說，這很棒，而且確實有效！ Competing against tech’s dominant culture 與科技的主導文化競爭 In the past two decades, because of the success of the Google/Meta product management cultures spreading into many new companies, we have now implemented OKRs across the industry. As a result, we have tens of thousands of product managers signing up for +5% increases in their metrics and aiming their hundreds of thousands of engineers at this problem. This dominant culture is one of data, of optimization, of taking small concrete steps to move metrics that are big and easy to understand. And you could argue that the success of many of the biggest tech products — the ones that don’t seem to evolve much at their core, but just inch along — are a success based on this strategy. 在過去的二十年裡，由於 Google/Meta 產品管理文化在許多新公司中的成功傳播，我們現在已經在整個行業中實施了 OKR。因此，我們有數以萬計的產品經理簽署了將其指標提高 5% 的協議，並讓數十萬工程師致力於解決這個問題。這種主導文化是數據文化、優化文化、採取具體小步驟來推動大型且易於理解的指標的文化。你可能會說，許多最大的科技產品的成功——這些產品的核心似乎沒有太大發展，只是緩慢發展——都是基於這個策略的成功。 What do you do if you are their competitor? For example if you are a startup competing with one of the FAANG companies? 如果你是他們的競爭對手，你會怎麼做？例如，如果您是一家與 FAANG 公司之一競爭的新創公司？ The reality is, you can’t race their race. If you are data-driven in the same way they are, have the same dominant PM culture, and make decisions as slow as them, you will lose. They have more resources, can move the ball inches forward each time, and it’s hard to win as a result. 現實是，你無法與他們競爭。如果你和他們一樣以數據為導向，擁有同樣的主導 PM 文化，並且做出決策的速度和他們一樣慢，那麼你就會失敗。他們擁有更多的資源，每次都可以將球向前推進幾英寸，因此很難獲勝。 Instead, you may have to do the opposite to compete: 相反，你可能必須做相反的事情才能競爭： mostly intuition rather than waiting for data 主要依靠直覺而不是等待數據 super fast decision-making rather than incremental decisions 超快速決策而不是增量決策 big moves rather than small ones 大動作勝於小動作 Of course, I will always advocate to optimize the things worth optimizing for — like signup flows and key notifications. But a startup’s core product strategy might need something more drastic, and intuitive. 當然，我將始終主張優化值得優化的內容，例如註冊流程和關鍵通知。但新創公司的核心產品策略可能需要更激進、更直覺的東西。 Data-driven vs data-informed vs data-ignorant 數據驅動、數據知情、數據無知 I write all of this with the deep pains and scars of using data in almost every product decision I’ve been apart of over the past decades. Of course I will always use the data. Yet in my twenties as a feral young founder, I erred towards relying on data too much — a desire to be data-driven, at all costs. But I found that often took me to local maxima. 在寫下這一切時，我經歷了在過去幾十年裡我參與的幾乎每一個產品決策中使用數據所帶來的深刻的痛苦和傷痕。當然，我會一直使用這些數據。然而，在我二十多歲的時候，身為一個狂野的年輕創辦人，我錯誤地過度依賴數據——不惜一切代價渴望數據驅動。但我發現這常常把我帶到局部最大值。 Later on, I learned to be data-informed, still having intuition but overlaying data on top. There’s a happy medium here. But I think there is also an argument — particularly for zero-to-one situations — to just be ignorant of the quantitative data, and instead, just train on intuition. Just make decisions based on your expert qualitative opinion about the market, your customers, and your competition. And only optimize based on data later on, when it’s more appropriate. 後來，我學會了以數據為依據，仍然有直覺，但將數據疊加在上面。這裡有一個折衷的辦法。但我認為還有一個論點——特別是對於零到一的情況——只是對定量數據一無所知，而只是根據直覺進行訓練。只需根據您對市場、客戶和競爭對手的專家定性意見做出決策即可。並且只有在更合適的時候才根據數據進行優化。 @andrewchen