In our eighth episode of 15 Minutes With we're talking to Luke Frake. Luke is the Experimentation Lead - User Growth at Spotify.
Luke's interests are in the field of optimisation and experimentation, which are becoming more and more recognised within mainstream digital commerce as vital to any long term experience strategy. As part of the User Research Team at Spotify, Luke is involved in optimising digital experience for Spotify users.
In this episode we talk through the challenges faced when a scientific mindset is applied into a business setting, valuing 'failures' as learnings and understanding that in some cases, the failures are more valuable than the successes.
OUR SEASON ONE COMPANION IS NOW AVAILABLE FROM AMAZON. GET YOUR COPY TODAY.
Contact the 15 Minutes With Team:
In our eighth episode of 15 Minutes With we're talking to Luke Frake. Luke is the Experimentation Lead - User Growth at Spotify.
Luke's interests are in the field of optimisation and experimentation, which are becoming more and more recognised within mainstream digital commerce as vital to any long term experience strategy. As part of the User Research Team at Spotify, Luke is involved in optimising digital experience for Spotify users.
In this episode we talk through the challenges faced when a scientific mindset is applied into a business setting, valuing 'failures' as learnings and understanding that in some cases, the failures are more valuable than the successes.
OUR SEASON ONE COMPANION IS NOW AVAILABLE FROM AMAZON. GET YOUR COPY TODAY.
Contact the 15 Minutes With Team:
On today's episode of 15 minutes with, we have experimentation and user growth lead at Spotify, Luke Frake joining us. Luke's interests are in the field of optimization and experimentation, which are becoming more and more recognised within mainstream digital commerce as vital to any long term experience strategy. As part of the user research team at Spotify, Luke is involved in optimising digital experience for Spotify users. Luke talks through some of the challenges that come from operating in an emerging field as well as how applying a scientific mindset allows us to value failures as learnings. In fact, it would seem the businesses can actually learn more from their experimentation failures than their successes.
Shelley:Luke, welcome.
Luke Frake:Hi. Hi, Shelley. Nice to chat with you. I'm good. Thank you. How are you?
Shelley:Fabulous, so good to have you. We wanted to get straight into it and ask you, what does good experimentation culture look like?
Unknown:I think there's quite a few different ways that you can summarise good experimentation culture, I think there's like a few different points that you can consider when looking at different companies and looking around. I mean, firstly, like, how data driven as a company, are they making their decisions, because somebody in some C suite office is saying that this is what they should do? Are they making decisions backed by measurable data, you know, user research, analytic data, anything like that? I think that's one of like the key ones that comes up first, I think people are encouraged to kind of think outside of the box, you know, people enabled to fail, make mistakes and try different things that kind of goes against the curve, what metric people are using? And like how, how are they measuring success in different areas of the business? Big one, I think, is democracy within experimentation culture, as well. So how are people allowed throughout the business to make these decisions that decisions and to make these mistakes? Are people C level making all of the experiment ideas? Or is it coming down to kind of every junior analyst and eveyone else around the company able to do these things? Continuing on - What happens with like results from experiments? And how do people perceive these sorts of things? I think there's like a huge list of like how people can consider and how people can think about these different like bits and pieces. It's a big one, it's a big question to unpack for sure.
Shelley:Absolutely, we could easily spend the entire episode talking about just that. In your experience, is that a utopian picture? Or is it achievable for businesses?
Luke Frake:Absolutely, I think it's completely achievable. I think it's very hard to change, though, that'd be my my crux of it. I think when you speak to companies that that do not have a good experimentation, culture don't have, you know, have fear of failure built in within the company, and how kind of top down decisions and don't look at data, I think it's very, very hard to change, you know, purely by the fact that if you have a top down culture, the top is deciding what's happening. If the top doesn't want to change, it becomes very difficult, but it's definitely possible. I mean, you know, I've worked with so many different companies and variety, like varying degrees at this scale of good culture and bad culture, if you'd like, quote unquote. But but it's definitely there are companies that are absolutely smashing this I mean, look at like Spotify, I think is a great company. But I think looking outwards, you know, booking.com, and companies like this, where they really are just churning out experiments, churning out learnings, but also making different from these learnings as well.
Graham:And would you suggest that, if a company wanted to do this, they start small? And if they're going to start small, where do they start? Because you could go, You know what, let's just both feet in and see what happens. But I guess with that, there's a risk that you'll try once fail, and never go back. Whereas if you start small, you can kind of build to a crescendo, per se.
Luke Frake:Yeah I think when it comes to running experiments, starting small is one thing, and I think it's important, but I think you touched on a really good point, then it's like, what is the outcome of these experiments? And how do people perceive that and that that is a fundamental part. And I think that's something that's hard to start small. I think that has to be kind of educated in all directions of a company. And what I mean by that is the, you know, the most successful experiments don't always come from the biggest uplift in a metric or something, an experiment that fails its hypothesis, you know, we think we think a is going to happen, but actually it doesn't. And this happens instead, that's, that's a learning. And that's what's important. That's the important crux of experimentation, you're learning. I love when one of my experiments increases a metric by 15%. And that's a huge win. And I give myself a pat on the back for that sort of thing. But at the end of the day, that's probably built on 20 other experiments that have decreased a metric that's built and 20 other learnings that have come from somewhere completely different that enabled us to get to that point, where do we get that huge win? So I think, I think that it's very good to start small in terms of getting people into experimentation, but really understanding that learnings and knowledge is what you've gained from experimentation rather than dollars. That's not the initial outcome at least.
Graham:What advice would you have for a business where experimental culture isn't part of what they do, but they're engaging with an agency or an external business that has that as part of the way they do things? And they potentially have two different versions of what success looks like. Ultimately, you would want to bring the company that doesn't have it closer to the company that does that. Is there sort of, you know, any advice on how to bring those two people closer together faster?
Luke Frake:Yeah, I think it's the same. If you have, there's two situations that are they're really the same here, you either have a company that doesn't have a culture of experimentation, you have a say, an agency that do or at the same time, you can have a company that doesn't have a culture, and you can have an internal CRO team that do. And that, that's something I've seen all sorts of different places, throughout my time. And it that is the bit that's hard, that's very hard to do, being being the small trying to change the big in these situations is very hard to do. But it definitely can be done. And I think one of the one of the techniques that I would say is really good for this is trying to get some common ground trying to get some commonality, really try to, you know, as the experimentation side, really trying to understand what are the goals and key metrics that the rest of the business, the rest of this company, whatever it is, are really trying to achieve? And how can we explain how experimentation can help us get there. So for instance, if you've got say, you know, say some engineering team that might necessarily not want to run experiments, they think it's a waste of product time explaining how small iterative experiments can actually help them do more more work in the long run, because what they deploy will be more valuable. If you've got a, I don't know, a marketing team who are really focused on their average order value, explaining how using experimentation and failing a few times can help kind of increase that. And therefore it will, it will make all of their metrics at the top of the funnel look much better. So I think that that's something that I've seen people fall into traps with before is always explaining experimentation is like number of experiments running. And not necessarily talking that kind of common language, that, that that piece of truth, if you'd like that everybody can kind of get on board with and agree with.
Shelley:From your experience, then it's, it's about the process. And by the sounds of it, it's also to do with context. So you can zoom in as close as you like to one particular experiment. And if that fails, it can essentially freak people out, right? Particularly if you don't have data to back it up, or you know, it doesn't, you don't have the learnings that you can apply, necessarily. But actually, if you step back and zoom out and appreciate the fact that that's part of this huge, huge context of all these other tests that are running these experiments that are running that are actually supporting, you know, a much, much broader end goal, then that's kind of the perspective that you're getting at. But I suppose from the from the angle of being data driven, you know, to try and influence leadership to change that culture. How do you do that? You know, when it's such a such a complex ecosystem?
Luke Frake:Yeah, I think, I think that's the hard bit because like you say, if you focus on a single experiment, as soon as you get your first failure, quote unquote, failure, you're in the position then where people, people don't understand the process, I think what you start to be able to do though, is you start to join up into stories, take that example a second ago, you know, we have a huge weigh in on one experiment, don't tell that story on its own don't signal that one experiment that was hugely successful out, you need to tell the story from the beginning, you need to explain the journey. And we did this, we tried this, and it didn't work. But what that taught us was x y z. So we compounded that on the experiment again, and again and again. And what we had at the end was this huge win. So make sure when you're telling your stories about how you got to this, you know, this, this, when at the end, you include that journey of pains and frustrations along the way, because realistically, that is the journey, that that's what happened. And that's how you start to get people away from this fear of failure, because they understand that nothing, you know, no failure is without learning, and we can move forward and take that into kind of the next and the next and the next and come up with something much better.
Shelley:Because we tend to understand the research process in an academic setting, you know, we get it and we go, Okay, leave the scientists to it, it's, you know, it's an iterative process, but then applying that mindset, within a business setting. It's sort of like, you know, we're not we're not joining, joining the dots fully. So I think that is such a brilliant explanation to actually say, you know, you need to look at the whole thing and don't even your own, how we present our own successes. We need to be, you know, more honest about it so that people actually understand that we're not undercutting ourselves, it makes perfect sense.
Luke Frake:Absolutely. I mean, we we so for instance, on our side, we have all of our experiments when they concluded they go into our knowledge base, and that's something that Spotify we share around the entire company everyone can see, that isn't just a pot of winning experiments, that is a pot of experiments that were run successful experiments, inconclusive experiments, failed experiments, because that is the learning. And we really push that as well.
Graham:So once you've got all the information, how do you get it out into kind of the wider teams on the wider business to get sort of other people excited about what's happening? Because having the information in a single place is one thing, but how do you get people to engage with it?
Luke Frake:Yeah, absolutely. I mean, I think that's, that's one of the most important parts of a CRO team, or in any kind of experimentation team, but often one of the hardest things to do, or most overlooked things to do. So for us, we, all of our analysis is compiled into some kind of deck. So each experiment has its own deck associated to it. And then that is contains all of the analysis and kind of all the details for that experiment. And then what we'll do after that is, if we, in this example, we have kind of a story of experiments, we might create something that's bigger that contains that whole story. So you can see you can drill down into the snippet into the singular, or you can kind of see the bigger, but having these things is only one part of it, right? If we have 1000 decks for the 1000 plus experiments that people run, then who's reading them and who's looking into them. So there's a few things we'd like to do around that. One is just sharing it as broadly as possible. So using kind of internal communication channels like email, or Slack, or Teams, or Workplace, or anything else that people are using, just to make sure that there's like a constant stream of this information being shared out to people, but also doing more, doing more like in person or virtually in person sessions, where you have people presenting these ideas, and really opening up to the floor of this isn't just me taking these ideas and giving them to you, this should be conversational. Challenge, the outcomes, challenge the questions like let's continue this learning and continue, there's bits and pieces of like, how can we improve, because out of those sessions, then you really start opening people up to there is no one right? That's all continually challenge and ask questions, which gets you back to kind of improving the overall culture and pushing those bits and pieces forward. So I think, trying to play as many channels as possible, but also trying to talk directly to people as well. And getting those outcomes is the best way to kind of disseminate that information.
Shelley:Well, definitely, I guess people in their different roles, respective roles within a business, they will have a completely different perspective of that business. And so to be able to voice their ideas, their questions and challenge exactly what you've been talking about experimentation, different learnings, is a way to enrich that process and also get them buying in to the whole to the whole process behind it.
Luke Frake:That's it. And I mean, that part earlier on trying to understand people's metrics, having those conversations opens up to that, because when you can say to somebody, this experiment has reduced the bounce rate of this page, and you can literally have the conversation, somebody can ask, well, what does that do for this metric that I care about, then you can start to explain it, you learn what they're interested in. And you can also explain kind of the joint between the two. So it just helps continue that process of conversation around experimentation, which ultimately leads to a better culture internally for experimentation to so I think it's very cyclical.
Shelley:Absolutely. And a better culture overall, when people are collaborating, communicating. So surely, that's part of how you also get senior leadership excited about this process as well.
Luke Frake:Exactly. Yeah, spot on.
Shelley:Luke, thank you so so much of your time, is there anything else in terms of tips and insights that you think people listening might be able to apply?
Luke Frake:I think I hope the main thing that people take from this is to is to communicate too many times that I go into different companies like when I when I was working within the agency, or when I'm working in companies and different teams aren't communicating. So I really hope the one main thing people take from this is talk to other teams, don't don't be that standalone silo, talk to your agency, talk to the team in your business, whatever it is, understand what people are thinking and how these different bits and pieces work and how they can come together to be something better than the sum. That's that's my main takeaway.
Graham:And that's good advice overall. So to be honest, outside of experimentation, we encourage everybody to talk to each other because it's, it's always those things that come up in surveys when you know, people ask about business satisfaction and employee engagement. Communication is always the thing that comes up is something that needs work on so yeah, we'll take that. Thank you.
Luke Frake:Most welcome. Yeah, most most welcome,
Shelley:Luke. It has been an absolute pleasure. Thank you so much for your time today, I know that this is going to be an episode that people can really really learn from. It may be a case of I imagine there will be loads of questions off the back of it. So I'm sure we'll get you back for a follow up episode at some point in the near future.
Luke Frake:Awesome. Lovely speaking to you, Graham and Shelley, thank you so much for your time. We really enjoyed it.
Graham:That was experimentation and user growth lead at Spotify, Thank you. Luke Frake. By shifting our mindset and not looking at experimental failures as failures, but more as learnings and then sharing them with the wider business in an open discussion, we're able to fully utilise the benefits of experimentation within our business. And remember, communication is everything. Thank you for joining us for this episode of 15 minutes with we look forward to having you along for the next one.