About This Episode
In this week’s Episode we talk with Matt Shaw, the CTO of Wave. Wave specializes in interactive, virtual concert experiences. Creating incredible visual performances with artists such as The Weeknd, Justin Bieber, Dillon Francis, and Alison Wonderland. For years Matt has worked in the game development industry. Working with development teams to create multiplayer experiences on a variety of systems. Matt gives us us Top 5 Tips on planning for success, failure, and troubleshooting along the way.
Matt’sTop 5 Leadership Tips:
Below is a summary of the Top 5 Leadership tips shared during the interview this week. Take a listen to the episode to learn more about the thoughts behind these tips –
- Plan For Success
- Plan For Failure
- Spread Out Your Workload
- Test For Bad Internet Access
- In Development Vs. In Production
- (BONUS TIP!) Have The Average User In Mind
We hope you enjoy the episode. You can find even more Full Stack Leader episodes here:
https://www.wondermentapps.com/fullstackleader/
Show Transcript
Ryan: Hello everyone. And welcome to this week’s episode of the full stack leader. I’m here this week with Matt Shaw of Wave based out of Austin. It’s a startup that focuses on creating interactive virtual concerts. So they’re doing some amazing stuff in this world. Matt, it’s great to have you with us.
Matt: Thanks. Glad to be here.
Ryan: Awesome. And you’re the CTO at wave, correct? Yes. And what does the CTO do at a startup like wave?
Matt: Well, in a small company. Sometimes the CTO role is kind of perfunctory, but I have a background of being a CTO at much larger companies. And part of the goal is as we grow larger, we will need more of a CTO role, but generally the difference between the CTO role and say, VP of engineering is that my focus is more, generally more upwards as far as business dealings and business negotiations. But at the same time, I partnered directly with our VP of engineering to work with the entire engineering staff to achieve all of the company’s goals.
Ryan: Awesome. Yeah. Tell us a little bit about some other organizations you’ve worked with in the past that kind of led you to this point.
Matt: Well, I started out as an engineer and I’m an extroverted person, which is unusual for engineers. That kind of ended up leading me to eventually become director of engineering at a company. And then as the company grew, they actually needed more of a CTO type role and became the CTO there. And that kind of naturally leads to people, wanting you to be CTO at other companies. And I’ve kind of gone back and forth and been, you know, going to larger in somewhat smaller groups on that. I find that smaller companies are actually sometimes more fun because you’re actually more involved with the Zack you’re actually building versus larger companies. The mix of my duties is almost all business and you don’t, it’s much less involved. What’s actually happening. Day-to-day.
Ryan: Yeah, absolutely.I think that, as the organizations get larger, that the actual kind of strategic elements become more and more important, in terms of the larger vision and you lose the actual experimentation on the ground playing with the engineering itself.
Matt: Right? Yeah. And I chose wave because of the interesting things it’s doing and that I wanted to be more involved with it. So that’s what made it attractive to me.
Ryan: Yeah. One of the things you just mentioned that I thought was, a great insight or maybe talking about is the idea of being an extrovert in the technology field and how that might adjust your career path or how you might work with that versus, maybe the kind of life of a coder or the life of an engineer. Specifically. You want to talk a little bit about that?
Matt: Certainly, one of the things that’s attractive to two people that end up being engineers is that computers generally do exactly what you tell them to do. And that’s, and that’s nice in many other circumstances and the inner life, things are much more gray. So what that means is that sometimes there may not be as good at personal interviews. but being extroverted, you have to be able to do that. The other part that’s important too, is that mindset wise as part of their director of engineering or CTO type role, you do have to have your feet in both camps. I mean, you have to understand the business. And understand the engineering needs, and you have to be able to talk to both groups and bridge that so that, there’s an understanding as to why the executive group wants this to happen. And the engineering group needs this and communicates that back and forth with.
Ryan: Yeah. And I think there is an idea that being that kind of person who can verbally express a lot and kind of connect the dots for people, allows for other people to maybe actually get down into the execution format.
Matt: Yes, absolutely.
Ryan: Do you try to cultivate any of the people on your team or any of your teams over the course of time? Have you, have you found people that you’re like, oh, I can cultivate this part of them. And see, and guide them more towards management roles or more towards these kinds of higher level roles versus maybe architects.
Matt: Absolutely. One of the things that I do have done commonly and it’s happened many times over the course of the year is you identify people like us.
Sometimes there are issues that people go through in their career. They may think there’s a glass ceiling if you don’t become a manager at that. They’re kind of stuck. And certain people based on their personalities and their desires are actually not good at management. And sometimes you can give them experiments where they can try it so they can actually see for themselves if they’re good or bad at it. But I have cultivated over the years, a number of people that have become. Effectively tech directors and become CTOs. Since I’ve been around for a while, I think last time I just kind of took a quick count. I think I have six people that are working for me that are now CTOs of companies. And I still talk to them all and get along well. So I, I sit here that a big
Ryan: plus. That’s amazing. Yeah. I think that’s something to be proud of, that you are able to mentor people into a position where they can see this path. That’s a really amazing path.
Matt: Yeah, and they’d liked it. And I liked it and that’s an important part. And I have ended up discouraging some people from going into management as well. Sometimes as I mentioned, you have to let them try it and let them fail a little bit. But usually you can give them good reasons why, but nowadays there’s less of a glass ceiling for that. And there’s good. Better pass them with companies for individual contributors to keep going up in level and further without having to become.
Ryan: Yeah, absolutely. I think, over the course of the history of working in technology, I’ve always seen the ultimate leadership paths for the engineering groups being moving into the CTO positions, or maybe VP of engineering, kind of more management oriented, emotional IQ oriented, or going down more of a path of a true, deep level engineer into. System architecture and really working on the tool sets themselves. What do you think are some of the qualities that you look for, of people that should guide into one of those two positions?
Matt: Well, a lot of it has to do with, if you tried to be introspective with yourself, what do you both, what are you good at and what you’d like to do, because you’re going to have the passion for what you like. I will say that to be fair, even for myself, there are times where I’ve gone. John, I really wish I had gone down more of the individual contributor rank or stayed there because to be fair, it’s really fun. And I feel that if you don’t do any of that, even as a CTO, then you’re kind of, you’re losing touch with what the rest of your group has to do.
So I try to take little tiny projects that are not in the critical path and keep keeping you abreast of all the technology and make sure I continue to do that. And when you go to choose it, that’s really what it’s about. I’ve talked to a number of the people that have gone through their career, and some of them had become CTOs of the company partially because I think the company just needed one. And they’re like, Hey, you’re our best engineer. You should become our CTO. And they turned out to be, they didn’t like it. And they ended up basically stepping back from it. And that’s hard to do because sometimes that means stepping back from a salary or position of leadership. But at the end of the day, they were happier when they did that.
Ryan: Yeah, that makes sense. I’ve noticed on your career path that you’ve worked a lot with. it looks like MMO gaming, , and obviously now a wave you’re working with, , virtual concerts. And it seems like there’s a lot of them. Kind of real-time live interaction built into the technology you’ve been building. It seems like an interesting area to work within and in an area that has its own unique challenges. What are some differences between let’s take an MMO game versus a console game in terms of what you have to manage on the technology front?
Matt: Well , it’s quite a large difference. For lack of, for whatever reason it turns out to be. If I look back at it, every, almost every, I think every game or every product I’ve worked on has been multiplayer, even on the very first games that I did commercially that were flight simulators, I spend most of our time working on the multi-player part of it is because no matter what you do in a single player game, the human element added into multiplayer just adds so much to the experience. But to be fair, it’s hard. Because on a particular, on a single game, you’ve got a computer effectively, whether it’s a console or a PC or a phone, and it’s instantly responsive as to what you do. If you call a function, it returns immediately or as fast as it can. Whereas almost everything you have to do when it’s multiplayer means that you have to send some communication. Data’s packed into the smallest data. You can send it over the internet or a network like back in the day doesn’t that wasn’t cable, and then how the other machine responds and comes back and does it. And so. Even with the speed of light, which is pretty fast that there is latency that happens. And you’ve got to try to make that experience as good as possible. And there’s we spend most of our time in multiplayer type scenarios, covering up for the speed of light, which sounds funny, but that’s what we were basically covering up for the fact that I’m telling a computer to do something and there’s, 300, five hundreds of Mies, a thousand milliseconds of time before I can see that happen. And, uh, that’s, I think that’s the real, the real, the real tricky, tricky part happens.
Ryan: That’d be a great title for your autobiography covering up for the speed of light you go. Right? It is an interesting phrase, but I mean, that is, the actual interaction. The speed of interaction and how close you can get to actual real-time is an important piece. And also like how much volume you can handle at once of real-time interaction. I know that becomes an issue as well, especially in the space you’re currently in. Definitely.
Matt: And it’s still happening. It still affects what we do at wave two, because this wave is creating interactive concerts. We’re basically making a multiplayer concert today, performer venue.
Ryan: I imagine that’s really challenging. What’s a, what’s an interesting challenge that you ran into, at some point in your career. That you faced was exceptionally difficult.
Matt: Well, there’s one particularly large one that I’ll talk about. And, I won’t mention any names to protect the innocent and the guilty. I guess at one point I was brought in to look at a really large project for a company, assess where it was, which is one of things you have to do as a CTO or director of engineering and the likelihood of it actually ever launching. So the project was really large and it was just part of an acquisition made by the larger parent company. And this project was a major part of that, the value of that acquisition. And immediately after acquisition, it was discovered it was in trouble because one of the things that happens when you do discovery on a company you’re purchasing is that you really only have limited access. And to be fair, they’re not likely to show you the words of what’s going on and you just kinda brace yourself and hope that they’re surmountable. Sometimes they’re not. , so I can come up with hundreds of specific things that need to be done to right the ship of this project. It kind of boiled down a certain few steps, which I think were useful to share. One of them is identifying exactly what you’re trying to accomplish with the project. That means like being, make sure the parent company tells you, this is exactly what we want from it. And then you start doing things such as interviewing nearly everyone on the team. Which is a good reason to be extroverted because it’s not easy to do. And you kind of try to suss out what’s going on and try to get them to tell you, like what’s working, what’s not working. It’s like
Ryan: A detective story, right?
Matt: It is. It is. And because some of them are going to feel good.
And they’re going to be, they’re not gonna really want to tell you what’s going on. So you gotta look for that too, but you need to carefully evaluate any anger or finger pointing. You hear from someone and try to turn that into really what if they’re angry about something, what is it that’s not working? What is working and turn that into, you not about the person, but it’s that the process or the tech that needs to be looked at. So the goal is don’t come in as the fixer, but you want to come in as like, Hey, I’m here to help us all succeed. And try to get, you know, their friend and be honest with them and identify the helpers. It’s almost like there’s a crisis, identifying and finding helpers in the group that really want the project to succeed. Again, use a block of salt, and, you know, gauge whether or not it is because you know what’s passionate and what one person thinks is important may not really be important or it’s important to them. But listen to them, find the natural leaders that are in the project. And that’s an example of where you can help mentor and support them. And this particular project, as we talked about earlier about bringing people up, I ended up identifying some people. That ends up becoming technical directors. They have the natural aptitude for it. They want to do the drive, and I could use them to help to be fair, helping me go through and find more issues. With it. So you also have to really be honest with the entire team and the company higher ups about where things are. You have to establish the trust between the two groups. Well, sometimes there isn’t one and try to come up with a plan to address any shortcomings, because you don’t want to go to the higher up team and go, here’s this huge bag of problems. See ya, you want to go like, Hey, here’s a big bag of problems and here’s where we have some solutions. Yes. And here’s some solutions and here are some options and some of these are painful, but I want to tell you that we thought about it thoroughly and try to bring that, to get that trust up. Because no matter what did happen before, because there’s always a tendency for any group that’s involved in that to be kind of antagonistic against each other and being want to blame somebody for it, go look, it doesn’t really matter. You don’t have a time machine. You can’t go back. What you need to do is make a plan to go forward. So if you give them the options to succeed, and if you believe you can do it, tell them, but also ask for a here’s what we actually needed to succeed and give them those balances to do.And if you can do that and this thing as quickly, this is not as simple or a quick process. If you can do that, Beginnings and how you actually correct the project because coming in, we don’t want to fire everybody unless the team, unless for some reason, the team is just ridiculously bad, which is unlikely. It is, you can usually do it like even on this huge project. We, I don’t think we let anybody go at the end. It was maybe out of hundreds of people. This may be a couple of people who end up leaving. But it just needed better direction. Yeah,
Ryan: We do. We do a lot of diligence at our organization as well. So we’ll be a third party diligence organization coming in and assessing acquisitions or investments, things like that. And one of the things that we see, and, I know some of the partners I work with on this, regularly is. People have very different perspectives of a problem and all of them are real, but like trying to define the perspective adjustment that’s actually going to make the most impact in the organization is some of the art of the process.
Matt: Absolutely. And you were correct that they’ll have different perspectives on it, but at the end of the day, it is possible to. Bridge those gaps and explain to the engineering team. The executive group is not a bunch of idiots and they think they are right, but here’s what actually is driving that, but here are the pressures they are under. And so I’m sorry. That’s what happens. The business leaders have pressures that you don’t normally have to deal with. And the engineers have pressures that the executive team doesn’t have to deal with. But if you can communicate with them. You can help.
Ryan: Yeah. And I think this goes to the concept of leading through change, right. Especially in these kinds of areas where there is like a major change happening in a case like you’re talking about, but in technology leadership, there’s constant change. Things are iterating like crazy, especially at startup environments. Do you find that you have to do more, Minor versions of this, even through, adjustments and sprints and the way that you guys are moving on, pivoting on the product, even a waiver or any of the ones that have worked at, oh,
Matt: Absolutely.Now coming into a new project where, people have add the circle groups is one thing, but if you’re coming into a, hopefully a company where everybody is gets along, you still have to do all these things because even in that scenario, Yeah, things change enough that you go, well, look, we should pivot out of not doing this part. Like this should, that we reprioritize. And there’ll be groups that have spent a lot of time and energy working on something and they’ll be really in love with what they’ve done and they don’t really want to put it down, even if it makes sense. So you have to kind of manage that as well. It’s like, look at the end of the day, we’re trying to make this product. And if we have to put this back on the backlog and take this off. Here’s why we’re doing it and, the best, but I found that the best companies, some of the things I really like working at wave are that the communication is really good and that we actually can talk to people. Now, sometimes it takes more time to be like, I’m going to explain to you why we have to do this versus just doing it. Right. But at the end of the day, if you tell someone and you explain to them why you’re doing it, they’re learning. And so they will make better judgments themselves, which means that you have to do less correcting.
Ryan: Yeah, that’s a really good point. All right. Thank you. That was a great overview of some of the challenges and experiences you’ve had as a CTO, and also evaluating things and helping people through change. It’s amazing. We’re now going to jump into your top five tips on leadership. I’m excited to hear these. I think you have a lot of great perspectives and these really interesting environments.
Break One
Ryan: Matt has a unique technical perspective on creating applications that involve group interaction. Because of this. I appreciated how much he displays the importance of considering the unpredictability of humans. Additionally, here reminds us that part of leadership is being open to the many different perspectives that exist on teams. One of the true skill sets of a CTO is hearing all of the different sides and making an assessment as to which is the most applicable in the current scenario. Even more than that, Matt knows that everyone’s experience in feedback is relevant. Even if it sometimes feels different than your own current take on things. So without further ado, let’s jump in. What’s your tip, number one for us.
Tip One (Plan For Success)
Matt: Okay. So my tips are about leadership, but it’s more about how to do leadership on big online products. Things that people just don’t tend to think about. So the first one is a plan for success. Plan for out, people are gonna start using your product from the initial sign up all the way to getting into it and look for the weak spots. What parts of that can you control and what don’t you control , pick an expected number of users and plan for five times that load and think about what’s going to be. Put in ways to limit that blow if necessary, because what you really want to do is have anything work, fail gracefully, and better to have a good experience for less folks than a crappy experience for everybody.
Ryan: This is an amazing tip. Having worked on projects, like what you’re talking about. And for anyone that is working on projects like this, listening to this is a really important thing. And where I’ve seen it really play out is in terms of handling, handling load across servers or planning for enough people to be able to access different, different experiences simultaneously, like planning for the largest number possible because you really don’t know. Right.
Matt: And what happens when you hit it? Right.
Ryan: Okay. And even then you may still go beyond that. Sometimes things surprise you.
Matt: They always
Ryan: often, often they do. Yeah. Anyway, I love that tip. It’s a great tip. How about tip
Matt: number two?
Tip Two (Plan For Failure)
Matt: Okay. So the counterpoint to that one is planned for a failure test. What happens when systems don’t work? What happens when a telemetry or a database call takes too long or it doesn’t work? Is it blocking? What is it, user experience? What is recovery? One part of the problem is that almost all code systems people, right? They expect everything to just work. When you make a rest call, what happens at that rest call doesn’t come back, hang the whole thing, things like this, because those things will happen. And that’s exactly, and I’ve done it. I’ve done things even including going through and ripped out network cables on servers while we’re doing a load test. See what happens. Just to make sure it’s like, what’s going to happen because that’s the best way. And that practicing for failure is one of the things that will help save you. Because when you have problems at a launch of a product, you don’t want to have thousands of problems. You can handle a few, but you can’t handle a thousand like jokingly using the wave analogy. If you have some waves that are here. It’s fine, but if you have a tsunami that knocks you over and drives your head in the sand, you’re not going to be happy.
Ryan: Do you think that when you’re making a list of potential risks for a project, you know, the potential failures that could happen, do you kind of have to curate the top most important ones? Or do you cover as deep as you can go with the resources that you have?
Matt: Oh, you definitely have to balance the top ones because those are the ones that are likely to hit you. Right. But, I tend to curate them by impact, and also by control, span of control. If you have systems you’ve all built, hopefully you can test them. But if you have a dependency on a third party system, What do you do if that doesn’t work? Because you have no way to correct that. And if for the sake of argument, you can’t get them on the home, on the phone or on our own slacked it to fix something. What are you going to do?
Ryan: Yep. Makes sense. All right. Thank you. How about tip number three?
Tip Three (Spread Out Your Workload)
Matt: All right. Chip number three is planned to spread the load. So one of the ways you kind of help yourself while doing this is think about ways to try to slow big spikes or to try to socially engineer them to not have. You can do that through presales that allow pre-sign up when there’s less people in there and even convert beta testers to real users, they don’t have to go through the whole flow again. There’s lots of ways you can socially engineer it because it’s very exciting. I’ll say we announced this and immediately this is available, but that’s terrible for load and you really don’t want it to build for such huge spikes if you can help it. So the best way to do it is to try. And if you can, you’ve got to work with marketing on this because they love the big events, but you’ve got to try to get them to spread the load out.
Ryan: Yeah, it’s a really good point because what I’ve found is a lot of business people want the biggest population possible, which is understandable because that is kind of the main KPI for success. And then a lot of times the system can only handle so. Fraction of that load. And so I think there’s a lot in the communication between the business and the tech as to what the expectations are and hopes are for business and the realities for the tech. All right. Tip number four.
Tip Four (Test For Bad Internet Access)
Matt: This one’s called a test for crappy internet access. So if you live in a large metropolitan area, which a lot of our tech folk do, you’d likely have better internet than most of the United States. You need to test systems if the internet is like, if it’s. What if you have, even down to dialing up or just bad connections or photos to connections, it’s amazing how the experience can change.
And many times, things will actually literally break just because your latency is too high. Even during a login process, you’ll get things at timeout. But of course, when you’re doing it in your development environment, it all works perfectly. And, you need to do this on a continuous basis. Actually Evanston critic, quick story. This, we actually had to think way back in the day where we were doing a game that actually was played a lot in the armed services in Iraq by the US army. And they couldn’t log in because of like a 40 millisecond addition to the login process that we couldn’t have produced until we artificially put it in there and we found it.
Ryan: Yeah. And also figuring out those edge cases is really interesting. Right? We’re working with your actual customers to hear, tell me exactly how you’re experiencing this. We had a similar thing early in the days of mobile gaming at a company I was working with, with subways. We couldn’t figure out why this kept happening and kept having these huge breaks and major cities with. And then we realized what it was.
Matt: Yeah. Then you can’t get good access all the time in every four as you are in the subway.
Ryan: Absolutely. Awesome. All right. And, finally tip number five.
Tip Five (In Development Vs. In Production)
Matt: This is commonly found, but I call this don’t trust dev environments. Now what I mean by this is in development versus production
Ryan: environments.
Matt: You know, they’re off, they often do not reflect the production capability or. Now what you’ll be told when you’re using a dev environment. If you do a load test, is that well, that’s normal. That fails. Oh, the real system is amazing. It has unlimited capabilities. It will never, you won’t have any problems. Believe me, but they’re lying. Don’t believe those promises because their things stay on low to test the hymns, they will be. Tip over points that will still fail almost every time that I’ve done. We’ve done things where we’ve had to use a development environment. When it goes into the production environment, the environment has changed.
Tip over points changed, like I mentioned, or distinct, still don’t scale. Like they said they would trust, but verify.
Ryan: Yeah. And that, it can be hard to recreate those environments to get the real perspective on it. And a lot of times people don’t want to make those investments until there’s light environments. So, yeah. Expensive. Yeah. It’s expensive. Yeah. And we see a lot, like in, trying to create, like you said, real time populations coming into something, also just low balancing across or load testing across a whole series of things and then, and investing in the balancing process, within that. So it’s always getting the business to understand that until they actually see the emergency happening. And then it makes sense right then. Oh yeah, we should invest. Exactly. You mentioned to me before we, before we started this, that you might have a bonus tip as well. And I wanted to offer that and see if you’d like to talk about that.
Tip 6 (Have the Average User In Mind)
Matt: Sure. This one’s called your developed machines are from the future. If you make a product that must run on a consumer machine, like a game, for example, unless it’s a console, which is the same for everyone. You’re developing machines are always the most powerful things that your company and afforded by you because they want your development speed to be fast. But it does mean that it’s so much more powerful than an average user. That every bit of experience you create or build or authorization you make may not apply to them. That’s why I stayed there for like, from the future. So it’s really important that you specify what a low end system is. Acquire them, have engineers work on them as well as QA and make that part of your test plan, because it’s the only way to optimize that and get QA to test it right. And get a pass, fail on those projects and things to actually make it so they’ll have a good experience. Otherwise it will only work on the highest end systems.
Ryan: This is really subtle. Kind of nuanced perspective on how to foresee issues in the future that if you’ve done it enough times, you’re like, oh, of course we were going to do that right up front, because that will prevent so many things. But, it’s a great tip for people who might be getting started early in their career and thinking about okay, how do I prepare? And it will circumvent a lot of things. If you think like this,
Matt: I’ve had times when I’ve been personally using the Intel analyzer and optimizing a very serious routine to make us, you know, 200 orcs and be able to be on the screen at one time. And I’m like, yes, they made these changes. Three milliseconds off the frame and it actually is slower on slower machines.
Ryan: Think that that change worked against you. Yes, this is amazing. Well, thank you for the amazing tips. Thanks for the rundown today. And I really love talking about the way that you build technology for people who are coming together in large group environments and thinking about multiple humans existing within. James within concerts within areas that they normally may not have existed in the past. Cause these are, you’re really blazing new trails for the way this works. Thank you. Yeah, it was really exciting to have you here today. Thanks for joining us. And we’ll talk to you
Matt: soon.
Ryan: I really liked how Matt’s tips specifically take into account the important approaches to working with online interactive communities. Whether games or concert platforms, products that involve quick bursting scale and unpredictable risk factors require leaders to rethink and re approach problems that might traditionally be solved. The nuance I appreciated most was how he put preparation into the driver’s seat and made common issues. A part of the actual development process.