Digital: Disrupted: An Inside Look at Game Developing

Episode Transcript:

Paul Muller: Alright, pop quiz. Which industry do you think makes more money, gaming or movies, magazines, or video on demand services? Well, it's a bit of a trick question because at 160 billion in 2020, gaming of all kinds—and I'm talking mobile, PCs, and consoles—generated more revenue than movies, magazine's, subscription, video on demand combined—just a humongous number. And according to industry watchers, that's only set to increase as the world's biggest media industry, TV is predicted to lose advertising dollars to the game developers. Well, in addition to creativity and gameplay design, one of the biggest technical challenges facing game developers is the sheer size of the code base of these new releases. If you thought the Lord of the Rings trilogy was big, imagine trying to develop some of these more modern games. It can literally take more than a day to compile some of these biggest releases. Think about what that means.

That means a three day wait between developing an idea, being able to test it, find any bugs, and then recompile and test again. So, the question is, what can be done to radically shorten the development testing cycle? Before we find out, I want to thank Rocket Software again for making today's episode possible. If you haven't already, check them out at rocketsoftware.com to see why over 10 million IT professionals rely on them every day to run their most critical business applications, processes, and data. Also, don't forget to subscribe to the show, and if you can give us a quick review, it really means a huge amount to me. Well, welcome to another episode of Digital: Disrupted with me, your host, Paul Muller. Today's guest is Reggie Yativ. He's the chief revenue officer of Incredibuild, a hybrid build acceleration platform for the software development process, focusing on game developers to enable them to dramatically speed up that build and testing process that I was speaking about before, helping free up their time to focus on creativity and quality. Welcome to the show, Reggie.

Reggie Yativ: Thank you. I'm very happy to be here, Paul.

PM: It's great to have you. Where do we find you today?

RY: Well, just sitting in my office, kind of gazing at the screen. Nothing special.

PM: Which part of the world are you guys in?

RY: Tel Aviv.

PM: Lovely part of the world. Hey, we're going to find out a bit more about this incredible world that is game development, but before we do, we've got something we call the lightning round, get an opportunity to get to know you a little bit better. You ready to play the game?

RY: Yep. Ready.

PM: Awesome. Let's do it. First question for you, Reggie, what would people say is your superpower?

RY: Well, people tell me that I can be in 10 places at the same time, 20 hours a day. I don’t know if that's a superpower, but that's what people tell me.

PM: Either that or you've got some clones running around the country. It's a great superpower. Alright, the most disruptive technology of all time?

RY: You know what? There's many of them. The first thing that jumps to my mind is cell phone technologies. It’s that thing that allows us to enter anything including games.

PM: Yeah, very true. The best quality a leader can have?

RY: If I have to be short about it, I would say agility combined with humility. So, you need to know when you are wrong and you need to adjust yourself. You need to stay agile so you can up your game every time. So, I mean, that's kind of it briefly.

PM: Yeah, no, they're both great answers, and I like the way you connected the two of them together. Next question, your advice to people starting their careers?

RY: Wow, I have a lot, but the one I would pick is to follow a plan. I think many people mistake their career as something that is just going to fly by and everything will turn out fine. But I think it's the biggest mistake that people make. You need to have a plan for yourself, you need to know the milestones, and you need to reach those milestones and critique yourself. If you are not reaching those milestones, if you set yourself on a path to execute that plan, your career will be really, really fulfilling and also eventful. So, I think from a very early age, I had a plan and I'm so happy I have been able to fulfill it.

PM: That was going to be my question is did you start with a plan and how old were you when you figured out what that plan was going to be?

RY: Ah, well, that's a good question. I was pretty young. I think I was just 22 when I sat down with myself. And I don't know what possessed me at that moment, but I was trying to imagine myself at the age of 45, and I said, okay, if I can plan myself out the next 20 years and kind of follow that path, I should be fine. Because I have a plan. I know what to study, I know what to ask people. I know how to learn, what do I need to do next in my career to guide me through to the objective. So, I think at a pretty early age, you need to have this plan, even an outline. It's okay if it changes every now and then, but you need to have the outline at a pretty early age.

PM: Yeah. Well, that's clearly worked well for you. Let's move on. The first thought that comes to mind when you think about gaming?

RY: I think for me and many other people that I talk to, I think gaming is a social experience. Yea, you can play a singular game with yourself, you can win against a machine. But I think for me, a very compelling part of gaming is the group you play in and also the community.

PM: Are you a gamer?

RY: Well, I used to have more time for this. I admit, not often right now, but when I was playing group games, I was trying to play many group games, casual games, games that included interaction. Sometimes even after the game, staying around on a channel and chatting about what happened or strategizing ahead of a game, I think it's part of the fun. 

PM: Wow. There you go. I've done, it's funny, I sort of had a brief stint of gaming, probably in my teens and my early twenties, but the bug never really stuck with me. My youngest daughter is a pretty avid gamer, but the other one seems to have virtually no interest in it. So, it's funny, it's, it's what the British would call marmite, you either love it or you hate it.

PM: Let's move on. If you could use technology to solve one world problem, what would it be and why?

RY: Well, I'm going to give a rather sad answer to that question. The one thing that I would want to see vanishing from the planet is cancer. So, if we could put in any technologies around that objective and eradicate that thing from the universe, I'll be the first supporter.

PM: That's a great answer, and I think you're the first person to have said that on this podcast in over two years. So, you are unique, Reggie, what can I say? Great answer. So, let's jump into the topic of gaming. Tell us a bit about your background, you obviously shared a bit about how you had a career path here. You've had several senior roles. I think the COO at Agora was probably one of your more recent ones. But tell us a little bit about how you came to be in the world of gaming.

RY: Well, yeah, you mentioned Agora. Agora has been active in gaming providing SDKs and KPIs for the gaming industry. And among that is audio and video SDKs. And before that, funny enough, I was part of running sales for Redis, which is the most commonly used database or no SQL database in all high-speed, high-performance environments. And as we all know, gaming is really a high speed, high performance, very intense technological environment or always at the forefront of things. So, the game developers are always playing with the edge of the cutting-edge technology. So, through Redis, I came across a lot of gaming use cases as well. I think I would say for pretty much all of my career, I'm not going to bore you with the entire history now, but all my recent history was in really quick, high performing, technologies. Also, by the way, very much attached to how developers empower those experiences through their own personal creations. I think throughout my career there has been a steady thread of working with developers, and that's been fun. I mean, it's still fun.

PM: Yeah, no, it's great. I think particularly in the last, probably decade or two as software development, software engineering practices, and some of the related concepts around things like SRE - site reliability engineering - and DevOps and others have really come into their own, I think as companies have realized that they're all in the software business no matter what they do, being able to speak to that developer is probably one of the most critical things that a business can have. So, you've landed in the right place or you've had the path plotted very well. So, let's talk a bit about the world of gaming, because I think, as I said in the introduction, I don't think people really understand just how enormous this industry is in terms of revenue generation. We usually think, as I mentioned, things like movies, and we just assume that movies are big because we are all experiencing, and gaming would be a fraction of the size, but it's completely the opposite, isn't it?

RY: Look, gaming as itself is gigantic. We serve basically, I'd say 80% of the gaming industry at Incredibuild. Most of them are all customers. And we see it's not just gaming, it's the gigantic ecosystem around gaming and gamification of content. So basically, there's a lot of adjacencies to gaming such as entertainment and education, and embedded technology even in cars. And everything is about gaming. At the end of the day, gaming is embedded into everything we do, which I think makes the challenge of the gamers even more extreme because they have to deal with combining all those very powerful, heavy duty technologies into many environments, into many forms of deployment, whether it's on-prem or cloud, they need to be very versatile in their knowledge, they need to understand game engines, the commercial game engines of course, like Unreal. And they also must understand the homegrown engines, which they stumble upon when they develop games. They have to be very versatile. So, their challenge is pretty big. And of course, all that translates into other more practical challenges. How do you create those big builds? How do you have all the machine power that you need, et cetera, et cetera.

PM: Let's talk about that then. So, when we talk about a build, because I guess even in my mind, and I consider myself to be pretty technologically aware at a nuts-and-bolts level, but I'd never really stopped to think how big are these releases? And we talk about these incredibly big compiled and test times. I mean, they're staggering. I remember back when Microsoft were talking, I think about releasing Windows NT or something similar, and Windows itself as a platform was going through something very similar. It was so mammoth. The compiling cycles just became these enormous efforts that slowed down the testing the exact opposite of what DevOps is, right? This short, small batch size cyclical thing.

PM: So how big are these releases? And ultimately, can they be unit tested or do they really need to be tested as a whole because it is this whole experience? They're two questions in one there, my apologies, Reggie.

RY: No, it's totally understandable. Those two are intertwined in a way. I think the key word here, Paul, is iterations. I think when gamers or game developers, companies, and studios, think about releasing a game, they go through a lot of iterations and those iterations allow them to make the quality of the product better. So, each time you do testing, you need to compile. And when you create those builds, waiting those long hours just delays testing. And now you add to this the other component, which is crunch time. So many times, studios will have specific timelines they need to deliver those games, and the pressure is really big on the teams, whether it's developers, DevOps, IT leaders, all of them have those pressures. And of course, revenue leaders, they'll need to deliver a title in time. So that puts  big pressure on testing.

So do the testing faster, and in order to do testing faster or more frequent, you need to finish those builds. And the more sophisticated those technologies become, the builds become bigger, as you described, correctly, heavier. They are comprised of many elements that have to come together. Of course, those things are prone to errors. And then you run your first build, you wait for it to finish, run the first iteration, you find out the problems, you go back, fix them, and then you want to run it again to check that you actually solved the problems. Each iteration like this, if it takes long hours, and that's what we're seeing with folks coming to us, is that it takes so many hours to create those builds and run those tests. It just takes away time. It takes away from the creativity because they want to focus on the actual game.

The build wait times are sometimes just annoying for them. It's just sitting and waiting for something to finish. People work from home, they have other limitations, they have restrictions that come from working away from powerful networks. They don't always have, especially during COVID-19, we've seen the shortage of massive machines. Not everyone can get a threadripper just at will and all of those combined, they're pretty challenging. If you think about a team of developers and DevOps teams that really work every day to simplify and streamline that process, I think that's the biggest challenge is how do I streamline the go live process through streamlining of use of resources, the available resources, cutting the build short, go live with a better quality product. Because at the end of the day, all the game metrics speak to quality, you want to release something that's quality. We're not going to go through all the metrics, but a lot of the metrics are a manifestation of quality.

PM: So how is this any different than any other software development exercise? So, what's different? What is unique about the game developer's situation that is any different from releasing a large complex software application? So why is Inredibuild tailored toward the gamer or is it just another build release platform?

RY: Well, Incredibuild is built for several segments in the market and we have customers from many segments in the market. But it started from gaming. And one of the reasons why game developers and studios adopted Incredibuild quickly is because of the scope of a release in a game. So, you have to combine very complicated technologies, which means builds will take a long time to compile. If you're talking about a standard game, even in a standard game release, you will have much bigger graphical elements. You will need much more demanding technologies combined into the art. You're not looking just at linear text or anything you'd see in a standard application. With a standard application you would see graphics, you would see frequent use of GPUs, you would see audio and video technologies, you would look at metaverse use cases. Once you start embedding all of those concepts into the gameplay, which is much more rich, much bigger, you end up with builds that are longer to create and heavier. Meaning the developers would need more fire power.

They would need more resources at their disposal to create those builds. And this is why creating a build in a gaming environment will normally take a longer time and those builds will be heavier. This is why the first adopters for Incredibuild were gaming companies, but today we're seeing the same phenomenon happening also in other segments. Like I mentioned embedded, I mentioned the auto industry, you mentioned banks, we work with a lot of them, which also have long wait times and long builds. But I think the challenge is accentuated in gaming because of the graphics, because of the art that's involved in the creation of the game. Now going back to your initial question about the sequence. In games, the feedback of users could make or break a game. That's why the reaction, if you compare it to traditional software releases, the reaction time of the studios has to be much faster to what the users want. So, iterations, you won't have so much time between the first release and subsequent releases. You'll have to react quickly.

PM: Do you have any specific data to give us an idea of the type of timeframe we're talking about here? Is there any sort of market research or data that talks about the latency between gamer feedback and developers and the publishing houses responding to that? I'm curious.

RY: Well, I don't have it on hand right now, but I can tell you from talking to hundreds of customers, I'm not talking about problem solving. I'm talking about game developers or studios looking at users. Retention goes up if they respond quicker to requirements of players or you're looking at sessions cutting short. If something in the quality of the game is not to their liking, people walk away from the game quickly. It's a much more dynamic environment than many of the other more traditional software where you have feedback and people understand it takes time to implement. I think the gaming industry is very, very dynamic. We probably provided with some more data around this, but we clearly see that the feedback cycles are very short, the speed is high, and you need to deliver constant crunch times and constant releases.

PM: So, let's talk then about what you've done to try and crack the back of that problem. So, what are you doing to help try and break the back of this really long compile, test, recompile, retest cycle?

RY: Well, maybe we can look at it in three dimensions. One is resources, creating a build requires resources. So first of all, we optimize the consumption of resources on your network. So maybe to describe a little bit about the process, you take this build if you want, graphically speaking, distribute it across the network, and while you distribute that build, you basically harvest any cores transforming any core on the network, making your computer almost like a supercomputer. So, harvesting power that resides on the network and then you collect back the build much faster.

PM: Like SETI@home or I guess even Bitcoin mining, just using spare compute capacity in your organization to help accelerate that. Have I got that roughly right?

RY: Yeah, so there's three dimensions to it. One, you don't always have to use high-end machines to run those builds because you can leverage a lot of idle cores or cores that reside in the network no matter where you are. If you're at home or you're in the office, you will have idle cores to use. Second, you can instigate more resources if you kind of run out of any other resources, automatically it would instigate new resources to enable you to quickly finish the build, so you’re saving money and cost.

PM: And what do you mean by instigate new resources? I'm not sure what that means?

RY: For example, if you're working on prem and you ran out of resources on prem, you can burst to cloud and you could use it in a hybrid way, you could use resources that sit on the cloud to complete the mission at hand. So, in other situations you would just wait or sometimes even get stuck until more resources have been provided and you see that problem accelerates as you're sitting for example, and you don't have access to all those high-end machines. So that's one way to look at it. The other way to look at it is if you are a studio and you don't want to invest in those high-end machines, but you have a lot of quick access to commodity machines. So, if you basically aggregate the power of all those cores that you can harvest from many smaller commodity style servers, you have superpower, but it does not require you to have very high-end, very expensive machines.

So even a small studio can quickly compile finished test and release without necessarily looking for those high-end machines spending the money. And sometimes they're not even available for you to buy. Even if you want to buy, they won't be readily available for you to obtain them. And I think the final element of it is shortening time. If you take all this power, you give it to the developer and then instead of waiting eight hours, you're now sitting and waiting 40 minutes or instead of three hours, you're waiting 20 minutes, all of a sudden you basically saved a lot of money for the organization. That's one. Second, you saved yourself a lot of time. You can run more tests, improve the quality of the game and release faster. And of course, not to mention the aggravation of sitting and waiting, which many developers tell us jokingly that ever since they installed Incredibuild, they became thinner.

And then I'm asking them why. And then they say we eat less donuts waiting for the builds to complete. So, it's basically also frustration, people waiting to get to the next step. In terms of creativity, it kind of interrupts your creative train of thoughts when you have to wait constantly. So, there's a lot of elements that get streamlined and not to mention the CFO is normally happier when you are spending less resources on the same build. By the way, except for acceleration, we also have caching. So caching is a different way of streamlining the development process. Basically, you cache once and you reuse many times. So, what's the point in recreating? And again, spending those resources to recreate the same, if you could cache a specific piece of content or specific set of deliverables and keep reusing it. Many times, if you don't have to change, you can just reuse it. And that again, saves time, efforts, resources, it all translates to money and time at the end of the day. And of course, if you can create more, you can test more. If you test more, quality is better. So that's the paradigm, kind of the process.

PM: We’ll wrap up quickly here. Where do you see the game industry heading toward? I mean obviously there has been a huge push toward mobile, there is a lot of hype around Web3 and the metaverse. Yeah, I do not want to live in a world where I don't have any legs, but where do you see the gaming industry heading in the next couple of years?

RY: To try and summarize it quickly, I'd say richness. We spoke about richness of graphics, and you mentioned metaverse. I think metaverse is a big trend. I mean there's many definitions for metaverse, but I'd say people spoke a couple of years ago about AR and VR and now a lot more about metaverse. I think the world is going to continue to evolve very quickly into combining real elements into games and gaming elements into real experiences. I think those bidirectional trends are hot and are going to continue to be very hot and as a result will create more complexities. It will require higher resolutions, more sophisticated graphic capabilities. And of course, all of that will be resource intense. I think those trends are there to stay. There's also a bit of a saturation in the junction between gaming, entertainment, and all the adjacencies.

I think game developers and studios will have to keep staying on top of their game, staying unique so they can shine out from the cloud of offerings and appeal to users. And I think maybe the number one thing to look at right now is, because everyone's talking about the economy and what's happening in the economy, being frugal, the concept of developing games and software in general while making sure that you are keeping your bottom line tight. So, making sure you don't spend money when you don't need to. Make sure you understand your game metrics and your development cycle supports those game metrics like DAU, MAU, and ARPU and all of those things that game developers look at in terms of game adoption. I think that managing resources carefully will ensure that game studios can continue to be profitable and continue to scale. I think all the market is now talking about what's happening out there and I think we are part of a set of tools that allow people to stay efficient. So, I think efficiency maybe is the best word to describe that. So that is going to continue to stay with us for a while.

PM: Yeah, I think you're right. I think we're already seeing it in the huge layoffs across a lot of the tech industry. I think getting lean and focusing on being frugal with the resources you have is going to be the watchword for the next couple of years. Alright, where can people go to learn more if they want to learn about simplifying their build processes, accelerating them?

RY: Well, they can go to Incredibuild.com, they can go to our LinkedIn that has a lot of information. Inside incredible.com, there's a lot of developer resources and documents that will explain more about that process. But also, generally speaking about, as you said, lean development cycles that help people stay on top of their game, but saving costs and be successful. I think those are the main places I would send people to look for more information on that. Of course, I'm here to answer any questions.

PM: Alright, so the show sponsor Rocket Software has a set of values that they talk about that matter to them, part of their value system, which is their empathy, humanity, trust, and love. So, my question for you, Reggie, is what matters to you right now?

RY: I think right now what matters to me is that people through these times will continue to support each other, basically help each other create a prosperous economy and help everyone stay successful. I think it's crunch time out there, so I want people to support each other.

PM: Very, very noble of you. Well, let's wrap up. Thank you Reggie for joining us. Really appreciate you taking the time and thanks again to Rocket Software for bringing us another episode of Digital: Disrupted and thank you for listening in. If you like what you've heard do give us a thumbs up on iTunes, Spotify, whichever pod catcher you happen to be listening on. You can also reach out to me at Twitter while it still exists, I make that only slightly jokingly, @Xthestreams or our show sponsor @Rocket with any questions you've got for our guests such as Reggie or ideas for future topics you'd like to hear covered on the show. With that, we'll see you all next week. Stay disruptive for everyone. Thank you.