Using Joyent to Scale Your Online or Mobile Games

You can download the podcast here…
http://www.indiegamepod.com/podcasts/gdco-joyent.mp3

Or listen to it here…

[wp_youtube]Nvt9BQAXtT0[/wp_youtube]

Show Notes:
Interviewer: I’m here at Game Developers Conference Online and with me today is a special guest. How about you introduce yourself?

Adrian: Hi, I’m Adrian Ludwig. I’m a VP in Marketing for Joyent.

Interviewer: What’s Joyent about?

Adrian: We’re the premier provider of application services, so infrastructure and the server as a service and platform as a service, especially targeted at game developers who need really high performance, very little latency and a network that gives them a world wide presence for their games so people can get at them.

Interviewer: OK. So, this is a service that’s perfect for developers who are doing online games or social games or mobile games, and they need a back end, say, on the infrastructure for that service.

Adrian: Absolutely, yes. A lot of our big customers and even hundreds of our smaller customers are building social games that tie into Facebook, Yahoo, My Space and are hoping that their game will go from being relatively small to being really, really big.

So, we see a lot of independent game developers who anticipate needing to go through scale issues and move from tens of users or hundreds of users today to millions of users in the near future. So, we help them design the architecture for their application. Then we actually provide the presence in data centers in North America as well as in Europe and Asia so that those games can successfully scale up as they get more and more users.

Interviewer: How is your service then different from Amazon Web Services or Rackspace? Why should smaller developers consider you guys compared to AWS for scaling or Rackspace for scaling? What’s the difference?

Adrian: Sure: The big difference is we’re really an application oriented company. Our origin and all of our customers have always been building large scale or trying to get to a point where they were large scale applications. And so, we know what it means to take an application that’s built in Java or PHP or Ruby start from small and go to big.

We see companies like Amazon or Rackspace really being about running a data center efficiently. They’re about scale for themselves as opposed to scale for your application.

And so, we sit down with a lot of companies and help them understand, here’s where the bottlenecks are, or how do you need to change your application, or when do you need to think about sharding your database? When should you be considering no SQL solutions for your data store as opposed to Sequel based solutions?

So, we really partner to deliver that infrastructure, make sure that it’s optimized for your particular application, and then help you to be successful. So, it works very, very well for small companies. In some instances, those small companies become big companies. So, we’ve got a lot of examples like that.

Country Life, for those who don’t know, one of the top ten Facebook games for quite a while now. Millions of users, as many as ten million monthly, active users. They were literally at that point, just three people inside of the organization that had built the game. We helped them to scale it, and so building that up.

Similarly with companies like, what used to be called Watercooler. It’s now Kabam. They started using us when they were very, very small, and now they’re one of the top Facebook game developers that’s out there. We’ve seen lots of examples of this and gone through a lot of companies, especially in the gaming space building that out.

Interviewer: Can we talk about… So, is it a consulting service that you are offering when these developers are small and trying to scale, and what’s the cost of that? I know that’s people intensive. Is that even something that small developers can consider or even think about or is the pricing so high that they have to go to AWS or something else?

Adrian: No. It’s definitely something that we make available for smaller folks. Our hope and the reason that we’re willing to put that upfront is to help them get big. For the really small, we’ve had partnerships with Facebook, Yahoo, My Space and the other social networks to provide free infrastructure which is sort of an on ramp for somebody building an application and getting it to a point where they can test it. That’s been available for years through those developer portals. We’ve got tens of thousands of developers that are working with it that way.

As companies get a little bit bigger, we do have some packages that we make available. This week we’re actually announcing two new packages specifically around testing and publishing where it’s a prepackage architecture. It’s got load balancing, data store, compute and then a discount for people when they get bigger.

Our experience has been a little bit of consulting, a little bit of support wrapped up with those services which help people to be successful. And so, it pays dividends on the back end for us. We bundle that basically for free for companies that need the assistance in getting bigger.

Interviewer: So, let’s visit this whole services package concept. So, you’re saying that you’ve got packages, computing packages where smaller developers can just use them to automatically, say, scale the front end or scale the back end. Does that mean they don’t even have to consider all the issues of having multiple My SQL databases and all that other stuff, or do they still have to consider all that?

Adrian: No. Somebody has to consider it.

Interviewer: I wasn’t sure if that was your service that takes care of that or is it… For example, with AWS they have the RDS service or they have the SimpleDB service where they can just call some kind of API or something else and use those no SQL solutions without having to really consider a lot of the logical details.

Adrian: You can do that. What we’ve seen is that when people use those services whether it’s Amazon’s or anybody else’s and haven’t thought through the architectural implications in their application, that’s when their apps begin to break. The measure we always have is if you multiply your number of users by 10 or by 100, do your costs go up by 1,000 or 10,000. We see that’s really common for people who are using APIs without thinking about what the architecture should look like.

For example, what is a database… What is the transfer cost associated with putting something into S3? What do you do if all of a sudden you need to go across multiple availability zones? So, those are the kinds of things that we help somebody to think through. There’s absolutely a need to do those kinds of things, and that’s true even if you’re using an API based service. So, that’s just something that we sit down with them.

We help people. A lot of time people are saying, we’re integrating S3 for our long-term storage into an application architecture where most of the compute is on Joyent. We definitely have people who are doing those kinds of hybrids where they’re using multiple service providers. We’ve talked to a few people.

Interviewer: In the pre-interview you mentioned that you guys have a special package or service for just handling My SQL or being able to scale that. Can you talk about what that is and how developers can use that?

Adrian: Sure. At Joyent what we provide, so that a unit of purchase is what we call Smart Machine. It really is a virtual instance. On those virtual instances what you can get on a daily basis so you can scale up, scale down your application horizontally based on that. We have a number of partners that have provided software solutions that we’ve optimized for that environment.

The classic example for us is to use load balancer and traffic manager. It has very sophisticated ability to do things like read-write, splitting between your application to your new database tier. Some of the most popular ones are preconfigured database solutions. So, My Sequel is one of those. And then, another one that we have is Bashoo which provides REAC.

Both of those, whether it’s a Sequel based or no Sequel based solution, we’ve worked with the vendor to optimize it for the particular environment having preconfigured architectures and can help you to make a decision about which of these are right for me based on the needs that I have for my application.

We’re also working on one specifically around some of the application language, like a preconfigured PHP, a preconfigured Ruby. So, it’s really optimized and tuned for execution of the particular development languages.

Interviewer: So, you do have services where folks can just load balance. So, they can easily roll out something that allows for handling a high bunch of front end hits from a whole bunch of people who are visiting your game.

Adrian: Exactly. And that’s how somebody goes, like Back Stage Games, how they back in March of this year went from being a few thousand visitors today to being the fastest growing game on Facebook with over a few million visitors per day by the end of March. In less than 30 days they’ve gone through that ramp.

We’ve seen that a bunch of times. LinkedIn, obviously not in the gaming space, but their bumper sticker game is now over a billion page views every month. It’s built in Ruby and runs on Joyent. And so, there are a lot of examples like that where people are doing tiered databases, tiered load balancing. They might have geographic load balancing. Some of their requests are going to one data center. Some of the requests are going into another data center.

The part that’s exciting for us is LinkedIn is a name you’ve heard of. Kabam is a name you’ve heard of. But a lot of times it’s somebody who was small to start with. Kabam wasn’t a name that you heard of six months ago or a year and a half ago. That’s the part that’s exciting for us, sort of helping somebody who is small get to be big.

Interviewer: When they have to scale to something that big, do they have to then change from My SQL to SimpleDB, or how do you even make that decision, when do you make that decision? When I say SimpleDB, I mean no SQL solution.

Adrian: We haven’t necessarily seen that. It’s definitely something that people talk about and think about. But we’ve seen a lot of instances where they’ll do what we think of as a vertical scale. So, there’ll be an instance of My Sequel running on a relatively small machine, maybe, a 2 gig Smart Machine. And then, it’ll slowly get to be a 4 gig, an 8 gig, a 16 gig. We can help them make that particular instance get larger.

And that’s usually the first level of scaling. Then, they’ll start doing read, write splitting. It can do read-white splitting in a relatively simple manner or with traffic manager you can do read-write splitting across 10, 15 different instances. And then the synchronization gets happened, gets used.

So, actually our biggest customers at this point are still running on My SQL with read-write splitting using Zeus as their intermediate layer.

Interviewer: And they haven’t had, I mean, have there been a lot of issues with the scaling as they ramped up? I mean, obviously Joyent can help with that, but any other surprises or…

Adrian: Yeah. It’s never perfect. I think the reality is as somebody’s going through one of these transitions, they need to make a decision one day. I need to be now on a larger instance. How do I move to a larger instance? What do I need to do with people who are currently on an active session?

With Kabam, for example, we sat down with them and said, “Here is how you need to sub-segment your virtual world in order to be able to stand up in a new instance and have those isolated from one another so that you can optimize the user experience around the particular architecture.

So, anybody who tells you you’re going to go from zero to millions of users with no problems whatsoever is not telling you the truth or just doesn’t know that they’re not telling you the truth. That’s more often the case. They really haven’t done it before, and they haven’t seen it. What I can say though is we’ve been there. We’ve helped the people through it, and we learned something from it every single time.

One of the things that we’re doing with one of our new customers, THQ, is helping them understand because right now they’re dominant in the console space. They’re relatively new in the online space, but they understand that they’re going to need to be prepared to go through that scaling. So, they’ve built their architecture from the get go to go through that as seamlessly as possible and as painlessly as possible. And so, that is something that we help them to do.

Interviewer: Do you have white papers on how to do that, that seamless transition?

Adrian: So, we have white papers around performance and scalability. We have a very good white paper on application architectures and recommended architectures. It goes into the details on how to do database sharding, some of the considerations that you’d want to take and some of the architecture using things like I said, you know, read-write splitting using Zeus.

So yeah, we’re starting to do that. We’re working on a whole bunch of benchmarks as well to help people understand, based on how many users they have, what’s the right technology for me to be using and the forecaster on that.

Interviewer: Let’s talk about pricing then.

Adrian: Sure.

Interviewer: What is the pricing or the business model that you guys have for a small developer who wants to consider you guys?

Adrian: Yeah. Traditionally, our pricing has been extraordinarily simple. We charge by the gigabyte of Ram, and we throw in a fixed amount of storage with that, dedicated storage, long-term storage, actual storage not something that’s sort of synthetic, that’ll go away if your machine has to reboot. That’s thrown in there for free. Unlimited bandwidth is thrown in for free.

We have CPU bursting. That helps with people, especially as they go through scale. That’s usually the biggest challenge that people have is I need more performance right now, not 30 seconds from now, not a minute from now but right now, or things are going to start to break. And so, bursting helps to provide for that.

And then, in addition to that sort of core, we have software packages like I mentioned, Reac, Zeus, a few others where the software developer charges an additional amount on top of that. So, in those instances it’ll be a little bit more expensive on a per gigabyte basis.

Specifically in the gaming space, we’re really focused on gaming. We’re announcing today, actually, here at GDC a new package specifically for game developers basically half off. It’s a little bit better than 50 percent discount on a core architecture that involves load balancing, database layer, application layer and then connectivity into our CDM partners.

We use CD Networks, connectivity into application performance management that New Relic can provide. So, helping you to have the core things that you need in place to be successful.

The way the program works is there’s a fixed fee to participate in the program. You pay up front, and then you have that application architecture for a full year. And then, if you were to grow, then that’s when we would begin to start to make a little bit of money. And we have a discount on our normal prices, but you have access to our folks to be successful and to get access to the growing infrastructure that you need.

Interviewer: And where can developers then find out more information about your service and get started?

Adrian: Yeah. So, joyent.com has all the information, specifically, the game studios. You can go to joyent.com/gamestudio. We’ve got a ton of information about what people have been doing, how they’ve been building in our Wiki, in our discussion groups.

We’ve been in this business of providing infrastructure for applications for six years now. We often point out that we were a Cloud before Amazon even started using the word Cloud and it became popular. We’ve got that knowledge, both internally but also in the community. People have been building this for this long. There’s a ton of information up there, joyent.com.

Interviewer: Thank you very much.