Fight My Monster Architecture in 5 mins!

We computing engineers live in exciting times. The Internet and its supporting infrastructures really have come of age, and all of a sudden things can, and mostly importantly should be done in new ways. Some small teams are successfully scaling enterprises to massive scale on tiny budgets, and the new generation of applications and games can be delivered across multiple platform formats. With these opportunities come responsibilities and we need to choose our technical strategies and platforms wisely, as these decisions have deep impact on developing tech businesses.

Fight My Monster has grown rapidly and will soon pass 1 million user accounts. But although we recently completed a funding round with some fantastic investors, the business was originally bootstrapped on a very limited budget. For those following the project, and for those who are considering working for Fight My Monster  — as you may have heard, Fight My Monster is expanding and has some great opportunities for talented engineers right now — I wanted to provide a quick overview of our architecture, and the reasons we chose it. As you will see, it’s not that conventional.

We’ve gone for a simple but scalable three level architecture, which we host in the cloud. I hope this overview of how our system works proves interesting.

RIA Client

We have a Flash client with a difference. It is full screen, uses a flowed layout, and looks and behaves like a normal website, except in those areas where highly interactive game sparkle is required. The client implements an underused technique called deep linking which allows individual virtual pages within the “site” to be addressed by URL, and browser integration via Javascript allows the simulated site to respond to clicks on the browser’s forward and back buttons and change page. The client has a complex state, but if the browser window is refreshed, the client is able to restore its previous state from the server. We developed the client mainly in Flex using a framework we developed in house. It’s one of the more sophisticated Flex applications out there at the moment.

Application Layer (Starburst)

Our client communicates with the application layer using a proprietary remote method calling framework built on top of the RTMP (Real Time Media Protocol) protocol supported by the Flash player. RTMP allows for multiplexing of a remote method call stream, video and other data over single TCP/IP connection, in which remote calls pack their parameters using AMF (Action Message Format). The protocol also allows server code to call into functions that the client has registered. The framework we created allows call results to be returned asynchronously in a different order to which the calls were made, such that server methods taking a little longer to generate their result do not block the return of results for following calls, thereby keeping client applications responsive. The framework also handles things like automatic reconnection, the queuing of events while a connection is broken, and the resetting of client state where data has been lost after cluster events or prolonged disconnection.

The application layer itself is comprised from a horizontally scalable cluster of servers running a proprietary Java software framework called Starburst, which we may eventually open source. Starburst uses the Red5 server as a container, which provides the base RTMP functionality and contains Tomcat too, which enables us to serve JSP pages (we only serve a few HTML pages, but it is a requirement nonetheless). The current setup may change, and we are looking at using Netty and even replacing RTMP with a bespoke protocol but that is another discussion.

I am limited in what I can say about how Starburst works right now, but basically a client connects by making an initial HTTP “directory” request to obtain the IP address of the cluster node it should connect to. The HTTP request is made to a load balancer, and can be forwarded to any running Starburst node in the cluster, which is completely symmetric in the sense that all nodes are the same and none play special roles. Once the targeted cluster node’s IP address is returned, the client makes a direct RTMP connection to that node. The node the client is directed to is chosen using consistent hashing (a really useful approach that you should Google if you have not come across before). The consistent hashing algorithm evenly distributes clients over the cluster nodes and, if a cluster node is lost, automatically evenly redistributes clients that were connected to it among the remaining nodes, thereby preventing load spikes and hot spots.

The cool bit about Starburst is that it completely abstracts away the multiplicity of client and server nodes for application code writers. Clients and server side connection contexts may register to receive events, and server code written to Starburst APIs may generate events on paths or for specific users that may or may not be online, but it is only ever necessary to write business/game logic, never to consider how different parts of the system are connected. I can’t go into the means by which this is achieved at scale now, but needless to say it makes a real difference to what we do.

Database (Cassandra)

There used to be only one way to make complex apps that scale: shard your database. Sharding is the process of dividing up responsibility for different ranges of your data amongst your servers. A simple sharding strategy might be, for example, to have 27 servers and then use the first letter of usernames to map responsibility for users to the servers. In practice, many organizations also shard functionality, so different servers have different types of function too. For example, Facebook works this way with MySQL databases. The problem with this approach is that your systems become very, very complex, and system complexity has a tendency to scale up with app complexity.

When we first started with Fight My Monster, we also began by thinking about a sharded architecture using MySQL for storage. However, Fight My Monster is a pretty complex app, and it quickly became apparent that sharding was going to require too much work given the resources we had at that time. We then experimented with different ideas, and it was about this time that the NoSQL movement first came to our attention. For all the described reasons, swapping the relational database model for a less complex one that could scale without sharding immediately stood out as a bargain. We began using HBase, which is a very solid system that offers a single logical database that is horizontally scalable, but which is actually quite complex to maintain and administer. Because of that complexity, and for other reasons including throughput, we moved over to Cassandra, which has proven to be a good decision for us.

Without wishing to stir up NoSQL flame wars, Cassandra has amazing qualities. Some of the highlights are as follows:-

• Clusters provide a single logical database that is almost infinitely scalable
• The cluster is symmetric i.e there are no special types of node to manage
• Nodes organize themselves using a P2P system, and adding new nodes is simple
• Performance is best in category, with write performance better than read performance
• Data is safely replicated across nodes (we use a Replication Factor of 3)
• A number of nodes can go down without affecting cluster operation

We access Cassandra using an open source Java library we originally created called Pelops. Because our data processing is highly transactional, and Cassandra does not provide locking mechanisms itself, we need to handle the distributed serialization of database operations initiated from the various Starburst nodes. For this we use another open source library that we also created called Cages in conjunction with Hadoop’s ZooKeeper system.

Using Cages and ZooKeeper has worked very well for us, but if there is a single theme running through Fight My Monster’s architecture it’s simplification! For this reason, I recently developed the Wait Chain Algorithm, which provides a way for Cassandra itself to be used as a distributed lock server, for example using the Pelops or Hector clients, thus providing a way for us to eventually do without ZooKeeper. Watch this blog for news of the first system using this algorithm.

Red5 cabin fever – advanced scope and room management

Red5 provides us with the abstraction of a hierarchy of scopes or “rooms”, to which clients can connect. In their most basic usage, scopes are simply used as rooms where for example shared objects used for chat applications or live media streams are located. However in the development of advanced MMOs and other applications, we want to push our usage of scopes a little further. This post aims to share some tips that can help those wishing to use Red5 scopes.

First a quick recap of the scope system. Red5 assigns each application a default scope, known as the “WebScope”. This scope exists whether or not anyone is connected to it (more on scope lifetime later). Once clients connect to scopes, or we otherwise create them, you get a tree structure.

WebScope
WebScope->Offices
WebScope->Offices->QueensPark
WebScope->Offices->Sausalito
WebScope->Seminars
WebScope->Seminars->Red5
etc

Now to the tips! Note: This post is not meant an exhaustive discussion of Red5 scopes. It simply offers up some useful pointers for advanced usage. Don’t forget to read the documentation 😉

1. The basics: Scope connection sequence

Flash/Flex clients connect to scopes using code such as:

var nc: NetConnection = new NetConnection();
// ... assign event handlers etc.

// connect to scope /Offices/QueensPark inside myApp
nc.connect("rtmp://someserver.com/myApp/Offices/QueensPark");

You will likely have created your Red5 application by extending the ApplicationAdapter class. When you override the connect() callback method of this class,  this will be called when clients connect. It is important to realize that it will be called for the entire scope hierarchy that the client is connecting to. So the following code:

@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    System.out.println("Connecting to: " + scope.getPath() + "/" + scope.getName() +
        " (" + scope.getContextPath() + ")");
}

Will produce this output in your Eclipse console:

Connecting to: /default/myApp ()
Connecting to: /default/myApp/Offices (/Offices)
Connecting to: /default/myApp/Offices/QueensPark (/Offices/QueensPark)

In the above output, first the client is connected to /default/myApp your WebScope, then down through successive parent scopes to the target leaf scope. At any stage, you can prevent connection by calling rejectClient(<REASON>), or throwing a ClientRejectedException(<REASON>) deep in your stack. The client will receive the reason for the rejection in their error handler e.g.

@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    return true;
}

2. Beyond the basics: Authenticate in connect()?

Those new to Red5, or those not having to build scalable applications, may be tempted to stop reading now. They might now think they know all they need to know to implement an authentication scheme e.g.

Client:

nc.connect("rtmp://someserver.com/myApp/Offices/QueensPark", "dominic", "abracadabra");

Server:

@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    if (!Accounts.authenticate(params[0], params[1])
        rejectClient("Invalid login details");
    return true;
}

Unfortunately there is a little more to it than that. First of all, you need to notice the innocuous synchronize statement inserted before the ApplicationAdapter.connect method.

If you think about it, this little statement will have the effect of serializing execution of the connect() callback, and thus serializing the process of connecting to your application. Therefore if, as is most likely, your authentication code involves a database lookup to retrieve the password for the user, if there are multiple clients trying to connect at once many will spend ages in a “connection” queue – database lookups take *time* so you don’t want to serialize operations!

The next thing you should therefore do, is remove the synchronized statements from your callbacks by deriving your application from MultiThreadedApplicationAdapter rather than ApplicationAdapter as is typical i.e.

@Override
public boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    if (!Accounts.authenticate(params[0], params[1])
        rejectClient("Invalid login details");
    return true;
}

But does that solve things??? Unfortunately not!

If for the sake of experimentation (1) change your connect() implementation to the following code (2) open a couple of Flex clients in your browser (3) make them connect simultaneously (4) have a look at the output in your Eclipse Console window… you will notice that the calls to connect() are *still* serialized.

@Override
public boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    System.out.println("connect() starting work " + (new Date()).toString());
    Thread.sleep(4000);
    System.out.println("connect() finishing work " + (new Date()).toString());
    return true;
}

The problem is that somewhere deep inside the Red5 architecture is a global lock which is serializing calls to connect() for you. The only way around this performance problem is to use lazy authentication, as described in the next section.

3. Lazy authentication on demand

Now it maybe that you have a simple application, where the user authenticates themselves when they connect, and thereafter connects to scopes and performs actions as an authenticated user. Fine.

But what if you want to make things a little more complicated, by having for example two scopes, one of which contains publicly available functionality, and the other which contains functionality only authenticated users can access e.g.
WebScope
WebScope->signup
WebScope->game
Note: I apologise if these examples get rather contrived, but they will help illustrate some points!

The solution is to add an authentication method to your WebScope. This is really easy to achieve, simply add a *public* method to the application class you derived from MultiThreadedApplicationAdapter e.g.

public boolean authenticate(String username, String password) {
    // do your stuff
    return true;
}

When the client receives a notification about successful connection, it can then call authenticate e.g.

    nc.call(
        "authenticate",
        new Responder(
            function() { game.proceed(); } // chain forwards!
            function(fault: Object) { Alert.show("Authentication failed\n"+ObjectUtil.toString(fault)); }),
        "dominic",
        "abracadabra"
    );

4. But I want to authenticate in my own scope, not WebScope!?!?

As you will have no doubt realized, there is a problem with this approach. In order to call authenticate(), your client needs to be connected to the default root scope (the WebScope). Furthermore, you need a way of preventing your client from connecting to protected scopes without the necessary privileges.

You might hope the solution is to (1) mark clients as authenticated using attributes, and (2) provide another method that effects dynamic server-side navigation to your scopes. So we now have inside our application class:

public boolean authenticate(String username, String password) {
    // do your stuff
    Red5.getLocalConnection().getClient().setAttribute("Authenticated", true);
    return true;
}
public boolean setScope(String[] path) {
    // change scope
}
@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    if (scope.getContextPath().length != 0
        && conn.getClient().getAttribute("Authenticated") != true)
        rejectClient("Authentication required");
    return true;
}

5. Damn, I can’t make my setScope method work!

This method kind of works, but there are a bunch of gotchas related to dynamically changing scope, which I shall iterate to finish of this post.

1. You change a client’s scope, by joining it connection to that destination scope.
2. When you join the connection, this will cause Red5 to create a completely new Client object, even though the actual (remote) client is unchanged

Therefore, the following code will *not* work:

public boolean setScope(String[] path) {
    IScope destinationScope = getDestinationScope(path); // your own method
    IConnection conn = Red5.getConnectionLocal();
    IClient originalClient = conn.getClient();
    conn.connect(destinationScope);
    IClient newClient = conn.getClient();
    newClient.setAttribute("Authenticated", originalClient.getAttribute("Authenticated"));
}

The problem is that deep inside Red5 your call to conn.connect above will result in your overridden version of the connect application callback being called. Unfortunately, it will be called with a new client object that does not have the Authenticated attribute you set, and which will therefore cause your code to call rejectClient.

6. Forget authenticating in connect and all is sweet…

The solution is to take your overridden application connect callback method out of the authentication process completely. You should now check whether someone is allowed to connect to a scope inside setScope.

public boolean authenticate(String username, String password) {
    // do your stuff
    Red5.getLocalConnection().getClient().setAttribute("Authenticated", true);
    return true;
}
public boolean setScope(String[] path) {
    IConnection conn = Red5.getConnectionLocal();
    IClient client = conn.getClient();
    IScope destinationScope = getDestinationScope(path); // your own method

    // Now see whether the client is allowed to connect to the destination scope.
    if (destinationScope.getAttribute("RequiresAuthentication") == true) {
        if (client.getAttribute("Authenticated") != true)
            return false;

    // We got this far, so we must be allowed to connect
    conn.connect(destinationScope);

    // Carry attributes from original client object to new client object
    IClient newClient = conn.getClient();
    newClient.setAttribute("Authenticated", client.getAttribute("Authenticated"));
}

7. Miscellanea

If you’ve read this article and like the idea of setting up your own Scope structure and advanced authentication scheme, read on for just a couple of moments. These two tips could save you some more time!

Firstly, you may have used your application startup callback to initialize your scope structure. For example:

@Override
public boolean appStart(IScope app) {
    if (!super.appStart(app))
        return false;

    if (!app.createChildScope("signup"))
        return false;

    if (!app.createChildScope("game"))
        return false;

     IScope gameScope = app.getScope("game");
     gameScope.setAttribute("RequiresAuthentication");

    return true;
}

You find that the first time you connect a client to your scope, everything works fine, but the second time you connect your client to your scope, you get an error saying the scope doesn’t exist.

The reason is that by default, Red5 tears down scopes when the last client disconnects from them. If you wish to setup standard scopes, and have them permanently in memory, for example preserving state irrespective of whether clients are connected, you need to pin them there.

The simplest way to pin your standard scope is to make your application “listen” to events on them. For example:

public class MyApp extends MultiThreadedApplicationAdapter implements IEventListener {
...
@Override
public boolean appStart(IScope app) {
    if (!super.appStart(app))
        return false;

    // setup signup scope
    if (!app.createChildScope("signup"))
        return false;
    app.getScope("signup").addEventListener(this);

    // setup game scope
    if (!app.createChildScope("game"))
        return false;
    IScope gameScope = app.getScope("game");
    gameScope.setAttribute("RequiresAuthentication");
    gameScope.addEventListener(this);

    return true;
}
...
}

Finally, you might have realized from the above that once a client has connected to a scope, they will no longer have access to the setScope method we defined and will therefore be unable to dynamically change scope again!!

The solution here is to package your setScope function into a special handler that you will add to every scope. For example:

public class MyAppStandardMethods {
public boolean setScope(String[] path) {
    IConnection conn = Red5.getConnectionLocal();
    IClient client = conn.getClient();
    IScope destinationScope = getDestinationScope(path); // your own method

    // Now see whether the client is allowed to connect to the destination scope.
    if (destinationScope.getAttribute("RequiresAuthentication") == true) {
        if (client.getAttribute("Authenticated") != true)
            return false;

    // We got this far, so we must be allowed to connect
    conn.connect(destinationScope);

    // Carry attributes from original client object to new client object
    IClient newClient = conn.getClient();
    newClient.setAttribute("Authenticated", client.getAttribute("Authenticated"));
}
}

This handler can be added to every scope. For example our appStart might become:

public class MyApp
    extends MultiThreadedApplicationAdapter
    implements IEventListener {
...
@Override
public boolean appStart(IScope app) {
    if (!super.appStart(app))
        return false;

    // setup signup scope
    if (!app.createChildScope("signup"))
        return false;
    IScope signupScope = app.getScope("signup");
    signupScope.addEventListener(this);
    signupScope.registerServiceHandler("api", new MyAppStandardMethods());

    // setup game scope
    if (!app.createChildScope("game"))
        return false;
    IScope gameScope = app.getScope("game");
    gameScope.setAttribute("RequiresAuthentication");
    gameScope.addEventListener(this);
    gameScope.registerServiceHandler("api", new MyAppStandardMethods());

    return true;
}
...
}

If you have dynamically created scopes, you can also register MyAppStandardMethods as a service handler in the relevant application callback.

Now wherever your client is connected within your scopes tree it can move around any time it wants using setScope e.g.

    nc.call(
        "api.setScope",
        new Responder(
            function() { game.renderBattlefield(); } // chain forwards!
            function(fault: Object) { Alert.show("Changing scope failed\n"+ObjectUtil.toString(fault)); }),
        "battlefield",
    );

Hope that helps!

Quick Red5 install on Mac OS X

These steps work fine with version 0.9.0 RC2 at least…

1. Download the latest dmg from http://code.google.com/p/red5/
2. Open the package, drag Red5 to your Applications folder, then start the app (nothing happens visually, it’s a server!)
3. Open http://127.0.0.1:5080/demos/ in a browser, which should hopefully now show you the demo page

The next step is to install the demos you want to try using a flash control panel that comes packaged with the distribution. Now, be warned, as of version 0.9.0 RC2, the distribution is a little flaky. So when you click the link to “install” demos on the page you just loaded, it won’t work.

So if the link doesn’t work for you, try http://127.0.0.1:5080/installer/ instead! Go back to the first page to access the demo you installed.

Finally use a terminal session to browse inside your app folder (i.e. inside /Applications/Red5.app) and have a poke around the underlying code and configurations.

/Applications/Red5.app/Contents/Resources/Java/webapps is a good place to start. You’ll notice that the system packs Tomcat, and the distribution acts as a web server.