Dominic Williams

Occasionally useful posts about RIAs, Web scale computing & miscellanea

Archive for the ‘Flash’ Category

Fight My Monster Architecture in 5 mins!

with 2 comments

We computing engineers live in exciting times. The Internet and its supporting infrastructures really have come of age, and all of a sudden things can, and mostly importantly should be done in new ways. Some small teams are successfully scaling enterprises to massive scale on tiny budgets, and the new generation of applications and games can be delivered across multiple platform formats. With these opportunities come responsibilities and we need to choose our technical strategies and platforms wisely, as these decisions have deep impact on developing tech businesses.

Fight My Monster has grown rapidly and will soon pass 1 million user accounts. But although we recently completed a funding round with some fantastic investors, the business was originally bootstrapped on a very limited budget. For those following the project, and for those who are considering working for Fight My Monster  – as you may have heard, Fight My Monster is expanding and has some great opportunities for talented engineers right now – I wanted to provide a quick overview of our architecture, and the reasons we chose it. As you will see, it’s not that conventional.

We’ve gone for a simple but scalable three level architecture, which we host in the cloud. I hope this overview of how our system works proves interesting.

RIA Client

We have a Flash client with a difference. It is full screen, uses a flowed layout, and looks and behaves like a normal website, except in those areas where highly interactive game sparkle is required. The client implements an underused technique called deep linking which allows individual virtual pages within the “site” to be addressed by URL, and browser integration via Javascript allows the simulated site to respond to clicks on the browser’s forward and back buttons and change page. The client has a complex state, but if the browser window is refreshed, the client is able to restore its previous state from the server. We developed the client mainly in Flex using a framework we developed in house. It’s one of the more sophisticated Flex applications out there at the moment.

Application Layer (Starburst)

Our client communicates with the application layer using a proprietary remote method calling framework built on top of the RTMP (Real Time Media Protocol) protocol supported by the Flash player. RTMP allows for multiplexing of a remote method call stream, video and other data over single TCP/IP connection, in which remote calls pack their parameters using AMF (Action Message Format). The protocol also allows server code to call into functions that the client has registered. The framework we created allows call results to be returned asynchronously in a different order to which the calls were made, such that server methods taking a little longer to generate their result do not block the return of results for following calls, thereby keeping client applications responsive. The framework also handles things like automatic reconnection, the queuing of events while a connection is broken, and the resetting of client state where data has been lost after cluster events or prolonged disconnection.

The application layer itself is comprised from a horizontally scalable cluster of servers running a proprietary Java software framework called Starburst, which we may eventually open source. Starburst uses the Red5 server as a container, which provides the base RTMP functionality and contains Tomcat too, which enables us to serve JSP pages (we only serve a few HTML pages, but it is a requirement nonetheless). The current setup may change, and we are looking at using Netty and even replacing RTMP with a bespoke protocol but that is another discussion.

I am limited in what I can say about how Starburst works right now, but basically a client connects by making an initial HTTP “directory” request to obtain the IP address of the cluster node it should connect to. The HTTP request is made to a load balancer, and can be forwarded to any running Starburst node in the cluster, which is completely symmetric in the sense that all nodes are the same and none play special roles. Once the targeted cluster node’s IP address is returned, the client makes a direct RTMP connection to that node. The node the client is directed to is chosen using consistent hashing (a really useful approach that you should Google if you have not come across before). The consistent hashing algorithm evenly distributes clients over the cluster nodes and, if a cluster node is lost, automatically evenly redistributes clients that were connected to it among the remaining nodes, thereby preventing load spikes and hot spots.

The cool bit about Starburst is that it completely abstracts away the multiplicity of client and server nodes for application code writers. Clients and server side connection contexts may register to receive events, and server code written to Starburst APIs may generate events on paths or for specific users that may or may not be online, but it is only ever necessary to write business/game logic, never to consider how different parts of the system are connected. I can’t go into the means by which this is achieved at scale now, but needless to say it makes a real difference to what we do.

Database (Cassandra)

There used to be only one way to make complex apps that scale: shard your database. Sharding is the process of dividing up responsibility for different ranges of your data amongst your servers. A simple sharding strategy might be, for example, to have 27 servers and then use the first letter of usernames to map responsibility for users to the servers. In practice, many organizations also shard functionality, so different servers have different types of function too. For example, Facebook works this way with MySQL databases. The problem with this approach is that your systems become very, very complex, and system complexity has a tendency to scale up with app complexity.

When we first started with Fight My Monster, we also began by thinking about a sharded architecture using MySQL for storage. However, Fight My Monster is a pretty complex app, and it quickly became apparent that sharding was going to require too much work given the resources we had at that time. We then experimented with different ideas, and it was about this time that the NoSQL movement first came to our attention. For all the described reasons, swapping the relational database model for a less complex one that could scale without sharding immediately stood out as a bargain. We began using HBase, which is a very solid system that offers a single logical database that is horizontally scalable, but which is actually quite complex to maintain and administer. Because of that complexity, and for other reasons including throughput, we moved over to Cassandra, which has proven to be a good decision for us.

Without wishing to stir up NoSQL flame wars, Cassandra has amazing qualities. Some of the highlights are as follows:-

  • Clusters provide a single logical database that is almost infinitely scalable
  • The cluster is symmetric i.e there are no special types of node to manage
  • Nodes organize themselves using a P2P system, and adding new nodes is simple
  • Performance is best in category, with write performance better than read performance
  • Data is safely replicated across nodes (we use a Replication Factor of 3)
  • A number of nodes can go down without affecting cluster operation

We access Cassandra using an open source Java library we originally created called Pelops. Because our data processing is highly transactional, and Cassandra does not provide locking mechanisms itself, we need to handle the distributed serialization of database operations initiated from the various Starburst nodes. For this we use another open source library that we also created called Cages in conjunction with Hadoop’s ZooKeeper system.

Using Cages and ZooKeeper has worked very well for us, but if there is a single theme running through Fight My Monster’s architecture it’s simplification! For this reason, I recently developed the Wait Chain Algorithm, which provides a way for Cassandra itself to be used as a distributed lock server, for example using the Pelops or Hector clients, thus providing a way for us to eventually do without ZooKeeper. Watch this blog for news of the first system using this algorithm.

Written by dominicwilliams

February 28, 2012 at 8:32 pm

As3Crypto RSA “padding function returned null” death bug fix :)))

with 5 comments

UPDATE: Unfortunately it transpired the fix I had only worked for small amounts of data I was encrypted/decrypting. It was masking a problem I could see in the code, but did not work with larger amounts of data. This post therefore has changed into a report of the problem. Sorry for any disappointment.

WHAT I FOUND: Firstly, this really is a horrible bug, and I’ve had to move on after spending much more time on it than I intended!

For those that what to grab the baton, here are the main points. When you encrypt/decrypt some data, the results will actually appear non-deterministic as illustrated by the code immediately following.

This is the first thing you need to eliminate to debug, and this achieved by disabling the Random class in Random.as by returning 0xff in the first line of nextByte(). Thereafter, you will the encryption operations that fail will be deterministic, unlike in the following code:

	protected function onRunTest(event:MouseEvent):void
	{			
		var privKey: String = "-----BEGIN RSA PRIVATE KEY-----\n" +
			"MIIBOwIBAAJBAM7U/mEVLiXKC4vGjpDcQ3qiAi/paF2Qp03Y6JXvcuYZG5/vHRgD\n" +
			"qWpVX+oFtXq7uhjcl+m4fx9zSfklcwdmknUCAwEAAQJAFN7PMFKnzm5dzePiPOHM\n" +
			"+VHhsJ33xwEysJtDlOWNjYQqhrpYRNgkx/fRKzEbY9ymwihRhO1IfBHpOYL5WXgQ\n" +
			"3QIhAPZKg4j+osWdRmv33Ql8bJdls5u5OqgX5k7clvbawwzrAiEA1vxFt/o9BMds\n" +
			"DJDxfDl1eYmUsYFUP2vMavjHgz3Phh8CIQCK9ClX7koJch1cJuCnTHK7zB5UWmH0\n" +
			"ml9O2Pe3WF85dwIgNhEn74cNhYAp2lcxhE5nDvPc429lIrYXqOd8NbN7130CIQCo\n" +
			"JUINRE5bjANnPIIviuVUqJDAmc1ZS29dyJiz/Qr+3w==\n" +
			"-----END RSA PRIVATE KEY-----";
		
		var pubKey: String = "-----BEGIN PUBLIC KEY-----\n" +
			"MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAM7U/mEVLiXKC4vGjpDcQ3qiAi/paF2Q\n" +
			"p03Y6JXvcuYZG5/vHRgDqWpVX+oFtXq7uhjcl+m4fx9zSfklcwdmknUCAwEAAQ==\n" +
			"-----END PUBLIC KEY-----";
						
		var dummyTxt:String =
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790";
				
		var rsaPublic: RSAKey = PEM.readRSAPublicKey(pubKey);
		var rsaPrivate: RSAKey = PEM.readRSAPrivateKey(privKey);

		for (var i: int=0; i<500; i++) 
		{
			var message: String = dummyTxt.substr(0, Math.floor(Math.random() * 400));
			
			if (!doTest(rsaPublic, rsaPrivate, message))
			{	
				if (doTest(rsaPublic, rsaPrivate, message))
				{
					Alert.show("Arrghh. Nondeterministic! "+i);
					break;
				}
			}
			trace(i);
		}
	}
	
	protected function doTest(rsaPublic: RSAKey, rsaPrivate: RSAKey, message: String): Boolean
	{
		try {
			var src:ByteArray = Hex.toArray(Hex.fromString(message));
			var encrypted:ByteArray = new ByteArray();
			rsaPublic.encrypt(src, encrypted, src.length);
			
			var decrypted:ByteArray = new ByteArray();
			rsaPrivate.decrypt(encrypted, decrypted, encrypted.length);
			var original:String = Hex.toString(Hex.fromArray(decrypted));

			if (original != message) {
				Alert.show("Corruption: shouldn't happen!");
			}
		} catch (e: Error) {
			return false;
		}
		return true;
	}

Of course in many ways it would have been nice if the result continued to be nondeterministic without Random being disabled. It would mean some global state was accidentally disabled which would be relatively easy to find.

Instead, I’m afraid this looks like a bit of a demon bug buried in the complexities of bit shifting and maths with big integer values. If you are reading this, and you are an encryption specialist, realize that it is your responsibility to sort this out ;-) since it will take a non-specialist a very long time.

For those feeling brave, you’ll notice that in the following function from RSA.as, when chunk is calculated in such a way as to cause only 63 bytes to be written to dst, then you’ve got a problem. You can partially mask the problem by having BigInteger construct from 63-bytes, which works, but this does not solve the problem when a concatenation of big integers is written out and some entry but the last is shortened.

	private function _encrypt(op:Function, src:ByteArray, dst:ByteArray, length:uint, pad:Function, padType:int):void {
		// adjust pad if needed
		if (pad==null) pad = pkcs1pad;
		// convert src to BigInteger
		if (src.position >= src.length) {
			src.position = 0;
		}
		var bl:uint = getBlockSize();
		var end:int = src.position + length;
		while (src.position<end) {
			var block:BigInteger = new BigInteger(pad(src, end, bl, padType), bl, true);
			var chunk:BigInteger = op(block);
			chunk.toArray(dst);
		}
	}

Good karma in advance for anyone who is brave enough to take this on!

Written by dominicwilliams

June 4, 2010 at 4:02 pm

Flex font embedding with Spark and Halo made easy!

with 7 comments

I’m going to keep this one really brief. There is so much stuff written about fonts and Flex on the Web that the subject seems incredibly complex. It is not(!) and I’m going to quickly outline what I think is the best real-world production approach to take with fonts when using Flash Builder 4.

Firstly, understand that you often *should* use embedded fonts inside your complex Flex projects. The reason is simple: you probably don’t your text to look slightly different depending on what computer it’s running on. For example, a Windows machine and Mac will have different versions of the Arial font. Unless you embed the font your are going to use, you do not know exactly what the font will look like to the user, and the differences can be very significant from the design perspective. Better to embed fonts, so it always looks the same.

So, you need to embed fonts. How do you do this? The most convenient way is to use a style declaration in the root of your project e.g.

       
<fx:Style>
        @font-face {
            font-family: "MyriadPro";
            src: url("assets/MyriadPro-Regular.otf");
            embedAsCFF: true;
	}
</fx:Style>

Notice that we embed as CFF, which provides benefits such as advanced anti-aliasing. But, you howl, Halo (the stuff starting with mx) doesn’t support this! Don’t worry, this is covered in a moment.

Before moving on, we’ll optimize this some by only embedding the characters from the font we are likely to use, which keeps file size down.

<fx:Style>
        @font-face {
            font-family: "MyriadPro";
            src: url("assets/MyriadPro-Regular.otf");
            embedAsCFF: true;
            unicode-range: U+0020-U+007D;
	}
</fx:Style>

So now everything looks the same on each system, perhaps we have been able to use an exotic font, and we haven’t blown our SWF file size up too much. But there’s a problem, Halo’s mx controls show blank spaces where text should be.

No worries, this is simple too. You just need to override the class of the text renderer object used by the Halo controls.

<fx:Style>
        @namespace mx "library://ns.adobe.com/flex/mx";
        @font-face {
            font-family: "MyriadPro";
            src: url("assets/MyriadPro-Regular.otf");
            embedAsCFF: true;
            unicode-range=U+0020-U+007D;
	}
        mx|Alert, mx|Button, mx|Form, mx|FormHeading, mx|FormItem, mx|FormItemLabel {
            font-family: MyriadPro;
            textFieldClass: ClassReference("mx.core.UIFTETextField");
	}
</fx:Style>

Nice! There you have your production font strategy. Of course, the more of those sorry old mx controls you are using, the longer your list of mx|,… will be :)

Written by dominicwilliams

May 27, 2010 at 12:23 pm

Follow

Get every new post delivered to your Inbox.

Join 66 other followers