As3Crypto RSA “padding function returned null” death bug fix :)))

UPDATE: Unfortunately it transpired the fix I had only worked for small amounts of data I was encrypted/decrypting. It was masking a problem I could see in the code, but did not work with larger amounts of data. This post therefore has changed into a report of the problem. Sorry for any disappointment.

WHAT I FOUND: Firstly, this really is a horrible bug, and I’ve had to move on after spending much more time on it than I intended!

For those that what to grab the baton, here are the main points. When you encrypt/decrypt some data, the results will actually appear non-deterministic as illustrated by the code immediately following.

This is the first thing you need to eliminate to debug, and this achieved by disabling the Random class in Random.as by returning 0xff in the first line of nextByte(). Thereafter, you will the encryption operations that fail will be deterministic, unlike in the following code:

	protected function onRunTest(event:MouseEvent):void
	{			
		var privKey: String = "-----BEGIN RSA PRIVATE KEY-----\n" +
			"MIIBOwIBAAJBAM7U/mEVLiXKC4vGjpDcQ3qiAi/paF2Qp03Y6JXvcuYZG5/vHRgD\n" +
			"qWpVX+oFtXq7uhjcl+m4fx9zSfklcwdmknUCAwEAAQJAFN7PMFKnzm5dzePiPOHM\n" +
			"+VHhsJ33xwEysJtDlOWNjYQqhrpYRNgkx/fRKzEbY9ymwihRhO1IfBHpOYL5WXgQ\n" +
			"3QIhAPZKg4j+osWdRmv33Ql8bJdls5u5OqgX5k7clvbawwzrAiEA1vxFt/o9BMds\n" +
			"DJDxfDl1eYmUsYFUP2vMavjHgz3Phh8CIQCK9ClX7koJch1cJuCnTHK7zB5UWmH0\n" +
			"ml9O2Pe3WF85dwIgNhEn74cNhYAp2lcxhE5nDvPc429lIrYXqOd8NbN7130CIQCo\n" +
			"JUINRE5bjANnPIIviuVUqJDAmc1ZS29dyJiz/Qr+3w==\n" +
			"-----END RSA PRIVATE KEY-----";
		
		var pubKey: String = "-----BEGIN PUBLIC KEY-----\n" +
			"MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAM7U/mEVLiXKC4vGjpDcQ3qiAi/paF2Q\n" +
			"p03Y6JXvcuYZG5/vHRgDqWpVX+oFtXq7uhjcl+m4fx9zSfklcwdmknUCAwEAAQ==\n" +
			"-----END PUBLIC KEY-----";
						
		var dummyTxt:String =
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790"+
			"0123456790012345679001234567900123456790012345679001234567900123456790012345679001234567900123456790";
				
		var rsaPublic: RSAKey = PEM.readRSAPublicKey(pubKey);
		var rsaPrivate: RSAKey = PEM.readRSAPrivateKey(privKey);

		for (var i: int=0; i<500; i++) 
		{
			var message: String = dummyTxt.substr(0, Math.floor(Math.random() * 400));
			
			if (!doTest(rsaPublic, rsaPrivate, message))
			{	
				if (doTest(rsaPublic, rsaPrivate, message))
				{
					Alert.show("Arrghh. Nondeterministic! "+i);
					break;
				}
			}
			trace(i);
		}
	}
	
	protected function doTest(rsaPublic: RSAKey, rsaPrivate: RSAKey, message: String): Boolean
	{
		try {
			var src:ByteArray = Hex.toArray(Hex.fromString(message));
			var encrypted:ByteArray = new ByteArray();
			rsaPublic.encrypt(src, encrypted, src.length);
			
			var decrypted:ByteArray = new ByteArray();
			rsaPrivate.decrypt(encrypted, decrypted, encrypted.length);
			var original:String = Hex.toString(Hex.fromArray(decrypted));

			if (original != message) {
				Alert.show("Corruption: shouldn't happen!");
			}
		} catch (e: Error) {
			return false;
		}
		return true;
	}

Of course in many ways it would have been nice if the result continued to be nondeterministic without Random being disabled. It would mean some global state was accidentally disabled which would be relatively easy to find.

Instead, I’m afraid this looks like a bit of a demon bug buried in the complexities of bit shifting and maths with big integer values. If you are reading this, and you are an encryption specialist, realize that it is your responsibility to sort this out 😉 since it will take a non-specialist a very long time.

For those feeling brave, you’ll notice that in the following function from RSA.as, when chunk is calculated in such a way as to cause only 63 bytes to be written to dst, then you’ve got a problem. You can partially mask the problem by having BigInteger construct from 63-bytes, which works, but this does not solve the problem when a concatenation of big integers is written out and some entry but the last is shortened.

	private function _encrypt(op:Function, src:ByteArray, dst:ByteArray, length:uint, pad:Function, padType:int):void {
		// adjust pad if needed
		if (pad==null) pad = pkcs1pad;
		// convert src to BigInteger
		if (src.position >= src.length) {
			src.position = 0;
		}
		var bl:uint = getBlockSize();
		var end:int = src.position + length;
		while (src.position<end) {
			var block:BigInteger = new BigInteger(pad(src, end, bl, padType), bl, true);
			var chunk:BigInteger = op(block);
			chunk.toArray(dst);
		}
	}

Good karma in advance for anyone who is brave enough to take this on!

Flex font embedding with Spark and Halo made easy!

I’m going to keep this one really brief. There is so much stuff written about fonts and Flex on the Web that the subject seems incredibly complex. It is not(!) and I’m going to quickly outline what I think is the best real-world production approach to take with fonts when using Flash Builder 4.

Firstly, understand that you often *should* use embedded fonts inside your complex Flex projects. The reason is simple: you probably don’t your text to look slightly different depending on what computer it’s running on. For example, a Windows machine and Mac will have different versions of the Arial font. Unless you embed the font your are going to use, you do not know exactly what the font will look like to the user, and the differences can be very significant from the design perspective. Better to embed fonts, so it always looks the same.

So, you need to embed fonts. How do you do this? The most convenient way is to use a style declaration in the root of your project e.g.

       
<fx:Style>
        @font-face {
            font-family: "MyriadPro";
            src: url("assets/MyriadPro-Regular.otf");
            embedAsCFF: true;
	}
</fx:Style>

Notice that we embed as CFF, which provides benefits such as advanced anti-aliasing. But, you howl, Halo (the stuff starting with mx) doesn’t support this! Don’t worry, this is covered in a moment.

Before moving on, we’ll optimize this some by only embedding the characters from the font we are likely to use, which keeps file size down.

<fx:Style>
        @font-face {
            font-family: "MyriadPro";
            src: url("assets/MyriadPro-Regular.otf");
            embedAsCFF: true;
            unicode-range: U+0020-U+007D;
	}
</fx:Style>

So now everything looks the same on each system, perhaps we have been able to use an exotic font, and we haven’t blown our SWF file size up too much. But there’s a problem, Halo’s mx controls show blank spaces where text should be.

No worries, this is simple too. You just need to override the class of the text renderer object used by the Halo controls.

<fx:Style>
        @namespace mx "library://ns.adobe.com/flex/mx";
        @font-face {
            font-family: "MyriadPro";
            src: url("assets/MyriadPro-Regular.otf");
            embedAsCFF: true;
            unicode-range=U+0020-U+007D;
	}
        mx|Alert, mx|Button, mx|Form, mx|FormHeading, mx|FormItem, mx|FormItemLabel {
            font-family: MyriadPro;
            textFieldClass: ClassReference("mx.core.UIFTETextField");
	}
</fx:Style>

Nice! There you have your production font strategy. Of course, the more of those sorry old mx controls you are using, the longer your list of mx|,… will be 🙂