Dominic Williams

Occasionally useful posts about RIAs, Web scale computing & miscellanea

Archive for the ‘Uncategorized’ Category

Red5 cabin fever – advanced scope and room management

with 24 comments

Red5 provides us with the abstraction of a hierarchy of scopes or “rooms”, to which clients can connect. In their most basic usage, scopes are simply used as rooms where for example shared objects used for chat applications or live media streams are located. However in the development of advanced MMOs and other applications, we want to push our usage of scopes a little further. This post aims to share some tips that can help those wishing to use Red5 scopes.

First a quick recap of the scope system. Red5 assigns each application a default scope, known as the “WebScope”. This scope exists whether or not anyone is connected to it (more on scope lifetime later). Once clients connect to scopes, or we otherwise create them, you get a tree structure.

WebScope
WebScope->Offices
WebScope->Offices->QueensPark
WebScope->Offices->Sausalito
WebScope->Seminars
WebScope->Seminars->Red5
etc

Now to the tips! Note: This post is not meant an exhaustive discussion of Red5 scopes. It simply offers up some useful pointers for advanced usage. Don’t forget to read the documentation 😉

1. The basics: Scope connection sequence

Flash/Flex clients connect to scopes using code such as:

var nc: NetConnection = new NetConnection();
// ... assign event handlers etc.

// connect to scope /Offices/QueensPark inside myApp
nc.connect("rtmp://someserver.com/myApp/Offices/QueensPark");

You will likely have created your Red5 application by extending the ApplicationAdapter class. When you override the connect() callback method of this class,  this will be called when clients connect. It is important to realize that it will be called for the entire scope hierarchy that the client is connecting to. So the following code:

@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    System.out.println("Connecting to: " + scope.getPath() + "/" + scope.getName() +
        " (" + scope.getContextPath() + ")");
}

Will produce this output in your Eclipse console:

Connecting to: /default/myApp ()
Connecting to: /default/myApp/Offices (/Offices)
Connecting to: /default/myApp/Offices/QueensPark (/Offices/QueensPark)

In the above output, first the client is connected to /default/myApp your WebScope, then down through successive parent scopes to the target leaf scope. At any stage, you can prevent connection by calling rejectClient(<REASON>), or throwing a ClientRejectedException(<REASON>) deep in your stack. The client will receive the reason for the rejection in their error handler e.g.

@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    return true;
}

2. Beyond the basics: Authenticate in connect()?

Those new to Red5, or those not having to build scalable applications, may be tempted to stop reading now. They might now think they know all they need to know to implement an authentication scheme e.g.

Client:

nc.connect("rtmp://someserver.com/myApp/Offices/QueensPark", "dominic", "abracadabra");

Server:

@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    if (!Accounts.authenticate(params[0], params[1])
        rejectClient("Invalid login details");
    return true;
}

Unfortunately there is a little more to it than that. First of all, you need to notice the innocuous synchronize statement inserted before the ApplicationAdapter.connect method.

If you think about it, this little statement will have the effect of serializing execution of the connect() callback, and thus serializing the process of connecting to your application. Therefore if, as is most likely, your authentication code involves a database lookup to retrieve the password for the user, if there are multiple clients trying to connect at once many will spend ages in a “connection” queue – database lookups take *time* so you don’t want to serialize operations!

The next thing you should therefore do, is remove the synchronized statements from your callbacks by deriving your application from MultiThreadedApplicationAdapter rather than ApplicationAdapter as is typical i.e.

@Override
public boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    if (!Accounts.authenticate(params[0], params[1])
        rejectClient("Invalid login details");
    return true;
}

But does that solve things??? Unfortunately not!

If for the sake of experimentation (1) change your connect() implementation to the following code (2) open a couple of Flex clients in your browser (3) make them connect simultaneously (4) have a look at the output in your Eclipse Console window… you will notice that the calls to connect() are *still* serialized.

@Override
public boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    System.out.println("connect() starting work " + (new Date()).toString());
    Thread.sleep(4000);
    System.out.println("connect() finishing work " + (new Date()).toString());
    return true;
}

The problem is that somewhere deep inside the Red5 architecture is a global lock which is serializing calls to connect() for you. The only way around this performance problem is to use lazy authentication, as described in the next section.

3. Lazy authentication on demand

Now it maybe that you have a simple application, where the user authenticates themselves when they connect, and thereafter connects to scopes and performs actions as an authenticated user. Fine.

But what if you want to make things a little more complicated, by having for example two scopes, one of which contains publicly available functionality, and the other which contains functionality only authenticated users can access e.g.
WebScope
WebScope->signup
WebScope->game
Note: I apologise if these examples get rather contrived, but they will help illustrate some points!

The solution is to add an authentication method to your WebScope. This is really easy to achieve, simply add a *public* method to the application class you derived from MultiThreadedApplicationAdapter e.g.

public boolean authenticate(String username, String password) {
    // do your stuff
    return true;
}

When the client receives a notification about successful connection, it can then call authenticate e.g.

    nc.call(
        "authenticate",
        new Responder(
            function() { game.proceed(); } // chain forwards!
            function(fault: Object) { Alert.show("Authentication failed\n"+ObjectUtil.toString(fault)); }),
        "dominic",
        "abracadabra"
    );

4. But I want to authenticate in my own scope, not WebScope!?!?

As you will have no doubt realized, there is a problem with this approach. In order to call authenticate(), your client needs to be connected to the default root scope (the WebScope). Furthermore, you need a way of preventing your client from connecting to protected scopes without the necessary privileges.

You might hope the solution is to (1) mark clients as authenticated using attributes, and (2) provide another method that effects dynamic server-side navigation to your scopes. So we now have inside our application class:

public boolean authenticate(String username, String password) {
    // do your stuff
    Red5.getLocalConnection().getClient().setAttribute("Authenticated", true);
    return true;
}
public boolean setScope(String[] path) {
    // change scope
}
@Override
public synchronized boolean connect(IConnection conn, IScope scope, Object[] params) {
    if (!super.connect(conn, scope, params))
        return false;
    if (!scope.getContextPath().startsWith("Offices")
        && !scope.getContextPath().startsWith("Seminars"))
        rejectClient("Requested server scope not recognized");
    if (scope.getContextPath().length != 0
        && conn.getClient().getAttribute("Authenticated") != true)
        rejectClient("Authentication required");
    return true;
}

5. Damn, I can’t make my setScope method work!

This method kind of works, but there are a bunch of gotchas related to dynamically changing scope, which I shall iterate to finish of this post.

1. You change a client’s scope, by joining it connection to that destination scope.
2. When you join the connection, this will cause Red5 to create a completely new Client object, even though the actual (remote) client is unchanged

Therefore, the following code will *not* work:

public boolean setScope(String[] path) {
    IScope destinationScope = getDestinationScope(path); // your own method
    IConnection conn = Red5.getConnectionLocal();
    IClient originalClient = conn.getClient();
    conn.connect(destinationScope);
    IClient newClient = conn.getClient();
    newClient.setAttribute("Authenticated", originalClient.getAttribute("Authenticated"));
}

The problem is that deep inside Red5 your call to conn.connect above will result in your overridden version of the connect application callback being called. Unfortunately, it will be called with a new client object that does not have the Authenticated attribute you set, and which will therefore cause your code to call rejectClient.

6. Forget authenticating in connect and all is sweet…

The solution is to take your overridden application connect callback method out of the authentication process completely. You should now check whether someone is allowed to connect to a scope inside setScope.

public boolean authenticate(String username, String password) {
    // do your stuff
    Red5.getLocalConnection().getClient().setAttribute("Authenticated", true);
    return true;
}
public boolean setScope(String[] path) {
    IConnection conn = Red5.getConnectionLocal();
    IClient client = conn.getClient();
    IScope destinationScope = getDestinationScope(path); // your own method

    // Now see whether the client is allowed to connect to the destination scope.
    if (destinationScope.getAttribute("RequiresAuthentication") == true) {
        if (client.getAttribute("Authenticated") != true)
            return false;

    // We got this far, so we must be allowed to connect
    conn.connect(destinationScope);

    // Carry attributes from original client object to new client object
    IClient newClient = conn.getClient();
    newClient.setAttribute("Authenticated", client.getAttribute("Authenticated"));
}

7. Miscellanea

If you’ve read this article and like the idea of setting up your own Scope structure and advanced authentication scheme, read on for just a couple of moments. These two tips could save you some more time!

Firstly, you may have used your application startup callback to initialize your scope structure. For example:

@Override
public boolean appStart(IScope app) {
    if (!super.appStart(app))
        return false;

    if (!app.createChildScope("signup"))
        return false;

    if (!app.createChildScope("game"))
        return false;

     IScope gameScope = app.getScope("game");
     gameScope.setAttribute("RequiresAuthentication");

    return true;
}

You find that the first time you connect a client to your scope, everything works fine, but the second time you connect your client to your scope, you get an error saying the scope doesn’t exist.

The reason is that by default, Red5 tears down scopes when the last client disconnects from them. If you wish to setup standard scopes, and have them permanently in memory, for example preserving state irrespective of whether clients are connected, you need to pin them there.

The simplest way to pin your standard scope is to make your application “listen” to events on them. For example:

public class MyApp extends MultiThreadedApplicationAdapter implements IEventListener {
...
@Override
public boolean appStart(IScope app) {
    if (!super.appStart(app))
        return false;

    // setup signup scope
    if (!app.createChildScope("signup"))
        return false;
    app.getScope("signup").addEventListener(this);

    // setup game scope
    if (!app.createChildScope("game"))
        return false;
    IScope gameScope = app.getScope("game");
    gameScope.setAttribute("RequiresAuthentication");
    gameScope.addEventListener(this);

    return true;
}
...
}

Finally, you might have realized from the above that once a client has connected to a scope, they will no longer have access to the setScope method we defined and will therefore be unable to dynamically change scope again!!

The solution here is to package your setScope function into a special handler that you will add to every scope. For example:

public class MyAppStandardMethods {
public boolean setScope(String[] path) {
    IConnection conn = Red5.getConnectionLocal();
    IClient client = conn.getClient();
    IScope destinationScope = getDestinationScope(path); // your own method

    // Now see whether the client is allowed to connect to the destination scope.
    if (destinationScope.getAttribute("RequiresAuthentication") == true) {
        if (client.getAttribute("Authenticated") != true)
            return false;

    // We got this far, so we must be allowed to connect
    conn.connect(destinationScope);

    // Carry attributes from original client object to new client object
    IClient newClient = conn.getClient();
    newClient.setAttribute("Authenticated", client.getAttribute("Authenticated"));
}
}

This handler can be added to every scope. For example our appStart might become:

public class MyApp
    extends MultiThreadedApplicationAdapter
    implements IEventListener {
...
@Override
public boolean appStart(IScope app) {
    if (!super.appStart(app))
        return false;

    // setup signup scope
    if (!app.createChildScope("signup"))
        return false;
    IScope signupScope = app.getScope("signup");
    signupScope.addEventListener(this);
    signupScope.registerServiceHandler("api", new MyAppStandardMethods());

    // setup game scope
    if (!app.createChildScope("game"))
        return false;
    IScope gameScope = app.getScope("game");
    gameScope.setAttribute("RequiresAuthentication");
    gameScope.addEventListener(this);
    gameScope.registerServiceHandler("api", new MyAppStandardMethods());

    return true;
}
...
}

If you have dynamically created scopes, you can also register MyAppStandardMethods as a service handler in the relevant application callback.

Now wherever your client is connected within your scopes tree it can move around any time it wants using setScope e.g.

    nc.call(
        "api.setScope",
        new Responder(
            function() { game.renderBattlefield(); } // chain forwards!
            function(fault: Object) { Alert.show("Changing scope failed\n"+ObjectUtil.toString(fault)); }),
        "battlefield",
    );

Hope that helps!

Written by dominicwilliams

March 9, 2010 at 5:07 pm

Posted in Authentication, Red5, Scopes, Uncategorized

Tagged with

Quick install HBase in “pseudo distributed” mode and connect from Java

with 10 comments

Platforms used: Ubuntu karmic, hadoop-0.20, hbase-0.20.3, Java client

This post will be useful for those wishing to setup HBase on a single server machine in pseudo distributed mode. The advantage of running the database in this mode is that it can then be accessed over the network, for example to allow a bunch of developers to start crafting Java code against it. It is also a nice stepping stone to fully distributed mode, either on your own servers, or somewhere like EC2.

On first reading of the HBase documentation, setting up pseudo distributed mode sounds very simple. The problem is that there a lot of gotchas, which can make life very difficult indeed. Consequently, many people follow a twisted journey to their final destination, and when they finally get there, aren’t sure which of the measures they took were needed, and which were not. This is reflected by a degree of misinformation on the Web, and I will try and present here a reasonably minimal way of getting up and running (that is not even to say that every step I take is absolutely necessary even, but I’ll mention where I’m not sure).

Step 1: Check your IP setup
I believe this is one of the main causes of the weirdness that can happen. So, if you’re on Ubuntu check your hosts file. If you see something like:
127.0.0.1 localhost
127.0.1.1 <server fqn> <server name, as in /etc/hostname>

get rid of the second line, and change to
127.0.0.1 locahost
<server ip> <server fqn> <server name, as in /etc/hostname>

e.g.
127.0.0.1 localhost
23.201.99.100 hbase.mycompany.com hbase

If you don’t do this, the region servers will resolve their addresses to 127.0.1.1. This information will be stored inside the ZooKeeper instance that HBase runs (the directory and lock manager used by the system to configure and synchronize a running HBase cluster). When manipulating remote HBase data, client code libraries actually connect to ZooKeeper to find the address of the region server maintaining the data. In this case, they will be given 127.0.1.1 which resolves to the client machine. duh!

Step 2: Install Hadoop Packages
Hadoop is quite a big subject – hell the book has over 500 pages. That’s why it is great that there is a company making pre-packaged distributions called cloudera. So my recommendation here is to go with those packages. Perform the following steps, but check the important notes before proceeding:
a/ If you are on Debian, you need to modify your Apt Repository so you can pickup the packages. In the instructions following, if you are running a recent Ubuntu distro like karmic, then configure your cloudera.list to pickup the packages for “jaunty-testing”. Make sure you choose hadoop-0.20 or better.
Instructions http://archive.cloudera.com/docs/_apt.html
b/ Install the packages setting up hadoop in standalone mode
Instructions http://archive.cloudera.com/docs/_installing_hadoop_standalone_mode.html
c/ Install the package the sets up the pseudo distributed configuration
Instructions http://archive.cloudera.com/docs/cdh2-pseudo-distributed.html
IMPORTANT NOTES
You should begin the above process with your system in a completely hadoop-free state to be sure the steps will work correctly. For example, if you have an entry for a hadoop user in your /etc/passwds file that is different to the one the config package wants to install, installation of the config package can fail. Furthermore, old items may have the wrong permissions which may cause later steps to fail. To find everything on your system you need to remove, do:

cd /
find -name "*hadoop*"

and

cd /etc
grep -R hadoop

Step 3: Prepare user “hadoop”
We are going to make it possible to login as user hadoop (or rather, do a sudo -i -u hadoop). This will make it possible to easily edit for example configuration files while keeping their owner as hadoop. We are also going to run HBase as user hadoop.
Change the following entry in /etc/passwd

hadoop:x:104:112:Hadoop User,,,:/var/run/hadoop-0.20:/bin/false

to

hadoop:x:104:112:Hadoop User,,,:/var/run/hadoop-0.20:/bin/bash

There is a lot of talk on the Web about setting up ssh for the hadoop user, so that hadoop can ssh to different nodes without specifying a password. I’m not sure that this is necessary any more, but the weight of recommendation (including here http://hadoop.apache.org/common/docs/current/quickstart.html) persuadesd me to do this anyway. So next:

# sudo -i -u hadoop
hadoop$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
hadoop$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Step 4: Configure hadoop
Open /etc/hadoop/conf/hadoop-env.sh and make sure your Java home is correctly set e.g.

export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.15

Step 5: Test hadoop
Checkout the Web-based admin interfaces e.g.
http://hbase.mycompany.com:50070/
and
http://hbase.mycompany.com:50030/

Step 6: Install the HBase package
You need to download and install the latest version from http://www.apache.org/dyn/closer.cgi/hadoop/hbase/. Proceed as root as follows:

# cd ~
# wget http://apache.mirror.anlx.net/hadoop/hbase/hbase-0.20.3/hbase-0.20.3.tar.gz
# tar -xzf hbase-0.20.3.tar.gz
# mv hbase-0.20.3 /usr/lib
# cd /etc/alternatives
# ln -s /usr/lib/hbase-0.20.3 hbase-lib
# cd /usr/lib
# ln -s /etc/alternatives/hbase-lib hbase
# chown -R hadoop:hadoop hbase

Step 7: Configure HBase
Now, login as hadoop and go to its conf directory

# sudo -i -u hadoop
hadoop$ cd /usr/lib/hbase/conf

a/ Then update the Java classpath in hbase-env.sh, just like you did for hadoop-env.sh
b/ Inside hbase-site.xml, configure hbase.roodir and hbase.master. The result should look something like below, notes following:

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:8020/hbase</value>
<description>The directory shared by region servers.
Should be fully-qualified to include the filesystem to use.
E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR
</description>
</property>

<property>
<name>hbase.master</name>
<value>23.201.99.100:60000</value>
<description>The host and port that the HBase master runs at.
</description>
</property>
</configuration>

NOTES
1/ hbase.rootdir must specify a host and port number exactly the same as specified by fs.default.name inside /etc/hadoop/conf/core-site.xml. Basically it tells HBase where to find the distributed file system.
2/ hbase.master specifies the interface that the HBase master, or rather the Zookeeper instance HBase will start (more later) will listen on. It must be externally addressable for clients to connect. This is a good point to double-check the IP setup step at the beginning of this post.

Step 8: Start up HBase
If you have not already started hadoop, then start it e.g. as described by cloudera:

for service in /etc/init.d/hadoop-0.20-*
do
sudo $service start
done

Next, start HBase as the hadoop user:

hadoop$ /usr/lib/hbase/bin/start-hbase.sh

Step 9: Check HBase is up and running
Open up the HBase Web UI e.g. http://hbase.mycompany.com:60010

Step 10: Open HBase shell, create a table and column family
You need to login to hbase, and create a table and column family that will be used by the Java client example:

me$ /usr/lib/hbase/bin/hbase shell
hbase(main):001:0> create "myLittleHBaseTable", "myLittleFamily"

Step 11: Create Java client project
Create your sample Java client project with code, as described for example at http://hadoop.apache.org/hbase/docs/r0.20.3/api/index.html.
Next, you need to add a special hbase-site.xml file to its classpath. It should specify the ZooKeeper quorum (the minimum running ZooKeeper instances, in this case, your hbase server). The client contacts ZooKeeper to find out where the master, region servers etc are (ZooKeeper acts as a directory for clients, but it also acts a definitive description of the HBase cluster and synchronization system for its own nodes). Based upon the foregoing, the contents will look something like:

<configuration>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hbase.mycompany.com</value>
<description>The host and port that the HBase master runs at.
A value of 'local' runs the master and a regionserver in
a single process.
</description>
</property>
</configuration>

Step 12: Yeeeha. Build your Java client, and debug
Watch your e.g. NetBeans Ouput window for a trace of what is hopefully happening… welcome to the world of HBase

Written by dominicwilliams

January 28, 2010 at 12:21 pm