Free Connectionist Q-learning Java Framework - Tutorial on using the Q-learning framework

Main Menu

Home

Documentation

Tutorial on using the Q-learning framework

Applying the QCON Framework to your project requires just a few steps:

1. Download the jar file and add it as a library to your project.

In these boxes I'll refer to the QWanderBot example, which is available as an applet and whose source code can be also downloaded .

2. Create a class extending the framework class: pl.elsy.qconf.Perception.

See wanderbot.brain.MyPerception class.

3. In this class implement required methods:

public void updateInputValues() - here you will allow your agent to observe the world by setting input values for the Brain module. To set an input value, use the method setNextValue().
public double getReward() - here you can reward or punish the agent for the action he executed in previous step. For example - if he hits a wall, the method should return -1. If he reaches a goal position, return +1. Please, keep the returned value between -1 and +1. Otherwise the learning process may not converge.

In updateInputValues() WanderBot observes 9 points in front of him (marked with blue dots in the applet):
    protected void updateInputValues() {
        for (int d = 1; d <= 3; d++) {
            for (int a = -1; a <= 1; a++) {
                double xPerc = xPerc(d, a);
                double yPerc = yPerc(d, a);
                setNextValue(player.getWorld().pointInObstacle(xPerc,yPerc) ? 5: -5);
            }
        }
    }
The getReward() method is simplier - it returns 0.1 if the agent moves forward, -0.5 if a collision occurred and 0.0 otherwise:
    public double getReward() {
        if(player.movingForwardWell()) return 0.1;
        if(player.collides()) return -0.2;
        return 0.0;
    }

4. Think what actions the agent can perform. For each action create one separate class extending pl.elsy.qconf.Action.

Here we've got 3 classes: MoveForward, TurnLeft and TurnRight, all placed in the wanderbot.brain.actions package.

5. In these classes implement required method:

public int execute() - here you describe in the details what the agent should do when performing the action. The returned value is up to you. You can access it later by calling the Brain method: getExecutionResult().

Each Action class has its own execute() method. These are respectively:
    public int execute() { //MoveForward
        return player.moveForward(Player.STEP_SIZE);
    }
    public int execute() { //TurnLeft
        player.turn(-Player.TURNING_ANGLE);
        return 1;
    }
    public int execute() { //TurnRight
        player.turn(Player.TURNING_ANGLE);
        return 1;
    }

6. In you agent's class add a field - instance of pl.elsy.qconf.Brain. Initialize it in the agent's constructor with the following arguments:

an instance of class extending pl.elsy.qconf.Perception (you prepared it earlier)
an array of instances of classes extending pl.elsy.qconf.Action (also prepared earlier).

The Player constructor In our example looks like this:
    public Player(World world) {
        this.world = world;
        r = 10;
        Action actionArray[] = new Action[3];
        actionArray[MOVE_FORWARD] =    new MoveForward(this);
        actionArray[TURN_LEFT] =    new TurnLeft(this);
        actionArray[TURN_RIGHT] =    new TurnRight(this);
        perception = new MyPerception(this);
        brain = new Brain(perception, actionArray);
        brain.setAlpha(0.2); // the lines below only set specific Q-learning parameters instead of default ones
        brain.setGamma(0.9);
        brain.setLambda(0.8);
        brain.setUseBoltzmann(true);
        brain.setTemperature(0.02);
    }

7. In the main loop of the program, where the player decides what to do, include the following lines:

        ...
        myPerception.perceive();
        brain.count();
        brain.executeAction();
        ...

In our example it is almost the same. See Player.count() method.

8. That's all. Now, just check, whether everything goes well. Good luck!

If you have any questions or comments, please let me know , so I could improve the framework.

Comments

Only registered users can write comments.
Please login or register.

< Prev		Next >