Features need to know

As you start to use Sikuli, you will realize that there are a wide variety of things that you can do with it. This section will teach you all about the most commonly performed tasks and most commonly used features found in Sikuli.

The basics

Firstly, let's run through some of the basic capabilities found in Sikuli. These types of activities are the ones that you'll use every day with Sikuli.

Waiting is one of the most common tasks you'll find in a Sikuli script. Let's go to some of the other options for waiting, and how to use them. Sikuli provides two modes for waiting, one using the wait()method, which shows how to wait for something to appear. Sikuli also provides a mechanism to wait for something to disappear, which is invoked using waitVanish(). The following snippet is waiting for the title bar of a document to disappear:

wait() and waitVanish() both take an optional duration parameter. This can be number of seconds (the default is 5), or the global keyword FOREVER, which will wait until something happens.

Another important feature is typing and keyboard input. In many instances, it is simpler and faster to drive your application through the use of the keyboard.

The basic type() command is quite simple. You can just pass it a string.

If you need to provide modifier keys, Sikuli also makes this easy by providing a collection of constants to make them easy to use. Here's an example of using multiple modifier keys at the same time:

Here's a complete list of the available KeyModifiers:

Control	KeyModifier.CTRL
Shift	KeyModifier.SHIFT
Alt	KeyModifier.ALT
Meta	KeyModifier.META
Command	KeyModifier.CMD
Windows Key	KeyModifier.WIN

There are also lower-level functions for working with the keyboard. Here's a snippet of code, which shows how these work. This script types the letter a, and then uses the left arrow key and a carriage return to move it down a line (see keyupdown.sikuli).

You can even operate modifiers using keyDown and keyUp, which allows you to do more complicated operations when needed:

Clicking is the last introductory skill we need to cover. You can also specify a bunch of details about the search image.

Try clicking on one of the search images, and you will get the Pattern Settings dialog. Here's the one for Don't Save click action.

The File tab shows you a preview of the image, the path of the image file being used for matching, and the filename:

The Matching Preview tab shows you where your search is found on screen. The matches are indicated using a red transparent highlight. You can also adjust the similarity, which will allow you to match more or less exactly (by default Sikuli looks for a 70 percent match). If your script is clicking on places you don't expect, that often means you need to adjust the similarity, so that the script will only find the correctly matching items. You can use the Matching Preview tab to make sure it's going to select the items you expect. Though, sometimes the items on the screen you're interested in will be transient, and you'll need to use a process of trial and error to figure out the correct similarity value. If you've changed the similarity, this will be indicated in the editor by a number in the upper-right corner of the screenshot. For the Don't Save button, this is what it should look like:

The maximum number of matches limits the number of items that findAll() will locate. The default maximum value is 10.

Finally, we have the Target Offset tab. This lets you manage where on the screen you want to click. By default the target offset is set to the center of the image, but this can be changed. What this lets you do is to change the click position to someplace other than the center of the image. This is extremely useful when you want to click on something that won't be easy to find, say an arbitrary spot in a large blank area of the screen, or on a particular button which is similar to many others. What you can do instead is find a unique element on the screen, and use an offset to click on the one you want instead. Here, we've just set the offset a bit off to the right of the button. If the offset is set, it will be indicated by a little red cross inside the editor.

The offset marker will be positioned related to where the offset is set. So, if the offset was to the left, the cross would be on the left. In a large image, the offset can also appear where the target actually will be.

There are also convenience functions for performing the doubleClick() or rightClick() actions. Similar to the type command, these can take a modifier option. So, say you needed to press Command and rightClick() at the same time, you can do that as well.

You can get pretty far with Sikuli before you start getting into its more advanced features. Through the simple combination of starting up application, waiting for things to appear/vanish from the screen, being able to type, and being able to click on objects, you can produce a rich collection of interactions with applications. In the next section, we'll start introducing some of Sikuli's more advanced capabilities, which will let you handle more variation in your programs.

Finding things on the screen and how to use regions

Finding things on the screen and interacting with them is another common operation in Sikuli. We've already seen this to some extent with the click() and wait() commands. Behind the scenes, these are actually combined with a find operation. But, sometimes you need to look for things explicitly.

find() and findAll() let you find matches on the screen. Generally speaking, we won't use find very often, but it is handy for doing things like setting up regions to allow for optimization of your scripts. findAll() is used a lot when trying to operate on a bunch of similar items on the screen, and we'll discuss it further in a moment.

To show how to use find, we'll add a region to a script to limit the area of the screen where Sikuli is scanning when running our script.

So, let's discuss this for a moment. The new elements are the use of a find() call to get a reference to the title bar and the creation of a region based on the title bar's location. We also make some use of variables (titlebar = and r =), so we can re-use some of our collected data.

Another useful tidbit is the highlight() method of Region. This allows you to draw a box around a portion of the screen corresponding to a region. You can try this out in any of our scripts up to this point as well by callinghighlight() directly.

Notice that the entire screen is briefly marked with a red box showing the current region (the whole screen). All of the commands (wait(), find(), click(), and others) can be used on a variable containing a region (r) as it will restrict the search area, and help speed up your scripts. Setting up regions is also helpful if there are similar items found on the screen, but you only want to work with the ones in a particular place. Note that we're using the r region instead of calling click() and wait() directly.