Sikuli - One of the first tools I researched for Mobile Testing
When I started researching tools and approaches for mobile testing about 4 months back, the first tool I came across and had lot of discussions was Sikuli.
A group of really smart students from MIT, built this GUI automation tool based on a Computer vision algorithm. So basically you can automate anything which has a GUI interface immaterial of which OS version it is, which application it is etc.
COMPONENTS INVOLVED-
1. Sikuli IDE - really intuitive
2. Sikuli Scripts - A visual scripting API based on Jython.
3. There is also a command line interface present for this to do manipulation via the command prompt.
INSTALLATION-
It was really easy, piece of cake. Seriously!!!!! As long as you have the basic components. More information can be found here
http://sikuli.org/download.shtml
MY RESEARCH-
I downloaded the tool and started playing around with it. I found the following basic things which the tools does.
1. This is based on automation using screenshots. Now what do I mean by "screenshots" (I know it is confusing right). To make it more easier I will compare this with "MS Paint". So, in "Paint" application, we can select a picture or portion of the picture and cut it using a snapshot tool. Similarly, in Sikuli you have an option called "Take Screenshot" and you can select any GUI component you want to use in the automation.
For example - Say, I want to click on the "Chrome" browser icon on my desktop. What I do is, click on "Take Screenshot" icon in the IDE and then select the Chrome icon on the desktop. This then appears in the IDE automatically. As simple as that.
2. You can programatically do the GUI automation using Jython. So, once the necessary screenshots are taken you can manipulate them using Jython and give different conditions and what not.
I like this feature as it gives more control to the user and you can insert conditions and do any kind of manipulation you want.
Check out the screenshot of the IDE below and now everything will make sense :-)
I was playing around with the tool and thought about the following scenarios-
a. How can I do automation using this just on the desktop for different test cases
b. How can I used it in mobile testing.
Let me discuss what I did one by one-
a. How can I do automation using this just on the desktop for different test cases
There is no use in blindly using a tool, for me everything should have a charter/a goal and I don't want to deviate from the charter. This helps to make your research more focused and prevents you from going into a rabbit hole.
These were my charters-
1. Open "Google Chrome" from my desktop and enter the URL "www.google.com".
Rationale behind this: I just wanted to see how easy it is to automate and see how the tool works.
2. Do a "Click Test" to simulate number of click around the edges of a button and finally click the button.
Rationale behind this: Quite often in the mobile space, users come across rendering issues. A particular
web page may look good on an iPhone but on an iPad the page is extended or a button gets moved to the side etc.
So, if I can have a test to simulate a number of clicks in and around the edges of a button, if the button has moved my test would detect it and immediately trigger an event when this happens, letting the tester know "Hey dude your button has moved".
3. Do a "Position Change Check" test for a button.
Rationale behind this: Pretty much the same goal as the 2nd charter, but my test is designed differently to achieve the same goal.
4. Do GUI automation on my company website and note down Pros/Cons.
Rationale behind this: This was my primary job :-), So, I have to do it even if I don't like it.
5. Do GUI automation using an iPhone emulator (which comes as part of XCode)
Rationale behind this: Wanted to see how the tool works with emulators.
6.. Do GUI automation on mobile devices - Android
Rationale behind this: This was my ultimate goal.
7. Do GUI automation on mobile devices - iPhone
Rational behind this: Refer 6th charter above
The above were my charters and I started my research in a focused manner, to prevent any deviation from my goals.
My findings-
1. Open "Google Chrome" from my desktop and enter the URL "www.google.com".
This is was pretty simple. I took a screenshot of the Google Chrome desktop on my browser and automatically got added to the IDE.
Then I found that there are so many in built functions already present as part of the IDE to mimic commonly used user actions like type(), paste(), click(), doubleClick(), dragDrop() etc.
You just double click on them on the left side panel and they get added to the script window. It is as easy as that.
So, I used the type() function entered the URL and I was done. First Success!!!!!
2. Do a "Click Test" to simulate number of clicks around the edges of a button and finally click the button.
I know how to code atleast to a basic level (I did my Undergrad and Masters in IT related domains, so I ought to know atleast some amount of coding/programming languages). With that being said, I started off with an obstacle of not knowing Python (remember I said this is based on Jython).
Thanks to Tim Berners Lee (giving us World Wide Web), this obstacle was easily overcome through the numerous websites which gave me the syntaxes of the language.
As I was saying....
I created a sample HTML with a Firstname and Lastname field (this was used for my testing) and I wrote the following code to simulate user clicks in Jython in the sikuli IDE
code:
doubleClick("on the HTML file on my desktop - This was a screenshot)
//my HTML file was as shown below
//Then I wrote the below code
It worked and it was awesome to see the power of this tool. You run the program by clicking on the "Run" or the "Run in slow motion" button. I liked the latter option as it highlights with red spirals where it is clicking and this gives you more information about the clicks and can ensure your code is working properly.
Second Success!!!!!
3. Do a "Position Change Check" test for a button.
For this test I had to simulate a button's position being changed. So, what I did was created 2 HTML files - 1) The html page you see above 2) the same HTML page with the button moved to the right a little bit.
Then I get the x coordinates of the button from both the files and do a compare to detect a position change.
So in real time, this would be just one piece of code running on just one HTML page and if the page is rendered improperly when opened in Kindle Fire or iPad i.e if the button was moved a bit due to a rendering issue my code should catch it.
Code below:
This was an Amazing Success. Now, I found that the tool is just not a random tool and it is pretty useful. One thing to note here is, you can modify the tolerance level of the images you capture.
So, if you double click on the image, you can set a level called "tolerance level". So basically you are telling Sikuli, "Hey, recognize this image and it is OK if the image is slightly blurred or you can tell Sikuli, I want you to click on an image which looks exactly like this to the pixel level".
So, it does pixel level image validation. (Awesome right!!!!)
Third Success!!!!
4. Do GUI automation on my company website and note down Pros/Cons.
I wrote automation scripts to go through a happy path flow through different options in my company website and the tool did a pretty decent job of recognizing "images" (see what I did here, I did not use the word "Objects", because for Sikuli it does not care what a particular entity is, it consider everything as an image and can match it to the pixel level. So, unlike say QTP - Quick Test Professional, you cannot do manipulation at the Object level)
There were some occasions where I faced a problem where the same script worked a majority of times but sometimes the same image used in my script wasn't recognized and had to re-run the script or recapture the image.
But overall it was OK. Not bad
I cannot give code or screenshots here as those things are Company Secrets!!!! (Just kiddin, I just did not want to reveal all the details here as my other charters which I have explained in this blog post, would have given sufficient amount of information in case you get inspired by my research and want to start working with the tool :-))
Fourth Success!!!!
Now coming to the main focus of my research and the reason why I started this research
b. How can I use it in mobile testing.
My 5th, 6th and 7th charters cover this point
5. Do GUI automation using an iPhone emulator (which comes as part of XCode)
The tool did work properly for the iPhone emulator. The only problem I faced was simulating scrolling on the phones Up/Down/Left/Right scrolls. For this I did a lot of manipulation by detecting the coordinates of the left corner of the simulator, then the right corner of the simulator then the bottom right and goes on and on.
It was messy!!!! so, except for the scrolling aspect, everything else was good
Fifth Success!!!
I will cover 6th and the 7th charter together as this is where I failed :-(. 6th and 7th Failure!!!!!!
6 and 7th Charter- Do GUI automation on mobile devices - Android and iPhone.
I found that, in order to run your tests directly on the mobile device you either use a simulator or VNC.
For android, there is a VNC viewer and you can use that to run the tests on the mobile device but it is VERY slow. Check this link out. You can try it Yourself!!!
http://i-miss-erin.blogspot.com/2010/01/automated-test-in-android-by-sikuli.html
On the iPhone, there is no VNC viewer or a remote client like this unless you have JAILBREAKED your iPhone. As I was using my company devices and my phone. This was not an option.
SUMMARY-
Overall, I found this tool really interesting and had good potential. In the world where there are no standardized tools (I generally don't like this word "standardized" as they often for some reason people associate it with QTP) for doing mobile testing, we need to start exploring tools like this and see which feature of the tool would be useful for your project purposes.
I had lot of fun doing this research and hope atleast some of the information I gave is useful to folks who are doing research like me.
Hit me up with questions if you need more info or share your experiences here :-) Always, glad to help.
More stuff and fun research on tool/approaches to come........