I'm just curious if anyone here is using Selenium for testing their site.

For those who don't know, it is a web testing framework that can remote control a browser to make it navigate, fill out forms, and validate a web site. It works with practically every browser on most operating systems.

It also has a Firefox plug-in that can record and playback tests. It's a major time saver to use Firefox (with the Selenium IDE) to record a test, export it into your preferred programming language, and add/edit from there.

Quick question as it wasn't answered on their homepage either: What kind of testing does it do? Visual layout, valid XHTML, XSS vulnerability, SQL Injection attacks, ... ?

It can be used like a unit testing tool (though at that level it's really integration testing).
You can validate that data exists on a page, navigate pages, fill out forms, verify links, etc.
Testing layout can be done but, its a bit tricky. I've written code to take screen-shots after each navigation. That way my testers can quick scan the screen-shots (much faster than click and wait).

It's a web tester that drives a browser so, if you want to try SQL injection, you're going to be setting fields in form fields.

The easiest way to try it is with the Selenium IDE (firefox plugin). With it you can record/playback navigation, form submits, etc. You can also add asserts.
Though, the best way to use it, is as part of an automated test framework.
In my project, whenever a code changed in version control, a new build of the project is kicked off. The last part of the automated build is deploying to a test server and running the selenium tests.

I've used it in the paste, but I've been a bit lazy on the test part lately.
I'm learning Django at the moment, so it will stay a bit behind of my priorities for a while.

But it's a really great tool!

We use to use Selenium all the time.. Its strength is in "Functional" testing of web sites and services.

It has some weaknesses related to javascript execution though.. we had all sorts of issues getting it to play nice with AJAX enabled pages and development controls.

Visual Studio Team Test is a far superior product in my opinion, plus the designer and reports are way better.. But its not free, so obviously its not for everyone.

Originally Posted by JeremyMiller

Quick question as it wasn't answered on their homepage either: What kind of testing does it do? Visual layout, valid XHTML, XSS vulnerability, SQL Injection attacks, ... ?

Not really ... if you can think of how to test your site against a particular vulnerability, you'll know how to code against an attack of that sort. This is more for testing "does this block of code do what I expect it to, having just written it?"

I don't use Selenium. I've actually been building something very much like it myself, and I'm surprised to see how similar the macro language is. I guess I shouldn't be.

Originally Posted by RabidSniper

We use to use Selenium all the time.. Its strength is in "Functional" testing of web sites and services.

It has some weaknesses related to javascript execution though.. we had all sorts of issues getting it to play nice with AJAX enabled pages and development controls.

Visual Studio Team Test is a far superior product in my opinion, plus the designer and reports are way better.. But its not free, so obviously its not for everyone.

You are right, it is "functional testing". I always confuse the terms.

My main reason for it over VSTT is the need to run on multiple platforms and to be able to drive the tests automatically (in a continuous integration system) with Maven.

I have it run the tests by connecting to various flavors (and languages) of windows (2K,XP,etc) as well as a Mac and Linux box (Firefox, Safari, Konqueror).

I'm surprised that so few developers here are running automated tests...

BTW, how do you like working with VSTT?

Thanks for the clarifications. If I understand correctly, you have to go in and set the thing up to do a test and then it's advantage is that you can re-run the test as much as you want, in a variety of browsers. If that's the case, I think I'll stick to just testing things myself as I usually just code something and go test it. Success and I move on, fail and it's back to the code. I don't understand why I would want to test the same code in multiple browsers since the PHP will work the same way regardless. I only check multiple browsers (and then, usually only IE and Firefox) for CSS verification which this doesn't do (and probably can't as it'd require a human to see if something weren't as visually intended).

Hope I'm not off in left field.

you can re-run the test as much as you want, in a variety of browsers.

Hmmm, not sure. Don't selenium as a browser plugin exists just for Firefox ?
Edit: I just looked up, and seen that no, it can do any browser from a specific application. Cool !

I think I'll stick to just testing things myself as I usually just code something and go test it.

Selenium is oriented to the interface, and it's used more intensively on the javascript side.
The advantage of using it, is that you define your test case once, and you can replay anytime later.
Particularly, if you re-write a module of your application, you can re-run the test suite that focus on this particular module to see if all the tests are still valid.
If not, you know that you might have a regression, or you have to update your tests to follow your UI evolution.
It's the "test driven development" idiom applied to the web.

CSS verification which this doesn't do (and probably can't as it'd require a human to see if something weren't as visually intended).

Not sure, I have heard of robots that where comparing the color variations in screenshots to evaluate if the rendering of the interface had changed.
But I don't think Selenium can do that, no.

I think testing is becoming more and more integrated into the typical development process.

To the person who said you were going to keep testing it as you have been. I mean, thats obviously up to you.. But the thing I like best about automated testing is exactly that.. its automated.. so if you are clicking around in your app that you just made and everytime you change something you have to go back and click everything again.. just to do proper regression testing.. well I mean, if that floats your boat.. have at it! If I have an application or bit of functionality that depends on things that could change.. (like controls or templates or whatever).. automated regression testing on hundreds of possible "clicks" is a life saver..

I love VSTT for two main reasons... 1. It creates GREAT Functional and Load tests.... those tests are part of a project SOLUTION.. so in they essentially become first class citizens in a software project. 2. It plays really nice with just about everything weve thrown at it.. and it integrates in MSBUILD quite nicely.. (CruiseControl.NET or NANT I think both offer support for VSTest harness projects..)

Also.. for HTML/JAVASCRIPT testing.. an automated test tool like Selenium or VStest helps find browser based javascript issues and other things like missing images, oversized script and graphic files.. slowly loading pages.. etc..

really neat Idea and more people should practice it.

Um, I'm glad that you like to have your testing automated, but don't think that my method of testing things is inferior or more time consuming than yours.

I structure my code in such a way that a change of template changes zero functionality. It's called MVC and I employ it on all of my stuff.

Fact of the matter is that there's nothing which can substitute for a human eye viewing the results of inputting data and nothing substitutes for data entered by an "average" user.

I write many small applications and a number of very sophisticated applications which span 10's of 1,000's of lines of code. Testing is part of my development, of course, and I am open to new ideas, but a system which makes me do the test and then records it for me to replay doesn't seem worth the effort.

Perhaps I'm just an efficient tester of efficient code?

Something else to consider is that my code is my code. No one else is allowed to so much as touch it. Some of the people on here work on teams. In a team environment, you never know what the dope on your right will add in which will affect your code, so I can see greater value in such an application for those systems.

Final point I'd like to make, to completely defend myself, is that NO amount of testing will be sufficient to have 100% guarantee of no bugs. It can just help reduce the likelihood of a bugs remaining in the final product. As an example, I once had a guy write me an email where he said that my code had worked for him for the last 2 years just fine, but had suddenly broke. The reason? He changed the XML data being input from what was assumed in the original specification.

Originally Posted by RabidSniper

Also.. for HTML/JAVASCRIPT testing.. an automated test tool like Selenium or VStest helps find browser based javascript issues and other things like missing images, oversized script and graphic files.. slowly loading pages.. etc..

Wanted to add that a 3 second glance at firebug will do the trick for this.

Something else to consider is that my code is my code. No one else is allowed to so much as touch it.

That might be the difference.
Where I work, code is collectively owned AND assumed.
As we are a team, anyone of the team should be able to go in the code to fix it if it's needed, like during a weekend, where there is a picket.

In that case, agile development method, and automated testing are a must.
Otherwise, the code is almost locked on 1 person.
If that person is in holidays, have an accident or such, what to do ?
Rewrite the buggy part because it will be faster than going through an unknown code to find the bug, or wait until that person comes back ?

What you described Jeremy, is feasible in a small team of 5~10 persons.
When the team is around 100 persons, you cannot work that way anymore.
You have to enforce rules about how the code must be structured, how the objects must be designed, and put the road map on the table first.

Otherwise, it would be like working an anarchistic hell.
And even if I feel myself close to the Anarchistic ideology (not iconography though), I would not like to work in an environment that much chaotic.

Just to illustrate that, yesterday, I have refused an extension a Front-end developer asked me to implement in the DB, because even if it was shortening his development time, it was going against all the way the DB stored procedures are designed.
Efficiency and speed are not the 1st qualities to look for, conformance and standardization are.

Well I never meant to suggest your way was inferior..

In fact if your doing MVC.. then my way is probably inferior.. lol! Functional tests are not 100% accurate you are right.. they never give you ALL the results you need.. for sure.. but they help in a lot of areas..

Oh yeah.. i was sold on FireBug a long time ago.. thanks though! Love that tool.. for all sorts of reasons! but the VSTT will do it for entire user scenerios(page a, eneter something, page b etc etc etc).. not just one page at a time. Not to mention, most of the time Im not even running the test.. my Build automation server does it for me.. I just wait for the Green light, red light condition of the entire build.

Actually.. Id be curious to know.. Jeremy.. since your on the MVC.. do you practice TDD when you write code? Do you use mock objects and the like to simulate data models etc?

Ive been thinking about trying the whole MVC thing out.. I can see the TESTING advantages.. because the PageController model in .NET is kinda hard to test web forms with.. But there are also several things Id have to relearn and sacrifice.. so jsut wanted to get your take on it.

PHP will be the same but, what about browser bugs and differences? Browsers suck
Some pages may work fine in one browser and not in another. Especially when you add javascript and ajax.

Testing manually can be tedious and error prone. If I have a data driven site, testers may need to make hundreds of navigations with many validations on each page. That can take a human many hours to perform. Why not go get some coffee and let the lazy computer do the work.

It is true that the machine can't test everything. But, why not let it do as much as possible/practical?

No amount of testing means that code is 100% bug free. But, when you find bugs, you can create a test to reproduce the bug. Then add to the automated suite. Then you can prevent the bug from showing its face again. This gives testers time to look for new bugs instead of verifying that old ones have not resurfaced.

Call me lazy but, if I can get the machine to do the boring stuff, then I get to spend more time of the fun stuff.

Originally Posted by RabidSniper

Actually.. Id be curious to know.. Jeremy.. since your on the MVC.. do you practice TDD when you write code? Do you use mock objects and the like to simulate data models etc?

Ive been thinking about trying the whole MVC thing out.. I can see the TESTING advantages.. because the PageController model in .NET is kinda hard to test web forms with.. But there are also several things Id have to relearn and sacrifice.. so jsut wanted to get your take on it.

Um, I'm lost here. What's "TDD"? I don't use mock objects, though occasionally I stub stuff that I plan on filling in later.

My steps to development are:

1) Interview the client extensively and relevant employees. Basically, the closer to a full systems analysis, the better.

2) Model the database and normalize to 3rd normal form.

3) Break 3rd normal form from step 2 where such action will substantially decrease joins and compose stored procedures to ensure data integrity. Transactions are fairly new to me (my degree is in math w/ a minor in CIS, with most of my programming self taught), but I am researching them now for implementation at this stage.

4) Model the structure of the files on the system. Actually, I have a rubber stamp here that I tweak. It's this which keeps the views from the controllers and completes the model.

5) Fill it all in with PHP.

Now, that's assuming, of course, a ground-up creation. If a pre-packaged item out there does the job, I'll start with that and tweak as necessary.

I'm interested in further elucidation on the subjects you mentioned, however, as I'm always trying to improve my skills.

Originally Posted by willcode4beer

Your PHP will be the same but, what about browser bugs and differences?

Some pages may work fine in one browser and not in another. Especially when you add javascript and ajax.

I have packaged up my ajax routine which I've tested on all the browsers I care to support, so it stays functioning just fine (though IE 8 has broken it, so I need to change that -- perhaps it was the page transitions that were broken, however, as the data is transferred, but I digress...)

My javascript usually works just fine across browsers as I try to make it as close to 100% DOM compliant as possible.

Originally Posted by willcode4beer

Testing manually can be tedious and error prone. If I have a data driven site, testers may need to make hundreds of navigations with many validations on each page. That can take a human many hours to perform. Why not go get some coffee and let the machine do the work.

I'm not saying that your method is wrong in the slightest. I'm just saying that for me, it takes more work.

If I have 100 pages, I don't need to test them all if they contain static content as I have the navigation and display routines packaged up into objects, so if 1 works, there's a very high probability that they all work.

Now, if all 100 are dynamic, then yes, I do have to check them all, but I also have to code them all.

Usually as I'm coding, I check the page as I build. I'm not good enough to write out a 500 line script without testing first. Well, most of the time. I have done it and am delighted when they all work as intended, but that's rare.

Originally Posted by JeremyMiller

so if 1 works, there's a very high probability that they all work.

Now if I could convince my boss that a "very high probability" was good enough.... especially after making changes.

Fortunately for me, we have a QA dept that does the testing. Granted, developers still have to do unit tests (jUnit for me) and make sure things work.

QA is checking the boundary conditions, weird data from the CMS, strange user behavior, etc.

Well, now, I'm not talking about me being stupid, but if we have code like

PHP Code:

<?php
global $lang;

$page_name = preg_replace('/[^a-z0-9_]/i','',$_GET['page']);

if (strlen($page_name) > 0 && file_exists($file_location.'.php')) {
  require_once($file_location.'.php');
} else {
  $page_body = $lang['page does not exist'];
}
?>

And a driver which includes the $lang array, the code above, and then includes a template script which echoes out the template and $page_body in the right spot, which is populated in the required file or via the error message shown above, how many pages would you say I'd have to check to have a "very high probability" of success that this script will correctly read future pages? Now, the first thing someone will come back with is "How do you know that the included files aren't buggy?" That would be done when creating a script to compose these pages and by testing it with funky data, we should be good to go.

It really depends on the nature of the project. I doubt anyone on this forum would take the time to use any of their fancy tools on a simple 5 page site. If I had a 1000 page site of slightly-sophisticated code, I wouldn't be the only tester I would use either. The principle I'm following is to use the tool which is most effective in the given context. But, context matters.

Well that just goes to show that the world of "testing" is so vast.. there isnt just one way, and in some cases you need several types of validation..

TDD is Test Driven Development.. the idea being that if you are going to write a function or some sort of algorithm, that you first write a test that will expect the proper value in return.. (or some condition or another). and then you write code to make your test "pass". I only ask because from what Ive heard, read and seen the MVC pattern lends itself to TDD very effectively on the web because of a forced separation of business logic and presentation/form.

I can see though, that the type of scenario you describe that you would need to test, is usually not the same type of thing that I test..