Is there a way to setup continous integration for apex tests?
There are a couple of decent Dreamforce presentations here: Team Development: Possible, Probable, and Painless and Continuous Integration in the Cloud.
We ran into some issues with this in practice and there was no way to get true automation (i.e., set it and forget it). We were also setting it up with Selenium.
Here were the issues that I remember.
Some features aren't supported in the metadata API and cannot be moved via the ant migration. If you have any unit tests that work with those features, you have to manually work on your CI org.
Deletions are harder to maintain. You have to manually update and apply a destructiveChanges.xml file or replicate the deletion in the CI org.
We ran into a situation where some metadata XML files had 'invalid' data in them. The suggested solution was to build a post checkout script that manipulates the offending XMLs into valid XMLs. Not ideal.
On projects, we wanted to track just our changes and push out just our changes in source control. In theory, this would allow much easier rebaselining. This would have required more manual maintenance of XML files (e.g., 2 new fields added on Account and only want to push those 2 fields not all (*) fields).
My conclusion is that it is worth doing if you can get it set up, but if you are working on shorter term projects and don't have a decent amount of time budgeted in for it, it probably isn't worth setting up.
Although it isn't CI, check out http://developer.force.com/cookbook/recipe/automated-unit-test-execution. You could set it up to run every hour or something like that.
We're trying to set up the same, and while we still have to get some configurations right, I do think we have gotten the overall setup.
Using github plugins to jenkins you can poll for repository changes. We're using ssh keys between jenkins and github as authentication.
Jenkins uses ANT, and so does the force.com migration tool, you can use the deploy xml file you'd write for the migration tool to run with ant from jenkins, this can include that all tests need to be ran.
I believe this is the setup used by quite some teams already, yet no one seems to have written a full description on how to set it up publicly (at least, I've not found one). I may give it a try once we have our system fully running
I'm doing a presentation at Dreamforce '12 on this exact subject: Team Development and Release Management for ISVs
In the session I'll outline going through a complete CI life-cycle (develop, version control, build, package, deploy to AppExchange).
This session is mostly geared to AppExchange releases but could easily be a good building block for anyone whom wants to do CI. I'm hoping to get the Github repo up which will include a link to the Amazon AMI preconfigured like the one we use to manage multiple devs and builds with Jenkins at Bracket Labs