Steven Cherry’s Blog

Random ranting and musing

Interesting site about open source continuous integration

Just stumbled across this site that lists a bunch of open source continuous integration products. It’s interesting because we use the Zed Builds And Bugs Continuous Integration Server that is advertised on the site. But everything else on the site appears to be completely open source. Good information, and a useful link.

December 1, 2009 Posted by | Uncategorized | , , | Leave a comment

Automating Eclipse Headless Builds with Zed Builds and Bugs

I was reading through the Zed Documentation and found a nice guide on how to use Zed to implement Eclipse headless builds using Zed. We’ve been using Eclipse since the start of the Shadow Studio, and this looked like a nice solution since we also use Zed to manage our automated builds. So, I decided to give it a shot.

Our application consists of roughly 27 Eclipse plugins, and since this was going to be a change to our existing config/build system I decided to start with a small subset of our application to test the theory with. We have a new Web Services Consumption component that is under development and looked to be a perfect candidate to run the test with.

Some key points for the application that made it a good fit:

  • It was new, so changing things wouldn’t break any existing builds
  • It was nicely self-contained with its own feature definition, and just a few dependencies on the main plugin that defines our product
  • It is not enormous – only about 11 plugins in total, so very manageable

With this small project we easily met the recommended prerequisites from Hericus, so I started to proceed.

First I updated our existing source tag and checkout to include the WSC modules:

svn checkout file:///var/svn/studio/eclipse/tags/${BUILD_TAG}/com.shadowstudio.wsc
svn checkout file:///var/svn/studio/eclipse/tags/${BUILD_TAG}/com.shadowstudio.wsc.doc
...

Next I added the following steps to our build definition:

Create genbuildxml file. Following the guidelines this is a simple Ant build XML file that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<project name="buildBuildFiles" default="all" basedir=".">
        <target name="all">
                <eclipse.buildScript elements="feature@com.shadowstudio.wsc.feature"
                        buildDirectory="${WORKDIR}"
                        baseLocation="c:/eclipse34/"
                />
        </target>
</project>

The ability to use the Zed replacement variables like “${WORKDIR}” comes in very handy for creating files on the fly like this.

Generate build.xml files. Now we use the above created build file to use the eclipse Ant task to automatically create the required build.xml files for the feature. Here’s the command:

c:\jdk1.5.0_05\bin\java -cp c:\eclipse34\plugins\org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar org.eclipse.core.launcher.Main -application org.eclipse.ant.core.antRunner -Dws=win32 -v -file ${WORKDIR}/genbuildxml.xml

Looking at it, I really should have defined an build environment variable for the location of the eclipse launcher so that when (not if!) it changes we only have to do it in one place. Note to self – do this. TODO.

Fix autogenerated defaults. The eclipse ant tasks that create the build.xml files have some defaults that they inject into the build.xml files that I don’t like. It’s trivial with Zed to go through these and update them to what I want them to be:

  • Change javacFailOnError from “false” to “true” – I like builds to fail when the compiles fail.
  • Change javacSource from “1.3” to “1.5” – we require a 1.5 JDK for our product.
  • Change javacTarget from “1.2” to “1.5” – same reason.
  • Also look for our version number eye-catcher and update it to the correct setting using ${BUILD_FULL_VERSION}

Run the feature build. At this point, I’m ready to run the feature build, so I added a command step with the following:

c:\jdk1.5.0_05\bin\java -cp c:\eclipse34\plugins\org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar org.eclipse.core.launcher.Main -application org.eclipse.ant.core.antRunner -Dws=win32  -file ${WORKDIR}/com.shadowstudio.wsc.feature/build.xml clean build.jars

Again, here’s where that environment variable would have come in handy so I don’t have the eclipse launcher hard-coded in each step. TODO.

Build the WSC Package. Now, this really could have been done in the step above, but there are a hand-full of things that we add in after eclipse has built all of the jars but before the final package is assembled. These typically have to do with native libraries or DLL’s that we need to add to the package that eclipse isn’t aware of. So the final command is:

c:\jdk1.5.0_05\bin\java -cp c:\eclipse34\plugins\org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar org.eclipse.core.launcher.Main -application org.eclipse.ant.core.antRunner -file ${WORKDIR}/com.shadowstudio.wsc.feature/build.xml zip.distribution

And with that, we have a nice WSC feature jar file all packaged up and ready for inclusion in our final packaging steps.

Following the guide from the Hericus documentation made things really a snap. I just need to circle back and get that eclipse launcher reference out of each of the commands and use a single environment variable to make it easy to maintain.

May 15, 2009 Posted by | Uncategorized | 1 Comment

Programmatically creating DTP Driver and Profile definitions

This took me too long to figure out, and all of my google searches turned up mostly empty.  Thanks to Brian Fitzpatrick‘s posts on DTP, I was able to put enough pieces together to accomplish what I needed to do.  Since there was nothing else that I found directly, I thought I’d share my efforts here so that others may benefit from my exploration and not have to go over the same ground I’ve been through already.

The Goal: Create a DTP driver definition and connection profile completely within code… No GUI’s allowed!

The Rationale: Our code needed to automatically create certain DTP driver definitions and connection profiles for the user so that we could guarantee that they existed, and so that the user wouldn’t have to walk through the (somewhat) annoying DTP wizards to get this done.  

Actually the only major annoyance of the DTP wizards is that we would have had to ask the user to find the appropriate jar files manually.  We ship all of the jar files required, but they’re not exactly in obvious places.  And… they move😮  Every time our software is updated, the list of plugins that we supply to Eclipse changes, so we need to programmatically manage this for the user so that they just don’t need to care.

Step one: Ensure that you have DTP installed in your environment and add the datatools plugins as dependencies to your project.  In our plugin.xml editor, that meant adding the following dependencies:

  • org.eclipse.datatools.connectivity
  • org.eclipse.datatools.connectivity.apache.derby
  • org.eclipse.datatools.connectivity.apache.derby.dbdefinition
  • org.eclipse.datatools.connectivity.apache.derby.ui
  • org.eclipse.datatools.connectivity.db.generic
  • org.eclipse.datatools.connectivity.db.generic.ui
  • org.eclipse.datatools.connectivity.dbdefinition.genericJDBC

Step two: Import the following into your class:

import org.eclipse.datatools.connectivity.IConnectionProfile;
import org.eclipse.datatools.connectivity.ProfileManager;
import org.eclipse.datatools.connectivity.drivers.DriverInstance;
import org.eclipse.datatools.connectivity.drivers.DriverManager;
import org.eclipse.datatools.connectivity.drivers.IDriverMgmtConstants;
import org.eclipse.datatools.connectivity.drivers.IPropertySet;
import org.eclipse.datatools.connectivity.drivers.PropertySetImpl;
import org.eclipse.datatools.connectivity.drivers.jdbc.IJDBCConnectionProfileConstants;

Step three: Use the GUI!

I know, I know, I said “No GUI’s allowed”.  But that’s just for the end-result.  We’re allowed to use a GUI during development to see what happens under the covers:-)

So, go and create a Driver definition and a connection profile that work.  Follow the DTP Getting Started as a good reference if you haven’t already been through this.

Step four: Look under the covers

Now you have at least one driver definition, and one connection profile.  Add some code to walk them and output their settings so that you’ll know how to recreate them as required:

To list the driver definitions DTP knows about do this:

DriverInstance[] list = DriverManager.getInstance().getAllDriverInstances();
for(int i = 0; i < list.length; i++){
   System.out.println("=============" );
   System.out.println("Driver ID: " + list[i].getId() );
   System.out.println("Driver Jar List: " + list[i].getJarList() );
   System.out.println("Driver Name: " + list[i].getName() );
   list[i].getPropertySet().getBaseProperties().list(System.out);
}

 
To list the connection profiles DTP knows about do this:

IConnectionProfile[] plist = ProfileManager.getInstance().getProfiles();
for(int i = 0; i < plist.length; i++){
	System.out.println("=============" );
	System.out.println("Profile Name: " + plist[i].getName() );
	System.out.println("Profile Provider ID: " + plist[i].getProviderId() );
	System.out.println("Profile Provider Name: " + plist[i].getProviderName() );
	plist[i].getBaseProperties().list(System.out);
}

 
With that, you can now start to plan your requirements to build the driver definitions and connection profiles with code.

Step five: Create the driver definition

To create the driver definition, set up the following Properties object:

Properties baseProperties = new Properties();
baseProperties.setProperty( IDriverMgmtConstants.PROP_DEFN_JARLIST, jarList );
baseProperties.setProperty(IJDBCConnectionProfileConstants.DRIVER_CLASS_PROP_ID, "org.apache.derby.jdbc.ClientDriver"); //$NON-NLS-1$
baseProperties.setProperty(IJDBCConnectionProfileConstants.URL_PROP_ID, driverURL);
baseProperties.setProperty(IJDBCConnectionProfileConstants.USERNAME_PROP_ID, username);
baseProperties.setProperty(IJDBCConnectionProfileConstants.PASSWORD_PROP_ID, password);
baseProperties.setProperty(IJDBCConnectionProfileConstants.DATABASE_VENDOR_PROP_ID, vendor);
baseProperties.setProperty(IJDBCConnectionProfileConstants.DATABASE_VERSION_PROP_ID, version);
baseProperties.setProperty(IJDBCConnectionProfileConstants.DATABASE_NAME_PROP_ID, databasename);
baseProperties.setProperty( IJDBCConnectionProfileConstants.SAVE_PASSWORD_PROP_ID, String.valueOf( true ) );
baseProperties.setProperty( IDriverMgmtConstants.PROP_DEFN_TYPE, "org.eclipse.datatools.connectivity.db.derby102.clientDriver");

Of course, you’ll have to determine what to use for jarList, driverURL, username, password, vendor, version, databasename as appropriate for your own situation and requirements. Also make sure that the PROP_DEFN_TYPE is correct. The above is proper for a 10.2 instance of the Derby client driver, but you’ll need to adjust for your own scenario.

Then create a property set and provide the above Properties object as the base properties

IPropertySet ips = new PropertySetImpl("Our Driver Name", "Our Driver ID");
ips.setBaseProperties( baseProperties );

 
Finally, hand the result to the DriverManager and tell it to add a new instance.

DriverInstance di = new DriverInstance( ips );
DriverManager.getInstance().addDriverInstance(di);

Step six: Create the connection Profile

When you create the Connection Profile, use the ProfileManager and ensure that your profile name does not already exist. In our case, the name was unique enough that we could look for it, delete it if found, and then replace it with the new version:

ProfileManager pm = ProfileManager.getInstance();
/* Now that we have the driver definition above, create a profile that references it. */
IConnectionProfile icp = pm.getProfileByName("Our Connection Profile Name");
if(icp != null){
	pm.deleteProfile(icp);
}

Then we are ready to create the connection profile. Use the same baseProperties object as created above (it has all of the right settings in it) and add the driver definition reference. Then use the correct providerID to create the profile and you are done:

baseProperties.setProperty("org.eclipse.datatools.connectivity.driverDefinitionID", "Our Driver ID");
String providerID = "org.eclipse.datatools.connectivity.db.derby.embedded.connectionProfile";
pm.createProfile("Our Connection Profile", "Our Profile Description", providerID, baseProperties);

Wrap Up

Some of the settings may not be obvious and most of the doc that you’ll find on the net is geared towards the GUI side of DTP, but it is possible to automatically create driver definitions and connection profiles for your users within DTP.

Now that you know how, go find ways to make your user’s lives easier by automating things they shouldn’t have to do by hand:-)

April 24, 2009 Posted by | Uncategorized | 3 Comments

Software idea – story planning by locality

An idea struck me the other day, an idea for a software product to help writers organize their thoughts for stories that they work on. I did a quick search, and as usual there are very many examples of book writing software including directories of writing products and even your typical book writing software is evil responses to the existence of such software.

Dramatica seems to be the leader of the pack, with many others in the arena giving it healthy competition. This being the case my idea doesn’t seem so killer any more, but I have to remind myself that the existence of competition really doesn’t mean you can’t make money with a product. It just means you have to build a better mouse-trap. It also validates the market and proves that there are people who will pay money for this type of product.

So what’s the idea, you ask? I was staring at a white board, an empty white board, while thinking about the flow of some of my favorite books and the idea hit me that the author’s overall sweep of the story really moved the characters around in their world quite a lot. I tend to enjoy books where you have to keep referring to the map on the first (or last) page (don’t your books have maps in them?) so that I can visualize where everything is taking place and how the characters have moved around throughout the story. As I did this the idea struck me that if you lay the chapters and locations out on a grid like this:

Locality\Chapter Chapter 1 Chapter 2 Chapter 3 Chapter 4
Village 1
Mountains
Forest
City 1
City 2
Castle 1
Ocean

you can start to see the overall motion of the story by placing the characters inside the grid corresponding to chapter and location. As the chapters progress, you move the characters from location to location in order to create the motion in the story. When you need grand sweeping chapters that describe your world and provide rich detail and background for your characters, you separate them and let them move in very different directions. When you need action and conflict then you bring one or more groups together into the same places and describe more about their actions, thoughts, and emotions.

Once you plan the overall motion of your story then this becomes your blueprint for what is happening in each chapter, defines where you need to get your characters to move from and to, and allows you to visually see the way that your story will rise and fall as characters come together, interact and explore their world.

Of course you would need utilities to help with character development, outlines, plots, etc. You’d need example story skeletons, sample worlds that could be adapted to the writers needs and all of the rest. That’s a given.

After thinking about it for a while, I also realized that this would also work for those stories that are not based on travel throughout an entire world. Fantasy/Adventure stories usually have the characters traveling all over the place across a huge range of locations. What about drama, though, or mystery and intrigue? Take the old standby game Clue. The entire story takes place in a single house. That doesn’t mean that there’s no motion or locality of the characters to follow, though. In fact, mysteries and dramas would work just as well with this type of layout, because it would help the author to keep track of where everyone is while the story unfolds. It would just look more like this:

Locality\Chapter Chapter 1 Chapter 2 Chapter 3 Chapter 4
Foyer
Parlor
Study
Kitchen
Servants Quarters
Master Bedroom
Guest Bedroom

You can still watch your characters move from place to place. You can still describe your setting by separating your characters and exploring the environment through their eyes. You can still create action and conflict by moving them from place to place, putting them together and keeping them guessing where the others might be. The idea still works, and holds up in different genre’s.

So what’s the difference between this idea and a simple spreadsheet to keep track of it all? Well, that’s what makes a product a product, now isn’t it! What’s better about this product idea than the others already out on the market? Well, that remains to be seen. Ideas aren’t always implemented, but you never know:-)

April 24, 2009 Posted by | Uncategorized | Leave a comment

   

Follow

Get every new post delivered to your Inbox.