Ruby on Rails Makes Me Happy

Last year I got to work on my first Enterprise Rails application and although there were some brief moments of coolness, that project was punctuated with a lot of things that just made it suck overall. In fact, I was doubting that bringing Rails to the enterprise would be a good thing. There were just to many other things wrong that had nothing to do with the technology.

But half a year later our enthusiasm for Ruby on Rails hasn't been seriously diminished despite a less than stellar project, and surprisingly our commitment is growing. From three developers doing Rails we've grown it to five and my managers have fully accepted it and are pushing it forward for most new projects.

Excitingly for me, I was recently assigned a small but very high profile project. And this time I think I've been able to do things the best way I know how in a very short period of time. The controllers are RESTful, my routes are appropriately nested, the layouts are clean table-free HTML with CSS and the Javascript is unobtrusive. I really dig Prototype and the app is sprinkled with just the right amount of AJAX and the Scriptaculous effects are not gratuitous.

Quite honestly I'm pretty thrilled with this app. It's only been a week or two worth of effort but it's been thoughtful and extremely efficient. So efficient that I think the app will be fully functional before the TQA and Production Tomcat servers I requested will be provisioned by the server team (remember I'm in a big enterprise). And yes, you read that right I'm deploying to a JEE web container. I'm past any trepidation I may have once harboured about JRuby vs MRI Ruby. To me it's a proven part of my deployment strategy that I can depend on.

Anyway I'm a happy developer again. I'm getting apps written and deployed for my users and I love the look of my code. The cool thing is that Rails keeps getting better. Rails 2.3 is just around the corner. I like the application templates and the integration of Rack. I still have to bring up my testing game but give me another week and I think my tools of choice (RSpec, Remarkable, and FactoryGirl) will have me cranking out better test coverage than I've ever had before.

Anyway let me wrap this up with some of the things I think are wicked about Ruby and Rails and the various gems:

• pluralize
• belongs_to, has_many :through (declarative ORM)
• Model.all
• Model.paginate (will_paginate gem is awesome)
• link_to_if
• select, collect, each (collection methods are cool)
• statement if expression (statment modifiers rock)
• distance_of_time_in_words_to_now
• model_url
• before_update, before_delete
• validates_presence_of, validates_length_of (declarative validation is sweet)
• Ruby, a REAL language, in my view templates!
• jruby script/console oracle (interacting with Oracle through my models via JDBC)
• rake war:clean war (Thank you warbler)
• script/autospec (even better on my Mac when integrated with Growl)
• ActionMailer with layouts (my HTML emails never looked so good)
• Ajax.Request, Ajax.Updater (Yay Prototype)
• case (yeah just a nice simple case statement)
• auto_discovery_link_tag and builder (yeah I added an RSS feed for good measure!)
• Railscasts (an exemplary example of a great community, Ryan Bates you rock!)

JRuby & Rails 2.2

We just started converting things over to Rails 2.2.2 and I thought I should say that the one feature that really kicks some ass is its thread safety. JRuby 1.1.5 (very soon to be 1.1.6), which takes advantage of Java's native threads, combined with this latest version of Rails finally makes for a very good deployment story.

Rails' traditional single threaded model, while conceptually nice for development, never really made things easy for deployment. In MRI-land, you always needed multiple processes, translating into multiple instances of Mongrel. In JRuby we needed multiple instances of the JRuby runtime. While it worked, it was definitely a bit of a memory pig. With Rails 2.2 we can now run with only one JRuby runtime in a JEE web container that was already multi-threaded. Now memory consumption is several fold smaller than what it was before and we now have a much better story to tell everyone. Thanks to everyone on the JRuby and Rails teams!

Now, I shouldn't just end with a big hurrah to everyone without mentioning that we did run into a problem with some thread unsafe code in our own app. It took awhile to find it (a week) so the one word of caution is be careful about any shared state in your application (in our case it was a rather obvious global variable that we seemed to keep overlooking).

In any case, Rails and JRuby are a great match for enterprise web app development. I've been hesitant to say that in the past because of a few rough edges like the one I mentioned before but now I think it's about time everyone should look at this fantastic combination.

UPDATE:

Thought I'd add a link to a recent Ruby shootout. It's reporting on the performance of JRuby vs other Ruby implementations and the importance of threading.

Heroku, RubyGems and Git

My daughter and I started a little Ruby on Rails project and I thought it'd be nice to host it somewhere for her to play with. I have an account on Heroku that I haven't used in months and thought it'd be nice to try it out again. I signed back in (after resetting my forgotten password) and was impressed to discover that they've integrated Git. After creating the project and navigating to the Rails start page I found some instructions:

gem install heroku
heroku clone stuffies
cd stuffies
[..local edits..]
git add .
git commit -m "local changes"
git push

So I ran off and installed the heroku gem. But had trouble with step 2. Fortunately a couple of google searches had me back up and running. One to deal with "Permission denied (publickey)" upon heroku clone. And another to figure out how to create the RSA key. Now I have a local Git clone of my project that I can work on in NetBeans and then I can quickly publish the changes to Heroku with a commit and a push. Sweeeet!!!

Lessons from my first Rails project

I’m finally being paid for working on an “Enterprise” Ruby on Rails pilot project. I think I helped persuade our IT group to give Rails a try. But unfortunately I wasn’t available to help the project get started. Instead I joined the project several months in. So while it’s cool to finally be doing Ruby and Rails full-time it’s also pretty painful to be living with some of the poor decisions made before I got here. I can’t say that if I’d been involved from the get-go that we would have avoided some of these problems (sometimes it runs deeper than the technology) but what I can offer is a list of things I’d suggest you watch out for (in no particular order):

Be realistic
In the early days of Rails I remember people saying things that made it sound that Rails was ten times more productive than Java. That’s a wild exaggeration and everybody should know it. There’s no way you’re going to be 10X more productive, end of story. Hope for “10% more productive” after you’ve got one or two projects under your belt. At least that’s realistic and achievable and if you get more than that then great! But for the love of God don’t start off with that idea. A big part of the software game is managing expectations. Don’t dig yourself a hole you have to try and climb out of. Despite what you read on the web, your first Rails project will probably take you more time, not less. New language, new tools, new framework, new infrastructure, new bugs, and new deployment problems. My project started with unrealistic expectations and now we’re over budget and over schedule. That’s not a great way to get management on your side.

Deploy early and often
That’s an agile thing and not unique to Rails, but getting over the hurdles of deployment shouldn’t be taken for granted. We’re primarily a Java shop so when people saw JRuby and WAR deployment, they figured everything would be easy-peasy. Not so. Part of being “Enterprise” is that quite often you’re several months, if not years, behind the technical curve. The first thing we needed was to get past Java 1.4 and an old version of Weblogic. We needed at least Java 6 and Tomcat 6. Trust me, getting “comfortable” IT infrastructure people to upgrade their technology can be very painful and slow.

Now assuming you got the basic app server running on a reasonably recent version of Java, go and figure out how to build the WAR (warbler gem), get to Oracle (activerecord-jdbc gem) through JNDI, get your Rails log files out of the exploded war directory (logging gem), integrate with ActiveDirectory for Authentication and you’ve got your hands full of a bunch of other problems that will eat away at your time. Budget for it.

Now once you got all that, deploy often! Our project has periodically deployed something outside of our dev environment but not frequently enough to gather any kind of momentum to satisfy our users. Without gaining the users’ trust you get into a whole other set of problems.

Unit test (RSpec seems pretty awesome)
Sounds like a reasonable thing to do... but we didn’t. I won’t go into all the reasons why testing is good, but to fail to take advantage of the testing infrastructure that Rails gives you out of the box is pretty ridiculous. Even if all you’re doing is executing the code and never asserting the results are correct at least you’re executing the code and will find runtime problems that a language like Java would have given you at compile time. I started using RSpec for my stuff and I really like it, but it sucks when the other developers won’t add to the tests or even run yours. But my strategy is to lead by example. Maybe sooner or later the other will pick it up too. If you can get buy-in early and be ruthless about keeping your testing effort on track.

Figure out the non-standard Rails stuff
Beyond the deployment stuff I talked about earlier, figure out the other places you must configure rather than adopting the Rails conventions. For example, we have a rule that says you must access the database from an account other than the schema owner. That means we needed to call set_table_name(‘schema.table’) in all of our models. We also needed to create the permissions for the non-schema owner. And don’t forget proper referential integrity constraints in your migrations. These are all things you can ignore for a while but if you have any hope of putting you application in your users’ hands you’re going to have to figure it out. Personally, I hate surprises like this late in the project. My recommendation is to sort them out early.

Don’t underestimate security (a.k.a., read the requirements doc)
We have users who don’t trust anyone and have dreamed up one of the most elaborate security mechanisms you can think of. Unfortunately we went through five months of development and not one of the developers before me even seriously looked at it. Sadly, I can understand why. Every Java developer (including me) thinks they got that one figured out and just skips that part of the requirements doc. Except that someone should have actually read ours and said Woah! It’s not always easy to retrofit something like this if you just assumed it was an orthogonal concern. If someone wrote a requirements document for you (no matter how badly written) be sure to read through the whole thing. You never know what problems may be lurking.

Read ‘The Rails Way’
In general, get some training. Or in lieu of that, get some good books. I bought “The Rails Way” by Obie Fernandez (great book btw) after I got assigned to the project. To my way of thinking the project should have already bought a whole bunch of Rails books and made them available to the developers. But it’s up to you to be prepared. If there are parts of the framework that are still a bit fuzzy start reading some books (go to the source if you need to).

Evaluate your editor
So like I said before, our shop is primarily a Java shop, so when I got on the project people were using IntelliJ to edit their code. Now IntelliJ may be a kick-ass Java IDE but from what I’ve seen it’s not great for doing Rails. So when I started, I used NetBeans. Now NetBeans has never in my mind been a great Java IDE, and quite frankly it’s not my dream Ruby IDE either but Sun hired the JRuby dudes and Tor Norbye, the principal NetBeans guy at Sun seems to understand Ruby and the needs of Rails developers pretty well. As near as I can tell, unless your a Mac-toting Textmate user, NetBeans is the best game in town for Rails development (I even use it on my Mac). So my recommendations is to get off your butt and seriously evaluate the tools you’re using. Maybe IntelliJ is the right thing for you, but you don’t really know unless you look around.

Use an issue tracking system
We didn’t have an issue tracking system when I started on this project. So not only could we not tell how far we were by test/spec status, we weren’t even tracking the change requests and bugs properly. Wow. In my mind this is part of the scope problem we’re having. The users just ask for something and without visibility to the whole team and management, it just gets magically added and the schedule slips.

Web 1.0
AJAX is pretty attractive to a Web 1.0 developer. But food is awfully attractive to a starving man too, but that doesn’t mean you should let him eat himself to death. Take it slowly. IMHO, do it old school on your first (and maybe your second or third) Rails project. Throw in the odd autocompleting field if you want but this AJAX stuff and all the pretty visual effects can get out of hand pretty quickly.

Use people who want it to succeed
Java and .NET are pretty typical technologies in IT and unfortunately a lot of the developers who’ve invested a lot of time learning this stuff seem to forget that the technology is only a means to an end and not the end itself. They’ll be threatened by new technology. They’ll be threatened by Ruby and Rails. So if you can, staff your project with the developers who are open minded and wouldn’t mind if a Rails application succeeded. If you push these hostile developers onto a Rails project their negativity will bring everyone else down.

Understand RESTful Rails
I don’t remember when exactly David Heinemeier Hansson really started pushing the idea of a resourceful/RESTful way to develop web applications with Rails, but it’s damn good stuff and you should really figure out what it all means before you start building a whole bunch of stuff the old way. Figure out Rails’ routing system and what map.resources does and then you’ll be able to go with the flow and let Rails do a lot more of the work for you. I’ve seen more than one developer write a lot more code than they need to.

Separate HTML, CSS, & Javascript
I can’t believe these so called IT web developers who still use all the old FONT, BORDER, WIDTH, ONCLICK etc. HTML elements in their templates. C’mon! HTML is mark-up, CSS is styling and JS is behaviour. Separate these aspects of your view layer the same way you separate models, views, and controllers. I’ll make some allowances for surgical uses of RJS but if you want things to be maintainable sort this out already. (I know this isn’t just about Rails but it bugs me so much I figured I’d rant about it anyway ;-))

If you’re about to embark on a Rails project, get ready to enjoy yourself, just don’t lose your head in all the hype and follow some good common sense.

JRuby and Hibernate

The beautiful thing about Rails on JRuby is that you have access to Java. Perhaps you have existing code you'd like to reuse or maybe there's some functionality in a Java library you'd like to use. Whatever the reason, it's good to know how to do it.

So today I created a new Rails app to demonstrate how to use Rails' controllers and views with some model classes from one of my earlier Java projects. My old app used the typical HibernateUtil singleton (section 1.2.5 of the Hibernate docs). I also liked the ActiveRecord pattern so my Java domain classes had some handy "find" methods. So in order to make this work in a Rails app, I installed the goldspike plugin:

script/plugin install http://jruby-extras.rubyforge.org/svn/trunk/rails-integration/plugins/goldspike

and then ran the rake task war:standalone:create

I copied the classes and lib directories from my old apps' WEB-INF directory to the WEB-INF directory in my Rails app and then generated a controller and modified it to look like this:

require 'java'
import 'com.example.domain.Person'
import 'com.example.util.HibernateUtil'

class PersonController < ApplicationController
  def list
    with_transaction do
      @people = Person.find_all.to_a
      @people.sort!{|p1, p2| p1.name.downcase <=> p2.name.downcase}
    end
  end
  
  def with_transaction
    begin
      tx = HibernateUtil.session_factory.current_session.beginTransaction
      yield
      tx.commit
      HibernateUtil.session_factory.current_session.close
    rescue
      tx.rollback
    end
  end
end

Finally, I added a list.rhtml template for the "list" action and voila! I have a Rails-ish app that uses Hibernate instead of ActiveRecord.

Ruby Cocoa

RubyCocoa and the Scripting Bridge are two software development technologies bundled with Leopard. The app presented here demonstrate how to create a native Mac interface written in Ruby that communicates with iTunes via Applescript. It does nothing more than display the iTunes version number.

1. Create a "Ruby-Cocoa Application" from the "New Project" dialog in XCode
2. Add a "Ruby NSObject subclass" to the project called "ITunesController"
3. Add an ib_outlet for a text field and an ib_action called "show_version" to the ITunesController ruby class.
4. Generate an Objective-C header file that scripting bridge can use to communicate with iTunes. Use the two terminal command, sdef and sdp, within your project folder like this: "sdef /Applications/iTunes.app | sdp -fh --basename iTunes"
5. Add the header file to your project.
6. Add "ScriptingBridge.framework" to your project.
7. Open "MainMenu.nib" in Interface Builder and add a button and a label to the Window. Also instantiate an NSObject subclass.
8. Set the NSObject's class to "ITunesController". (You may need to select the "Read Class Files..." menu item to get ITunesController to show up as a valid option)
9. Wire the button to your controller's "show_version" method and wire the controller's text_field outlet to the label.

Here's what my project looks like:


Here's what my Ruby "ITunesController" class looks like:

require 'osx/cocoa'

include OSX
class ITunesController < NSObject
    ib_outlet :text_field
    def show_version(sender)
        iTunes = SBApplication.applicationWithBundleIdentifier:'com.apple.iTunes'
        @text_field.setStringValue("iTunes version: #{iTunes.version}")
    end
    ib_action :show_version
end

Here's what the app looks like:


Now you should be able to run the project and get the iTunes version number displayed in your own application. Yeah, the end result is kind of lame but the journey was pretty cool. The interesting bits for me were:

1. My Ruby class extends Cocoa's NSObject class.
2. The class methods ib_outlet and ib_action are parts of the Ruby DSL for doing Cocoa development with Ruby.
3. The RubyCocoa framework makes it really simple to swap out Objective-C for Ruby when doing Cocoa development.
4. Everything else is the same. I still use XCode for my controllers and model and still use Interface Builder to graphically build my GUI and bind it to my code. The only difference is that I get to use Ruby and all of its beauty to get the job done.
5. The downside is that I don't get anything too useful in the way of code completion while in XCode. Still might be worth it if you really dig Ruby.

JRuby and Rails in the Enterprise

At work, management is finally getting the message about dynamic languages like Ruby and starting to recognize that frameworks like Rails may have productivity benefits.  We're actually going to look at doing a pilot project.  I like to think that I've had some influence in that but you never know...

Anyway, I would love to make this work but this is a Java shop through and through.  We can only deploy to a Java web container, specifically Weblogic 8.1 (the transition to Weblogic 10 is happening but it's months away).  I've used JRuby (works great BTW) and the JRuby extras to help me deploy an app to Tomcat, but I can't manage to get JRuby and Rails working in Weblogic 8.1.  Any ideas?

Secondly, Groovy and Grails is the other contender.  I like the idea behind Groovy and Grails but I can't help feel that it's a pale imitation of Ruby and Rails.  Anybody have enough experience to compare and share?

Finally, I'm not naive enough to think that this will cure all the ills of working in the "big enterprise".  When I think about all the things that bug me, the technology is probably the smallest thing that causes me pain.  But on the other hand, this would be a wonderful distraction. :-)

Marching on...

I used to develop applications with PowerBuilder (PB) for about 8 years.  I started with PB 1.0.  In the beginning it was new and exciting.  Windows was new.  Client/Server was new and even object oriented development in a mainstream tool was relatively new (at least in the corporate software development scene).  But as technology progressed I began to see exciting things elsewhere.  At the time Delphi was the cool new product.  The speed, the beauty, the elegance... ahhh I was smitten.  But the other PB developers around me didn't see it.  I fought for Delphi and even did one or two project with it.  But unfortunately Delphi wouldn't become my mainstream development tool.  PB was entrenched and the developers around it protected it with a passion.  It was disappointing. 

Then Java came along.  To my eye Java shared a lot in common with Delphi.  I liked it too.  But still the PB developers around me couldn't seem to appreciate this new tool. In fact, a lot of them were actually pretty hostile to anything that might threaten the sacred cow that was PB.  Fortunately for me the hype of the Internet and the .COM boom turned Java into something people couldn't ignore.  And I decided to ride the wave and finally leave PowerBuilder behind.

Now another 8 years has gone by and I find myself in the middle of a community of Java developers who are extraordinarily similar to that crowd of PowerBuilder developers.  They only see Java.  They've built a career around it, they're entrenched and they protect it with a passion.  For me, I see the appeal of other languages and tools.  In particular Ruby is beautiful and Rails elegantly solves many of my Java web development headaches.  But many of the people around me don't see it and quite frankly I doubt that they've even looked.  And that's the disappointing thing.  They're just like the PB guys with the blinders on.  They're hostile to anything that isn't Java.

In my current group there seems to be a new love forming for GWT.  Personally, the Google Web Toolkit and its unmistakable Swing/AWT flavor of Web development just seems wrong.  It's definitely web development for Java developers and I can see why they like it, but to me it's an artificial abstraction that doesn't sit well with me. Ruby and Rails isn't perfect either, but even a couple years after I started looking at it I still think it's better than GWT.  Rails embraces the browser technology of HTML, CSS, and Javascript and makes it easy to work with.  GWT on the other hand, puts me behind a Java facade where I can pretend to be developing Swing and it will generate the HTML and Javascript for me.  Ick.  No thanks Google.

Anyway, my point is that I think I'm getting to another point in my career where I need to migrate to a new place with like-minded people.  I'm tired of trying to get others to see what I think is self-evident if only they'd look...

Java & Ruby HTTP Clients: Part 2

Several months ago I wrote an article about how to create an HTTP client in Java or Ruby. I included examples in both languages for getting by BASIC and FORM based authentication. I also showed you how to resubmit the value of an HTTP cookie that many websites use to store state.

What I didn't mention is that quite often web applications store state not in a cookie but within the HTML itself. In order for you to programmatically interact with the website, you'll need to get that data out of the HTML and put it in your next request.

Java

So the basic recipe is to to use HttpClient (the way I demonstrated last time) to get the raw HTML. Then feed that text into NekoHTML, an HTML parser. It can correct the various problems you see in old-school HTML, namely unbalanced tags, missing parent tags, and mismatched elements.

Neko returns a standard Document object, but personally, I don't care much for the standard XML API. Instead I prefer to use the simpler API of DOM4J. So the next step is to take the XML that NekoHTML provides and feed it into DOM4J so that I can use XPath expressions to find what I need.

Now the question becomes what data do you need to post. Well my answer is to simply look through the FORM you're trying to submit, and resubmit everything. Look for 'hidden' tags, input tags of type 'text', 'password', and 'select', gather the names and values of all those tags. Override the ones where you need to provide the information (like id and password for example) and then do the POST.

Ruby

The Ruby approach is exactly the same: get the raw HTML, parse it, use XPath to get the name-value pairs of the FORM elements, override some of the values, and resubmit. The only additional rubygem from my last article is hpricot. It provides the same functionality of NekoHTML and DOM4J in Java. The typical script might look like this:

require 'net/http'
require 'rubygems'
require 'hpricot'

res = Net::HTTP.new('myserver', 80)
# res.set_debug_output $stderr #uncomment this to get console debug info
res.start do |http|
#go to the first page
get = Net::HTTP::Get.new('/home.aspx')
response = http.request(get)
                                                    
#collect the cookie information
cookies = ''
response.response['set-cookie'].split(';').each{|c|
cookies += c.split(/path=.*?,/).last.strip + ';'
}

#collect the existing form data
doc = Hpricot(response.body)
form_data = {}
['text', 'password', 'hidden'].each{|t|
elements = doc.search("//form[@name='loginForm']//input[@type='#{t}']")
elements.each{ |e|
  form_data[e['name']] = e['value'].to_s
}
}

#override some of the values
form_data['username'] = 'my_username'
form_data['password'] = 'my_secret'

#login
post = Net::HTTP::Post.new('/login.aspx')
post.set_form_data(form_data)
post['Cookie'] = cookies
puts http.request(post)
end

In this case all I did was print out the resulting HTML. At the very least you'd probably do the hpricot thing one more time to retrieve the data in which you're interested.

Other

Finally one last gotcha... At least half the websites I've tried to scrape do something "interesting" with Javascript to set various form elements. Since we don't have a Javascript engine executing you should expect that you'll have to parse the HTML and Javascript yourself to figure out what's going on, and set the fields manually in your script. I highly recommend Firefox and the "Web Developer" and Firebug plugins for inspecting the HTML, JS files and the HTTP Requests that the browser submits.

P.S.: I was a bit lazy this time and didn't provide any Java code. If you're really having trouble and can't get it working, leave me a message and I'll put together an example. Secondly, there are other Java HTML parsers out there that may work just as well or maybe even better than Neko, but since I don't have any personal experience with them I didn't mention them. If you like something else, leave a message.

Aptana RADRails & JRuby

I thought I'd go and take a look at what's been happening on the RADRails project lately. As it turns out, quite a bit! Aptana has taken over the project and are making quite a bit a progress integrating it with their Eclipse based IDE. The most impressive thing I saw was refactoring support! Fantastic! It's great to see RADRails getting some attention again. Go download the beta.

Then I went over to the JRuby project to see what they've been up to. JRuby 0.9.9 is out and looks very good. The Java integration is quite good. But I've been mostly interested in seeing support for deploying a Rails app in a JEE Web Container. Well nestled in beside the JRuby TAR file is a file called sample-rails-warfile.tar.gz. I downloaded it and pretty quickly had a rails application running in Tomcat. The app doesn't do much of anything but it's still pretty impressive to see it. My hope is that this kind of integration will help to get Ruby & Rails adopted into some of Java-only environment I tend to work in. The next thing I did was build my own Rails app and deploy it in Tomcat. I also tried deploying it in WebLogic but didn't have as much luck. Obviously there's still some work to do. Check out the JRuby WIKI for details.

Ruby OLE

The other day someone came over to my desk and asked me how they could programatically invoke the "Find Computer" function of the Windows shell. Normally you just press CTRL-WINDOWS-F and up pops an Explorer window ready for searching. I didn't have a quick answer for him... so I immediately took up the challenge to figure it out.

After a couple dead ends I started looking at Windows Script Host (WSH). It allows you to write VB or JavaScript to access the ActiveX API of a Windows application. After a bit of googling I found the "Shell.Application" documentation on MSDN. After futzing around with Javascript for a few minutes I started thinking that I should be able to do the same things with Ruby. So with a little more searching I discovered that by adding "require 'win32ole'" at the top of my ruby script I could instantiate an instance of "Shell.Application".

The only problem left was finding the right API call. A three line Ruby script like this gave me a list of methods:

require 'win32ole'
wsh = WIN32OLE.new('Shell.Application')
wsh.ole_methods.collect{|m| m.to_s}.sort.each{|m| puts m}

A quick change and my final script looked like:

require 'win32ole'
wsh = WIN32OLE.new('Shell.Application')
wsh.FindComputer()

or in JavaScript:

wsh = WScript.CreateObject("Shell.Application");
wsh.FindComputer();

If you want to take this scripting stuff to the next level I suggest you go check out David Mullet's "Ruby on Windows" blog. He describes some of the same things I just demonstrated above and goes into a lot greater depth about how to use Ruby to script think like Word and Excel.

And in case you're wondering why I had to use Ruby to get a list of methods instead of just looking it up in MSDN, well just try looking at the MSDN page I referenced above with Firefox instead of IE.... That's right... the site navigation only works correctly in Internet Explorer.

Java & Ruby HTTP Clients

In a mythical IT universe designed around a service oriented architecture (SOA) you could assemble several loosely coupled, autonomous services into a business solution. Each service would communicate in a platform and technology agnostic manner and XML would be the lingua franca. For example, if you needed to integrate data from several business partners you could call various SOAP or RESTful services, get structured XML, and then transform the results into something you could use.

But unfortunately that's not what happens in the real world. Instead of nice composable services you usually get web sites targeted at people not machines. That means you need to automate what would typically be browser conversations with various websites to get the data you need. And then you need to deal with the format of the resulting data. If you're lucky, the data may be structured in a comma separated value (CSV) text file. But undoubtedly you'll have to parse unstructured text representations of a report or get the data out of an excel spreadsheet or even a PDF. It's not pretty.

But lets forget that unpleasantness for the moment and deal with the first problem you'll encounter in trying to automate a browser conversation, getting by the various authentication mechanisms. Let's look at BASIC authentication first. It's pretty common and well supported in the Java-based Jakarta Commons HttpClient library and the Ruby Net::HTTP Standard Library.

BASIC Authentication

To start things off I created a simple Java-based "Dynamic Web Project" using the Eclipse Web Tools Project (WTP) plugins. To keep things simple I created a servlet that returns a string. Then I configured the application to protect the url for that servlet with basic authentication in web.xml. Then in Tomcat's server.xml I modified the context element for my web app to include a reference to the default Tomcat in-memory user database:

<Context docBase="MyWebApp" path="/MyWebApp" [...]
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
    debug="0" resourceName="UserDatabase"/>
</Context>

With that in place, and the server running we're able to write a couple of methods to access the servlet. In Java:

public static void basicAuthDemo()
  throws HttpException, IOException{
HttpClient client = new HttpClient();
List<String> authPrefs = new ArrayList<String>();
authPrefs.add(AuthPolicy.BASIC);
client.getParams().setParameter(
  AuthPolicy.AUTH_SCHEME_PRIORITY, authPrefs);

client.getState().setCredentials(
  new AuthScope("localhost", 8080, "localhost:8080"),
  new UsernamePasswordCredentials("tomcat", "tomcat")
);

GetMethod get = new GetMethod(
  "http://localhost:8080/MyWebApp/myservlet");
get.setDoAuthentication(true);
client.executeMethod(get);
System.out.println(get.getResponseBodyAsString());
get.releaseConnection();
}

In this example I limited HttpClient's default authentication mechanism to BASIC. I know what my target system uses so why complicate matters with DIGEST or NTLM? Then it was a simple matter of defining the credentials and telling HttpClient to automatically use them and then executing the Http GET method. It looks surprisingly similar in Ruby:

def basic_auth_demo
url = URI.parse('http://localhost:8080/MyWebApp/myservlet')
get = Net::HTTP::Get.new(url.path)
get.basic_auth('tomcat','tomcat')
response = Net::HTTP.new(url.host, url.port).start do |http|
  http.request(get)
end
puts response.body
end

The difference between the two implementations is that the Java HttpClient is doing some housekeeping for you. You define a scope for your authentication and as long as you GetMethod is configured to do authentication it will automatically pick up any necessary credentials from the HttpClient instance. In Ruby you need to set the credentials on the GetMethod explicitly.

FORM based authentication

The next most common method is form-based authentication. When you make a request for a web resource, the response contains a cookie that identifies your session on the server. If that session indicates that you haven't been authenticated yet, then you're redirected to a form to enter your id and password. You fill in the values and then submit the form. Now assuming you entered the right credentials your session on the server will indicate that you're authenticated and every subsequent request (which includes the cookie to identify your now authenticated session) will execute normally. There are variations on this theme that may add more than one cookie so just be sure to capture the cookies and continue to submit them on every request in your conversation.

In order to test this I modified the web.xml file for my Java web app:

<login-config>
<auth-method>FORM</auth-method>
<form-login-config>
  <form-login-page>/login.jsp</form-login-page>
  <form-error-page>/login-error.jsp</form-error-page>
</form-login-config>
</login-config> 

and added the requisite JSP pages. The login.jsp contains a form that looks like this:

<form method="POST" action="j_security_check">
Username:<input type="text" name="j_username"><br/>
Password:<input type="password" name="j_password"><br/>
<input type=submit value="Login">
</form>

So the Java code to access the servlet using form based authentication looks like this:

public static void formAuthDemo()
  throws IOException, HttpException {
HttpClient client = new HttpClient();

// make the initial get to get the JSESSION cookie
GetMethod get = new GetMethod(
  "http://localhost:8080/MyWebApp/myservlet");
client.executeMethod(get);
get.releaseConnection();

// authorize
PostMethod post = new PostMethod(
  "http://localhost:8080/MyWebApp/j_security_check");
NameValuePair[] data = {
  new NameValuePair("j_username", "tomcat"),
  new NameValuePair("j_password", "tomcat")
};
post.setRequestBody(data);
client.executeMethod(post);
post.releaseConnection();

//resubmit the original request
client.executeMethod(get);
String response = get.getResponseBodyAsString();
get.releaseConnection();
System.out.println(response);
}

The Ruby code looks like this:

def form_auth_demo
res = Net::HTTP.new('localhost', 8080).start do |http|
  #make the initial get to get the JSESSION cookie
  get = Net::HTTP::Get.new('/MyWebApp/myservlet')
  response = http.request(get)
  cookie = response.response['set-cookie'].split(';')[0]

  #authorize
  post = Net::HTTP::Post.new('/MyWebApp/j_security_check')
  post.set_form_data({'j_username'=>'tomcat', 'j_password'=>'tomcat'})
  post['Cookie'] = cookie
  http.request(post)

  #resubmit the original request
  get['Cookie'] = cookie
  response = http.request(get)
  puts response.body
end
end

Again, the two implementations are remarkably similar. The biggest difference is that the Java HttpClient library is again doing the housekeeping, by tracking and automatically resubmitting the cookies for you. In the Ruby code you have to fetch the cookie yourself from the response header and set the HTTP header for all future requests.

So there you go, you're past the website authentication and are ready to make whatever requests you need to get the data you require.

Ruby on Rails demo

Yesterday I gave my co-workers a lunch time presentation about Ruby on Rails. Since these guys haven't even seen Ruby, never mind Rails, I kept it pretty basic and loosely based it on DHH's build-a-blog-in-15-minutes screencast at RubyOnRails.org. I coded live and in person and even the mistakes I made were good because they highlighted not only Rails' good error reporting but also the quick edit-and-refresh-the-browser style of coding. No Ant scripts, no compiling, no restarting the app server, no waiting for the VM, etc.

In the end I think I made the impact I wanted. I showed people some other ways of developing web applications that are fundamentally more productive than the endless configuration hell we Java developers tend to work in. Unfortunately, I also heard unfortunate statements like "that's cool but we're a Java shop and we'd never be able to deploy something like that". While that was disappointing, I kept my composure and plugged JRuby as a possible future solution (I can't wait to be able to a build a Rails app into a WAR and deploy it in WebLogic).

I planted the seed. Now all I have to do is continue to nurture the idea of Rails development, continue the education and hope for a brighter future.

Code Generation with Ruby

I just watched some parts of a Google Tech Talk Video about code generation. It was a videotaped presentation by Jack Harrington, the author of "Code Generation in Action" to some of the developers at Google. I didn't find most of the presentation to be that interesting, but there was one snippet of Ruby code that I liked. It looked something like this:

require 'erb'

File.open('./test.txt', 'w+') do |f|
  name='Darcy'
  erb = ERB.new(File.new('template').read)
  f.write(erb.result(binding))
end

My template file was a single line that contained this: "Hello <%=name%>". I think you can probably figure out what the result would look like ;-)

The real example he used in his presentation was slightly more elaborate in that he had a source of data, an XML file, and he used REXML (a great XML library in Ruby) to read the source to generate a SQL file.

I'm not a big advocate of code generation but if you find yourself in the unenviable position of needing to do it then this is a great place to sprinkle in a little Ruby magic.

RadRails Refresher

So I started working a bit harder on a rails project this past weekend (several hours a day) and discovered a few things about Ruby/Rails and my IDE that I thought I'd share (in no particular order):

1. RadRails has the ability to automatically run your unit tests whenever you modify certain files. The AutoTest functionality is great but only if you realize it's there. I had been staring at a big red 'X' in the toolbar occassionally wondering "I wonder what that's for" and then suddenly saw a nice big green checkmark after I fixed a bunch of tests. Duh!
2. ctrl-space is your friend. In the absence of any true "intellisense"-like functionality, RadRails' templates will save you a bunch of typing. In fact if you're a TextMate refugee then try Corban Brook's textmate-like templates for RadRails.
3. Rake is cool, but I always forget all the tasks I can run. Yes even the simple ones like "rake migrate VERSION=3" or "rake db:fixtures:load" seem to continually elude me. Well RadRails has a Rake view. Just make sure you're in a Rails project and then Bob's your uncle.
4. I like watching the development log. And lo and behold RadRails allows me to right click on the .log file in the Rails Navigator and select "Tail". This allows me to watch the end of the file in a console window. I recommend creating a new console view, displaying the tailed log file, and then pinning it. Then drag it somewhere where you can always see it.
5. The servers view allows me to set up a mongrel view and then start, stop, and restart it with the click of a button. It also has a handy little globe button for opening a browser within RadRails that goes to your default URL. The browser's functionality is a bit anemic, but it's still a nice touch.
6. Autotest is awesome but if you want to run a single unit test, just open it in an editor, right click and select "Run As -> Test::Unit Test"
7. When I use Eclipse to do Java editing I often like to click the button in the toolbar that only shows me the selected element in my editor. That way I can focus on a single method at a time. RadRails has this button too but it seems to be permanently disabled. But if you have the outline view kicking around, try clicking on a method and watch what happens.
8. I like to keep the Data Navigator view around to see the table structures. Comes in handy when I can't remember some detail about my data model.
9. My console view doesn't work. Something abut the readline library on Mac OS X I think... Regardless, if it works for you either use it or keep a terminal window open. The console rocks.
10. Thanks to the "has_many :through" blog I discovered that I can add a block to associations in ActiveRecord classes and add methods to these associations. For example, I added a find method to one of my has_many relationships. The cool thing is that the find method is scoped by its outer has_many definition. Not sure if I really understand what's going on under the covers (still figuring this Ruby thing out) but it seems like an elegant way to express something like this: user.blog_entries.find_all_by_tag( tag ).
11. Now if I could only remember what the RadRails shortcut keys were for navigating between models and tests and navigating between controllers and views I'd be happy...(UPDATE: CTRL-SHIFT-V to go between controller and view, CTRL-ALT-T to go between model/controller and test)

Digg, Ruby, & Cocoa

I recently discovered that Digg allows you to simplify story submission for readers by placing a "digg this" link on your website. That sounded like a reasonable thing to do for my blog entries and after reading Guy Kawasaki's blog about it, I decided to give it a whirl. It wasn't long before I found myself wanting to automate the generation of the anchor tag. So I immediately ran off to my Ruby editor and generated a little script:

require "erb"
include ERB::Util

title, description, url, topic = ARGV[0], ARGV[1], ARGV[2], ARGV[3]
puts <<EOS
<a href="http://adigg.com/submit?phase=2
&url=#{url_encode(url)}
&title=#{url_encode(title)}
&bodytext=#{url_encode(description)}
&topic=#{url_encode(topic)}">
<img src="http://digg.com/img/badges/91x17-digg-button.gif"
width="91" height="17" alt="Digg!" border="0" />
</a>
EOS

That worked reasonably well, but then I started thinking back to all the stuff I learned about Mac OS X software development with Cocoa and decided that I should really wrap up my cute little Ruby script with a Cocoa GUI!

So I immediately opened XCode, generated a Cocoa application, and started messing around with Interface Builder. Here's what I came up with:



From Interface Builder I stubbed out my controller class (MVC after all!) and wired the button to an action method. I also decided to use Cocoa Bindings to wire the text fields to NSString properties in my model without any code. Now since my model only needed to expose four string properties (one for each field in my window) I chose to cheat a bit and just added the four properties to the controller.

Since Objective-C doesn't have automatic memory management, you typically manage the memory allocation and deallocation in the accessors. Fortunately Accessorizor came to my rescue and generated that code for me. So the only thing left for me to code was to pass my four text values to my Ruby script. A simple Google search resulted in this page on Cocoa Dev Central that describes how to call the 'ls' unix command from Cocoa. I quickly adapted it for calling my Ruby script. The one change I made was that I added my Ruby script to my XCode project as a resource so that it would be included in the application bundle at build time. So I added this line for finding it:

NSString *path = [[NSBundle mainBundle] pathForResource:@"digg" ofType:@"rb"];

Finally I decided it would be nice to just put the generated anchor tag on the clipboard when I click on the "Digg It" button. So I perused my "Cocoa Programming For Mac OS X" book and added these three lines:

NSPasteboard *pb = [NSPasteboard generalPasteboard];
[pb declareTypes:[NSArray arrayWithObject:NSStringPboardType] owner:self];
[pb setString:string forType:NSStringPboardType];

and voila:

Ruby User Group Meeting

Last night I attended my first Ruby user group meeting. I've been a member of a lot of user groups and so my expectations were (perhaps unreasonably) high. My initial impressions weren't that great and I left the meeting unsure whether or not I'd return the following month.

But then I started to think about it:

1. Here were a bunch of geeks (me included) getting together to talk about a nascent technology in a city largely dominated by Java and .NET. The fact that these people cared enough about fostering the growth of the language and framework to set all this up is commendable.
   
2. The presentation was actually quite good. I came out of there knowing something about Capistrano that I didn't know before I went in there. It inspired me enough to go looking at the Capistrano and Mongrel websites.
   
3. The group was roughly 20-25 people. Not bad when you consider very few of us are actually getting paid for writing Ruby.
   
4. There were door prizes (a couple books from O'Reilly and no, I didn't win). The people in charge are trying to entice people to come and learn and share.

So I reconsidered. I'll probably continue to attend and hope to see the technology and user group flourish. Only with growth (and perhaps some corporate sponsorship) can we eliminate poor first impressions like the one I had. In the interim, I'll continue to learn, make contacts, and maybe if I learn enough I'll be able to give a presentation one day too.

JRuby on Rails

It was only a week ago I was waxing philosophically about Enterprise Software and mentioned that Ruby was in the early stages of running on the JVM and .NET CLR and just today Sun announced that they're hiring the JRuby developers to continue their work full time at Sun. Wow! That's big news. Maybe mainstream Ruby on Rails development won't be far behind... Ahh the winds of change.

P.S. I tried JRuby a few weeks ago after Rails went to 1.1.6. The Gem system actually worked to fetch rails and I was able to create a skeleton rails app and start webbrick. It didn't work much more than that (which was still very impressive) and it was definitely slow but that proof of concept gives my high hopes for this project.

What is Enterprise Software?

What is "enterprise software"? And the follow up question what is an "enterprise software developer"?

The reason I ask is that I've been thinking about the Java vs Rails question again and the reaction I get from my fellow Java developers when I talk about Ruby and Rails. Generally, they turn their noses up. Ruby doesn't have the breadth they say, it's immature, it's slow, it doesn't scale, it's not an "enterprise" solution. So what exactly are they talking about?

"Breadth": Java has it and Ruby doesn't. There I said it. And quite frankly I agree with it. If I need some oddball service in my app, I can probably find an open source Java framework to help me out. I may not be so lucky in Ruby. But on the other hand do I care? Not really. If I'm building a web app (which describes nearly every single Java project I've worked on) then I don't really give a hoot about most of the stuff that's in Java. As a matter of fact Ruby's ability to create domain specific languages (DSLs) is the power that Rails exploits in its framework, allowing me to express the complexities of my projects very succinctly. It's not about writing gobs of code it's about efficiently expressing a maintainable solution to a problem in as few lines of code as possible. And this is where Ruby and Rails succeeds.

"Ruby/Rails is immature/slow": While it's true that Ruby's been around a long time it certainly hasn't had the commercial success that Java enjoyed and therefore it hasn't received the research dollars to improve it's language runtime. However Rails is getting a lot of attention and by extension so is Ruby. Ruby is in the early stages of running within the .NET CLR and Java Virtual Machine and by the sounds of things Ruby 2.0 has some virtual machine tricks of its own. So at that point at least the discussion of performance can be evenly leveled at all the virtual machines of the world.

Immaturity has also been expressed to me in terms of a lack of an IDE. While I lament the inability to easily refactor my Ruby code as easily as I refactor my Java code with the assistance of something like Eclipse, I console myself with the knowledge that refactoring tools in the Java world are relatively new too. It wasn't that many years ago that I was doing this the hard way in Java. Personally I think this particular complaint is pretty weak. I think developers who are clutching on to this one are doing themselves a real disservice. If nothing else they are depriving themselves of a fresh perspective to object relational mapping and web development because they don't have all the fancy tools.

"Scalability": Well that's a tough one to tackle because of the differences in strategy between Java and Rails. Java likes distributed shared caching (think EJB). Rails likes a Shared Nothing Architecture (SNA). So which is better? Well both have their successes. But to me, the SNA architecture is simpler to understand, simpler to implement and well-proven by the Googles of the world. Fewer moving parts certainly attracts me. There's less to go wrong.

"Enterprisy": Well I dunno how to address that one at all. Because first of all, I don't know what the term "Enterprise" means. I've developed a lot of applications for a lot of users who believe their apps are critical to the success of their business. So does that mean I've developed "enterprise" applications and as a result am an "enterprise developer". Maybe? So am I qualified to say whether or not Rails is ready for the enterprise? Maybe... maybe not. But in any case I can say that I think the IT shops I work for could definitely benefit from Ruby and Rails. I've seen a lot of crazy Java solutions. They tend to be overly-complicated and inconsistently designed and constructed. There is way to much configuration, way too much thinking about "what-if" and not enough thinking about keeping it DRY. Rails comes with a lot of opinions about how things ought to be done and if you can just accept them and go with the flow, you can be very productive.

But it's funny, the "enterprises" that could benefit the most from this technology are probably the people who will be the last to adopt it. Enterprises are big and difficult to steer. Once they get some momentum in one direction, it's hard to get them off that path. Take for example my latest attempt to move from Java 1.4 to Java 1.5. You wouldn't believe the number of departments and the amount of bureaucracy I have to cut through to try and get this approved. All I want to do is use the latest version of Java that was released two years ago so I can take advantage of it's language features so I can write cleaner, more maintainable code. What are the chances that this particular enterprise will ever use Rails? And they claim they're an agile development shop too. LOL.

Watch that SQL!

Long before I used object relational mapping tools like Hibernate or Toplink in Java or ActiveRecord in Ruby on Rails I used to write a fair amount of SQL (some real nasty queries too!). And in a lot of the shops where I worked it was standard practice to create an explain plan just to make sure that the database could execute the SQL efficiently. If it couldn't, I would dutifully restructure the query, add indexes, or go ask for some DBA help.

But nowadays in this wonderful land of ORMs people just let the framework crank out the queries and never ever look at them. To me, that's just plain wrong. If you're using Hibernate turn on the logging. If you're using Rails just go take a look at the development.log file and go watch what's going on! You may be surprised.

I started a new job a few weeks ago and just wanted to see what Hibernate was doing. And lo and behold, a JSP page was causing 69 queries to run in order to generate a simple list of rows from a table. The main problem was that every lazily instantiated relationship for each object in the collection was being traversed after the initial fetch.

So what could have been done? Well in this case, Hibernate has several options:

1. Hibernate's query language is quite expressive and something like select e from Employee e left outer join fetch e.department d would have caused both objects to be fetched in a single query.
2. The fetch keyword I used in the first example has one limitation (for good reason too). It can only be used to fetch one collection. So take advantage of Hibernate's caching to prefetch related objects instead. Hibernate will associate any newly instantiated objects with objects that already exist in cache. Even if you're not using something like EHCache you still have some caching built into your session so don't discount this tip right away.
   
3. Use a DTO-like object and go old-school and only select the fields you need with something like: select new EmployeeListDTO( e.firstName, e.lastName, d.name) from Employee e left outer join e.department d

Now if you're using ActiveRecord you don't have the double edged sword of a cache so the second option I described above isn't available, but the other two are possibilities:

1. Employee.find :all :include => :department
2. Employee.find_by_sql "select e.first_name, e.last_name, d.name from employees e left outer join departments d on e.department_id = d.id"
   

The second option shows one distinct design difference between Hibernate and ActiveRecord. Hibernate gives you a SQL-like query language but does away with having to specify join constraints because that information is in the mapping metadata. Whereas ActiveRecord says why reinvent SQL and just let's you have at it. Oddly enough I like both approaches...

Finally, be sure to use the ad-hoc query tools at your disposal to try out your queries before you embed them in your app. If you're using Hibernate 2.x then go get a copy of Hibernate Console (part of the Hibernate Tools). It can be a bit of a pain to get it setup but I really appreciate being able to work out the query before running it as part of my app. If you're using Hibernate 3.x and Eclipse then go get the Hibernate Tools plugin. If you're using Ruby on Rails then you already have all the tools you need. Simply drop to the command line, cd to your project directory and then type script/console. Open up a second shell window (if you're running unix) and type tail -f log/development.log from your project directory and you can instantly watch what SQL Rails generates when you execute those find methods.

Now go out there and watch what you're doing for database access! Don't let all that ORM goodness make you lazy.