So why do you do it?

That was the question a member on my team asked me when I said "even though our team has a bright Ruby and Rails future it doesn't seem like anyone else in Calgary does".

You see he's not a Java developer or a Ruby/Rails developer but rather a data warehousing/ETL expert, and after hearing us rave about all things Rails he thought it might be a good idea to pick it up too. I didn't want to discourage his exploration but I also didn't want to give him the impression that in these tough economic times that being a Rails guru was really going to turn him into a hot commodity. That's plainly not true. In Calgary, it seems like the wind is in the sails of the Microsoft .NET community.

And while I don't despise C# (or Java for that matter) I don't have the affection for those technologies that I do for Ruby and Rails so his question threw me off. And he has a point, if it's not going to improve my marketability as a developer outside of my current contract then why in the world would I choose to develop non-mainstream skills?

Well for me the answer is easy. I've worked with Java for about ten years and done the odd C# project or two so I feel pretty confident in my abilities to use either one if I have to (C# just isn't that different from Java). But that's the point. My current environment doesn't make me use those technologies so I choose not to because I know I can be more productive with Ruby and Rails. In other words I can provide more value to my client for the same amount of money they're spending to have me sit there. I can deliver software to my users faster than I can with the other mainstream choices, and I can do it with less code that is better tested and I think more maintainable. And although I know there are developers who like to squeeze every billable hour they can from their client, I prefer to deliver so that my users have a positive experience. My theory is that the positive experience will make them think of me the next time a potential project rolls around.

That's why I do it! And oh yeah, I love it too. :-)

Ruby on Rails Makes Me Happy

Last year I got to work on my first Enterprise Rails application and although there were some brief moments of coolness, that project was punctuated with a lot of things that just made it suck overall. In fact, I was doubting that bringing Rails to the enterprise would be a good thing. There were just to many other things wrong that had nothing to do with the technology.

But half a year later our enthusiasm for Ruby on Rails hasn't been seriously diminished despite a less than stellar project, and surprisingly our commitment is growing. From three developers doing Rails we've grown it to five and my managers have fully accepted it and are pushing it forward for most new projects.

Excitingly for me, I was recently assigned a small but very high profile project. And this time I think I've been able to do things the best way I know how in a very short period of time. The controllers are RESTful, my routes are appropriately nested, the layouts are clean table-free HTML with CSS and the Javascript is unobtrusive. I really dig Prototype and the app is sprinkled with just the right amount of AJAX and the Scriptaculous effects are not gratuitous.

Quite honestly I'm pretty thrilled with this app. It's only been a week or two worth of effort but it's been thoughtful and extremely efficient. So efficient that I think the app will be fully functional before the TQA and Production Tomcat servers I requested will be provisioned by the server team (remember I'm in a big enterprise). And yes, you read that right I'm deploying to a JEE web container. I'm past any trepidation I may have once harboured about JRuby vs MRI Ruby. To me it's a proven part of my deployment strategy that I can depend on.

Anyway I'm a happy developer again. I'm getting apps written and deployed for my users and I love the look of my code. The cool thing is that Rails keeps getting better. Rails 2.3 is just around the corner. I like the application templates and the integration of Rack. I still have to bring up my testing game but give me another week and I think my tools of choice (RSpec, Remarkable, and FactoryGirl) will have me cranking out better test coverage than I've ever had before.

Anyway let me wrap this up with some of the things I think are wicked about Ruby and Rails and the various gems:

• pluralize
• belongs_to, has_many :through (declarative ORM)
• Model.all
• Model.paginate (will_paginate gem is awesome)
• link_to_if
• select, collect, each (collection methods are cool)
• statement if expression (statment modifiers rock)
• distance_of_time_in_words_to_now
• model_url
• before_update, before_delete
• validates_presence_of, validates_length_of (declarative validation is sweet)
• Ruby, a REAL language, in my view templates!
• jruby script/console oracle (interacting with Oracle through my models via JDBC)
• rake war:clean war (Thank you warbler)
• script/autospec (even better on my Mac when integrated with Growl)
• ActionMailer with layouts (my HTML emails never looked so good)
• Ajax.Request, Ajax.Updater (Yay Prototype)
• case (yeah just a nice simple case statement)
• auto_discovery_link_tag and builder (yeah I added an RSS feed for good measure!)
• Railscasts (an exemplary example of a great community, Ryan Bates you rock!)

JRuby & Rails 2.2

We just started converting things over to Rails 2.2.2 and I thought I should say that the one feature that really kicks some ass is its thread safety. JRuby 1.1.5 (very soon to be 1.1.6), which takes advantage of Java's native threads, combined with this latest version of Rails finally makes for a very good deployment story.

Rails' traditional single threaded model, while conceptually nice for development, never really made things easy for deployment. In MRI-land, you always needed multiple processes, translating into multiple instances of Mongrel. In JRuby we needed multiple instances of the JRuby runtime. While it worked, it was definitely a bit of a memory pig. With Rails 2.2 we can now run with only one JRuby runtime in a JEE web container that was already multi-threaded. Now memory consumption is several fold smaller than what it was before and we now have a much better story to tell everyone. Thanks to everyone on the JRuby and Rails teams!

Now, I shouldn't just end with a big hurrah to everyone without mentioning that we did run into a problem with some thread unsafe code in our own app. It took awhile to find it (a week) so the one word of caution is be careful about any shared state in your application (in our case it was a rather obvious global variable that we seemed to keep overlooking).

In any case, Rails and JRuby are a great match for enterprise web app development. I've been hesitant to say that in the past because of a few rough edges like the one I mentioned before but now I think it's about time everyone should look at this fantastic combination.

UPDATE:

Thought I'd add a link to a recent Ruby shootout. It's reporting on the performance of JRuby vs other Ruby implementations and the importance of threading.

Heroku, RubyGems and Git

My daughter and I started a little Ruby on Rails project and I thought it'd be nice to host it somewhere for her to play with. I have an account on Heroku that I haven't used in months and thought it'd be nice to try it out again. I signed back in (after resetting my forgotten password) and was impressed to discover that they've integrated Git. After creating the project and navigating to the Rails start page I found some instructions:

gem install heroku
heroku clone stuffies
cd stuffies
[..local edits..]
git add .
git commit -m "local changes"
git push

So I ran off and installed the heroku gem. But had trouble with step 2. Fortunately a couple of google searches had me back up and running. One to deal with "Permission denied (publickey)" upon heroku clone. And another to figure out how to create the RSA key. Now I have a local Git clone of my project that I can work on in NetBeans and then I can quickly publish the changes to Heroku with a commit and a push. Sweeeet!!!

Calling Java from JRuby to do the dirty work

I recently had to try and read a UTF-16LE-BOM encoded text file from Ruby. I couldn't figure out how to get Ruby to deal with the double byte characters in the file. But I did know that Java has very good UTF support so I decided to let Java do the heavy lifting for me. I ended up with something like this:

require 'java'

import java.io.InputStreamReader
import java.io.ByteArrayInputStream
import java.io.BufferedReader

data = ''
File.open("text.txt", 'rb'){|f| data = f.read}

#strip the BOM (Byte Order Marker)
data.slice!(0..1)
#let Java deal with the UTF-16 encoding
reader = BufferedReader.new(
 InputStreamReader.new(
   ByteArrayInputStream.new(data.to_java_bytes), 'UTF-16LE'))

while ((s = reader.read_line) != nil)
 puts s
end

It may not be the perfect solution (I'd like to see a regular Ruby solution) but I suppose leveraging the Java libraries was the whole point of JRuby in the first place wasn't it!

(Executing File.open("text.txt", 'rb'){|f| puts f.read} looked fine on my Mac, except for the BOM, but looked terrible in the console on Windows. The solution above actually converts the text from double byte to single byte characters)

Really Achieving Your Childhood Dreams

By Randy Pausch. Great lecture from Carnegie Mellon's School of Computer Science. Here's the iTunes link and YouTube as well. Watch out for the head fake. Great stuff.

Lessons from my first Rails project

I’m finally being paid for working on an “Enterprise” Ruby on Rails pilot project. I think I helped persuade our IT group to give Rails a try. But unfortunately I wasn’t available to help the project get started. Instead I joined the project several months in. So while it’s cool to finally be doing Ruby and Rails full-time it’s also pretty painful to be living with some of the poor decisions made before I got here. I can’t say that if I’d been involved from the get-go that we would have avoided some of these problems (sometimes it runs deeper than the technology) but what I can offer is a list of things I’d suggest you watch out for (in no particular order):

Be realistic
In the early days of Rails I remember people saying things that made it sound that Rails was ten times more productive than Java. That’s a wild exaggeration and everybody should know it. There’s no way you’re going to be 10X more productive, end of story. Hope for “10% more productive” after you’ve got one or two projects under your belt. At least that’s realistic and achievable and if you get more than that then great! But for the love of God don’t start off with that idea. A big part of the software game is managing expectations. Don’t dig yourself a hole you have to try and climb out of. Despite what you read on the web, your first Rails project will probably take you more time, not less. New language, new tools, new framework, new infrastructure, new bugs, and new deployment problems. My project started with unrealistic expectations and now we’re over budget and over schedule. That’s not a great way to get management on your side.

Deploy early and often
That’s an agile thing and not unique to Rails, but getting over the hurdles of deployment shouldn’t be taken for granted. We’re primarily a Java shop so when people saw JRuby and WAR deployment, they figured everything would be easy-peasy. Not so. Part of being “Enterprise” is that quite often you’re several months, if not years, behind the technical curve. The first thing we needed was to get past Java 1.4 and an old version of Weblogic. We needed at least Java 6 and Tomcat 6. Trust me, getting “comfortable” IT infrastructure people to upgrade their technology can be very painful and slow.

Now assuming you got the basic app server running on a reasonably recent version of Java, go and figure out how to build the WAR (warbler gem), get to Oracle (activerecord-jdbc gem) through JNDI, get your Rails log files out of the exploded war directory (logging gem), integrate with ActiveDirectory for Authentication and you’ve got your hands full of a bunch of other problems that will eat away at your time. Budget for it.

Now once you got all that, deploy often! Our project has periodically deployed something outside of our dev environment but not frequently enough to gather any kind of momentum to satisfy our users. Without gaining the users’ trust you get into a whole other set of problems.

Unit test (RSpec seems pretty awesome)
Sounds like a reasonable thing to do... but we didn’t. I won’t go into all the reasons why testing is good, but to fail to take advantage of the testing infrastructure that Rails gives you out of the box is pretty ridiculous. Even if all you’re doing is executing the code and never asserting the results are correct at least you’re executing the code and will find runtime problems that a language like Java would have given you at compile time. I started using RSpec for my stuff and I really like it, but it sucks when the other developers won’t add to the tests or even run yours. But my strategy is to lead by example. Maybe sooner or later the other will pick it up too. If you can get buy-in early and be ruthless about keeping your testing effort on track.

Figure out the non-standard Rails stuff
Beyond the deployment stuff I talked about earlier, figure out the other places you must configure rather than adopting the Rails conventions. For example, we have a rule that says you must access the database from an account other than the schema owner. That means we needed to call set_table_name(‘schema.table’) in all of our models. We also needed to create the permissions for the non-schema owner. And don’t forget proper referential integrity constraints in your migrations. These are all things you can ignore for a while but if you have any hope of putting you application in your users’ hands you’re going to have to figure it out. Personally, I hate surprises like this late in the project. My recommendation is to sort them out early.

Don’t underestimate security (a.k.a., read the requirements doc)
We have users who don’t trust anyone and have dreamed up one of the most elaborate security mechanisms you can think of. Unfortunately we went through five months of development and not one of the developers before me even seriously looked at it. Sadly, I can understand why. Every Java developer (including me) thinks they got that one figured out and just skips that part of the requirements doc. Except that someone should have actually read ours and said Woah! It’s not always easy to retrofit something like this if you just assumed it was an orthogonal concern. If someone wrote a requirements document for you (no matter how badly written) be sure to read through the whole thing. You never know what problems may be lurking.

Read ‘The Rails Way’
In general, get some training. Or in lieu of that, get some good books. I bought “The Rails Way” by Obie Fernandez (great book btw) after I got assigned to the project. To my way of thinking the project should have already bought a whole bunch of Rails books and made them available to the developers. But it’s up to you to be prepared. If there are parts of the framework that are still a bit fuzzy start reading some books (go to the source if you need to).

Evaluate your editor
So like I said before, our shop is primarily a Java shop, so when I got on the project people were using IntelliJ to edit their code. Now IntelliJ may be a kick-ass Java IDE but from what I’ve seen it’s not great for doing Rails. So when I started, I used NetBeans. Now NetBeans has never in my mind been a great Java IDE, and quite frankly it’s not my dream Ruby IDE either but Sun hired the JRuby dudes and Tor Norbye, the principal NetBeans guy at Sun seems to understand Ruby and the needs of Rails developers pretty well. As near as I can tell, unless your a Mac-toting Textmate user, NetBeans is the best game in town for Rails development (I even use it on my Mac). So my recommendations is to get off your butt and seriously evaluate the tools you’re using. Maybe IntelliJ is the right thing for you, but you don’t really know unless you look around.

Use an issue tracking system
We didn’t have an issue tracking system when I started on this project. So not only could we not tell how far we were by test/spec status, we weren’t even tracking the change requests and bugs properly. Wow. In my mind this is part of the scope problem we’re having. The users just ask for something and without visibility to the whole team and management, it just gets magically added and the schedule slips.

Web 1.0
AJAX is pretty attractive to a Web 1.0 developer. But food is awfully attractive to a starving man too, but that doesn’t mean you should let him eat himself to death. Take it slowly. IMHO, do it old school on your first (and maybe your second or third) Rails project. Throw in the odd autocompleting field if you want but this AJAX stuff and all the pretty visual effects can get out of hand pretty quickly.

Use people who want it to succeed
Java and .NET are pretty typical technologies in IT and unfortunately a lot of the developers who’ve invested a lot of time learning this stuff seem to forget that the technology is only a means to an end and not the end itself. They’ll be threatened by new technology. They’ll be threatened by Ruby and Rails. So if you can, staff your project with the developers who are open minded and wouldn’t mind if a Rails application succeeded. If you push these hostile developers onto a Rails project their negativity will bring everyone else down.

Understand RESTful Rails
I don’t remember when exactly David Heinemeier Hansson really started pushing the idea of a resourceful/RESTful way to develop web applications with Rails, but it’s damn good stuff and you should really figure out what it all means before you start building a whole bunch of stuff the old way. Figure out Rails’ routing system and what map.resources does and then you’ll be able to go with the flow and let Rails do a lot more of the work for you. I’ve seen more than one developer write a lot more code than they need to.

Separate HTML, CSS, & Javascript
I can’t believe these so called IT web developers who still use all the old FONT, BORDER, WIDTH, ONCLICK etc. HTML elements in their templates. C’mon! HTML is mark-up, CSS is styling and JS is behaviour. Separate these aspects of your view layer the same way you separate models, views, and controllers. I’ll make some allowances for surgical uses of RJS but if you want things to be maintainable sort this out already. (I know this isn’t just about Rails but it bugs me so much I figured I’d rant about it anyway ;-))

If you’re about to embark on a Rails project, get ready to enjoy yourself, just don’t lose your head in all the hype and follow some good common sense.

My first JRuby contribution

I've reported lots of bugs before but this is the first time I submitted a fix back to an open source project.  The fix was to the JRuby-Extras ActiveRecord-JDBC project to allow you to do something like this for a Rails model class against Oracle:  

class Person

  set_table_name "department.people"

end

Accessing the database from someone other than the schema owner and therefore fully qualifying the table with schema is a rule I needed follow in order to deploy a webapp in my current IT department.  

I posted the original fix quite a while ago but nobody really took notice. Then Jesse Hu came along and made it better (and probably independently I might add) and also added a couple other fixes for things I hadn't noticed.  

But unfortunately neither one of us seemed to really know how to submit the fix so it got ignored for awhile.  Originally I created a patch and just pasted the text into the body of my comment.  Jesse attached the entire Java file.  A few days ago when Thomas Enebo asked for things we'd like to see fixed in the next release of JRuby I jumped at the chance.  After I got a hint about what to do from Charles Nutter, I got the latest version of the project from SVN, created a .patch file from Jesse's source, and then submitted it back to the JRuby-Extras Tracker and then finally back to JRuby's JIRA issue.

Today I got notice from Nick Sieger that it'll be in the next version.  Very cool.

Unfortunately I have to admit that I still a need a bit of work on my testing skills because I wasn't able to figure out where or how to insert some unit tests to prove that my fix worked.  Something to save for the next time. I'm just happy I can finally get rid of my custom version of this library.

Rails on Windows

I've been working on a Rails project at work for the last few weeks.  We've been targeting JRuby for deployment and therefore I've been working primarily in JRuby for development.  This week I thought it'd be nice to get running with the regular MRI with MySQL instead of Derby.  I installed cygwin and tried to get back to doing things at the command line (I've been using NetBeans)...  I was getting tired of the performance hit I was taking every time I ran a rake task or a generator or started up Mongrel.  I was under the impression that the MRI ran as well on Windows as it does on my Mac.  Not even close.  Performance still sucks and managing the differences between my JRuby installation and my cygwin hosted Ruby isn't much fun.  I'm about ready to bail and just go back to JRuby.  Now I know why people buy Macs to do Rails development.  It rocks.  Rails development on Windows on the other hand truly sucks.  Thank goodness JRuby and NetBeans are there to hand me a few niceties.  Otherwise I'm not sure I'd have the patience for it.

The iPhone SDK

I don't own an iPhone and since I'm in Canada I can't even buy an iPhone from Apple even if I wanted to, but that hasn't tempered my enthusiasm for the new iPhone SDK.  Like so many thousands of others I downloaded the SDK as soon it was available, installed it, and then started coding away.  I'm sure more experienced Cocoa developers were able to quickly bang out something cool, but not me.  It took me a bit longer to get the mandatory "Hello World" running, but nonetheless, my previous Objective-C/Cocoa investigations definitely came in handy.  I think one of my mistakes was thinking that the videos Apple put up would help me.  IMHO, don't bother.  Go right for the developer documentation in XCode instead.

Alas, I can't really say that much about the SDK.  I clicked through one of those license agreements and I'm sure that there's something in there that forbids me from revealing anything too juicy.  But I think it suffices to say that I'm still enthusiastic about where this is going.  I think the iPhone (even though I don't have one yet) is a great bit of technology and once in the hands of people in the enterprise who suffer at the other craptaculous mobile devices out there, they'll definitely start demanding stuff.  And that's where I fit in.  I'm excited to see how this all evolves and hope I'm ahead of the curve at least momentarily. ;-)

Javascripting with ExtJS

Back in the early days of my work in Java I adopted the typical Java developer's bias against Javascript. Whenever someone would mention Javascript I would cringe. Afterall, why would I choose Javascript when I had Java? But I always felt I was missing something.

So last year I decided to do something about it and integrated the Prototype Javascript library into my Struts application.  I quickly learned that I didn't hate Javascript but hated the inconsistencies in the browser DOMs and learned to properly direct my hate  (Internet Explorer, I'm looking at you!).  

Despite a couple hiccups, that project was really quite enlightening and I went on to expand my knowledge with a great series of videos at the YUI Theatre. Douglas Crockford has three series of videos that I highly recommend to anyone wanting to learn JS. He goes over the history of Javascript and plainly spells out Javascript's problems and strengths (e.g., it supports functions as first class objects with support for closure, hashes are all powerful, prototypical inheritance is different than classical inheritance but is quite workable, and be careful of the default global namespace).

So now armed with a proper understanding of the language, I started a building new web application in January and realized that there were requirement that would be best served by AJAX and some nice browser widgets.  In enters ExtJS. After a couple weeks writing a spike through Struts and ExtJS (still using Prototype underneath), I made the choice to adopt ExtJS as a the library for my application's user interface.  Two months later, I'm more than satisfied with the choice.  So are my users.  

In the end, I've shed a bias I really shouldn't have been harbouring, I embraced a "new" language and added the ability to better server my users.  I still won't choose Javascript for a lot of things but I'm much happier knowing that I can properly apply it as part of an overall solution.

JRuby and Hibernate

The beautiful thing about Rails on JRuby is that you have access to Java. Perhaps you have existing code you'd like to reuse or maybe there's some functionality in a Java library you'd like to use. Whatever the reason, it's good to know how to do it.

So today I created a new Rails app to demonstrate how to use Rails' controllers and views with some model classes from one of my earlier Java projects. My old app used the typical HibernateUtil singleton (section 1.2.5 of the Hibernate docs). I also liked the ActiveRecord pattern so my Java domain classes had some handy "find" methods. So in order to make this work in a Rails app, I installed the goldspike plugin:

script/plugin install http://jruby-extras.rubyforge.org/svn/trunk/rails-integration/plugins/goldspike

and then ran the rake task war:standalone:create

I copied the classes and lib directories from my old apps' WEB-INF directory to the WEB-INF directory in my Rails app and then generated a controller and modified it to look like this:

require 'java'
import 'com.example.domain.Person'
import 'com.example.util.HibernateUtil'

class PersonController < ApplicationController
  def list
    with_transaction do
      @people = Person.find_all.to_a
      @people.sort!{|p1, p2| p1.name.downcase <=> p2.name.downcase}
    end
  end
  
  def with_transaction
    begin
      tx = HibernateUtil.session_factory.current_session.beginTransaction
      yield
      tx.commit
      HibernateUtil.session_factory.current_session.close
    rescue
      tx.rollback
    end
  end
end

Finally, I added a list.rhtml template for the "list" action and voila! I have a Rails-ish app that uses Hibernate instead of ActiveRecord.

Ruby Cocoa

RubyCocoa and the Scripting Bridge are two software development technologies bundled with Leopard. The app presented here demonstrate how to create a native Mac interface written in Ruby that communicates with iTunes via Applescript. It does nothing more than display the iTunes version number.

1. Create a "Ruby-Cocoa Application" from the "New Project" dialog in XCode
2. Add a "Ruby NSObject subclass" to the project called "ITunesController"
3. Add an ib_outlet for a text field and an ib_action called "show_version" to the ITunesController ruby class.
4. Generate an Objective-C header file that scripting bridge can use to communicate with iTunes. Use the two terminal command, sdef and sdp, within your project folder like this: "sdef /Applications/iTunes.app | sdp -fh --basename iTunes"
5. Add the header file to your project.
6. Add "ScriptingBridge.framework" to your project.
7. Open "MainMenu.nib" in Interface Builder and add a button and a label to the Window. Also instantiate an NSObject subclass.
8. Set the NSObject's class to "ITunesController". (You may need to select the "Read Class Files..." menu item to get ITunesController to show up as a valid option)
9. Wire the button to your controller's "show_version" method and wire the controller's text_field outlet to the label.

Here's what my project looks like:


Here's what my Ruby "ITunesController" class looks like:

require 'osx/cocoa'

include OSX
class ITunesController < NSObject
    ib_outlet :text_field
    def show_version(sender)
        iTunes = SBApplication.applicationWithBundleIdentifier:'com.apple.iTunes'
        @text_field.setStringValue("iTunes version: #{iTunes.version}")
    end
    ib_action :show_version
end

Here's what the app looks like:


Now you should be able to run the project and get the iTunes version number displayed in your own application. Yeah, the end result is kind of lame but the journey was pretty cool. The interesting bits for me were:

1. My Ruby class extends Cocoa's NSObject class.
2. The class methods ib_outlet and ib_action are parts of the Ruby DSL for doing Cocoa development with Ruby.
3. The RubyCocoa framework makes it really simple to swap out Objective-C for Ruby when doing Cocoa development.
4. Everything else is the same. I still use XCode for my controllers and model and still use Interface Builder to graphically build my GUI and bind it to my code. The only difference is that I get to use Ruby and all of its beauty to get the job done.
5. The downside is that I don't get anything too useful in the way of code completion while in XCode. Still might be worth it if you really dig Ruby.

Yahoo Address Book & Leopard

Leopard's new address book allows you to sync with your Yahoo! contacts.  It's not intuitive so here goes:

1. Open the general tab of Leopard's Address Book preferences.
2. Check "Synchronize with Yahoo!"
3. Click on the Configure Button and enter your Yahoo! ID and Password
4. Open iSync's Preferences (here's where it gets weird)
5. Check "Show status in menu bar"
6. Select "Sync Now" from the new menu bar icon on the right
7. Enjoy!

Of course you should back up everything before you try this.  I've been using Yahoo! Mail for years so it's my most correct and complete contact list.  I'm glad I can now benefit from this in Leopard.  Now if Yahoo! would just be as good as Google and give me IMAP support for free or if Apple would give me a GMail Address Book Sync, I'd be able to use Apple's mail client most of the time.

Now that I think about it, it'd also be nice to have iCal sync up with either Yahoo's or Google's calendar too!

Leopard Installed

It's my second day running with Leopard and just about everything has gone smoothly.  I installed it without too much trouble on my MacBook and my PowerMac G5.  I say not too much trouble because not everything was perfect.  

Twice since installing Leopard on my G5 it's not gone to sleep on demand.  Instead it goes into a coma and starts snoring.  By that I mean all the USB devices shut off, the video signal goes away but the computer doesn't quite sleep.  Instead, all the fans start to spin up. For anyone that's never heard a G5's fans go nuts... it's like a small jet engine.  The only recourse is a hard restart.  I installed the first Leopard update and it hasn't repeated this stunt but I'm keeping an eye on it just in case.

The second thing is that MySQL won't start.  I've read a few things on the Internet and it just doesn't look the MySQL project is ready for Leopard. They had bug reports before 10.5 shipped but decided to just wait until it went gold. I guess I'll just have to patiently wait for an update.  No biggie.  I'll just use the built-in SQLite instead.

The third thing that I wasn't prepared for was the retirement of Classic OS 9 support.  I don't personally use any OS 9 apps  but my kids used to play the occasional game.  Sorry Rolly McFly.  Your days are over.

The last thing that isn't too big of a problem but a big disappointment is that the Java 6 SDK is MIA.  I guess I never really saw official mention of it, but I really expected Java 6 to ship with Leopard.  C'mon Apple!  I was there when Steve Jobs and Avie Tevanian stood up at JavaOne and said they'd have the best Java implementation on any platform.  You're definitely letting me down on this one.

Now onto the good news.  Everything else just works!  Mail is cool.  Safari is blazingly fast.  Dropped network connections don't mean a frozen finder anymore. Quick Look is sweet.  And Ruby is built-in.  I just deleted my old custom Ruby installation and everything (other than MySQL) continues to work.  My IDEs function properly (Eclipse & NetBeans) and despite the lack of Java 6, Java 5 on my Mac is still noticeably faster than Java 5 on Windows at work.  I don't mind the translucent menu bar and Cover Flow in the finder is nice but mostly unnecessary for me.  

The biggest shock to the system was the developer tools.  I've built the occasional Mac app, and while I'm no wizard at it, I got along just fine thanks.  So naturally I wanted to try the Ruby/Cocoa integration.  Then I opened Interface Builder.  Yikes!  That certainly  got a major overhaul!  Looks better, but now I see that I need to spend some time trying to figure out what's what.

So if someone asks whether they should upgrade or not.  I'd definitely give it the thumbs up.  Just make sure you don't need classic and if you're a geek like me, be prepared for a little bit of adjustment to the changes under that spiffy new exterior.

JRuby and Rails in the Enterprise

At work, management is finally getting the message about dynamic languages like Ruby and starting to recognize that frameworks like Rails may have productivity benefits.  We're actually going to look at doing a pilot project.  I like to think that I've had some influence in that but you never know...

Anyway, I would love to make this work but this is a Java shop through and through.  We can only deploy to a Java web container, specifically Weblogic 8.1 (the transition to Weblogic 10 is happening but it's months away).  I've used JRuby (works great BTW) and the JRuby extras to help me deploy an app to Tomcat, but I can't manage to get JRuby and Rails working in Weblogic 8.1.  Any ideas?

Secondly, Groovy and Grails is the other contender.  I like the idea behind Groovy and Grails but I can't help feel that it's a pale imitation of Ruby and Rails.  Anybody have enough experience to compare and share?

Finally, I'm not naive enough to think that this will cure all the ills of working in the "big enterprise".  When I think about all the things that bug me, the technology is probably the smallest thing that causes me pain.  But on the other hand, this would be a wonderful distraction. :-)

Leopard

I think it's fair to say that I could be characterized as a Mac "fanboy" from time to time.  I've been known to spout pro-Mac statements occasionally.  I take some unfounded pride in the fact that Apple's market cap is now bigger than IBM or Intel. And I love the fact that Apple just had yet another record quarter....

Perhaps it's just the validation of what I've felt in all the years that I've used a Mac, it's just plain better than Windows!  So now I sit here waiting for my pre-order of Mac OS X - Leopard to get to my house.  It's supposed to arrive tomorrow but the status already changed to "shipped".  To fill some of my waiting time I watched Apple's guided tour. I read some of the reviews (evolutionary not revolutionary) but I still can't help but be excited.  It's like a little bit of Christmas in October.

I'm sure I'll be underwhelmed a bit once the glow of a new OS has worn off but for awhile I'll revel in my fanboyism.  So if you too are a bit excited about the next Mac OS release, roll over to AppleInsider and feel free to take part in the unbridled Mac love fest and read some of the "Road to Mac OS X Leopard" series. Some of them are quite well written and will give you a good perspective of where Apple and NeXT have been and how the features in Leopard came to be.  Ignore the fact that some of the statements are a little too biased toward Apple and just enjoy the moment...

Marching on...

I used to develop applications with PowerBuilder (PB) for about 8 years.  I started with PB 1.0.  In the beginning it was new and exciting.  Windows was new.  Client/Server was new and even object oriented development in a mainstream tool was relatively new (at least in the corporate software development scene).  But as technology progressed I began to see exciting things elsewhere.  At the time Delphi was the cool new product.  The speed, the beauty, the elegance... ahhh I was smitten.  But the other PB developers around me didn't see it.  I fought for Delphi and even did one or two project with it.  But unfortunately Delphi wouldn't become my mainstream development tool.  PB was entrenched and the developers around it protected it with a passion.  It was disappointing. 

Then Java came along.  To my eye Java shared a lot in common with Delphi.  I liked it too.  But still the PB developers around me couldn't seem to appreciate this new tool. In fact, a lot of them were actually pretty hostile to anything that might threaten the sacred cow that was PB.  Fortunately for me the hype of the Internet and the .COM boom turned Java into something people couldn't ignore.  And I decided to ride the wave and finally leave PowerBuilder behind.

Now another 8 years has gone by and I find myself in the middle of a community of Java developers who are extraordinarily similar to that crowd of PowerBuilder developers.  They only see Java.  They've built a career around it, they're entrenched and they protect it with a passion.  For me, I see the appeal of other languages and tools.  In particular Ruby is beautiful and Rails elegantly solves many of my Java web development headaches.  But many of the people around me don't see it and quite frankly I doubt that they've even looked.  And that's the disappointing thing.  They're just like the PB guys with the blinders on.  They're hostile to anything that isn't Java.

In my current group there seems to be a new love forming for GWT.  Personally, the Google Web Toolkit and its unmistakable Swing/AWT flavor of Web development just seems wrong.  It's definitely web development for Java developers and I can see why they like it, but to me it's an artificial abstraction that doesn't sit well with me. Ruby and Rails isn't perfect either, but even a couple years after I started looking at it I still think it's better than GWT.  Rails embraces the browser technology of HTML, CSS, and Javascript and makes it easy to work with.  GWT on the other hand, puts me behind a Java facade where I can pretend to be developing Swing and it will generate the HTML and Javascript for me.  Ick.  No thanks Google.

Anyway, my point is that I think I'm getting to another point in my career where I need to migrate to a new place with like-minded people.  I'm tired of trying to get others to see what I think is self-evident if only they'd look...

Java & Ruby HTTP Clients: Part 2

Several months ago I wrote an article about how to create an HTTP client in Java or Ruby. I included examples in both languages for getting by BASIC and FORM based authentication. I also showed you how to resubmit the value of an HTTP cookie that many websites use to store state.

What I didn't mention is that quite often web applications store state not in a cookie but within the HTML itself. In order for you to programmatically interact with the website, you'll need to get that data out of the HTML and put it in your next request.

Java

So the basic recipe is to to use HttpClient (the way I demonstrated last time) to get the raw HTML. Then feed that text into NekoHTML, an HTML parser. It can correct the various problems you see in old-school HTML, namely unbalanced tags, missing parent tags, and mismatched elements.

Neko returns a standard Document object, but personally, I don't care much for the standard XML API. Instead I prefer to use the simpler API of DOM4J. So the next step is to take the XML that NekoHTML provides and feed it into DOM4J so that I can use XPath expressions to find what I need.

Now the question becomes what data do you need to post. Well my answer is to simply look through the FORM you're trying to submit, and resubmit everything. Look for 'hidden' tags, input tags of type 'text', 'password', and 'select', gather the names and values of all those tags. Override the ones where you need to provide the information (like id and password for example) and then do the POST.

Ruby

The Ruby approach is exactly the same: get the raw HTML, parse it, use XPath to get the name-value pairs of the FORM elements, override some of the values, and resubmit. The only additional rubygem from my last article is hpricot. It provides the same functionality of NekoHTML and DOM4J in Java. The typical script might look like this:

require 'net/http'
require 'rubygems'
require 'hpricot'

res = Net::HTTP.new('myserver', 80)
# res.set_debug_output $stderr #uncomment this to get console debug info
res.start do |http|
#go to the first page
get = Net::HTTP::Get.new('/home.aspx')
response = http.request(get)
                                                    
#collect the cookie information
cookies = ''
response.response['set-cookie'].split(';').each{|c|
cookies += c.split(/path=.*?,/).last.strip + ';'
}

#collect the existing form data
doc = Hpricot(response.body)
form_data = {}
['text', 'password', 'hidden'].each{|t|
elements = doc.search("//form[@name='loginForm']//input[@type='#{t}']")
elements.each{ |e|
  form_data[e['name']] = e['value'].to_s
}
}

#override some of the values
form_data['username'] = 'my_username'
form_data['password'] = 'my_secret'

#login
post = Net::HTTP::Post.new('/login.aspx')
post.set_form_data(form_data)
post['Cookie'] = cookies
puts http.request(post)
end

In this case all I did was print out the resulting HTML. At the very least you'd probably do the hpricot thing one more time to retrieve the data in which you're interested.

Other

Finally one last gotcha... At least half the websites I've tried to scrape do something "interesting" with Javascript to set various form elements. Since we don't have a Javascript engine executing you should expect that you'll have to parse the HTML and Javascript yourself to figure out what's going on, and set the fields manually in your script. I highly recommend Firefox and the "Web Developer" and Firebug plugins for inspecting the HTML, JS files and the HTTP Requests that the browser submits.

P.S.: I was a bit lazy this time and didn't provide any Java code. If you're really having trouble and can't get it working, leave me a message and I'll put together an example. Secondly, there are other Java HTML parsers out there that may work just as well or maybe even better than Neko, but since I don't have any personal experience with them I didn't mention them. If you like something else, leave a message.

Aptana RADRails & JRuby

I thought I'd go and take a look at what's been happening on the RADRails project lately. As it turns out, quite a bit! Aptana has taken over the project and are making quite a bit a progress integrating it with their Eclipse based IDE. The most impressive thing I saw was refactoring support! Fantastic! It's great to see RADRails getting some attention again. Go download the beta.

Then I went over to the JRuby project to see what they've been up to. JRuby 0.9.9 is out and looks very good. The Java integration is quite good. But I've been mostly interested in seeing support for deploying a Rails app in a JEE Web Container. Well nestled in beside the JRuby TAR file is a file called sample-rails-warfile.tar.gz. I downloaded it and pretty quickly had a rails application running in Tomcat. The app doesn't do much of anything but it's still pretty impressive to see it. My hope is that this kind of integration will help to get Ruby & Rails adopted into some of Java-only environment I tend to work in. The next thing I did was build my own Rails app and deploy it in Tomcat. I also tried deploying it in WebLogic but didn't have as much luck. Obviously there's still some work to do. Check out the JRuby WIKI for details.