Just Say No to Specialization

One of the topics of our standup meeting this morning was the communication problems with the database group. At the eleventh hour they wanted to change the names of tables, columns and constraints. While I suppose there's nothing too much wrong with caring about consistent naming conventions, I question a lot of the specialization that happens in larger IT shops nowadays.

I read this quote on Apple's Pro site:

Where these big shops have tiers and tiers of people focused on something specific and get bogged down by their infrastructures, we can win a huge campaign with just two artists and a Mac.

...and I can't help but see the parallels with IT development. The last few shops I've worked in (with one exception) have had Business Analysts, System Analysts, Technical Architects, Senior Java Developers, Junior Java Developers, Product Managers, Software Project Managers, HTML developers, Art directors, DBAs, QA managers, QA testers, Release Engineers, etc. etc.

As a result, even the simplest project ends up involving at least 10 to 15 people. If you then consider the communication overhead this introduces, there's not much wonder that productivity suffers. The funny thing is that all these shops actually think that they're agile.

Anyway, back to the database group's requests for change. WTF!? I've long since come to the view that the database model and the object model are just two perspectives on the same thing. But in a lot of circumstances I see the DB design and the object model design begin to drastically diverge as the product evolves. And naming conventions tend to be the worst offenders.

Usually a glossary of terms is developed (even informally) as the dev team and the business people talk about the business problem, and usually the object model's class, attribute and method names are changed to better reflect the real world problem space. But for one reason or another, the database is left to decay. And then when the all-knowing database group comes along to make the problem worse and suggests/enforces naming conventions, I just give my head a shake...

For me it's all this skill specialization that is destroying the holistic view of the software systems that we build. Smaller teams of people with a broader set of skills and responsibilities are what we need to get the job done.

That is just so awful...

I was watching a video about Django, a Python powered web framework, when the presenter, Jacob Kaplan-Moss, made the following comment (at around 29:22):

I've actually seen a system, I shit you not, that has every single URL on the site calls a stored procedure in an Oracle database that generates html in the stored procedure and returns it to the web for display. I'm not joking! That is just so awful I can't even start to begin...

That comment took me back a few years where I encountered just such a mess. I was brought into the organization to introduce Java and J2EE but the developers that lived there already had this Oracle solution in place. For reasons that I still don't understand to this day, they couldn't see that this was one incredibly awful idea. In any case, the next time you find yourself looking at some ugly bit of Java or Ruby or whatever, just remember it could be a lot worse!

Billion Dollar Boondoggle

The "Canadian Firearms Program" more widely known as the "Canadian Gun Registry" was a program instituted by the former Liberal government of Canada under Prime Minister Jean Chretian to license and register certain types of firearms across Canada. Bill C-68 was introduced in February 1995 and has been controversial ever since for a number of reasons but undoubtedly the issue that has caught everyone's attention has been the cost of the program, now over $1,000,000,000, and growing.

So leaving the politics behind, I was interested in the technical details behind this program from a software developer's perspective. What exactly did these guys do wrong that would result in such astronomical cost overruns? What technologies did they use and what could be learned from their mistakes?

Well it was surprisingly difficult to find information but finally I came upon a two-year-old article in eWeek titled "Canada Firearms: Armed Robbery". The system was originally comprised of an Oracle 7 database and a PowerBuilder application built by EDS. Given what I know about those technologies, I doubt that the problem was the technology. Instead, it appears that the biggest problem was incorrect initial assumptions and ballooning, ever-changing requirements.

But after eight years of mismanagement what was the solution? That's right! They decided to rebuild the application with a different set of technology and consulting resources (CGI)! Maybe the implementation really was poor, but I suspect some incompetent management types were deftly pointing fingers elsewhere and some unwitting bureaucrat believed them. Ahhh that old story...

DAOs and ORMs

The discussion that this article on the ServerSide generated reminds me of an article I wrote a little while ago. The original poster gets a bit confused about the problem because he assume the use of an ORM means that you need to use the "Open Session In View" pattern. But if you get by that and read some of the comments, you discover that many people (like me) don't believe that the DAO pattern is necessary when using an ORM solution.

A valid concern about ditching DAOs is the coupling that may happen between your domain objects and your persistence mechanism. But if you like ActiveRecord or Rich Domain Models or Annotations or XDoclet then you've already made the choice of productivity over ultimate framework plugability and should just get over it.

But if the coupling is still nagging you and you're concerned about sprinkling framework API calls throughout your code then maybe you want to consider looking at EJB3. Instead of using the vendor's API (e.g., Hibernate, Toplink, etc.) you can reference standard annotations and classes that are free of implementation specifics. At least that way you've provided some hope of switching out implementations.

In the end, question why you're using the DAO pattern and then ask yourself if you've already done something in your code that has violated that pattern. Think about your use of OpenSessionInView filters or the navigation of lazily instantiated collections in your view layer. Maybe DAO and ORM really don't mix...

XML Glue

When I discovered XDoclet I quickly stopped editing my hibernate mapping documents by hand. From the XDoclet website:

XDoclet lets you apply Continuous Integration in component-oriented development. Developers should concentrate their editing work on only one Java source file per component.

This approach has several benefits:

1. You don't have to worry about out dating deployment meta-data whenever you touch the code. The deployment meta-data is continuously integrated.
2. Working with only one file per component gives you a better overview of what you're doing. If your component consists of several files, it's easy to lose track. If you have ever written an Enterprise Java Bean, you know what we mean. A single EJB can typically consists of 7 or more files. With XDoclet you only maintain one of them, and the rest is generated.
3. You dramatically reduce development time, and can concentrate on business logic, while XDoclet generates 85% of the code for you.

But unfortunately I still find myself editing a ridiculous amount of Spring XML by hand. Spring sounds nice in theory but after using it for awhile I don't think there's anything wrong with the factory and singleton patterns. This unholy quest for ultimate flexbility by gluing components together with XML has to stop. Most of the time you simply don't need it.

On Frameworks and Top Management

I couldn't have said it better myself:

Frameworks

Top Management

MS Mac Lab

A friend of mine just sent me a link to a tour of Microsoft's Mac lab. Thanks Aaron... Thought I'd file it here under the category of "Holy Macs!".

Embracing Change

When I consider the "agile software development methodology", a key principle that comes to mind is "embrace change". The theory is that users don't really know what they want. They'll only be able to steer you to the goal once they see something working. So get ready for change. They'll change their minds a bunch of times as the software evolves. In the end they'll appreciate your flexibility to adapt the software to their changing vision and you'll be a happy developer because you've created something that will actually be used and appreciated by the end users.

But what happens when it's your team leader who keeps changing the technologies under your feet? Is it just another kind of change to be embraced? Let's see...

The elaboration phase of the unified process can be defined as: "This phase attempts to create an executable architecture that realizes architecturally significant use cases." In other words, we should select the appropriate technologies that will be needed to create a solution for the most difficult known use-cases and then actually prove that our chosen technologies will do the job by creating executable software. Then during the construction phase, we should be ready to tackle the problems (known and yet to be discovered) with the technologies in hand.

A second core principle of the "agile software development methodology" is "You aren't going to need it", which means you shouldn't overengineer something because you think you might need it in the future. Keep it simple. The idea is that you save time by not doing it now and you make your code better and more maintainable because it's simpler. If someone ever says to you "if we build it in now, it'll save us time later" and you already have too much other work to do right now, then tell them to forget it.

I'm currently in an environment where every "senior" resource is on a mission to turn the management spotlight on themselves and the way they try to achieve that goal is to way overengineer the solution with as many Java frameworks as possible. By doing so they can prove their technical merit with a complex solution. To give you an idea of what I mean consider this:

We have a framework for dependency injection, we have one for database persistence, we have one for role based security, and one for binding XML, we have one, no wait two (or maybe three) for doing UI. We're in the process of adding yet another Java web container (without retiring the previous two) and have added every conceivable open source code measurement and style checking software. We have two continuous integration servers and in addition to the JUnit testing framework, we're adding Ruby/Watir testing too...

Individually and maybe with some intelligent selection I don't have anything against most of these frameworks (except for Spring) but after six weeks of development with 9 programmers with only a login screen to show for it, I say enough is enough. Java web development may not be as productive as it is with Ruby on Rails, but it's not this tough!! Come on! Look at the requirements and choose proven frameworks that address problems you've identified and don't cost more to implement, learn and maintain than it would cost to create a custom solution. For example, if you have to spend more than a couple weeks implementing a role based security framework when nobody has actually said you need to worry about different roles, then you're probably doing something very wrong. Keep it elegant, efficient and simple.

Apple Listens to Customers

Well don't ever let it be said that Apple doesn't listen to its customers.  They released a new beta piece of software called "Boot Camp" that will allow people to dual boot Windows XP and Mac OS X on Intel based Macs.  I can't imagine that many self-respecting Mac owners would prefer to use Windows over Mac OS X, but there are undoubtedly occasions when it would be beneficial to do so.

Now Apple isn't being completely selfless here of course.  They make a lot of money selling hardware and if they can convince Windows users to buy a Mac instead of a Dell they'll be more than happy to accomodate them.  Maybe this is like a Trojan horse to get people to experience a good OS.

Next thing I'd like to see is the ability to run Windows in some kind of virtualization environment within Mac OS X...

Fast and Furious

I'm sure I have goofy speech patterns and favourite catch phrases of my own but I also have a talent for picking these colloquialisms out of someone else's speech as well. I have discovered over the years that the best time to capture these beauties within an IT environment is during the inevitable "status meeting". It's almost like a freestyle rap battle where two or more contenders try to outdo each other for status meeting supremacy. Fortunately for me, the agile process inspired daily standup meetings on my new project have become a veritable gold mine for these things. We have a few people on the project who are black-belt kung-fu masters of the colloquialism spinning these out with speed and grace... Here's a sample of the ones I've captured so far:

up to speed	high level design
make it real	snapshot
visionary thing	anecdotal evidence
identify the need	lay the groundwork
cycles available	disparate user experience
bottom feeder	look and feel
production capacity	nebulous
context	numerous permutations
drivers	shock effect
more or less	impact
grease the skids	fresh set of eyes
steer going forward	engage you
take forward	externalize
put the spin on it	socializing it
draft rollout plan	show stopper
iterative process	mitigate risk
right hand man	extra pair of hands
duck is cooked	messaging protocol
key artifacts	validation effect
verbage	sooner rather than later
put lipstick on it	technical components
bring to the table	consider the options
test bed	full meal deal
iterative cycle	lock it down in stone
instant gratification

Misplaced Priorities

I've worked on quite a few software projects in my career and I always wonder at the people who get hung up on their favorite features regardless of importance to the actual users of the system.

A few years back I worked with a developer who became analyst. At some point in one of his past projects he had created a feature for allowing a user to press some key combinations to "hot jump" from screen to screen. Although he never really elaborated I suspect this feature was more than just the typical shortcut keys you see in most desktop applications. But that's not the point. He became convinced this was THE feature to add to our app. Over the next few weeks he became fixated on "hot jumps". We kept hearing about "hot jumps" over and over. This was going to be the "most used feature" in the system.

But the other developers and I didn't agree with him... We felt "hot jumps" had absolutely no value unless the destinations of these hot jumps actually worked! And that was the problem and the point of this blog entry. We had tight deadlines, and a lot of actual business functionality to develop long before features like "hot jumps" should ever have entered anyone's mind. I'm not saying "hot jumps" would have been completely worthless, but at the time, it was undoubtedly a misplaced priority.

So it's with some surprise (and frustration) that I find myself in a similar situation today, only it's not "hot jumps", it's directory structures, eclipse project files, and (lord help me) Ant scripts. Under normal circumstances I would definitely say these things all have some importance, but what about actually developing the application? We're three weeks into development with five developers but under the direction of our "leader", I find myself spending my entire day renaming things and begging the "gods of Ant" to actually let me get this damn build.xml file right so it will do what I want. And gentle reader this is not just your regular run of the mill ant script. This is the mother of all ant scripts, utilizing every possible ant task available. And let me clarify, I did not write it. I inherited it and have been instructed to "make it work".

But my point again is this, what good is this "hot jump", oops, I mean Ant script if it doesn't have anything to build? At the end of the day, will the users be impressed with the quintessential Ant script or would they actually like to have the features they asked for?

DHH Interview

I just found a SYS-CON.TV interview with David Heinemeier Hansson that had some real quotable moments:

In his description of Ruby On Rails:

...we like to consider ourselves the Apple of open source web application development...

...it's extremely conventional. Most of the ideas in Ruby on Rails are the same ideas people have been building web applications on for the last ten years. They just hurt a lot less...

On Convention over configuration:

...In Java we call it XML sittups. You can produce just as much XML configuration as application code and that's just insane in our mind...

Does it scale?

...unquestionably yes. simply because Ruby doesn't invent a new architecture. We use exactly the same architecture as people who have been building the biggest sites on the internet.... shared nothing...

On the value of Rails.

[People think] it's so valuable to have extreme performance when there are tones of other attributes that are more valuable: programmer productivity, how maintainable your application is over time, ...

Ruby jEdit Plugin

I've been a big fan of jEdit for quite a few years now and I just downloaded a new plugin that I just simply felt compelled to write about, the jEdit Ruby Plugin. It's not the simplest thing to install but the integrated code completion and documentation features (including Ruby on Rails) are quite good. For example, when I'm editing a model object I can type in "has", hit ctrl-space and get a popup with the options "has_one", "has_and_belongs_to_many", and "has_many" options, and if I'm displaying the Ruby Docs window (currently docked to the right side of my jEdit window), it will display the documentation for each option as I select each one. Very nice. Perhaps some competition for RadRails?

Agile or Traditional?

I was recently speaking with a friend I haven't seen in months and she mentioned that she was reading a book I had lent to her. The book is called "Balancing Agility and Discipline" by B. Boehm and R. Turner. The book's main theme is the attempt to balance agile software development methods with traditional methods. The idea is to try and assess your project by looking at several different characteristics and then choose the processes from each method that will allow you to best manage it.

So that idea got me thinking about my most recent project, a Java rewrite of an existing application, and how desperately we were trying to use a pure agile methodology when it didn't really fit.

Here's a quote from a presentation available on the "American Institute of Aeronautics and Astronautics" website:

Problems characterized by change, speed, and turbulence are best solved by agility.

• Accelerated time schedule combined with significant risk and uncertainty that generate constant change during the project.

Is your project more like drilling for oil or like managing a production line?

• Oil exploration projects need Agile processes.
• Production-line projects are often well-served by rigorous methodologies.

Our project was undoubtedly a production-line application with very low risk:

• Although I didn't really like the technology we used on this project, it was being used to some degree of success on two other projects. So while it was undoubtedly not very productive, it was usable.
• The database design and user interface design were to be lifted untouched from the existing application.
• We were mandated with reproducing an existing production application so the requirements were well known if not well documented.
• Business users were not co-located and not easily reached for consultation (during the entire project I never spoke to a single person who would actually use it).

So what drove us to try and use an agile methodology? Well.... it's trendy. Agile is "in" and waterfall is "out". IT people would like to live in a world of absolutes, where we can say with certainty that A is better than B. But the reality is that we live in a world of spectrums where things are classified in terms of a position on a scale between two extreme or opposite points. Just like we shouldn't try to develop a simple project using every conceivable J2EE technology and framework available, maybe we shouldn't try to apply every agile process available where it just doesn't fit. Sometimes it's okay to create a plan and to write down requirements. Sometimes it's the best thing to do.

Suffering

One of the Pragmatic Programmer tips is:

"select" Isn't Broken.

Which means that's it is rare to find a bug in the OS or the compiler, or even a third-party product or library. The bug is most likely in the application, i.e., my code.

The problem with the project that I'm working on is that there is so many layers of code and abstraction that I don't even work with the framework anymore but some class ten times removed.

For example I have a JSP calling a JSF configured backing bean, calling a Spring configured service, calling another Spring configured EJB in a different process, calling another Spring configured service calling another Spring configured DAO which is (eventually) a subclass of Spring's HibernateDaoSupport class which attempts to hide Hibernate's API altogether.

So while I try to take the tip seriously and not blame the framework, there is so much stuff happening before it even gets to Hibernate (and let's not forget all that XML goo that glues everything together) that I can't help but lay some blame at the feet of these frameworks that encourage so much obfuscation.

To get real geeky I'll quote C3PO from the original Star Wars movie:

We seem to be made to suffer. It's our lot in life.

Bruce Eckel on Ruby and Rails

I just listened to the latest podcast by the Java Posse; an interview with Bruce Eckel.  Since it's the "Java" Posse there's a fair amount of discussion about Java, but the second part of the interview dives into Bruce's thoughts on Python, Ruby and Rails.  Although he gets some things wrong, like attributing the creation of Ruby to David Heinemeir Hannson (DHH) instead of Yukihiro Matsumoto (Matz), he largely says a lot of good things about Ruby and Rails and admits where he lacks sufficient knowledge to comment.

The one part in particular I agreed with was his assesment of the Rails supporters as being "loud", as in "loud-mouthed".  He says it's a lot like the early Java days.  As a matter of fact, my first taste of Rails was a presentation by DHH.  His rhetoric in that speech was a bit much to bear, but if you can see your way past the hyperbole, and just see that the guy is truly passionate about creating "beautiful" code, you get a good idea of why Rails is the way it is.

Rails is Boring and Ruby is a Toy

Chad Fowler did a presentation last month that I happen to be listening to at the moment. He's talking about his "evolution" from a Java developer to a Ruby developer. He talks about the various XML files you have to edit when doing J2EE, the number of lines of code that's needed to do just the simplest things in Java, etc.

What I find so surprising is that some of my recent comments seem to be echoing his statements. Unlike him however I was already on the path to trying to simplify my Java development and Rails has simply dropped in a great big exclamation point. Web application development doesn't need to be the morass we suffer through in Java. Ruby can make it fun and Rails can make it so easy to do. Geez now I'm starting to sound like an evangelist too! Oh well I use a Mac too... maybe it's in my DNA...

Hibernate vs ActiveRecord

Over the last 2+ years I've grown quite accustomed to the facilities of Hibernate (one of the most popular Java object-relational persistence frameworks). And before that I used TopLink for a couple of years. So now when it comes to my work with Ruby on Rails I have some pretty high expectations.

The out-of-box experience with Rails' persistence framework is undoubtedly different but similar enough to Hibernate to make life fairly comfortable. But recently I started watching the SQL that gets generated from Rails and decided I'd like to optimize them a bit. But unfortunately that isn't as easy as everything else in Rails. I found this article at TheServerSide that sums things up pretty well. I think Rails comes out on the losing end. Now the question becomes, are the things you lose worth the things you gain?

What I don't have to know

I was thinking recently about how someone with no knowledge of Java or Rails web development would choose which of the two technology stacks to learn. Then I started comparing the lists of technologies on both sides.

1. For the View you have JSP (Java Server Pages) vs ERb (Embedded Ruby). (We'll assume that (X)HTML, CSS, & Javascript are a given knowledge requirement regardless of framework choice.)
2. For the Controller you have JSF (Java Server Faces) vs ActionPack
3. For the Model you have Hibernate vs ActiveRecord. (Since both of these technologies are Object Relational Frameworks we'll assume that relational database knowledge, including SQL is a given requirement regardless of framework.)
   

So on the surface things look pretty similar. But let's dig a bit deeper.

First let's start at the View: In Rails we take standard HTML and embed Ruby to dynamically generate the content. In JSF-flavoured JSPs on the other hand we substitute HTML tags with custom JSF tags. You must also know JSF EL (Java Server Faces Expression Language) because it is used heavily in the JSF tags.

Secondly, the Controller: In Rails, controllers are written in Ruby. In JSF the controllers are thankfully written in Java but in order to make them work you have to configure the dependency injection framework with XML. Your JSF XML file must conform to a particular DTD. Now if you also believe in the value of dependency injection frameworks you may want to use Spring. Spring isn't mandatory, but it's popular enough to be part of a typical Java stack of technologies. Unfortunately it requires it's own XML DTD.

Finally let's move onto the Model: In Rails we use Ruby to define our model. The naming convention is usually sufficient to allow ActiveRecord to determine what classes map to which tables. We typically use snippets of SQL to tell ActiveRecord how to fetch data from the database. In Java, Hibernate is the common persistence framework choice. In Hibernate another XML DTD tells the framework how to map the classes to the tables. But since people grew tired of maintaining so much XML they turned to XDoclet. And since that became so popular there are now Java 5 annotations that allow you to annotate your domain model wth mapping metadata. Currently you can use any of the three choices. Hibernate also uses its own query language, HQL, to instantiate objects from the database. This query language is based on SQL and has quite a few benefits but is yet another syntactically different language to learn.

So let's add it all up. To do Rails web development you need to learn:

1. Ruby
   
2. the various Rails APIs (e.g., ActiveRecord & ActionPack).
   

To do Java web development you need to learn:

1. Java
2. JSF tag libraries
3. JSF EL
4. JSF's XML DTD
5. Spring's XML DTD
6. Hibernate's HQL
7. Hibernate's mapping DTD or XDoclet or Hibernate Java 5 annotations (and the DTD for configuring the framework)
8. The DTD for web.xml.
9. The APIs for Spring and Hibernate and JSF
10. (And I didn't even mention EJBs)
    

People may look at this list and say "wait a sec, you're being unfair, XML is XML and you're counting four different types of XML in the Java stack". Well all I can say is that when I have to edit one of these things I have to take my eyes off the code that I'm currently working with and make a contextual switch to something that looks very different. That switch is enough for me to penalize the Java list with a bullet point for each one. But even if you collapse the four XML items into one, my point still stands: there seems to be an awful lot more things you have to be comfortable with when developing Java web applications. Perhaps that's why I'm beginning to favor Ruby on Rails so much more. It's the Simplicity.

Manifesto

Everyone has a manifesto nowadays. I just read an interesting entry at the "Creating Passionate Users" blog that pointed to the the 37signals manifesto. Both recommended reading.