A Question of Design

So I'm working on this project at the office: It's basically a rewrite of an existing system that is nothing short of a mess: Poorly designed and poorly implemented based on shaky requirements...basically everything that qualifies a project for the title of "Complete and Utter Failure".

Of course, the powers-that-be don't want to call this a rewrite -- they prefer "refactoring" or something, anything, that doesn't use the word "rewrite" in it, for fear of upsetting the users of said shoddy system. The system was just put into service a few years ago and there's a fear that those in charge of the purse strings will freak out if they found out that their current system isn't exactly the best they could get for the amount of time and money that has already been sunk into the system.

So, here we are, doing a rewrite that isn't a rewrite.

I was specifically requested to be a part of this project, and to take a "lead role" (whatever the hell that is...I still haven't figured it out, yet) given my rather strong desire to build stuff "The Right Way", and the fact that I had knowledge of the existing system...

See, I helped build the current system. Yes, I contributed to the problems that we are currently trying to correct, but not before trying desperately to improve the situation while I was involved. I eventually left the project out of frustration (a topic for a whole blog post itself), since I was unable to make any headway and certain members of the team at the time didn't seem all that interested in improving the situation. Fortunately, those individuals are not involved now, which at least makes the move towards a better system easier to accomplish.

So anyway, I've basically been given the task of redesigning the system the way I think is best (with some restrictions, which I cover below), given my rather vocal disagreements on how things were done on the existing system and given the fact that I'm a rather smart guy (their words, not mine).

We've hammered out a bunch of requirements, pulling both from the existing system (which does meet some of the needed requirements of the users) and feedback from the users on how they currently work. So far so good.

With this information in hand, I've been working with some proof-of-concept code to get a general feel for how the system should go together -- experimenting with Spring MVC, which I'm just starting to get a feel for, playing around with Spring Web Flow, and so on. This has been going on for a couple months while the project manager and some other higher-ups have worked out a few additional changes that need to be made to the system to accommodate some coming business changes.

At this point, I basically have a good feel for how the system needs to be built. And, quite frankly, if given a few weeks of uninterrupted work I could probably hammer out a good portion of the system on my own.

The problem I'm encountering now is that the powers-that-be want to take a more...formal...approach to the system this time around. They basically want to have a design laid out first, then have that design implemented -- either by me or some other team members that can understand the design. And they want me (and a few others) to work out the design artifacts -- class diagrams, activity diagrams, and so on. Also, they've decided that I shouldn't be the only one designing the system (for fear that I might do something "wrong" or something, I guess), and so have teamed me up with another developer so that we can work out an agreeable shared design.

My issue with this is that the time it would take me to work out all of this stuff would be better spent actually...you know...building the damn system.

Now, lest this sound like I'm against any kind of design, I'm not. I understand perfectly the desire to have a solid design for the system in place, and making sure everyone understands what that design is and being on the same page.

The problem I'm seeing, however, is that there's this expectation that the design is going to be first, then the coding. Where for me, both tend to happen hand-in-hand: I'll code a bit, then analyze what I've created. I'll then refactor whatever I think needs changing in the interests of keeping the system simple and maintainable, reanalyze, and so on. The design is eventually exposed through my working of the code into a form that make sense.

Also, I personally tend to see the design artifacts -- class and activity diagrams and such -- as communication mechanisms to get people to understand the implementation, not as a means to an implementation.

There's also the small fact that it's hard to come up with a solid design when you are working with architectures/frameworks that you are not completely familiar with. For example, I wouldn't know where to start with trying to design a system running in Ruby when I've never used Ruby before. Yet this is the kind of approach they are asking me to take here: Design everything first, have agreement on that design, then move to implementation.

Yea...there's that whole Waterfall feel about it isn't there?

Unfortunately, I can't seem to get the powers-that-be to understand this: They seem to think that there can't be any code without a design first. They've been taught/conditioned/whatever to expect that I will work up a detailed design using whatever design tools they've given me and then they'll just hand it off to some other developers to implement. What they don't seem to understand is that I can actually write the code according to the design I've envisioned faster than I could actually draw up all of these detailed design documents.

There's also the fact that, given the approach toward design I'm being asked to use here, I'm being pushed into a role that effectively takes writing code completely out of my hands, which I seriously have no desire for. I like writing code. It's what I'm good at, and I see no reason to move into some other designer/architect/whatever role that takes me away from it.

There's also a bit of the fact that I've been writing code for damn near 10 years now, and I've never been in an organization where these explicit designer/architect/coder roles have worked out in practice. In my experience, all software developers take on all three roles during the course of a project, and in fact need to take on those roles from time to time as a project evolves. Taking those roles out of the hands of the people actively working on the code does nothing but slow a project down to a crawl, as now any question about the design or architecture that a coder might question necessarily involves other parties who very likely have no interest in having a lowly coder (who couldn't possibly know anything about good design/architecture in their view) question their judgment. It's been a roadblock toward good software in my experience.

Maybe this is just a case where I'm not seeing the light or something, but I'd be surprised if I'm the only one that looks at the the whole design/architecture/coding differently from what I've been encountering here recently.

It's not a case where I think proper design and architecture are bad. Far from it. Just that I think that perhaps the conventional wisdom regarding them as explicit roles to be filled by specific individuals is somewhat flawed.

In Which I Respect Larry Ellison Just A Bit More...

Larry Ellison isn't exactly the most-liked individual in the tech industry. Heck, I've never been a fan of the guy personally, and I've never found anything in particular to like about anything he's said in the past. And the war stories I've heard from former Oracle employees have only reinforced my dislike of the man.

But dammit, I can't help but respect him for what he says in the clip:



My favorite part is at the 3:54 mark, where Mr. Ellison says:

Our industry is so bizarre. You know, you just change a term and they think they've invented technology.

Getting the Job Done.

Seems like it's been an eternity since Joel Spolsky has written an actual full-blown post on his site, but his latest is a really good one.

I'm a huge fan of pragmatism over idealism, and as others have already pointed out you do have to be able to balance the two in order to be a successful developer in this day and age.

I don't have much more to add, other than I find it slightly amusing that in describing a certain kind of programmer, Joel has also actually provided a good reason why Java -- A language and community that Joel himself has focused some disdain -- rose to prominence the way it did.

It ain't pretty, but it get's the job done.

Broken Code Snippets

Google has decided to discontinue Google Pages and has started migrating users over to Google Sites. As a result of this (and some of my own messing around with the site design) most of the code snippets I have on this blog no longer display properly. I've updated my post on syntax highlighting in blog posts to warn people away from trying to use the approach I've described there, and which is the one I had be using for code on this blog.

I plan to address the problem soon enough -- when my schedule allows, that is.

I guess this is a good argument for hosting my blog on it's own server (which I've been considering), which would have obviated the need for the Google Pages approach.

Oh well...live and learn.

Grails Kicks Ass

God how I wish I had discovered this framework sooner...

I decided last month, following the suspension of my previous project, that I'd take a look at Grails while I wait for some more work to come my way. I've read and heard some good things about it, so I grabbed The Definitive Guide to Grails (one of the authors, Jeff Brown, was at the NFJS conference I attended last year) via my Safari Books account, and have spent the past several weeks working with the framework and just seeing what it can do.

And so far I'm loving Grails...I mean I'm really loving it.

Why do I love it so much?

Well, for starters, let's say I'm creating a new project from scratch using the typical Java stack -- we'll say Spring/Hibernate with a sprinkle of JSPs -- I'd usually set up a project more or less like this:

1. Create a web project in Eclipse.
2. Grab the Hibernate and Spring libraries and put then in the WEB-INF/lib directory for the project.
3. Create a skeleton Spring configuration in XML
4. Create one or more domain classes.
5. Create Hibernate mapping files for each of the above domain classes.
6. Set up some tables in the target database for the above domain classes.
7. Set up the DB connection info (more XML configuration, usually...)
   
8. Create some initial DAOs (interfaces and implementations) to service the domain classes.
9. Create a basic Service (interface and implementation, of course) to handle the DAO access.
10. Create a simple Spring MVC controller.
11. Wire up the DAOs, Service and Controller in the Spring configuration file(s)
12. Create a couple simple JSPs and maybe even a basic HTML page or two that works with my controller.
13. Test it all to make sure I didn't screw up any of the configuration.
14. Create some unit tests
15. And so on...
    

I've probably missed a few steps in here, but you get the idea: Getting up and running with a typical Java web stack takes a lot set up -- most of it ending in ".xml".

And this is just to get started. I don't know about anyone else, but once I've been rolling along on even a moderately-sized project that uses Java technologies I find myself getting into a pretty ugly tangle of XML and Java eventually. It's not hard to do.

Meanwhile, here's what it's like getting started in Grails-land:

   grails create-app foo
   grails create-domain-class bar
   grails generate-controller bar
   grails generate-views bar
(repeat last three commands as needed for each domain class...)

It just gets easier from there...

And the best part: You're still using Spring and Hibernate.

The Grails developers did the single smartest thing possible: Rather than reinvent the wheel, they leveraged existing Java frameworks under the covers and provide what has to be the cleanest interface to those tried-and-true systems you could come up with. The "convention over configuration" approach means that a bunch of boilerplate that rarely changes from project to project disappears, and when it comes to actually changing the configuration, using actual Groovy code rather than XML (or annotations) just looks a hell of a lot cleaner. Working with Hibernate and Spring is incredibly easy in Grails -- much easier than I even thought possible.

Grails also deploys just fine to most major app servers, making it super-easy to leverage in an existing Java infrastructure, so working with it doesn't require contorting your entire infrastructure to support it -- just make a war file and off you go!

I really can't say enough good things about Grails at this point. I've already been working on a proof-of-concept at work to see how well it operates within our existing infrastructure and so far it's passed with flying colors. Transitioning to this framework is a breeze.

I've also been comparing the level of effort required compared to the usual Java Spring/Hibernate stack using the traditional approach and I'm amazed at just how incredibly fast I've been able to work with Grails -- what would have taken me days to get working normally took me mere hours with Grails.

Given the choice between the usual Java stack and Grails, I will pick Grails hands down in a heartbeat...it's that good. You can bet your ass I'll be pushing hard to use this in my future projects.

Moment of Reflection

A little more than 20 years ago, I was getting my first taste of programming on my brother's Commodore 64. The only information available to me were a few books my brother had that contained code listings, written in BASIC, for a bunch of computer games. It was all the information I had available to me at the time. I must've read and re-read those books a million times...

About 15 years ago, I rediscovered the joy of programming and began looking at, and buying (when I could afford them), whatever books and magazines I could get my hands on at the local computer store that covered computer programming. Aside from the library, I didn't have many other avenues for information on the field of software development. And since going to college was a decidedly expense avenue for me at the time, these were my only options for learning.

Today -- after upgrading my Safari Books account a few weeks ago -- I now have immediate access to every book currently in their library, which has a total count of books somewhere in the 8,000 range, last I checked, and includes just about every book on software development that I might ever want to read.

And even when I can't find what I'm looking for on there, I have Amazon.com as backup for cases where the book I want isn't available via my Safari account.

And on top of that I have The Web/Internet, with any answer to just about any question I could think of being a mere Google away...

Technology is so cool sometimes...

Java IDE Performance

An interesting discussion popped up on DZone last week. In a discussion of this link on DZone, the author of the post in question mentioned the poor performance of most Java IDEs, and I think he has a valid gripe: Most (if not all) Java IDEs are performance hogs, sucking up a lot of CPU and RAM.

Before anyone gets all hot under the collar about their favorite IDE and starts flaming me, let me be clear: All of the major players in the Java IDE world -- Eclipse, Netbeans, and IntelliJ IDEA -- are fantastic pieces of software from a feature standpoint and I've used just about every one of them at one time or another, with Eclipse being the mainstay for my day-to-day work currently. Given the current state of the art in Java IDEs compared to my early days in Java development (late '90s) I will take what we have now without question.

But I don't think any Java developer out there can say that these IDEs don't suck up system resources like crazy. Regardless of the hardware I've run on, I still run into long load times, laggy auto-complete when I type, and just general slowness...to say nothing of memory consumption. And the larger a project gets, the worse the situation becomes.

As far as I can tell, performance is still an issue for most of the major IDEs in the Java development world.

Now, maybe it's not a terribly big deal for a lot of Java developers out there -- I don't see a lot of discussion about IDE performance in the Java community these days. Perhaps everyone's just learned to live with it. Or perhaps it's just an issue for those of us in software development that don't have the luxury of monster-sized Quad-Core systems with 8+ Gigs of RAM, and sporting a Solid-State Disk for maximum performance. In other words: All us lowly enterprise developers who are at the mercy of our IT departments.

At any rate, even if you're sporting the best development system money has to offer, odds are you've had to deal with IDE performance at some point or another when doing Java development. The question is what's causing all of our favorite IDEs to perform so poorly in terms of both speed and memory footprint?

A few thoughts come to mind...

Java
Of course, we can't have a discussion of performance without thinking of the underlying VM upon which the major IDEs run. Despite the heretical nature of the "Java is slow" argument within the Java community, this is certainly a valid argument to make here since all of the major players in this space are written in Java, as far as I can tell. We could spend days arguing over whether Java really is slow compared to native code (and there's been plenty of arguments about it since Java's inception), but the fact is that you are running bytecode in a VM, which no matter how you slice it is going to be slower than native code in a lot of cases. And given the relative size and complexity of the pieces of software in question, I think it's a valid point to consider when it comes to IDE performance.

Of course, a question I have here is this: Why aren't there any popular natively-compiled IDEs for Java? I know it flies in the face of Java's overarching premise of being platform-neutral, but wouldn't having some form of high-performance, compiled IDE be useful to developers? Couldn't the free software or open-source communities get something going along these lines, creating an open-source, natively-compiled Java IDE that could be ported to the various platforms?

Thinking about this further, Eclipse itself leverages the widget set of the underlying OS via SWT, which means it's effectively leveraging native code to improve performance already (and this was in fact a major motivation for SWT's creation to begin with), so why not just take it a step further and make a native-code IDE?

The closest thing I've ever seen to a resource-friendly IDE that wasn't written in Java is the Java Development Environment for Emacs (JDEE) which, despite being a nice and speedy environment to work in, is still somewhat lacking in its feature set and is a bit hard to just jump into for most developers -- especially when you need to learn Emacs in order to use it effectively.

Anyone that's familiar with Emacs, however, will point out something interesting here: The JDEE isn't a natively-compiled IDE either -- it's written in elisp -- which actually raises more questions about the possibility of whether the Java platform itself really is a performance problem or not. Since elisp is interpreted code, yet the JDEE performs exceptionally well, perhaps the whole interpreted code thing might not be such a big deal after all.

Architecture
This, of course, brings us to the subject of the underlying architecture upon which these IDEs are built. It is certainly possible that the architecture of the IDEs themselves are just lacking, making them inherently slow. And with the growing list of languages and technologies that many of them try to support, it makes it more likely that the underlying architectures of the various IDEs can only handle so many things before they begin to buckle under the weight of it all.

Based on my own look into the Eclipse RCP architecture, I'd have to say that this is certainly a possibility. It's designed to be as generic as possible and support a wide range of possible uses, and as a result may not be perfectly tuned for software development the way it once was. I haven't looked at the Netbeans architecture, and can only guess that it would face a similar situation as Eclipse in terms of design.

IntelliJ IDEA may be the one with the advantage in this regard as I don't see anything to indicate that it's tried to be anything more than a feature-filled IDE. But nevertheless, it's still trying to support a lot of uses, which can affect performance.

I'd also like to think that since most of the major Java IDEs have been around for a decade or more, they'd have at least some focus on performance in their respective architectures for at least part of that time and have kept that at the forefront of all of their design decisions. Still, there is the possibility (at least in the case of Eclipse and Netbeans) that they've overextended by trying to be too generic.

Java Project Complexity
Another possibility -- and one that I think warrants some attention -- is the very nature of most Java projects and their sometimes dizzying amount of complexity and the array of tools that can be required to easily navigate and maintain such projects.

When a "simple" web project is using the likes of Spring, with Hibernate linking the database, along with any number of JSPs, or perhaps even some GWT or other AJAX-like technology in the web pages, all tied together by a multitude of libraries, XML files, property files, i18n support, and various bits of technology, it's no wonder the IDE might start having spasms. There's a lot of parsing that has to go on.

Of course, this touches on a bit of the other two points -- if the IDEs are designed and architected right then they should be able to handle this sort of thing. But since these days a lot of these IDEs are not necessarily optimized for just one type of project, or even one type of language for that matter, this might not be something that's easily addressed.

Maybe the IDEs are a little over-generalized to support too many things at once...?

Etc...
I'm sure there's something else out there that I'm missing, so feel free to jump in and add in your own theories.

I don't really have any solutions here, just some thoughts about the current state of Java IDEs. I think all the development platforms we have now are great in their own ways, and give us plenty of options.

But I can't help but feel that maybe we've all just settled for what works, without thinking about if we could be getting more out of our IDEs in terms of performance. Have we just accepted that things are as good as they're going to get?

I actually managed to use the JDEE on a project a few years ago and found it immensely more productive compared to Eclipse -- aside from the production boost that I got just from using Emacs (which I do miss these days), the performance of the environment in general was considerably faster than Eclipse, and I felt that I was able to be a lot more productive on that project than had I been using one of the other popular IDEs.

Unfortunately, I had to abandon the environment when I switched to a web application project -- the JDEE doesn't do web app development very well from what I've seen. Nevertheless, I still miss the amount of productivity I had.

I guess the question is...does anyone else?

Maybe this is a case where a developer who hasn't explored Java outside of the popular IDEs, who has spent most (or all) of a career working with the likes of Eclipse and such, just doesn't know any better and thinks that Java development is as fast as it can be based on those environments -- They basically don't know what they don't know.

At any rate, I think this are all valid questions we should be asking: Can Java IDEs get any better on the performance side? Should we accept the current status quo? Would we be better served by a native IDE client?

I'm curious to know what other developers think about this.