Jigsaw vs. OSGi?

It’s been awhile, and things have changed again. JSR 277 is dead. JSR 294 is alive and well. IBM almost bought Sun. Oracle actually did. And Jigsaw is on the horizon and appears to be on a collision course with OSGi.

The multi-module-verse. Like the parallel universes idea, Sun’s introduction of Jigsaw seems to create a significant fork in Java, with OSGi bundles in one version of reality and Jigsaw modules in another, forever isolated. Which begs the question: which reality do you want to inhabit? Better decide fast and jump the right way; the future of your code may depend on it.

Are we really going to be forced to choose? The answer is unfortunately yes, as things stand today. Each module system has its own model for deploying, loading and resolving dependencies across classes, and they won’t know how to share. So if you package your classes in a Jigsaw module, but want to use classes packaged as an OSGi bundle, you’re SOL. And the same is true in reverse. Heavy sigh.

As Neil Bartlett pointed out, one solution to this problem could be an acquisition that simply kills off Jigsaw, replacing it with OSGi. And sure enough, an acquisition has been announced.

But it would be a mistake to jump to any conclusions at this point on the future of Jigsaw.

Not only is it way too soon, it is also not as clear a choice as some would have you believe: Sun does have some valid reasons for building a simple module system for the JRE itself.

Where things get really contentious is when that same module system becomes the default choice for all developers. And the “our way is right/we know better” arguments flare up. But there is no “right” way—as with most software, different situations require different trade-offs.

Of course, we do have a standard model for solving this problem: a common API that enables different implementations. And we should use it here.

I won’t predict the life expectancy of Jigsaw, but one thing is crystal clear to me: the Java platform needs an abstraction layer for module systems. One that makes it possible for different implementations to exist, simultaneously and/or over time. One that provides a meaningful way for those implementations to share classes, so developers are not forced to make a potentially risky commitment. And one where a JVM vendor, or anyone else, can craft an optimized solution without shattering the universe.

Even if we could all agree on a single module system, an abstraction layer is still the right thing to do to help future proof the platform that so many of us depend on. It's just good software engineering.

If there is anything I'm sure that we should kill, it's the "us versus them" divisiveness that has dominated this space for so long.

JSR 277 Interoperation

We all know that JSR 277 is supposed to "interoperate" with OSGi, but the details have not yet been specifically addressed in the spec or EG. And many view this as a critical missing element: Glyn Normington has gone so far as to file a bug which has gotten a lot of attention (Glyn also reminded all of us that OSGi is not only a de facto standard but a proper one as well). Peter Kriens wonders if the JSR has become a ghost, and Neil Bartlett chronicles some of the events surrounding the "missing strawman", both underscoring the growing frustration in the OSGi community.

We'll see some activity in the EG on this very soon. But I want to make something clear: the bulk of the interoperability story is already present in the spec. No, it isn't called out as such, and yes, there are certainly some missing details, but the essentials are all there.

What the heck am I talking about? Well, lets explore what "interoperation" actually means here...

First, plant firmly in your brain that JSR 277 provides a framework under which multiple module implementations, called module "systems" in the spec, may co-exist in the same process. OSGi is obviously the one that many think of first, but there are others and we need to ensure that they can all play nice with each other. So clearly interoperation is concerned with the interactions between module systems. But this is far too fuzzy.

Second, consider the general problem of resolving dependencies among individual module instances (I'm using the term "module" in the generic sense here, think "bundle" when we're talking about OSGi). A module system has some means of expressing dependencies, and it is up to the runtime to satisfy ("resolve") each of them by finding an appropriate match out of the available modules. The expressions may be very loose (e.g. "package 'com.acme.dynamite', any version") or extremely rigid (e.g. "module 'foo', version 3.14, created and signed by the Apache foundation, for a specific version of an OS and processor).

Given a list of dependency expressions for a given module, an implementation must do the following for each:

• Use the available lookup mechanisms to find candidate modules that satisfy the expression; fail if none.
• If more than one candidate exists, use some interesting algorithm to choose one.
• If that candidate has not itself been resolved, do so.
  

If successful, this recursive process results in a list of module instances (the "imports"), each of which has its own ClassLoader. And the module is now ready to satisfy class and resource lookups from its loader, by delegating to the imported loaders when required.

Ok, so it is relatively easy to see how this works within a single module system, where all of the modules are the same type and internal implementation classes can be assumed and used. But JSR 277 would not be very interesting or useful if modules could only resolve against other modules of the same type. And that, finally, is what interoperation means here: enabling dependency resolution and class/resource sharing across modules of different module systems.

And it highlights some obvious requirements for cross module system interoperation. For the dependency resolution steps, each implementation must be able to:

1. Map its dependency declarations into a standard runtime representation.
2. Support a standard search model over its stored modules, using the runtime dependency expressions.
3. Map its stored module data into a standard runtime representation that can be returned by the search.

Abstractions for these are already well defined: dependencies can be mapped into a Query composed of arbitrary constraints (name, version, version ranges, attributes, etc). Queries can be passed to Repositories to find ModuleDefinitions, each of which can then be instantiated as a Module with its own class loader.

A single Repository instance will (normally) support modules from only one module system, but searches can and will traverse multiple repositories. So, given a search that starts at an OSGi repository, for example, and which also visits a BryanModuleSystem repository, it is then very natural for a dependency expressed in an OSGi bundle to be resolved by a BryanModule.

Class and resource sharing requires that each implementation must:

4. Support a standard class loader delegation model.

For this, the public methods of ClassLoader itself can be used: iterate the list of imported Module instances, grab the loader from each and call the appropriate public method. We can of course do a much better search by keeping a simple index that maps package names to the module(s) that export that package, thus avoiding a linear search across the imports (and the consequent fan out that would entail). Given that class loading often dominates startup time, and that search dominates class loading, this optimization turns out to be well worthwhile. (It is also quite natural when the primary dependency expression uses package rather than module name, but that is a topic for another day.)

Finally, lifecycle management must be addressed. Each implementation must:

5. Support standard install/uninstall operations.
6. Support standard "start" and "stop" operations.

Here, Repository provides the interface for install and uninstall, ModuleDefinition provides the "start" operation (get instance method), and ModuleSystem provides "stop" operations (release/disable module methods).

So what's left? We need a solution to the "interesting algorithm to select among multiple candidates" problem. We may want a new loader delegation interface that refines/optimizes the contract. We need to ensure that the concurrency model in place during resolution works correctly across module systems. And I'm sure we'll uncover a few more such details as we gain experience with multiple implementations.

And we do need to nail down these issues as soon as possible.

But, overall, the "interop" story has all the main characters and much of their dialog already filled in.

JSR 277 Could Be Great for OSGi

Many OSGi advocates have said they wish JSR 277 would "just die". Why? Because 277, as currently defined, seems to thumb its nose at this mature, widely adopted and rapidly growing modularity solution.

OSGi is the de facto standard in this space, yet in JSR 277 Sun has defined an entirely different, apparently competing solution. Targeted at a future SE release, 277 appears to undermine OSGi by casting doubt about its future role in the Java universe.

So yeah, OSGi adopters are pissed. Understandably so.

But there is a another way to look at it. Really. If you squint your eyes a bit at the early draft spec, you can see it.

Big, deep breath... exhale slowly... Good. Ready?

The initial spec actually has two separate parts: an api/framework, and an implementation with a new distribution format. Unfortunately, these are presented in a way that seriously blurs the distinction. Worse, the new distribution format (".jam" files) often takes center stage.

The emPHAsis is on the wrong syLLAble.

The api/framework layer must be the primary focus of the JSR 277 specification, providing a coherent compile/runtime model that enables multiple implementations. Specific implementations, while required to surface framework/api issues, should be documented in appendices or even separate specs.

If the EDR spec had been written from this perspective, we probably would have avoided most of the current animosity. We can and should fix this in the next draft release. Implementations can then be seen on equal footing. More importantly, they can be left to compete on their own merits.

Not everyone wants or needs OSGi, and the new .jam implementation may be right for them.

But we know that there will be LOTS of bundles around when SE 7 ships, vastly outnumbering .jam files, so we need an OSGi implementation. ASAP. Built by OSGi insiders. Without it, we cannot have confidence that the api/framework abstractions are right or complete. With it, we not only gain spec validation but have a ready-made solution for using bundles on SE 7.


So how could JSR 277 be great for OSGi? By providing tight integration of bundles into SE environments. The benefits would surface in four main ways:

Canonical storage model. The Repository abstraction provides extensible installation and search mechanisms for any module system. The Query abstraction enables flexible, extensible, indexable searches, over both local and remote repositories.

Compile time dependency resolution. The ModuleDefinition and ModuleContent abstractions, coupled with Repository, enable the compiler to actually use the runtime machinery to perform dependency resolution, ensuring high fidelity with runtime environments. (These same abstractions enable interesting tools other than compilers, as well.)

Class/resource sharing across module implementations. The Module and ModuleSystem abstractions work together to enable sharing at the ClassLoader level, across different module systems. (And yes, the exact mechanics of this are still TBD; again, we need an OSGi implementation to help flesh this out.)

Command-line execution. Like the -jar option, a -module option will enable convenient execution of a main class from any module, backed by full version/constraint based dependency resolution. A -repository option provides flexibility in directing the search for the specified module.

And this could all work for existing OSGi bundles. Could be nice, huh?

Maybe there's hope yet.