Prediction #5: Open source ALM tools continue to gain market share, give the development manager a migraine

by Mik Kersten, February 8th, 2011

The influence of open source on software development is often measured by the impact of successful libraries and frameworks. It’s hard to imagine building a modern web application without open source components. A similar trend is now unfolding in the Application Lifecycle Management (ALM) space, driven by tools created by projects needing to support their own open source software delivery. While ALM tools are often associated with the heavyweight workflow characteristics of enterprise application development, successful open source projects are a great example of the transformation underway in ALM, towards lean methods and lightweight, developer-centric tools.

In contrast with the application development tools which we use for writing and debugging code, ALM tools assist us with an application’s evolution over time. At their core, ALM tools track tasks and changes, help manage builds and releases, and support the dialogue that defines an application’s evolution. This subset of ALM is sometimes referred to as Application Development Management (ADM). On top of this core feature set layer tools for project and product management. Large organizations add additional tools to the stack to handle program and project portfolio management.

Thanks to a combination of resource constraints, a preference for using open source technologies for open development, and the common desire for developers to sharpen and extend their own toolset, the past decade has delivered a critical mass of the open-source ALM tools. Consider the scale of ALM on Eclipse.org: 33M lines of code in the last coordinated release (733 installable features/components), 330K total Bugzilla reports (3K new each month), 15K Bugzilla users, 1K committers (400 active), 300 projects and 173 Hudson build jobs. Add to that dozens of interdependencies between Eclipse projects and other open source projects such as the popular Apache libraries. ALM on Eclipse is managed entirely with open source tools including Bugzilla, CVS, Git, Hudson, MediaWiki and Mylyn.

The 1,948 respondents to the 2010 Eclipse Community Survey provide an overview of the degree to which open source tools have percolated into commercial software development. Only a small fraction of the survey respondents were directly involved with Eclipse, and half were from organizations with over 100 employees. The striking pattern is that the core open source ALM tools, when combined, have the market lead in each of three key ALM categories visible in the figure below. In 2010 for these categories, open-source ALM has taken the market lead from closed-source solutions. While surveys of this sort are always skewed towards the type of developer who bothers to answer surveys, this result remains indicative of a shift in application development practices and open-source ALM market share. In 2011, I predict that this trend will continue and that open source tools will percolate into the ALM stacks of more conservative organizations. A degree or two of separation from their open source counterparts, many of those developers will not recognize the term DVCS before it is force fed to them.

The attraction to open source ALM is not just price point, but the amount of innovation that has been driven by open source developers building tools to support their own productivity patterns. The ecosystem of extensions that forms around popular open source projects is another key driver of adoption. Those ecosystems are also likely to produce the most interesting integrations, since open source has proven itself as the most effective mechanism for growing both the community and APIs needed for innovative extensions. Finally, organizations with long application and product lifecycles are attracted to open source ALM tools because a suitable open source license gives them confidence that they will be able to access and extend the knowledge base represented by their issue tracker ten years from now, when the landscape of ALM vendors will much different than it does today.

Open source ALM tools are built on developer-centric principles. Transparency is favoured over hierarchy, with every task and bug on Eclipse and Mozilla being editable by anyone. It relies on asynchronous collaboration and a consistent use of the task or issue tracker to capture all discussion relevant to changes in the software. It encourages lining up modularity and framework boundaries with team boundaries, allowing API layers to facilitate dependencies and collaboration. There are also things missing from the stack. Responsiveness to the community often takes precedence over planning, and after the fading away of XPlanner, there has been a distinct gap in project management features within the open source tool space. There is also no single integrated open source ALM stack, instead open source projects glue together their own best of breed solutions, and layer customizations on top, as is the case with the numerous tools built on the Eclipse Bugzilla repository. Integration with product, project and portfolio management tools is typically non-existent, as this is not something that even large open source projects need.

While open-source developers will continue working very happily with their increasingly slick tool set, this impedance mismatch with large-scale ALM implies major problems for organizations who are planning to get a lot of ALM functionality for free. There mismatch between both the toolset and the cultural aspects of open source ALM tools and what’s needed by the enterprise. Agile and lean development have the opportunity to bridge some of the cultural gap, but still have a considerable way to go in order to incorporate the lessons learned from open source. There is enough of a gap in the toolset that organizations already deploying open-source tools at the enterprise ALM scale have needed to set up their own ALM tool engineering teams. These teams create enterprise-level authentication and access control, provide third-party ALM tool integrations, and implement support for features such as linking existing Project Portfolio Management (PPM) tools. Due to the pace of change in open source ALM tools, they are fighting a losing battle. While wasteful, this exercise is currently necessary. Large organizations that fail to integrate already deployed open source tools into their existing ALM and PPM infrastructure will see a dramatic reduction in the predictability of their development process, since their process relies on a connectivity between development and planning tools that was present in the more traditional ALM tool stack.

There is hope. First, the benefits of open-source ALM tools are fundamental as the ease with which they allow developers to work makes us happier and more productive. The velocity of successful open-source projects demonstrates how good these tools are at supporting the delivery of high-value software that is responsive to the needs of the community and customer. On the flipside, Enterprise ALM tools provide management and planning facilities which are critical for predictability of delivery as well as the longer-term planning that is necessary for organizations to thrive. These two worlds must be integrated into a cohesive whole, especially as more Agile teams find themselves at the intersection of open source and enterprise ALM.

After I presented my keynote on open source ALM at W-JAX last November, a colleague from one of the world’s largest software companies commented that the same developers that worked on open-source projects were twice as productive as when they worked on closed source projects. We discussed the usual issues of motivation and incentive structure, but nailed down the key issue being the sheer friction generated by traditional ALM tools which has been removed from open source ALM tools. It is time to reconnect the software planning process to a new breed of open source ALM tools that support lean and high velocity delivery, connect them to the planning and management tools needed for software delivery at scale, and bring some of the benefits of open source development to the enterprise.

Watch Tasktop webinars

About Mik Kersten

Dr. Mik Kersten is the CEO of Tasktop Technologies, creator of the Eclipse Mylyn open source project and inventor of the task-focused interface. At Tasktop, Mik sets the strategic direction of the company as well as drives many of Tasktop's key partnerships and key customers accounts. He created Mylyn and the task-focused interface during his PhD in Computer Science at the University of British Columbia. Mik has been an Eclipse committer since 2002, is a 3-time elected member of the Eclipse Board of Directors and serves on the Eclipse Architecture Council. Mik's thought leadership on task-focused collaboration and improving the software economy makes him a popular speaker at software conferences, and he was voted a JavaOne Rock Star speaker in 2008 and 2009.

Tags: , , ,

11 Responses to “Prediction #5: Open source ALM tools continue to gain market share, give the development manager a migraine”

  1. Jilles van Gurp Says:

    There are some omissions (categorical ones) in your definition of what is ALM.

    First there is build tooling. The Ant and maven ecosystems are at this point highly complex ones.

    Then there is artifact management. E.g. nexus falls in this category. So does yum & rpm.

    Then there is deployment tooling. E.g. puppet is used commonly to manage deployments in large data centers.

    Personally, I think ALM is just the latest craze. When Borland (or whatever they call themselves these days) restyled themselves as an ALM company I personally though they were full of shit and were probably not going to be heard from again (so far, so good).

    Most of this tooling works around some fundamental issues in (particularly) the Java world where e.g. the lack of proper modules is compensated with build tools that simply zip artifacts in jars that then need to be managed because there is no notion of explicit provided and required dependencies or versions. All these tools are just layers of complexity that work around issues. In my view tools are part of the problem, not of the solution. OSGI (the main module system in eclipse) is part of this problem as well since it requires custom tooling and infrastructure to work at all.

    As ever there is good money to be made contributing to the problem and pretending to have silver bullet style solutions. Which in a nutshell is what the ALM industry is all about. Peel away the nonsense and you end up with some very convoluted ways of getting files from A to B.

    There are some very fundamental issues that ALM does not address, at all. Why do we compile code for example? Isn’t that a run-time concern these days? With compilation comes the need to worry about where the compiler output goes and how to package that and how to get those packages in the right place. Arguably it is just a (minor) performance optimization to compile ahead of time. Without it, deployment is just a git fetch. DVCS is all about change management. Deployment should just be choosing which branch/repository is the right one. If deployment actually worked like that, we wouldn’t need all this complex tooling to make it appear like that. Think about it, what’s so hard about taking a bunch of .java files, compiling them and making them available to a class loader? Why wouldn’t the classloader be able to encapsulate this job? Is that really so hard?

    Then what is all this bullshit about copying files around in this connected world? Deployment should just be publishing some resources centrally and invalidating their cache headers so that downstream nodes know that they are out of date. What’s so special about software that this requires elaborate rituals that involve packaging files into zip files and shuffling them around via scp? Files are so 1970′s. File based tooling and infrastructure are inherently flawed.

    ALM is not a solution, it is the symptom of a very deep problem.

  2. Javin @ Tibco RV Tutorial Says:

    Thanks for sharing information dude. Open source project has there own risk of using it until its not from Apache like foundation

    Thanks
    Javin
    How to detect deadlock in java

  3. Mik Kersten Says:

    http://tasktop.com/blog/eclipse/prediction-5-open-source-alm#comments

    > ALM is not a solution, it is the symptom of a very deep problem.

    Jilles, there is an eerie logic to your argument, and I agree that we need to be sensitive to solutions that handle accidental complexity with more layers of complexity. So the question boils down to what the fundamental complexities are in ALM, in other words, in tracking and managing the evolution of an application. It’s tempting to say that in the Java world, it’s been all layers of accidental complexity since the base object-oriented featured set and runtime provided by Smalltalk environments. With that relatively simple dynamic environment, developers enjoyed all the features that now require an incredibly elaborate set of tools and frameworks (compiler, JVM, debug protocols, hotswap, deployment infrastructure, virtualization, and IDEs that are 100x more complex). But this additional complexity has made it possible to distribute, deploy applications in a way that Smalltalk didn’t, which is why Java won. It’s similar with ALM. Yes, more complexity has been introduced and we should try to reduce it. But some is fundamental. I see the most fundamental concept as that of tasks, which track the change in the application over time. While it’s called by different names, this common thread of a task surfaces consistently in open source projects, many of which have thrived by adopting only the most basic set of ALM functionality.

    > Open source project has there own risk of using it until its not from Apache like foundation

    Javin, yes, “open source” has long ceased being a good in and of itself, and a range of quality exists in projects. It’s similar to blogs and publications, where the brands of both individuals and organizations help us separate the wheat from the chaff. But just as you will find a range of quality in the New York Times, you will also find a range of quality in the projects of very popular foundations like Apache and Eclipse. The business that drives some of the open source R&D can also be a key factor to consider, as we witnessed recently with the fall of the Symbian Foundation and the fallout of the Hudson/Jenkins fork. But suitable open source licenses and the ability for the community to influence the direction of a project do help reduce the risk of adopting key open source components into your stack.

  4. Tasktop Blog » Blog Archive » Prediction #5: Open source ALM tools … « Cache Says:

    [...] Here is the original post:  Tasktop Blog » Blog Archive » Prediction #5: Open source ALM tools … [...]

  5. Paul Says:

    Unless you have structure and backbone in place, you will never win, unless you’re in a small development team where you have a close interaction with the release management process. In the corporate world, with many developers, many release streams and multple test environments, you need a management system in place. Whilst you are correct to question the complexity of the system, you should appreciate that one is required. Open-source is clearly the way forward and the technologies out there complement the logical concepts intirely. What is currently missing though is integration to bring them all together. It is sure to happen soon, but as far as I know is not currently available. Please let me know should you know otherwise ( phooper0001@yahoo.com )

  6. Mik Kersten Says:

    Paul, I wholeheartedly agree with you, as I repeatedly speak with development managers at large organizations who are rolling open source ALM tools and encountering major problems because they are missing this kind of integration backbone. I will address this some in my next post. The game changer here has been that Mylyn’s framework becoming the basis for an interoperability layer between commercial and open source ALM tools, which something that we leverage heavily in the Tasktop Enterprise ALM federation solutions.

  7. Paul Says:

    Mik

    The company I tun is in the final throws of developing an open-source fully integrated ALM solution. When the website is released I’ll send you the link for review ifyoure interested. Fell free to email me directly to talk more should you have any rewuirements that you would like to see included.

  8. Tasktop Blog » Blog Archive » Tasktop 2.0 released with Eclipse Mylyn 3.5, brings sanity to your heterogeneous ALM stack Says:

    [...] tools are becoming favoured by developers and being adopted in a bottom-up fashion, but these are disconnected from both enterprise and Agile ALM tools. The resulting difficulty of modernizing and integrating the increasingly heterogeneous ALM stack [...]

  9. Java Tutorials Says:

    my observation is that it is better to donate the your product to Apache and then provide support from there. It streamlines the support process and brings more users.
    http://www.javaexperience.com/eclipse-java-and-web-content-assist/

  10. java67 Says:

    Like everything else, new technology takes some time to be adapted. Currently same thing is happening with Scala, which I see a friend of Java

  11. software companies indore Says:

    It is really a nice and helpful piece of info. I’m glad that you simply shared this helpful info with us. Please keep us informed like this. Thank you for sharing.

Leave a Reply

Notify me of followup comments via e-mail. You can also subscribe without commenting.