Tasktop

If you are involved in large-scale software development, you may be alarmed to hear leading analysts discussing the growing number of Agile rollouts that are failing. Several factors are contributing to this. Legacy and in-house ALM tools have proven to be sticky and hard to migrate from. Their content, customizations and workflows encapsulate product history. Agile tools have moved in, but have not been adequately connected to developer workflow or to enterprise ALM tools and processes. Lightweight open source ALM tools are becoming favoured by developers and being adopted in a bottom-up fashion, but these are disconnected from both enterprise and Agile ALM tools. The resulting difficulty of modernizing and integrating the increasingly heterogeneous ALM stack is a key cause of the failing Agile deployments and ALM stack modernization efforts.

For the past three years, the Tasktop 1.x train for releases was focused on providing developers with the integrations that they need to work with Agile and ALM tools within their home environment, the IDE. Our goal is to bring the productivity benefits of the task-focused collaboration paradigm to as many developers as possible. On the Mylyn side, this has meant collaborating with Agile and ALM vendors and contributors in order to grow the frameworks that have become the most widely extended APIs for ALM integration. With last year’s restructuring of Mylyn to become a top level Eclipse project, these frameworks grew to encompass SCM tools, Build and Continuous Integration tools and code reviews. The Tasktop 2.0 release extends this ecosystem with new Mylyn connectors and new features that support the range of our integrations, including Task Federation™ and cross-repository Agile planning.

New Mylyn Connectors

For the 1.x release, our obsession with improving the developer’s workday meant providing integration with the huge breadth of ALM tools in use today. This meant getting involved with each of the corresponding open source projects and partnering with the best-of-breed Agile and enterprise ALM vendors in order to enhance their web service APIs and establish the long-term support needed for this level of interoperability. The figure below highlights the new integrations made available as part of Tasktop 2.0 in green, with the most notable being:

		HP Agile Accelerator (Tasktop Enterprise): This builds on our existing support for HP Quality Center and ALM in order to provide both the task management and IDE-based Agile planning facilities. Learn more.
		Accept360 (Tasktop Enterprise): Brings requirements and other product management artifacts to the developer’s desktop in order to better connect development to planning activities. Learn more.
		SmartBear CodeCollaborator (Tasktop Enterprise): This is the first full-featured Mylyn connector for a code review tool, SmartBear’s popular CodeCollaborator. Learn more.
		Polarion ALM (Tasktop Pro): Polarion is the vendor behind the Eclipse Subversive SVN client project and we now have full support for the Track & Wiki and ALM solutions. Learn more.
		Hudson/Jenkins (Eclipse Mylyn): For the Mylyn 3.5 release we created a Hudson integration that layers on top of Mylyn’s new Builds framework. On top of this Tasktop provides traceability with tasks that exist in ALM tools such as HP Quality Center. Learn more.

Tasktop for Visual Studio

Many of our customers have been asking for support for their teams and colleagues who are based in Visual Studio. Organizations are building both Java and .NET based applications that need common lifecycle management tools. The Tasktop 2.0 release includes the GA of Tasktop for Visual Studio, which brings the ecosystem of our task connectors into the Visual Studio IDE. Whereas previously only basic integration existed for task management within Visual Studio beyond Microsoft’s TFS product, you now get Mylyn’s functionality including instant opening of tasks, offline access, the Task List and a full-featured and WPF-based task editor.

Supported Connectors (Tasktop Certified): Accept360, Atlassian JIRA, CollabNet TeamForge & ScrumWorks, Edgewall Trac, Google Gmail, HP Quality Center & ALM, IBM Rational ClearQuest, Microsoft Outlook, Microsoft TFS, Mozilla Bugzilla, ThoughtWorks Mingle, Polarion ALM, VersionOne. Adding support for the remaining Tasktop Certified connectors is underway.

Learn more about Tasktop for Visual Studio.

Task Federation, for Developers

The big news in Tasktop 2.0 is not the new connectors, but what we have created on top of this ecosystem in order to solve the problem of ALM stack heterogeneity. Tasktop 2.0 includes a Task Federation framework that provides a layer of insulation between the various stakeholders in the software lifecycle and the implementation details of the organization’s ALM stack. Task Federation manifests differently for each set of stakeholders.

Developers are often stuck working with multiple ALM tools. You may be working on two different projects, each of which has a different ALM tool. Or you may be adopting Agile, but still needing to use the company standard defect tracker. Those of us depending on open source frameworks and SDKs will have dependencies between our internal task and those managed in an external repository such as Eclipse.org or Apache. Tasktop 2.0 provides full support for linking dependencies between the various task repositories that define your software’s evolution, and new features such as cross-repository Agile planning and task synchronization build on this support.

Task Federation, for Agile Planning

The biggest single new feature in Tasktop 2.0 is the release of the Agile Planner that we launched as beta at the Agile 2010 conference. Just as we do not replace your existing ALM servers, we are not replacing your existing Agile planning tool. Instead, we augment it with our unique interoperability features. The Tasktop Planner provides:

		Task Federation: The most important feature of the planner is its ability to show plans that span repositories. Planning your user stories in an Agile tool but using HP Quality Center for defects or an open source issue tracker is no longer a problem. Depending on another team or an open source project? These dependencies now show right in your release plans since Tasktop can display and reconcile state across repositories. All of the cross-repository state is stored in your existing ALM tools as hyperlinks and metadata.
		IDE Integration: This is the first full-featured cross-repository Agile Planning tool within the IDE. You get a fully native and offline-capable Scrum style task board and a release planner. Relevant content from your web-based planning tool, such as burn down charts, is available when connected via our embedding of web UI gadgets. For product owners and project leads, the planner is also available in the Tasktop desktop edition.
		Developer Focus: Whereas most tools focus on project-level views, the Tasktop Planner supports developer centric workflows such as highlighting everything assigned to you for the current Sprint or populating your Task List with user stories assigned to you for an iteration. Everything you need for your daily Scrum or release planning is always at your fingertips.
		Automatic Time Tracking: The Tasktop Planner uses time values from Tasktop’s automatic time tracking facility to make Agile estimation dramatically more realistic
		Supported Tools: Agile plans can be displayed from HP’s Agile Accelerator and Rally. More tools will be supported in subsequent releases. All Tasktop Certified connectors are supported as targets for cross-repository plans.

Task Federation, for the ALM Tools Manager

In addition to providing developers and plan stakeholders with a unified view of the various tools and servers that define their planning activities, Task Federation can also be deployed to further unify an ALM stack. Say that you are deploying IBM Rational Team Concert (RTC) for Agile planning, but have realized that HP Quality Center (QC) is completely entrenched in your defect and quality management process. You can deploy a Task Federation setup that bi-directionally synchronizes all the key tasks and fields between RTC and QC so that RTC has the state needed for planning, while QC happily remains your system of record for quality management. Since each Tasktop client includes the full federation functionality and uses Mylyn’s hardened bi-directional task synchronization and conflict resolution interface, it is possible to deploy this solution in a scalable and secure fashion without requiring any new servers.

Learn more about Task Federation

Task-Focused Continuous Integration

As always, our commercial efforts in Tasktop have been driving framework and open source tool improvements in Eclipse Mylyn. The most notable of these is the new Hudson tooling, which also supports Jenkins. You can now work with Hudson day-to-day entirely from within the Eclipse IDE, monitor and run builds, and inspect results. Accessing build information is instant and available offline. One of the most useful automation features is the ability to run tests that failed on the server within your workspace, with just one click. But the most profound feature here is the first phase of our new support for ALM traceability. If your developers are using Tasktop, thanks to the magic of task activation and automatic change set tracking, you now have perfect traceability between your Hudson builds and the tasks that changed for a given build, providing an unprecedented level of traceability between your tasks, source and builds.

Learn more about Mylyn 3.5

Summary

With this release of Tasktop 2.0, we are taking a big step forward in bringing a new level of interoperability and sanity to your Agile deployment and ALM stack, helping developers focus on coding while giving teams a much more complete and real-time view of plans and progress.

		Read the Tasktop 2.0 New & Noteworthy
		Sign up for my webinar on Tasktop 2.0

Voting for the Eclipse Foundation board member elections ends this Friday. If there is something to be learned from this year’s Hudson vs. Jenkins duel, it is the importance of the governance model of popular open source projects. The Eclipse Foundation’s governance has been effective, continues to evolve, and is not something that we should take for granted. As the technology and business landscape around Eclipse change, the leadership and direction of the platform and ecosystem must adapt. What makes Eclipse’s governance so unique is that it is designed to combine both commercial and community interests in the tools and frameworks hosted on Eclipse. Vendors are able to help direct Eclipse by voting with their membership level and with their election of sustaining member representatives to the board. Community members vote with their contributions, and have the opportunity to elect their representatives to the board. The result is nearly two dozen board members with a wide diversity of opinions, guided and reconciled by Mike Milinkovich and functioning as exemplary not-for-profit software organization that helps shape the broad innovation network that is Eclipse. My vision for Ecipse is included below, and those of the other candidates are listed on eclipse.org.

Vision

Eclipse has matured and cemented its role as a platform for innovation. As its Sustaining Members, we play a key role in driving the direction of Eclipse. Our contributions in the form of code, evangelism, as well as Eclipse-based products and services have made Eclipse the vibrant multi-vendor ecosystem that it is today.

In joining Eclipse as one of the first non-IBM committers, I have been engaged with both the technology community and commercialization aspects of Eclipse since the outset in 2001. For the past year I have been your representative on the Eclipse Board of directors, previously served as committer representative, lead the now top-level Mylyn project, and am currently serving on the Architecture and Planning Councils.

I have worked directly with many Solution Member companies and watched some Eclipse business models flourish while others have failed. Starting a profitable company around Eclipse has given me a pragmatic perspective on how to participate in the platform while growing both product and service revenues on Eclipse-based offerings. For the past year I served as your representative on the Eclipse board, and for this next year I plan on continuing to apply my experience and ongoing dialog with members in order to help make your Eclipse-based efforts successful.

Succeeding in Eclipse means striking a balance between the member, committer and user communities. As your board representative, my priorities will be to:

		Facilitate reaching users: Marketing in a vendor-neutral open source ecosystem can be far from obvious, and reaching the very broad Eclipse user base is needed for both the commercial success of members and of Eclipse as a whole. I have been closely involved in improving the install story for Eclipse solutions and the upcoming Eclipse Marketplace client. These efforts will provide the next generation of opportunities to get your solution into Eclipse users’ hands.
		Lower the barrier to engaging with projects: With product releases and other pressures, too often the step of getting directly involved with an Eclipse project that you build on does not make the cost/benefit cut. But not getting involved has the longer-term cost of incompatibilities and overhead. Lowering the bar for direct participation will provide members with better access to the collaboration and contributor community that make Eclipse a great open source ecosystem. On the Mylyn project we have had over 900 bugs and feature requests resolved by community contributions. I will use my experience of making this work for Mylyn to help lower the overhead of participation with existing projects so that similar benefits are accessible to more members.
		Listen to members: My job will be to represent your needs on the board, and I will work with the other representatives to actively seek out feedback from all Solution Members. I helped make the needs of individual committers heard as a committer representative on the Eclipse board, and will ensure that your needs are represented in discussions and decisions made at the board level.

We made good progress on these priorities in the past year, and I hope to continue representing you and helping you make the most out of your investment and engagement in Eclipse in 2011.

About the candidate

Dr. Mik Kersten is the CEO of Tasktop Technologies and lead of the Eclipse Mylyn project. Mik is a popular speaker on Eclipse at conferences in the United States, Germany, and worldwide. As a long-time Eclipse evangelist, Mik is a frequent source for media and analysts on matters related to Eclipse and developer tools. Highlights of Mik’s contributions to Eclipse include:

		Creator and lead of the Eclipse Mylyn top-level project (since 2005)
		Active member of the Eclipse Architecture Council (since 2007)
		Member of the Eclipse Planning Council (since 2010)
		Elected committer representative on the Eclipse Board of Directors in (2008/2009)
		Only Eclipse evangelist to be voted a JavaOne Rock Star (2008 and 2009)
		Only Eclipse content author on the top authors of the decade list of IBM developerWorks Java
		PhD in Computer Science from the University of British Columbia that lead to the invention of Mylyn’s task-focused interface (2006)
		Co-creator of the Eclipse AspectJ and AJDT projects (2002) based on his work creating of the first Aspect-Oriented Programming (AOP) tools, Xerox PARC (2000)

Related publications

		Symbian Blog: Mik Kersten on Transparency
		Tasktop Blog: Growing open source ecosystems: the install story
		Tasktop Blog: Tasktop working with Microsoft to improve Eclipse on Windows 7
		How Software is Built: Interview with Mik Kersten (highest ranked in series)
		Tasktop Blog: Tips on paying for free software
		Tasktop Blog: Platform for Innovation, Part 1: Openness & Modularity

Read more about Mik Kersten on Wikipedia.

The productization of DOS by Microsoft in the 1980s kicked off the trend of platform vendors clamouring for developer mindshare. Apple’s App Store may only generate 1% of the company’s profit, but it drives the platform that sells Apple hardware. One of the best examples of the value of a developer platform is Microsoft’s Visual Basic (VB), which is yet to see a rival in terms of breadth and pace of adoption. In 2010, VB ranked as the 5th most popular programming language on the Tiobe index, still well ahead of iOS’s Objective-C. Microsoft’s investment in developers goes well beyond paying homage to the company’s roots, since Microsoft knows the sheer number of Windows license sales that can be attributed to each developer that they won over with VB.

Developers drive the success of platforms. Whenever a platform shift occurs, such as the shift from proprietary to open source or from desktop to mobile, interest in gaining developer mindshare is renewed. While we may all be overloaded with atmospheric-condition-related acronyms, we are in the middle of a fundamental shift in enterprise software platforms. The VMware hypervisor has replaced Dell blade server as the hot new technology. Application architectures are being transformed by REST-based services. Over the course of the year, support for multi-tenanting Java applications will result in more significant changes to the programming model that any of the incremental features coming in Java 7. The shift to cloud and Platform as a Service (PaaS) has ramifications beyond the change in the deployment model. One of the most important changes coming down the pipeline is the convergence of open source, PaaS and ALM. The continuous integration and delivery loop, becoming increasingly popular in Agile deployments, will put a new set of requirements on both ALM tools and the connection between the running application and the developer’s workbench.

All of this is good news for developers. There will be an increase in competition, choice, and budgets which are dedicated to bringing developers to new application hosting destinations and PaaS solutions. Over the course of the year, early adopters of PaaS will help to harden these new offerings in order to get them ready for the on-premise and private cloud primetime. If you are an early adopter and running a greenfield project, you’re bound to see some cool hosting and services thrown your way. But there are a few things you will need to watch out for when migrating to the bleeding edge. Beware the snake-oil pitch that moving your Java app into the cloud will be as easy as choosing a hosting provider. When migrating existing applications, you are likely to discover that achieving a scalable and cost effective architecture will involve significant efforts such as reconsidering your data storage, implementing multi-tenancy and porting the application to new ready-made services. This migration is similar to moving from a rural homestead to a downtown condo—it will take more than just a moving truck, your lifestyle will change, and you will spend less time worrying about your plumbing.

We have transitioned from thinking about applications as static entities that we build and deploy to living entities whose evolution is punctuated by releases and sprints. Given the turnkey automation promised by PaaS, developers will demand integration between the running application and the lifecycle tools used to manage it. This will drive one of the most important developer-centric trends in ALM, the convergence of application hosting and monitoring with ALM. The core ALM services of change management and continuous integration will become the hub between the developer’s desktop and the running application, with tasks as the unit of abstraction. An application failure or problem will automatically create a defect that includes the application state that is related to the problem encountered. The product owner will schedule that defect into the appropriate sprint, and when the developer picks it up, all of the code relevant to the defect will be instantly retrieved from the SCM system and brought into focus. The commit of the fix will automatically spin up a build in a staging environment, deploy it, and then pass it on to production or Ops when tests pass. This future needs a new level or coordination between the core services of the ALM stack and PaaS offerings.

Vendors that tie together the programming model, frameworks, virtualization, hosting and ALM will be the winners in the battle for developer mindshare. To capture a meaningful snapshot of a hosted application, the runtime must understand the programming model. To present this to the developer, the IDE tools must be capable of be effective in displaying that information alongside the code. And at each step of the way, it’s the ALM tools that will facilitate communication between the various stakeholders in the process: monitoring by Ops identifies a problem, that problem’s state is captured in the issue tracker, the issue pops up on the developer’s desktop, and then all subsequent changes are attached and captured on the corresponding task and reported in the build that was triggered. Workflows of this sort will require significant changes from ALM stacks, which have generally ignored the details of application build and deployment to date.

ALM services will also facilitate the migration of applications to the cloud. As we saw from the $212M acquisition of Heroku, supporting developer workflow provides a great on-ramp for application hosting. In the land of enterprise Java this is considerably more complex than a Git-based push and deploy of Ruby apps. But the need to bring this kind of simplicity to the Java application lifecycle is the same, and tasks provide the link necessary to scale continuous integration and delivery to large Java applications with a broader range of stakeholders ranging from dev, to Ops and QA.

Connecting the IDE to an ALM hub and using the hosted source and builds to drive deployment will provide a convenient on-ramp to application hosting offerings, and will significantly reduce the configuration and administrative burden currently placed on developers. While it can be fun to play around with configuring deployment environments and continuous integration servers, this level of busy work gets very tedious after a few months. A new level of automation is needed to help us focus on delivering application features and business value in the face of increasing platform complexity. Source, issue tracking and continuous integration will be a welcome gateway drug to cloud and PaaS application hosting. While having an app store claim 30% of all enterprise application revenues is not something we are likely to see this year, a Steve Jobs-like focus on turnkey simplicity of deployment and lifecycle management will help win the hearts and minds of developers making the move to the cloud.

Three years ago, in his presentation on the Git Distributed Version Control Systems (DVCS), Linus Torvalds provoked developers by declaring that “if you actually like using CVS… you should be in some mental institution.” Since then, awareness of and discussion around both DVCS and mental health have grown rapidly. For many considering a migration to DVCS, discussions of trade-offs are filling coffee breaks. After the first migration steps are taken, the conversations get more heated and circle around getting DVCS best practices under control. For teams and organizations modernizing their ALM stack, this year will involve deciding whether to begin moving to a DVCS system. This post discusses the fundamental benefits that DVCS will bring to your Agile ALM stack, and identifies the gaps in tools and best practices that need to be addressed in order to avoid having the inmates get overly annoyed with the management of their asylum.

In 2009, the Eclipse Community Survey showed that 20% of respondents used CVS and 58% used Subversion (SVN), with CVS coming in at 24% for companies over 5,000 employees. While DVCS usage has been growing, the portion claimed by CVS and SVN has also grown in the 2010 survey. The mental model of CVS and SVN is entrenched for the average enterprise developer and builds on decades of the knowledge, tools and best practices that have built up around centralized version control. Tools built on top of CVS and SVN, such as the Eclipse and Mylyn team support, have also modernized these dated protocols and servers. But what the rapid rise of Git in open source has made clear is the limitations of traditional file-based change tracking and the centralized model for commit access control. The current adoption rate of Git is sufficient evidence that DVCS works and scales for open source development. The question then turns to how those benefits can be realized by your teams and organizations.

The key benefit of DVCS is that the collaboration rules and conventions captured by the version control system align with the trust network that forms organically between developers. The centralized version control model relies on component boundaries and restrictions on committer rights for managing the division of labour required for software configuration management (SCM) of complex systems. This perspective falls into the same trap as thinking that a modular, object-oriented decomposition of software is sufficient to manage the complexity of a large system. It is not. Dependency injection, aspect-oriented and declarative programming are key components of the modern programming model. In other words, concerns that crosscut the hierarchical decomposition of any large system need to be captured in order to evolve the implementation of that system effectively.

Tasks, whether in the form of user stories, features or defect fixes, are what define the planning and lifecycle management of large applications. Tasks crosscut the system’s modularity by capturing the discussions between developers and end users around the software, not the API and component boundaries of the software. Aggregations of tasks into user stories, plans and releases drive the evolution of the software over time and define the boundaries that naturally form between developers. Centralized version control systems have focused entirely on changes and components, and ignored this more social dimension of coding. Managing a large system meant breaking a system into components and managing access control with commit rights. This aligned version control with the system’s modularity, not with the way that developers work on the system in terms of experimenting with and collaborating on changes to the system. The crosscutting nature of developer collaboration and review activities needed to be captured and facilitated, and DVCS provides us just that.

In the video above, a visualization of directory structure corresponds to the modularity of the open Mylyn project. After watching the video for a few moments, we see that the activity of the developers themselves layers on top of the software’s modularity and has a structure of its own. This structure corresponds to the tasks, change sets and patch reviews done by the Mylyn committers. By facilitating the common operations needed to support this workflow between developers, such as task-based branching and merging, DVCS changes the game by making it easy for developers to capture their activity within the SCM system itself rather than tracking it through external tools, or not capturing it at all. By expressing the collaboration, review and trust patterns common with open source developers, DVCS has fundamentally transformed SCM.

That’s the good news. The bad news is that it will take the better part of this decade for many organizations to consume the benefits of DVCS. That timeframe may seem pessimistic given that some large organizations are already in the midst of deploying Git and Mercurial. But DVCS is a discontinuous innovation. It both provides a transformational benefit over centralized version control and requires a fundamental shift in developer practices that previously used a considerably simpler model. Consider the introduction of object-oriented programming (OOP). The core ideas were already baked in the 1970s, but it took decades and truckloads of books to turn the ideas into a scalable development practice in the enterprise. OOP introduced polymorphism and type hierarchies, neither of which supported the older style search and grep-based code navigation. A whole new breed of tool support needed to be created in order for developers to get a handle on navigating OOP system structure.

While the switch from centralized to distributed version control is a smaller paradigm shift than the switch from procedural to object-oriented programming, the scope of the change should not be underestimated. Problems will arise in three categories. First, the mental switch required from the average developer, accustomed to committing to the main line of the code with branches being few and far between, will be a source of friction and costly in terms of developer training. Second, tool support for discontinuous integrations tends to lag the initial deployments. For the near future, developers accustomed to full-featured graphical or IDE-based clients will find themselves back in the command-line when working with Git and Mercurial. Those with some muscle memory for command-line version control operations will still need to retrain their brains, since Git’s command line options are more complex. Finally, the biggest challenge organizations adopting DVCS in 2011 will face is a lack of clear best practices. If you are launching a new project with the social dynamics of the Linux Kernel, you’re all set and don’t need to look beyond the best practices that Linus has forged. But if you are a medium or large organization, you will need to define the various practices around branching, merging, rebasing, reviewing and staging that are needed to effectively map tools like Git onto your development processes. The higher complexity of DVCS introduces a broader variability of best practices. Today we are using different Git practices for developing the Mylyn and the Tasktop codebases.

Organizations pushing the bleeding edge of Git adoption will discover that these problems cascade. Neither the change in developer mind-set nor best practices are encapsulated by the Git tool support available today. This means that many developers will struggle to get a handle on needing to gain a sufficient level of Git expertise and best practices, and these will have to be learned individually rather than being captured in the repeatable process provided by tool support. That approach scales to a few dozen motivated developers, but much less so with a few thousand. The top technical people at these organizations will invariably push for DVCS migration, provided they see the fundamental productivity benefits as attainable. But until DVCS is further along the technology adoption lifecycle, for most organizations the move must be made stepwise, starting with non-core and greenfield projects establishing the best practices and conventions. On Tasktop and Mylyn development we have migrated 1/5th and 1/3rd of the codebases respectively, allowing ourselves time to refine the best practices suitable for each of those, and add the EGit support needed for our developers and contributors to be productive.

By architecting version control around the social structure of the software development process, distributed version control has aligned the Agile and task-focused planning process with the versioning and change operations that we use to manage our software’s history. Since both the task-focused and the DVCS movement have focused on the natural cadence of coding and developer collaboration, there has been convergent evolution between tools like Mylyn and Git. In the not-too distant future, the developer will activate a task, the appropriate Gerrit change or topic branch will be created, stashed when the task is deactivated, and a continuous integration build for the topic branch spun up, with failures percolated back to the developer’s desktop as updates to the task. As DVCS best practices evolve, we will see more of this natural mirroring between the Agile planning loop and the DVCS mechanisms converging, lined up with tool-supported best practices.

Successful open source projects have a community and contribution driven planning loop, and can already reap the benefits of DVCS. But the enterprise software planning loop is more complex. In his dogmatic focus on open source software, when proclaiming that everyone using centralized version control was insane, Linus ignored the work required to bring the benefits of open source technologies to the software development community at large. The average developer’s training needs to be accounted for, best practices for incorporating DVCS into the planning loop must be defined, and tool-based automation is needed to bridge the gaps.

Development managers at large organizations with monolithic application lifecycle management (ALM) stacks once had it good. ALM components were well integrated, played nicely with one another, and when they didn’t, there was someone to call. But lightweight issue trackers started to move into the organization, popularized by the need for developer-centric collaboration facilities. At a cheap or free price point, these were often deployed without the corporate IT department’s knowledge or approval. Then the Agile project tracking tools moved in, and with the excitement around Agile, team and product leads pushed them through without consideration of standardizing across the organization. In the meantime, quality management tools like HP Quality Center remained so entrenched in their domain and so valued by management that they did not budge when the newer tools moved in. As a result, the large organization’s tool stack resembles an ALM history museum, ranging from the old in-house defect tracker that’s still running, to the freshly-installed Kanban tools intended to address disillusionment with Scrum.

No two ALM stacks look alike. The success of open-source ALM components has driven diversity into even the smallest software shops, and the new norm for the ALM tool stack is heterogeneity. While this brings the benefits of choosing your best-of-breed solution, it also means dealing with new complexity in terms of navigating the landscape of tools and options when considering how to modernize your ALM stack to support Agile. This post is a brief guide to helping you navigate that complexity by focusing on the options you have for modernizing the most core part of your ALM stack: the issue, project and change management layers. Tool support on this front boils down to capturing different stages and granularity levels of software development tasks, much as Salesforce has succeeded at capturing and tracking tasks across the various stages and stakeholders in the sales process. At the base level, we have tasks related to the code itself, consisting of defects, feature requests and tests. This is the realm of the issue trackers. At the next level, tools abstract over development-specific items to capture tasks relevant to management and planning, such as user stories and requirements. The new breed of Agile tools is popular here. Then, there are the enterprise ALM tools that can capture and track across products, releases and portfolios of projects.

In 2010, some ALM surveys included over 100 different issue trackers and change management tools to choose from. This enormous diversity has driven innovation, as vendors strive to differentiate their offerings in order to win over customers and to add value over the steadily rising bar of commoditization coming from open source. Last year we saw the price points of basic issue tracker support for small teams drop to dumping levels in an effort by vendors to displace their competition. In terms of feature sets, issue trackers are starting to look very much alike and, as with cars, their most visible differences are colour combinations and dashboard designs. In 2011, the issue tracker will be well on its way to becoming a commodity. The key thing to look for is that your issue tracker supports developer collaboration along with good ease-of-use for comments and updates, and that it is integrated with the workflow of the developer. The issue tracker also needs to be integrated with your planning loop. If you are a small team or organization, that may be as simple as managing a field with your release or iteration or setting up a wiki-based Kanban board with hyperlinks to the issues. But once you start scaling beyond a small team, tool-based planning features are required to scale your planning process across teams and geographical locations.

The next layer in the ALM stack is the project planning and tracking tool. With Agile starting its move into the enterprise, competition between vendors in this area is becoming fierce. Numerous ISVs are clamouring to lead the trends, implementing the latest Agile and Lean fashions as quickly as the books and blogs defining them are published. A few vendors, including Rally, VersionOne and ThoughtWorks Studios, have differentiated themselves by combining Agile tools with thought leadership and training. This helps organizations adopt the cultural aspects of Agile while bringing the vendor’s tool-based support for Agile practices along for the ride. These vendors have been key to many of the Agile rollouts to date, and have driven much of the management level adoption of Agile and Lean methodologies. New players have appeared in this space, introduce a wide range of expertise, and provide a spectrum of tools from simple layering of functionality on top of the issue tracker to full-blown enterprise ALM solutions. Picking the best solution here often depends on the size of your development organizations, how opinionated you want your Agile tool to be in terms of enforcing a particular Agile workflow, and selecting a tool that integrates with the rest of your diverse ALM stack. The latter often proves to be the greatest challenge.

If your development process needs to plug into a project, product and portfolio management loop, and also requires a connection to quality and requirements management, you need to reach one more level up the stack into the realm of enterprise ALM. Just as lighter-weight tools offer increasing benefit the closer you get to the developer’s desktop, the visibility, predictability and planning needs of a large organization with thousands of developers are addressed by these more heavyweight enterprise ALM tools. When you have a team of five developers, requirements management can often be handled by a shared understanding of the customer and the problem space. When you have a team of five thousand developers, many of whom are remote or outsourced, it’s almost impossible to be effective at managing delivery without this level of tool support. For large organizations the crux of the problem tends to be that the open source, lightweight issue trackers and Agile tools they have deployed do not provide this level of ALM support. This is the reason why enterprise ALM tools such as IBM Rational Team Concert (RTC), Microsoft Team Foundation Server, and more recently HP ALM, are providing support for the entire change management cycle. The trouble with getting the full benefits of those tools is, once again, the heterogeneity in your stack.

At each of these levels of the ALM stack, the key is ensuring that in the presence of heterogeneity, the code being produced and deployed from the development level is connected to the goals and product strategy determined at the planning and management levels. Failing to automate the connectivity between these layers means that the organization falls back to less reliable communication formats such as excessively long email threads, more tedious meetings, and the wearing new channels into the office carpets as you walk back and forth between cubicles.

What’s needed to bring sanity to ALM stacks is a “task federation” layer. At small organizations, the top-most system of record for the planning loop becomes the issue tracker. At medium-sized organizations it’s the Agile project tracking tool. Issue tracking facilities in Agile planning tools will increasingly displace the standalone tracker, or in the cases where the issue tracker is sticky or provides additional value, a task federation layer will provide linking between the tracker of choice and the Agile planning tool. At large organizations, task federation will become a critical part of the planning loop. For example, if you are deploying IBM RTC as your ALM tool you probably have HP Quality Center deployed already and do not want to lose its benefits for quality management. Task federation provides you with the bi-directional synchronization of tasks between the two systems, while ensuring that a tool like RTC has all the information needed for planning and that the quality management tool is the system of record for tests and defects. Or you could deploy HP ALM’s Agile features in combination with open source issue tracking and change management. What’s important is that task federation provides you with the options needed to modernize and streamline your best-of-breed stack, while ensuring that development is connected to your ALM solution of choice. While no two ALM stacks are alike, in 2011 we will start seeing the developer and manager stakeholders insulated from the intricate details of ALM stack implementation, with ISVs taking on the burden of integration.

The influence of open source on software development is often measured by the impact of successful libraries and frameworks. It’s hard to imagine building a modern web application without open source components. A similar trend is now unfolding in the Application Lifecycle Management (ALM) space, driven by tools created by projects needing to support their own open source software delivery. While ALM tools are often associated with the heavyweight workflow characteristics of enterprise application development, successful open source projects are a great example of the transformation underway in ALM, towards lean methods and lightweight, developer-centric tools.

In contrast with the application development tools which we use for writing and debugging code, ALM tools assist us with an application’s evolution over time. At their core, ALM tools track tasks and changes, help manage builds and releases, and support the dialogue that defines an application’s evolution. This subset of ALM is sometimes referred to as Application Development Management (ADM). On top of this core feature set layer tools for project and product management. Large organizations add additional tools to the stack to handle program and project portfolio management.

Thanks to a combination of resource constraints, a preference for using open source technologies for open development, and the common desire for developers to sharpen and extend their own toolset, the past decade has delivered a critical mass of the open-source ALM tools. Consider the scale of ALM on Eclipse.org: 33M lines of code in the last coordinated release (733 installable features/components), 330K total Bugzilla reports (3K new each month), 15K Bugzilla users, 1K committers (400 active), 300 projects and 173 Hudson build jobs. Add to that dozens of interdependencies between Eclipse projects and other open source projects such as the popular Apache libraries. ALM on Eclipse is managed entirely with open source tools including Bugzilla, CVS, Git, Hudson, MediaWiki and Mylyn.

The 1,948 respondents to the 2010 Eclipse Community Survey provide an overview of the degree to which open source tools have percolated into commercial software development. Only a small fraction of the survey respondents were directly involved with Eclipse, and half were from organizations with over 100 employees. The striking pattern is that the core open source ALM tools, when combined, have the market lead in each of three key ALM categories visible in the figure below. In 2010 for these categories, open-source ALM has taken the market lead from closed-source solutions. While surveys of this sort are always skewed towards the type of developer who bothers to answer surveys, this result remains indicative of a shift in application development practices and open-source ALM market share. In 2011, I predict that this trend will continue and that open source tools will percolate into the ALM stacks of more conservative organizations. A degree or two of separation from their open source counterparts, many of those developers will not recognize the term DVCS before it is force fed to them.

The attraction to open source ALM is not just price point, but the amount of innovation that has been driven by open source developers building tools to support their own productivity patterns. The ecosystem of extensions that forms around popular open source projects is another key driver of adoption. Those ecosystems are also likely to produce the most interesting integrations, since open source has proven itself as the most effective mechanism for growing both the community and APIs needed for innovative extensions. Finally, organizations with long application and product lifecycles are attracted to open source ALM tools because a suitable open source license gives them confidence that they will be able to access and extend the knowledge base represented by their issue tracker ten years from now, when the landscape of ALM vendors will much different than it does today.

Open source ALM tools are built on developer-centric principles. Transparency is favoured over hierarchy, with every task and bug on Eclipse and Mozilla being editable by anyone. It relies on asynchronous collaboration and a consistent use of the task or issue tracker to capture all discussion relevant to changes in the software. It encourages lining up modularity and framework boundaries with team boundaries, allowing API layers to facilitate dependencies and collaboration. There are also things missing from the stack. Responsiveness to the community often takes precedence over planning, and after the fading away of XPlanner, there has been a distinct gap in project management features within the open source tool space. There is also no single integrated open source ALM stack, instead open source projects glue together their own best of breed solutions, and layer customizations on top, as is the case with the numerous tools built on the Eclipse Bugzilla repository. Integration with product, project and portfolio management tools is typically non-existent, as this is not something that even large open source projects need.

While open-source developers will continue working very happily with their increasingly slick tool set, this impedance mismatch with large-scale ALM implies major problems for organizations who are planning to get a lot of ALM functionality for free. There mismatch between both the toolset and the cultural aspects of open source ALM tools and what’s needed by the enterprise. Agile and lean development have the opportunity to bridge some of the cultural gap, but still have a considerable way to go in order to incorporate the lessons learned from open source. There is enough of a gap in the toolset that organizations already deploying open-source tools at the enterprise ALM scale have needed to set up their own ALM tool engineering teams. These teams create enterprise-level authentication and access control, provide third-party ALM tool integrations, and implement support for features such as linking existing Project Portfolio Management (PPM) tools. Due to the pace of change in open source ALM tools, they are fighting a losing battle. While wasteful, this exercise is currently necessary. Large organizations that fail to integrate already deployed open source tools into their existing ALM and PPM infrastructure will see a dramatic reduction in the predictability of their development process, since their process relies on a connectivity between development and planning tools that was present in the more traditional ALM tool stack.

There is hope. First, the benefits of open-source ALM tools are fundamental as the ease with which they allow developers to work makes us happier and more productive. The velocity of successful open-source projects demonstrates how good these tools are at supporting the delivery of high-value software that is responsive to the needs of the community and customer. On the flipside, Enterprise ALM tools provide management and planning facilities which are critical for predictability of delivery as well as the longer-term planning that is necessary for organizations to thrive. These two worlds must be integrated into a cohesive whole, especially as more Agile teams find themselves at the intersection of open source and enterprise ALM.

After I presented my keynote on open source ALM at W-JAX last November, a colleague from one of the world’s largest software companies commented that the same developers that worked on open-source projects were twice as productive as when they worked on closed source projects. We discussed the usual issues of motivation and incentive structure, but nailed down the key issue being the sheer friction generated by traditional ALM tools which has been removed from open source ALM tools. It is time to reconnect the software planning process to a new breed of open source ALM tools that support lean and high velocity delivery, connect them to the planning and management tools needed for software delivery at scale, and bring some of the benefits of open source development to the enterprise.

How does a continuous integration (CI) tool named after a butler or two grab such a large market share when much more feature-rich and polished commercial counterparts exist? The naïve answer is that it’s free. If you dig a bit deeper, the success of Hudson can be seen as part of a larger trend in developer-centric application lifecycle tools. A growing number of open source tools have hit the mark in capturing developer collaboration and Agile lifecycle management practices. What has caused these tools to snowball in popularity is the rich ecosystems of extensions that they support. That combination has become lucrative to larger organizations wishing to increase the productivity of their developers. When an open source community starts smelling of money-making potential, a different breed of dog moves in. This can be a very good thing, as it is often the catalyst needed to move the project across the chasm to broad industry adoption.

The challenge for popular open source projects is to create a governance model that marries the interests of the project’s community with those of vendors supporting the project. The most successful open source projects are ones that manage this dichotomy without alienating either party. Spring, JBoss and MySQL won by having a single dedicated vendor managing commercial and community interests. Eclipse and Apache have created governance models that support the overlapping interests of multiple vendors. In the case of Hudson, with Kohsuke’s departure from Oracle, a split formed. In recent blog posts we’ve seen Kohsuke Kawaguchi, creator of Hudson, highlighting the community and the ecosystem of extensions that have made Hudson successful to date. On the public forums, Oracle’s Ted Farrell has emphasized the importance of versioning, consistency and stability, which are relevant for taking Hudson to the next level of enterprise adoption. Both of these concerns need to be addressed.

Over the coming year, extensible continuous integration is going to play a key role in both on-premise ALM stack modernizations and cloud-hosted ALM solutions. If you have a few grey hairs on your head, this may not sound new or noteworthy. A decade ago, we set up a CI build for AspectJ.org with CruiseControl and Ant and it is still running today. More modern Mylyn CI builds are very similar, but running Hudson and Tycho. What’s interesting is that over the past decade the continuous integration loop has become the underpinning of build automation in the Agile development process. Easy extensibility has made CI servers a convenient hub for plugging a variety of ALM tools into the Agile build loop. We are moving away from a world where realses, code metrics, testing tools and deployment destinations are being configured by each developer within his or her IDE. Instead, the CI server is becoming the hub that holds the authoritative build specs, with developers attaching to the portion of the ALM artefacts that they need to work on for the current release. Testing, quality, and metrics tools will increasingly use the CI server as the place to hook into the development process. In an upcoming prediction, I’ll discuss how combining task-focused workflow with continuous integration will prove to be a convenient abstraction for tool support that facilitates knowledge capture and sharing between Planning, Dev, Ops and QA.

All of this potential means that Hudson is a hot topic, and that Hudson’s consumers are scratching their heads on what to do about the Jenkins fork. Hudson’s success to date comes from the typical mix of passionate community leadership, open source extensibility and a vendor-sponsored marketing investment that helped create that community. The latter is often omitted from developers’ discussions, but now that we are in the later stages of open source maturity, a project’s access to marketing resources has become a key component to success. A case in point is the significant marketing budget of successful open source foundations, or that of the Mozilla Foundation’s, which is in the millions.

In open source, community-centric passions drive projects to their initial critical. Then comes the point at which enterprise support, stability, and the ability for other vendors to invest in a project become relevant. These help a project to graduate from an early adopter community to the pragmatists who can establish an open source project as a de facto standard. For that to happen, a project must be bigger than the cult of personality around its founder, just as any organization must be bigger than its leader to thrive. But with the heart of the contributor community in his hands, and his brain more wired to the technical vision of the project than any other person’s on the planet, the founder is often required to drive major downstream innovations.

In the year ahead, the existing consumers of Hudson will vote with their feet whether to go with the forked Jenkins codebase or with Hudson. The question will largely boil down to how stable the core of Hudson is, and whether upcoming changes to the core of Hudson that happen within Jenkins, and its key extensions, warrant a migration. In the closer-knit community there will be a tendency to Jenkins, as represented two hundred community votes, the majority of which were in favour of the fork. But in the much larger end-user community the default will be resistance to change in the absence of major and clear value add to making the move. We’ve seen this on Eclipse with the overly slow migration of many plug-ins to e4, which brings incremental value add and only benefit.

The large and more conservative consumers of Hudson will look to which corporate sponsor, Oracle for Hudson and the CloudBees startup for Jenkins, is providing better long-term assurance for their build infrastructure investment. There will be questions about Jenkins’ promise of neutrality. In the past, genuine open source project neutrality has only been achieved by large open source foundations with sufficiently diverse funding sources, and even then with mixed success. As such, while Jenkins is likely to see activity, Oracle’s Hudson is unlikely to go away and I predict that it will continue to be a major target of CI deployments through the year.

In the interim, as with the Emacs vs. XEmacs fork of the 90s, there will be some confusion and annoyance from those less interested in the drama. Thankfully John Ferguson Smart wrote his Continuous Integration with Hudson book in open source fashion, making it easier to refactor. If the divergence between the two projects is sufficient to fragment and fracture the community, there could be a large enough gap for a new open source CI system to join the scene, but it would take time for it to build up momentum. In the meantime, the usual API-level federation layer is likely to form so that plug-ins can support both tools, and we’re invariably going to be asked to create one in Mylyn. Most of the internet associates the name Jenkins with a certain Leeroy who went into a World of Warcraft battle wielding little more than an attitude and a chicken. I hope that in this battle less damage is done to the consumers of this otherwise winning CI technology.

A couple of years ago, a web-based code editor called Bespin was the rage at EclipseCon, and has been reborn in the form of the Cloud9 IDE for JavaScript. Eclipse re-embraced the push towards web-based IDE technologies with the introduction of a new code editor named Orion. While activity is again heating up on this front, do not expect the enterprise developer’s IDE to be running in the browser this year. What we will see is web-based advances in specific IDE use cases, ranging from code editors for DevOps to GUI builders for mobile, with most developers staying within the comfort and native UI of their desktop IDE.

Apart from the OS, the IDE may now be the most complex piece of software running on your desktop computer. Eclipse pushes the limits of the UI element handling of the desktop OS as well as the classloader model of common JVMs. What makes an IDE platform even harder to port into the browser is the need for extensibility, ranging from interaction with the filesystem to shared data models of the large variety of code and ALM artefacts that constitute an application.

Just as mobile applications are constantly pushing the limits of the hardware behind our 3.5” handheld screens, the IDE tool suite continues to push the limits of the hardware driving dual 24” displays. Third party plug-in developers need to be very aware of the performance of their extensions to Eclipse so that they don’t bring an Intel Core i7 CPU with 8GB of RAM to a crawl. The current generation of browser-based JavaScript runtimes is not up to the task of running the IDE platform as we know it. But we are starting to see the first crop of high-performance web-based code editors. Between the drive towards cloud hosting of applications and the push to lower the barrier to entry for mobile development, there will be plenty of incentive for vendors to provide SaaS offerings that incorporate key parts of the application development experience.

The 15M or so professional developers (Evans Data, 2009) will spend their day in Eclipse and Visual Studio, but these IDEs will begin to feel more like a connected Rich Internet Application (RIA) than a desktop-bound dinosaur. The data accessed by the open source developer, including source code, tasks and builds, is already cloud hosted. Eclipse extensions like Tasktop Enterprise mash up the web browser experience with the rich IDE experience. The professional IDE will start feeling more like a connected and offline capable Mozilla Thunderbird or Microsoft Outlook, with collaboration and planning facilities taking a front seat alongside social coding tools. Just as iOS has found the sweet spot between native experience and web service-based data sources, so will the IDE.

Outside of enterprise application development, domains where a simplified tool stack is applicable will start seeing web-based IDE features. Xcode is one or two order of magnitude less complex than a full-blown enterprise Java IDE. Palm’s Ares project was the pioneering example of how effective combining Bespin functionality with a JavaScript GUI builder for webOS could be for providing a simple user experience to hobbyist mobile developers. Even our Smalltalk forefathers would be impressed with an environment that supports authoring JavaScript widgets in a JavaScript-based GUI builder. Given the lack of a clear winner for JavaScript development environment on the Eclipse front, JavaScript will continue to be a natural target driving progress on moving IDE functionality into the browser. But the amount of work here should not be underestimated, as a lot of what already exists in the desktop-based IDE will need to be re-created to get to the sort of fidelity and breadth of tool support that’s now taken for granted.

Cloud ALM hosting with the connected IDE and Browser-based clients

On the cloud hosting front, key parts of ALM data such as source code, builds and tasks, are already starting to be hosted in a unified manner, as exemplified by the upcoming Code2Cloud offering. Some stakeholders in the development process, such as DevOps and QA engineers, will start to benefit from Eclipse Orion-style code editors accessing and fixing entries in Spring config files without launching the IDE. This will lead to easier management of production since the need for console access for these sorts of tweaks will be removed.

Looking further out, the increasingly connected IDE will have all of its configuration data hosted, making the experience of the first launch of Eclipse similar to that of the first launch Skype, where entering credentials causes the rich client to instantly provision the user’s desktop. The division of labour between web and desktop based access of software artefacts will improve, with some content that was previously browser-bound, such as Scrum task boards, being supported within the IDE. The mobile hobbyist, casual web developer, and specific stakeholders such as the DevOps engineer, will see nearer term benefits from the growing convergence of the browser and the IDE.

I miss the heyday of the IDE wars. Seeing David Intersimone’s demo of JBuilder at OOPSLA 1998 opened my eyes to the way that tools can multiply our productivity. The competition that ensued between Borland, Microsoft and others brought us code assist, refactoring, structured editing, GUI builders, and a far more integrated developer workbench than the windowing system and command line could provide. In 2001, the landscape changed dramatically when IBM killed a significant part of the developer tools business by open sourcing Eclipse (later re-entering the tools business by acquiring Rational). The open sourcing of Eclipse was a success and created an unprecedented platform for tool innovation. But Java developers started expecting their IDE to be free, and the IDE business was gone for all but the most inspired tool builders, such as JetBrains.

The base IDE feature set has now been standardized and commoditized. In other words, it has become a lot more boring. To get buzz on Twitter, vendors have to pull zany moves like running a thin subset of IDE functionality in a web browser. The bleeding edge aside, we won’t see any dramatic departures from the base IDE feature set anytime soon. However, we’ll continue getting many nice tweaks and features, as you would expect with mature tools like cars or can openers. As with other commodities that need to differentiate their extremely similar feature sets via slick dashboards and sophisticated cup holders, we will see some welcome improvements on the fronts of eye candy and usability. This kind of tweaking is already surfacing in Visual Studio 2010, which saw an investment in user experience and graphic design refresh that put it ahead of the curve.

While the base feature set of code editing, building, debugging and deployment solidified, tool innovation will now occur on top of the commoditized IDE platform. The major shift in the IDE over the last decade has been to shift from end-user tool to tool platform. And like any software platform, the IDE now needs to evolve more slowly because the innovation is happening on top of it. Beneath the IDE we’re swapping out the deployment model to be cloud-based, as with the SpringSource Tool Suite’s (STS) direct cloud deployment. Atop the IDE we are layering on all sorts of collaboration features, recognizing that developers spend as much time collaborating around code as they do hacking it. I can no longer imagine working without having rich and IDE-based access to a task board and release plan.

The IDE has multiplied productivity by integrating, then personalizing developer’s workbench

The maturity of the Eclipse 3.6 and Visual Studio 2010 platforms has made it possible to bring the developer’s entire coding and collaboration activity into a single workbench built on plug-in APIs and shared data models. In 2011, we will see the remaining APIs completed, such as Mylyn’s new APIs for IDE-integrated code reviews, distributed version control, continuous integration and documentation. Whereas last decade’s IDE was all about integration and helping developers stay in the flow of their work, the next decade of the IDE will be about personalization. We are shifting from the IDE being a view on the entire system to the IDE being a view of the parts of the system and planning that are relevant to the developer. The shift might sound less exciting, but its impact on developer productivity will be just as pronounced, and is necessitated by the steady growth of software complexity. The integration of Agile, ALM and a task-focused workflow will drive the IDE’s shift from an integrated workbench to a highly personalized one, incorporating the focus and flow needed to keep developers working on extremely complex software happy and productive.

Boeing’s 787 Dreamliner is reported to run 6.5 million lines of code, approximately the same number as the core Eclipse platform. The whole of the 2010 Eclipse Helios release train was 33 million lines of code. Add to this all of the third party extensions, and today’s IDE platform ranks with some of the most complex artefacts produced by humanity. Due to the scale of investment, marketing and ecosystem management, as with commercial aerospace, we will see the two big players dominate the IDE market. The small and niche players will continue to exist, ranging from the boutique IntelliJ to JDeveloper for database development. But for those rolling out an IDE across their organization in 2011, the number and availability of third party extensions will be a more important decision factor than the base IDE feature set. ISVs and cloud/PaaS vendors will increasingly use the IDE as an on-ramp for their offerings, with Eclipse and Visual Studio as the clear targets for their integrations. Following the user-centric success of the Embraer E-190, which provides a superior cabin experience to similar offerings by the two aerospace giants, a renewed focus on developer productivity and Agile collaboration needs to be the driver of change in the IDE over the coming years.

Back when we worked on systems that were tens of thousands of lines of code, Integrated Development Environments (IDEs) were a nice option, but often ignored by the alpha geeks living in Emacs and vi. Now that even mid-sized enterprise applications are close to a million lines of code when both the application and its dependent libraries are considered, the majority of professional developers see the structured editing and navigation facilities of an IDE as a fundamental requirement.

The initial evolution of the IDE and of Application Lifecycle Management (ALM) tools was independent. A decade ago, the OTI folks at IBM, master tool builders with their roots in the Smalltalk era, created an Eclipse integration layer that made CVS feel like a modern SCM system. That seemingly impossible feat taught tool builders an important lesson. Even if you had a grossly antiquated service, you could provide a modern and high fidelity experience in the client. A few years later, the initial Mylyn release signalled the next step of the convergence, by providing a modern task and issue tracking experience in Eclipse for Bugzilla. The migration from disparate web and command line UIs into an integrated coding-centric IDE was well underway.

Since that time, countless developers have recognized the time wasted by switching to a web browser in order to update an issue or to start investigating the latest must-fix defect. Application switching itself is relatively small time sink, adding up to an extra coffee break or two in the workday (i.e., 4% of the pie in the figure below). A far greater problem is that browser-based applications are ignorant of the system’s structure. This means that the burden of mapping between the code and the tasks driving our programming activity is placed on our already over-taxed brains. If a stack trace appears in a defect, we have to copy the stack frame, paste it into a search box, and iterate this process until we’ve found and fixed the issue. If this turns out to be something which must be handed off to someone else, then all of the research, analysis and other task-related state which has been built up during the exploration gets tossed out when we reassign the task, and the wasteful process is repeated. It all feels very 1990s. In my PhD research I measured the impact of this repeated searching, task-related context switching and redundant code navigation. These numbers were similar to the more recent ones visible in the figure below. Around half of a developer’s day is going to waste. That’s before accounting for meetings.

Time-based breakdown of a developer’s day, gray is waste

On the SCM side, we still have command line clients. If you’ve got time to burn, switching between the command line or shell extension to work with your SCM system may not pose a problem. If you’re a top-notch developer, you may even have the spare brain cycles available to work effectively in this way. But that doesn’t mean that it makes sense to do so. Many of the early pioneers of Smalltalk showed us the path towards integrated tooling long ago: everything should be explicitly present and directly manageable from within the developer’s workbench. Today, that workbench is the modern IDE, and the benefits of the IDE to the Agile and ALM workflow go far beyond having an embedded rich client. The deeper benefits come from the rich linking and data sharing that’s possible once the developer’s entire toolset is built on the same integration platform. Tasks, contexts, commits, builds and reviews can now all get automatically linked as we work, and all collaboration activity takes place alongside the code, where it belongs.

Similar benefits arise in moving developer-specific workflow from browser-based Agile project tracking tools into the IDE. The web browser has been a great vehicle for helping Agile tools take shape and for introducing them to the masses. The Web 2.0 trend helped issue trackers get more colourful with slick stylesheets and a visual refresh with the large and simple styling popularized by 37signals. However, there is a reason that desktop-based applications still dominate the productivity tools landscape. The modern IDE is capable of putting all but your desktop’s hardware, except the graphics card, to full good use. It is now common to access Twitter, email, and instant messaging have gone back to the future by providing a growing number of native desktop and mobile clients. Given the complexity of the way we collaborate on the application lifecycle front, the benefits of a native client are even more pronounced. . Over the course of the year, as Agile ALM tools continue to mature, IDE-based access will cease to be an afterthought and minor check-box on the ISV’s feature matrix, and high fidelity IDE tools will become the client of choice for developers accessing the Agile ALM stack.

Innovation on the browser-based application front will continue, and in the following predictions I’ll discuss the state of web-based IDEs. For 2011, we can safely consider the desktop-based IDE to be the place where most coding gets done. Web UIs for Agile and ALM will continue to serve an important role, for example, as the primary interface for management-level stakeholders and product owners. What’s exciting is that that we will also see additional convergence between Agile tools and the IDE. Minutes before the daily Scrum the developer will be able to instantly pop up an embedded task board showing all things assigned to them for the current sprint, and with a click activate a user story they were collaborating on with a colleague, automatically share progress and update time remaining, and come out looking like a rock star in the Scrum. Sadly it may take longer before we’re updating task boards with Xbox Kinect based Minority Report type gestures, but this is progress.