Steffen Pingel's Posts

Mylyn 3.9: Improved Code Review Tooling

Thursday, October 3rd, 2013

Two years ago I claimed that contributing to Eclipse is a tedious process and I’m happy to say that this no longer holds true. The migration from CVS to Git last year and the adoption of Gerrit for code reviews by many Eclipse projects, combined with the arrival of contributor license agreements (CLAs), has made it incredibly easy to contribute to Eclipse.

Gerrit is great news for contributors since anyone with an Eclipse.org account can now propose changes as Git commits. It’s no longer necessary to attach patches in Bugzilla; changes can be pushed directly to Gerrit. In Gerrit it’s easy to work with the changes and, if the CLA has been signed, merging contributed changes only takes committers a few clicks.

I have highlighted in previous posts how the Mylyn Gerrit Connector integrates code reviews in the IDE. When Eclipse.org added CLAs and upgraded to Gerrit 2.6 (which the Mylyn connector didn’t support at the time), I realized how much I rely on the seamless Eclipse integration. Thanks to Tomasz Zarna contributed over 20 changes and didn’t shy away from using the browser Gerrit 2.6 is now supported in the latest Mylyn service release and will soon be available as part of Kepler SR1.

In addition to supporting the latest Gerrit, Bugzilla and Hudson version, Mylyn, and the Gerrit the connector in particular, have seen a number of improvements in the last major release. The code review editors are now complemented by a Review navigator. The navigator view shows all comments in a structured view that simplifies keeping track of parallel discussion streams as code review progress.

Global comments are now embedded in the review editor, with support for spell checking, and comments can be saved as drafts and edited offline. Conveniently, the editor now also supports all patch set operations including rebase.

The Mylyn New & Noteworthy has an overview of all enhancements in the latest release which is available here.

Watch Tasktop webinars

Mylyn 3.8 released as part of Eclipse Juno

Tuesday, July 10th, 2012

We are happy to announce the availability of Mylyn 3.8 which was released as part of Eclipse Juno! This release includes new components for creating EPUB documents, Subclipse integration, user interface improvements and support for the latest Bugzilla, Hudson and Gerrit versions.



The Gerrit and Hudson connectors have come along way since we created the Mylyn Builds, Mylyn Reviews and Mylyn Versions projects two years ago, when Mylyn became a top-level project. I want to thank everyone who provided feedback in hundreds of Bugzilla tasks and helped evolve the connectors over three versions. With this release the APIs are reaching 1.0 and the projects have successfully graduated from Incubation status to mature projects.

To make these integrations that connect Eclipse to popular application lifecycle tools, available to a wider audience we added the Gerrit and Hudson connectors to several EPP packages. For instance, the RCP/RAP package now contains all components to complete a full Mylyn contribution cycle:

  1. Querying the Eclipse.org Bugzilla using Mylyn Tasks
  2. Activating a Bugzilla task using Mylyn Context
  3. Committing and pushing a change to Git using EGit and Mylyn Versions
  4. Monitoring the triggered build on the Eclipse.org Hudson using Mylyn Builds
  5. Completing the resulting Gerrit code review using Mylyn Reviews

I showed this at the Eclipse DemoCamp in Zurich and Java Forum Stuttgart and never left the IDE throughout the demo. It’s great to see the tooling coming together with the Eclipse 4.2 based Juno packages covering all key aspect of the development workflow out of the box.

The connector reference implementations that are part of Mylyn enable us to validate our own frameworks but feedback that we get from users and adopters building on the APIs is also incredibly valuable to drive development. I want to thank Kiu Leung for implementing an initial prototype of a Koji connector based on Mylyn Builds and Robert Munteanu for creating an integration for ReviewBoard based on Mylyn Reviews fueling the growth of the connector ecosystem.

Likewise, the number of Mylyn projects under the application lifecycle banner has been growing. The Model Focusing Tools project provides focus when working with models reducing information overload. The initial 0.9 release is part of Juno and contains connectors for EMF, GMF, Ecore Tools and UML. Manuel Bork and Carsten Reckord from Yatta Solutions recently published an article that provides a great overview of the underlying technology.

A lot has already been said about Mylyn Intent which brings documentation, code, models and other programming artifacts closer together: “Based on Donald Knuth’s Literate Programmingconcepts and allowing collaborative work around design tasks, Intent will try to reconcile developers with Documentation.” Through tight IDE and workflow integration, Intent ensures that code and documentation stay synchronized. Intent contributes its 0.7 release to Juno.

Vex, a visual editor for XML, is another quickly evolving project under Mylyn Docs which has been releasing milestones regularly. The tooling now supports XML schema and XHTML as well as basic rendering of images. Vex is not yet part of Juno but available from a separate update site and hopefully will join the coordinated release Kepler next year.

Last, but not least, there is Reviews for Eclipse (R4E) which is a project under Mylyn Reviews. R4E supports any artifact, including models, as a review source and has built-in capabilities to conduct formalized reviews that follow IEEE standards. The first 0.10 release was published shortly before Juno and R4E is now on its way towards graduation and joining Kepler.

It’s amazing to be part of the continuously growing Mylyn community and I would to thank all the individuals and companies that have contributed to this release.

Watch Tasktop webinars

Mylyn 3.7: Task List Search Index and Code Review Integration

Monday, March 26th, 2012

When Mylyn was still known as Mylar, our Bugzilla listed a few hundred tasks for the project. Six years later, my Task List has accumulated over 10K tasks for Mylyn alone and contains twice as many tasks in total. To help manage large task lists like that, the just released Mylyn 3.7 enhances the filtering in the Task List with an incredibly useful feature for quickly finding tasks: a Lucene based index. With the latest version, it is now possible to scope searches and instantly query over the full comment streams of all tasks.

The powerful new search options are easily discoverable through content assist (Ctrl+space). I particularly like the ability to filter based on date range. If I know for instance that the task I am looking for was commented on recently I can now find the relevant tasks with a few key strokes:

There has been a lot of talk about Gerrit, and Eclipse projects are increasingly adopting code reviews as part of their workflow. The Gerrit connector, which has been evolving in the Mylyn Reviews project, is now included in Mylyn releases, the Juno repository, and available from connector discovery bringing the first-class IDE integration that Mylyn provides for tasks to code reviews.

We have incorporated a lot of feedback from early adopters and continuously improved the connector. It now supports Gerrit 2.2 (earlier versions have been reported to work as well) and ships with support for a variety of authentication mechanisms including Open ID. The connector is tightly integrated with the latest version of EGit and extends the import wizard for simple importing of projects. When a Gerrit server has been configured, the wizard provides a listing of all Git repositories from that server. This enables simple cloning and importing of available projects.

Another noteworthy enhancement is the improved compare editor integration. The editor now shows the structure of the review. You can navigate through patch sets and add comments inline.

A key feature of Gerrit is the ability to stage multiple iterations of a change. In practice each iteration, referred to as patch sets in Gerrit, equals a commit that is tracked in a separate branch. As in the Gerrit web interface, the review editor can now show differences between patch sets to review incremental changes.

Sometimes it’s seemingly simple enhancements that add a suprisingly big benefit. We have long recognized hyperlinks as a universal way to link artifacts and made them clickable in the Task Editor. This linking now works across builds, tasks and reviews.

For example, Loading the JUnit results or reviewing the console output of a failed build that is linked from a review comment is now literally two clicks away from the review editor.

If you are at EclipseCon in Reston don’t miss the opportunity to see Mylyn 3.7 in action. Visit Tasktop at booth #25 and also attend one of the great talks that will highlight new features. A full list of new features are described in the New & Noteworthy section.

  greenbullet_icon Read the Mylyn 3.7 New & Noteworthy
  greenbullet_icon Meet Tasktop at EclipseCon

Watch Tasktop webinars

Tasktop at CeBIT

Wednesday, March 14th, 2012

Tasktop attended the Eclipse Foundation Island at CeBIT 2012, an event with massive attendance. Eclipse Island is part of the Open-Source forum which is one of a many booths in the hall which is one of out of two dozen halls, located a mere twenty minute walk from the main entrance. Still, a lot of folks were finding us to discuss Mylyn, Tasktop and Sync or ask where they can get the free Eclipse CDs.

When we weren’t busy demoing the latest Tasktop features we had a fun time explaining to 12 year olds that they need Git, Gerrit and Hudson if they want to be successful programmers. We’re watching for a flood of resumes around 2026. If you want to work with these tools now, check out our careers page for openings.

  Tasktop Booth at CeBIT   Tasktop Booth at CeBIT

Watch Tasktop webinars

November Mylyn events in Belgium and Germany

Thursday, November 10th, 2011

Eclipse Demo Camp

We had an awesome EclipseCon Europe with great talks and discussions that covered the latest developments in the Mylyn community. Fret not if you missed the conference. There will be more opportunities in November to see Mylyn in action.

I’ll be speaking at Devoxx in Antwerp, Belgium next week about connecting Agile, ALM and the IDE. The week after I’ll be heading to democamps in Germany to present the Git, Gerrit and Hudson integrations. The first demo is in Hamburg on November 23, followed by Kaiserlautern on November 24 and Kassel on November 29. Hope to see you there!

Watch Tasktop webinars

Experiences from migrating Mylyn to Git and Tycho

Thursday, October 27th, 2011

When Mylyn became a top-level project, the monolithic command line driven PDE build that was understood by one committer, executable on a single machine was no longer appropriate to support the modularization needs and distributed development of the project. Since we like to play with the cool kids we jumped onto the Maven/Tycho and Git bandwagon for a ride to the land where contributions merge without conflicts and sources build themselves. We are still on the road learning something new each time we hit the occasional bump. We already found out…

…and much more. While the new build system is not a silver bullet that made release engineering obsolete it’s a huge improvement over the previous configuration and enabled us to take advantage of tools like FindBugs or Gerrit that we could not easily integrate before.

If you are interested in learning from our experiences take a look at the video of the EclipseCon session.

Watch Tasktop webinars

Contributing to Mylyn through Gerrit Code Reviews

Monday, October 24th, 2011

If the length of the CC list is any indication of interest in a bug, a lot of committers are looking forward to Gerrit support at Eclipse.org. The webmasters are working hard on making that happen but for Mylyn we were so eager to integrate code reviews in our workflow that we went ahead and deployed a Gerrit instance outside of Eclipse.org. Although the server has only been running for a few weeks, over 50 reviews have already been opened by a dozen people. Read on for a description how to setup your workspace to start contributing to Mylyn.

The server setup with Gerrit 2.2.1 and Hudson 2.1.2 follows the configuration described in Alex Blewitt’s tutorial. Every time an update to a Gerrit change is pushed, a Hudson build is triggered, and the result is reported back on the code review. After having worked that way for a while I would not want to miss the benefits of quick validation of staged commits and the simplified feedback process for contributions and changes proposed by committers.

Registering a Gerrit Account

To use the review server a few steps are required:

  1. Register an account at review.mylyn.org using an Open ID provider such as Google
  2. Define a username
  3. Upload your public ssh-key for authentication

Once the account has been created you need to contact me to get the account added to the trusted users group which has the required permissions to create code reviews.

Adding Gerrit as a Git Remote

On the client end, Gerrit is simply added as another remote to the local clone of a Mylyn Git repository. For example, the .git/config file for the Mylyn Tasks git repository would specify these two remotes:

[remote "origin"]
  fetch = +refs/heads/*:refs/remotes/origin/*
  url = git://git.eclipse.org/gitroot/mylyn/org.eclipse.mylyn.tasks.git
  push = HEAD:refs/heads/master

[remote "gerrit"]
  url = ssh://spingel@review.mylyn.org:29418/org.eclipse.mylyn.tasks.git
  fetch = +refs/heads/master:refs/remotes/gerrit/master
  push = HEAD:refs/for/master

If you have cloned any of the Mylyn repositories already, you can simply add the gerrit remote section. No need to clone again. Note that Gerrit uses a special ref for the push configuration called refs/for/master. This ensures that changes are not directly pushed into the master but staged in a branch first.

Additionally, I recommend adding these settings to the .git/config file:

[gerrit]
  createchangeid = true

[branch]
  autosetuprebase = always

The createchangeid flag causes EGit to automatically generate a Change-Id header for each commit. This id is used by Gerrit to track updates to existing code reviews. The autosetuprebase flag will cause all new remote tracking branches to be configured for rebase automatically which is useful when updating task branches.

Creating a Code Review

To push a new code review to review.mylyn.org, first create a local branch to track your change. This is good practice and makes it easy to modify the change based on feedback from the code review later.

$ git checkout -t origin/master -b bug#191522-provide-full-text-search

Now go ahead and commit to the branch.

The next step is to create the code review. Simply run this command:

$ git push gerrit

Instead of pushing to git.eclipse.org (origin) this will push the change to review.mylyn.org (gerrit). Note that Gerrit creates one code review per commit. In case you committed multiple times see below how to merge commits.

Updating a Code Review

If an existing code review needs be updated with additional changes or if it needs to be rebased against the current master, checkout the branch and pull:

$ git pull

Then make the desired changes and instead of creating a new commit, amend the previous commit. This retains the change id and allows updating of an existing code review:

$ git commit -a --amend

To make the update visible on the server push again:

$ git push gerrit

This creates another branch that is automatically attached to the same code review.

Pushing Changes to Eclipse.org

Once a code review is approved in Gerrit, changes are automatically merged into the master. Here comes the quirk: In the case of review.mylyn.org this does not mean much. The Git repositories on review.mylyn.org are mirrored hourly from git.eclipse.org overriding any changes that were merged by Gerrit. For good reason, only committers are allowed to push to git.eclipse.org hence a committer needs to run this simple command eventually to make changes visible in origin/master:

$ git push origin

Contributors need to go through the standard Eclipse contribution process. Once a code a review is approved a patch needs to be created and attached to a bug report to track the IP in accordance with the Eclipse process. This is very easy using EGit’s drag’n'drop support for commits.

Squashing Commits

If you committed multiple times you can always squash commits later using interactive rebase, e.g. to merge the last two commits into one commit use this command:

$ git rebase -i HEAD~2

This opens an editor to modify the commit history. If you change “pick” in the second line of the editor to “fixup”, that commit will be merged with the previous commit when you exit the editor.

Accessing Gerrit through HTTP

If you are not able to connect to review.mylyn.org through SSH on port 29418 you can instead setup a password for HTTP access. In this case the url in the remote configuration needs to be modified slightly:

[remote "gerrit"]
  url = http://spingel@review.mylyn.org/p/org.eclipse.mylyn.tasks
  fetch = +refs/heads/master:refs/remotes/gerrit/master
  push = HEAD:refs/for/master

More Information

The Mylyn contributor reference has a lot more information on how to get started. I am looking forward to your code reviews and contributions to Mylyn!

Watch Tasktop webinars

Mylyn Galore at EclipseCon Europe

Wednesday, October 19th, 2011

November will kick off with EclipseCon Europe in Ludwigsburg, Germany. It’s an excellent conference and a great opportunity to connect with the community. But, it gets even better: This year’s program is loaded with Mylyn related talks that cover a range of topics about exciting developments in Mylyn!

Wednesday

Thursday

Friday

If you would like to meet us for a chat or grab a cold beverage, please connect to @bmuskalla or @steffen_pingel or catch us at one of the talks!

Watch Tasktop webinars

Integration Goodies for Git, Gerrit and Mylyn

Monday, October 3rd, 2011

It’s always a good time when you meet people in person that you otherwise only interact with through the Eclipse.org Bugzilla. Recently, Matthias Sohn organized a week long hackathon at the SAP offices in Walldorf bringing together committers from EGit, Mylyn, Gerrit and the community. From Tasktop, Benjamin Muskalla and I joined the hackathon. We enjoyed lots of free SAP coffee and had a great time working with the rest of the team. We started the first day with a brainstorm session which resulted in a task board full of story cards.

My first goal was to get the Mylyn Gerrit connector working with the latest version of Gerrit. Since the early days of Mylyn we have worked bootstrapped feeling the same pains as everyone else using our tools. We know how important it is to eat our own dogfood and now that code reviews in Gerrit have (almost) become routine in our everyday workflows. Updating the tooling to work with the server used by the Mylyn project was an obvious step.

I’m happy that we made good progress during the week in Walldorf and the connector now works with Gerrit 2.2 while maintaining backwards compatibility with Gerrit 2.1.

Since the Mylyn instance is configured for OpenID authentication I also added support for that. It was an interesting exercise to extend the tasks framework to allow authentication through a browser window in a way that works across Windows, Linux and Mac.

Once the connector was working with the latest Gerrit, I took the opportunity to work with Dariusz from EGit Sychronize view fame. Dariusz knew instantly what classes to look at to enable navigating through patch sets from the Synchronize view.

The feature is still a work in progress but basic functionality is already available in the latest weekly build: The compare action in the review editor fetches the patch set and opens the corresponding changes in the Sychronize view.

Another useful enhancement to the Gerrit Connector was committed by Sascha Scholz who added a field for specifying free form queries.

We had lively discussions with Stefan Lay, Matthias Sohn, Benjamin Muskalla, Manuel Donninger and others around automating task-based branching. Not surprisingly everyone in the room had slightly different ways of working when it came to Git and there was no straight forward answer how to best automate this. I believe we settled on a nice workflow that is not intrusive:

  1. On task deactivation, the checked out branches for all repositories connected to projects referenced in the task context are remembered
  2. On task activation, branches that were checked out on last deactivation (if any) are automatically checked out again restoring the same workbench state

To support that Matthias committed a change in EGit that accelerates creating new branches by making the “Switch To” menu top-level in the popup menu of the Git repositories view. Additionally, Manuel proposed an enhancement to pre-populate the branch name based on the summary of the active task literally reducing the click count for creating a task branch to 4 clicks.

Based on the progress that Manuel had already made, a number of additional changes to persist branch information per task for Mylyn were proposed that are currently under review. If you have any input, please comment on bug 309578.

A number of other features that are beyond the scope of this post were being worked on throughout the week including significant performance improvements for EGit. Of course we didn’t limit our activities to coding but socialising was a big part of the hackathon.

Thanks to Matthias for organizing the meeting. Hopefully we’ll get a chance to repeat this in the future!

Watch Tasktop webinars

Stage, Build, Review with Git, Gerrit, Hudson and Mylyn

Tuesday, July 5th, 2011

Getting a contribution accepted into Eclipse is not a walk in the park. In the Mylyn project, each patch that comes in through Bugzilla is reviewed by a committer usually going through multiple iterations until it makes its way into Git or CVS. Mylyn has great support for handling patches, but this can still be a tedious process for both the contributor and committer when it comes to larger changes, due to a lack of automation of the contribution workflow. Contributors need to keep patches up to date, possibly juggling several outgoing change sets which can easily lead to incomplete or over inclusive patches. On the receiving end committers are burdened with validating patches and running tests which requires a number of manual steps. Conversations around patches happen through Bugzilla comments making it difficult to refer to particular lines.

These problems are not limited open source contributions. In many organizations, development follows similar work-flows often requiring use of several applications that are not integrated. This article introduces the new Mylyn connector for the open-source Gerrit code review system that helps automate onerous parts of the code review process enabling developers to focus on collaborative aspects and code rather than nit picking broken patches.

Gerrit is a web-based code review server for the distributed version control Git. When combined with Hudson/Jenkins it forms a powerful system that stages and validates every change before other developers have even noticed. With the integrations available in Mylyn 3.6 and Mylyn Reviews 0.8 developers can now seamlessly integrate these tools in their day-to-day work-flows.

Alex Blewitt has written an excellent tutorial how to configure Git, Gerrit and Hudson. In this article we will assume that the following servers have been setup as described in the tutorial:

To get started we need to install the following tools from their respective p2 repositories into Eclipse:

To illustrate the interactions in a typical code review we’ll look at how Alice and Bob work on on a code change together. We’ll start with Alice who intends to implement an enhancement for an existing project that is already setup in Gerrit and Hudson.

Alice has cloned the Git repository that is managed by the Gerrit server into her workspace. In her environment the server is called dev.



By default EGit creates a configuration that pushes the local master branch to the remote master branch. For Gerrit, Alice needs to change the configuration to push the local master to refs/for/master instead. Gerrit uses that special location to stage changes for the master branch.



Alice is now ready to commit her enhancement and push it to Gerrit. The commit dialog has an option to automatically generate a Change-Id which is used by Gerrit to track the change. If Alice wants to update her commit later Gerrit can relate the changes based on that ID.



The confirmation dialog shows that the push to refs/for/master automatically created a new branch to track the changes for the code review.



In our setup we configured a Hudson server to monitor the review server for new branches. Every time a new branch is detected a build is automatically triggered. Alice now wants to verify that all tests pass in the build that was triggered by her push before involving Bob in the code review.



After Alice has configuring the build server in the Team Repositories view and subscribed to the corresponding Hudson job she can monitor the status in the Builds view without switching to the browser. Since Alice’s commit broke one of the integration tests the build status is red. This is not problematic since the commit is staged and not yet visible in the master branch.



It’s easy to drill down into the details of a failed build. Double clicking a job opens an editor that shows the failed tests, changed files and created artifacts. With a few clicks Alice can open test results in the JUnit view and re-run the failed test in her workspace.



Once Alice has fixed the bug she commits and pushes again to replace the previous change on the code review. To update the existing code review she amends the previous commit. That way, the commit message gets automatically populated with the Change-Id from the previous commit which identifies the code review. Through the ID Gerrit knows that the new commit replaces an existing change and is not a new code review.

Again, the build server picks up the changes and this time all tests pass.


Alice is now ready to pass the change on for review. She adds a task repository for the Gerrit server and a new query in the Task List.





The query is a personalized subscription to the review server. It brings in code reviews Alice is interested in into the IDE and enables her to work with them. Like any other Mylyn task, code reviews can be scheduled and activated embedding reviews seamlessly into the task-focused work day.

Code reviews open in an editor that is the hub for the conversation around a Gerrit change. The editor shows details about the work-flow state of a review, the people involved and the comments.

The review that is shown here has received one vote already. When the Hudson server executed tests it posted the results back to the review. A +1 in the Verified category indicates that the build passed.



To pass the code review a +2 is required in the Code Review category. To notify Bob that the change is ready for review Alice adds him as a reviewer in the people section.



Bob’s workspace is setup with the same projects and queries. When Bob task list refreshes a little popup notification is displayed and the review is decorated with a blue arrow in the task list.

Bob’s first step is to open the code review. The content under review is listed in the patch-set section. Each patch-set equals one commit to the Gerrit server and is tracked in a separate Git branch. Bob can now bring a patch set into his local workspace by using the fetch button under the patch set. This is useful to verify a fix or to try a feature through running changes locally. In this case Bob decides to review changes directly in the compare editor.



Double clicking a file in the patch-set section opens an editor that shows the current master on left and the proposed changes on the right. Comments can be added by selecting any line in the file and right clicking. Initially, comments are saved as drafts and are not visible to others.



After Bob has completed the review adding several comments he publishes them. The publish button opens a dialog that allows Bob to provide a rating.



With Bob’s +2 vote all requirements are satisfied and Bob submits the review as the last step. The most recent patch on the review is automatically committed to the master branch by Gerrit and becomes visible to all developers.

With these IDE integrations, Tasktop provides the same task-focused productivity benefits that developers already enjoy for tasks and SCM systems to code reviews and builds, capturing more of the developer’s workday and providing even better integration for mixed open source and commercial ALM stacks.

To see a demo watch the video at the end of Mik’s blog post on Mylyn 3.6.

Watch Tasktop webinars