Tasktop

Creating integrations is hard, but testing them is even harder. Every web service API has its own vocabulary, semantics, nuances, and bugs. Every release of a web service potentially involves breaking changes, syntactic and behavioral. When these web services are controlled by 3rd parties, it gets even harder. As creating integrations is our business, we set out to improve how we create integrations; to create a scalable, reliable method of creating high quality integrations taking into account the continuously shifting landscape of integration end-points. Here are the hard-won best practices that we’ve developed which you can apply to your integrations.

The Integration Factory

We created an Integration Factory. Factory is a great name for it since it involves a highly repeatable process building lots of things that are the same – but it begs the question, what is an Integration Factory? Though it shares some similarities to software factories, especially in the use of manufacturing techniques, it doesn’t apply the model-driven code generation techniques frequently associated with the term. Integration Factory is really just a fancy term for:

• The Integration Specification (or the “spec”)
• The TCK (Technology Compatibility Kit)
• Connectors, which are implementations of the specification (these are the integrations)
• A build and test environment, which enables testing integrations against 3rd party systems
• Reporting, which provides an incredible level of detail on correctness and TCK conformance
• A delivery process for evolving the spec, TCK and connectors
  • Continuous Integration
  • Code reviews
  • Build triggers

It’s a set of technologies, an approach, a methodology, a repeatable process for creating robust, high quality integrations.

Connectors

Having a common API for creating integrations is an essential step in creating a factory. Tasktop Sync uses Mylyn Tasks as the API for integrations. Mylyn Tasks is a fantastic API and framework developed originally for IDE integrations, enabling developers to bring ALM artifacts (tasks, bugs, change requests, requirements, etc.) into their IDE. At the core of this API is a common data model for ALM artifacts and an API for performing basic CRUD and search operations. We call implementations of this API “connectors”. We have lots of connectors; one for each of Atlassian JIRA, Microsoft TFS, IBM RTC, IBM RRC, HP Quality Center and HP ALM, CA Clarity, etc.

Figure 1: a typical Mylyn connector as it relates to API

While a common data model and API enables us to connect any end-point with any other end-point when synchronizing ALM artifacts, the API on its own is not enough.

We need to know that each connector implements the API correctly. We need to know the capabilities of the connector and of the web service, and how they’re different for each version of the web service, and if it changes as new versions of the web service are released. We need to know what works and what doesn’t and why; is it a shortcoming of the connector implementation, or a limitation of the web service? This leads us to the Connector TCK.

Connector TCK

During one of our innovation-oriented engineering Ship-It days, one of our engineers prototyped a set of generic tests that could be configured to run against any connector. Why not apply the concept of a TCK to connectors? Benjamin dubbed his creation the Connector TCK, and the name stuck. The Connector TCK would have tests that ensure that every connector is implemented correctly, and tests the capabilities of each implementation.

Figure 2: the Connector TCK

The kinds of tests that were added to the Connector TCK varied from the most basic (e.g. a connection can be established with a repository) to more detailed (e.g. a file attachment to an artifact can be created and retrieved correctly with non-ASCII characters in the attachment file name). The beauty of the Connector TCK is that it could be used to measure the quality and capabilities of every connector equally. It could be configured to run a connector against multiple versions of a repository, in fact we test as many versions as we believe necessary to ensure correct behaviour for any supported version of an integration end-point.

Figure 3: testing a connector with multiple versions of a repository

Having a Connector TCK is great – but we’ve missed the essential question: what tests should be in it? The only way to know for sure is to have a definitive contract, a specification.

A Specification

For some software engineers requirements aren’t glamorous, exciting or even really that interesting. When we sit in front of a keyboard, the first thing we want to do is to start hammering out code. This is analogous to framing a house without blueprints. Sure, it’s fun – but the house won’t be what we want in the end. The Integration Specification (the “spec”) is the blueprint that spells out the desired behaviour of integrations. The spec takes the following form:

• User Stories (US) – stories written from the user’s perspective that define the functionality of integrations
• Technical User Stories (TUS)- stories written from the technology perspective that map to the connector API
• Acceptance Criteria (AC) – criteria that must be satisfied in order for a technical user story to be considered complete

Here’s an example from the spec:

• US-2: Connector client can set up a connection to a repository
  • TUS-2.1: Connector client can establish a connection with the repository server given the URL, credentials and other necessary connection parameters
    • AC-2.1.1: Connector client can validate URL, credentials and other necessary connection parameters and receive feedback of successful connection
    • AC-2.1.2: Connector client receives meaningful feedback for invalid or missing URL
    • AC-2.1.3: Connector client receives meaningful feedback for invalid or missing credentials
    • AC-2.1.4: Connector client…
  • TUS-2.2: Connector client can…

Normal software development often involves building features to a spec (or without a spec) and moving on. In our case where we’re building many integrations that essentially do the same thing, we get a lot of mileage out of the spec. TUSs and ACs in the spec apply to every connector implementation, of which there are many. So we value the spec with a kind of reverence that is unusual for software engineers.

Pulling It Together

The magic in this process really comes to light when we pull it together. Using JUnit and its powerful TestRule concept, we are able to connect our tests with ACs from the spec using a simple annotation:

@Retention(RUNTIME)
@Target(METHOD)
public @interface Validates {
	/**
	 * Provides the IDs of the acceptance criteria.
	 */
	String[] value();
}

Here’s an example of the annotation in use:

	@Test
	@Validates("2.1.2")
	public void testMissingUrl() {
		// test it
	}

With this simple technique, we can report on test results within the context of the specification. The test report takes on a whole new significance: it’s now a report on TCK compliance and connector capabilities. We can now definitively say which features are working and which are not for any integration, and easily determine differences when testing against new versions of a web service.

Figure 4: TCK compliance reporting

What Comes Next?

In the API economy it’s hard, but possible to create high quality integrations. We’ve taken a look at some of the concepts behind an Integration Factory which make it a lot easier. In the next installment we’ll look at other aspects of an Integration Factory, including build and test environments and the delivery process.

In Eclipse 3.6 we worked with the Microsoft interoperability team to bring some major improvements for Microsoft Windows users, such as Jump Lists, taskbar progress indicator and taskbar overlay text and images. As part of Tasktop’s ongoing partnership with Microsoft, we’ve been working hard to bring you two more improvements this year: Desktop Search, and Glass.

Desktop Search

The idea behind Desktop Search is simple: enable developers to search for resources outside of the current workspace. Until now Eclipse developers have only been able to search for resources within their workspace. First requested in 2007, this feature will help developers using multiple workspaces or those who regularly work with non-workspace files or documents.

While Desktop Search works for users on any OS, Windows users will experience a much faster search as a result of tight platform integration with Windows Search. Windows Search provides a full-text search index over a user’s files. The powerful Windows Search “SELECT TOP” syntax is used to find the most relevant matches.

Workspace and non-workspace resources are displayed side-by-side in the search result.

We hope to have Desktop Search integrated as a Platform feature for Eclipse 3.8. In the meantime, Desktop Search is available from the Mylyn Incubator update site. To install select “Help -> Install New Software” from the Eclipse menu. Using the following site http://download.eclipse.org/mylyn/incubator/3.7 select “Mylyn Desktop Search (Incubation)”. If this is something that you’d like to see as an Eclipse Platform feature, please vote for it on bug 192767.

Aero Glass for SWT

Modern Windows applications can have transparency, commonly known as Glass. For the first time, with this latest improvement to SWT on Windows, both Eclipse workbench and Eclipse RCP applications will be able to look like modern Windows 7 applications.

As an example here is the Glass look applied to the Tasktop RCP application, which only took a few hours of effort:

In this screenshot we’ve updated Tasktop RCP to use Glass for the shell, toolbar and search widget.

RCP and Eclipse platform developers will be able to use Glass with the new TRIM_FILL style bit:

	Shell uiShell = new Shell(display, SWT.SHELL_TRIM | SWT.TRIM_FILL);
	uiShell.setText("Glassy World");

	Composite uiRoot = new Composite(uiShell, SWT.TRIM_FILL);

Looking forward we would like to see this new Glass support adopted by the workbench modernization effort in e4.

Glass support in SWT is nearing completion, under development on bug 325795: support Windows Vista and 7 Aero Glass shells.

Eclipse And Windows – Looking To The Future

Through our partnership with Microsoft, Tasktop is continuing our efforts to keep Eclipse looking fresh and modern on Windows. We are currently discussing the next round of improvements, so if you have any ideas or feedback about what you’d like to see next, please drop me a note at david.green@tasktop.com.

You can read more about these and related efforts from the Microsoft perspective:

• Windows Gets Eclipse Platform Improvements
• Eclipse shines on Windows 7: Microsoft and Tasktop partnering to contribute code enhancement to Eclipse
• Interoperability Bridges and Labs Center

Contributors

Many thanks go out to Felipe Heidrich, Scott Kovatch, Mike Wilson, Steve Northover, Silenio Quarti, Bogdan Gheorghe, Raymond Lam, and Shawn Minto for helping to make Glass with SWT a reality, and to Łukasz Milewski for his prototype. Thanks also go out to Raymond Lam, Shawn Minto, Steffen Pingel and David Green for creating the desktop search integration, and to Dani Megert for supporting integration into the Eclipse core platform. Also I’d like to give a special thank you to the Microsoft Interop team who have been driving a better experience for Eclipse users on Windows.

The Tasktop team is growing again! We have four new technical positions available. This is a great opportunity to join a fantastic team of world-class engineers who affect hundreds of thousands of people through open source and our proprietary innovations.

The Tasktop Team, Vancouver contingent

Before I joined Tasktop, I was astounded at the Tasktop’s velocity and the impact they’ve had on software developers through Tasktop’s “less is more” approach to developer tools and the task-focused interface. Now that I’ve seen it from the inside, I understand that it’s a combination of many factors: the caliber of the team, the extent to which Tasktop’s innovation is driven by community, the way that Tasktop works with its customers and partners, the connection to thought leadership both within Tasktop and at large, the way that Tasktop communicates and organizes, and Tasktop’s culture of listening to and empowering the individual.

If you’re passionate about software and people, love technology and tools, have a persistent hunger for learning, and want to work with some of the best minds in the industry, apply and join the Tasktop team!