Friday, 25 September 2015

Describing API Key Authentication in RAML

I've finally figured out how to say in RAML that API operations are protected by an API key query parameter:

Saturday, 19 September 2015

Retiring Kafka Web Console

I've been a busy bee the past few months. The lack of activity on my blog and GitHub is a testament to this. Given my current priorities, I've taken the decision to retire Kafka Web Console. Don't despair yourself! Kafka Manager appears to be a more sophisticated alternative to what I've developed, and besides, it's maintained by Yahoo.

Monday, 10 August 2015

Dynamically Create Rules using Drools & Rule Templates

Rules are used for a variety of stuff in the systems we build. Most often these rules are hard-coded in our application logic. The trouble is that sometimes we want to have the end-user the ability to define his own rules. Imagine an order processing system. The supplier wants to be notified on any range of events as they occur throughout the system but the notification rules are not known ahead of time. Such a rule could be for a late payment or a highly lucrative order event. In Java, the latter rule can be modelled as follows [1]:

Supporting conjoined conditions in a rule requires us that we tweak the previous example:

I consider it a risky proposition to write your own primitive rules engine to evaluate rules like the above. I much prefer a solution leveraging Drools 6 in combination with Rule Templates. Rule Templates is an awesome Drools feature giving you the ability to define abstract rules at design-time. At run-time, a Drools compiler runs through the rule template and evaluates expressions to generate concrete rules. Given an event type class (e.g., OrderEvent) and a Rule object (e.g., highValueOrderWidgetsIncRule), we can conceive the following rule template:

A couple of things to observe:
  • Line 1: Declares that the DRL file is a rule template.
  • Line 3-4: rule and eventType are template parameters.
  • Line 8: alertDecision is global variable which we write the outcome to should the rule evaluate to true.
  • Line 12: @{row.rowNumber} is an in-built expression that makes the rule ID unique. This is useful for situations when you don't know how many rules you're going to have ahead of time. Note that this doesn't apply to our example.
  • Line 14: @{eventType} and @{rule} MVEL expressions that are substituted with the template parameters at run-time.
  • Line 16: Sets the property doAlert to true to signal the application that the notification rule was fired.
Generating a rule from the template is a matter of instantiating ObjectDataCompiler and passing as parameters:
  1. A map consisting of a Rule object (e.g., highValueOrderWidgetsIncRule) and the name of the event class the Rule object pertains to (e.g., org.ossandme.event.OrderEvent)

  2. The template.drl file

Drools cannot evaluate a Rule object in its current POJO form. In order to evaluate it, we override the Rule class's toString() method to transform the POJO into a formal statement:

Before running the data through the template, Drools calls toString() on the template parameters. Calling toString() on highValueOrderWidgetsIncRule returns the statement: price > 5000.0 && customer == 'Widgets Inc.'. Going even further, if we apply the template to the statement and event type OrderEvent, we would get the following generated rule:

The last step is to evaluate the rule:

Finally, let's put this all together. Don't worry, a copy of the complete application can be found on GtiHub:

1: I'm ignoring the fact that most likely the rule is retrieved from a data store.

Friday, 15 May 2015

A Primer to AS2

Check out my latest guest post about AS2 on ModusBox's blog.

Tuesday, 16 September 2014

The Trials of Smooks

The fact that I'm a hard to please guy explains why I rarely show appreciation for a tool. I easily get frustrated when a tool fails to meet the challenges it's meant to solve. Smooks is one of the few tools I appreciate. It's an invaluable transformation framework in the integrator's arsenal. On a project I was on, I threw at Smooks [1] all manner of challenges, and one after another, Smooks overcame them without giving up a key requirement: maintaining a low memory overhead during transformation. A shoutout to Tom Fennelly and his team for bringing to us such a fantastic tool.

Trial I

The initial challenge I brought to Smooks was about taking a tilde delimited CSV file and map its records to POJOs:

You can see the file has an unorthodox header in addition to a footer. Using Smooks's built-in CSV reader, I wrote concisely the Smooks config doing the mapping to POJOs:

What's happening under the covers, and in general, is that the reader pulls data from a source (e.g., to go on to produce a stream of SAX events. The reader I'm using above is expecting the source data to be structured as CSV and to consist of 4 columns. Let's make things more concrete. Reading from products.csv, the reader produces the following XML stream [2]:

Listening to the stream of SAX events is the visitor. A visitor listens to specific events from the stream to fire some kind of behaviour, typically transformation. With the singleBinding config element, the CSV reader pre-configures a JavaBean visitor to listen for csv-record elements. On intercepting this element, the JavaBean visitor instantiates a org.ossandme.Product object and binds its properties to csv-record's children element content. You'll notice that I left Product's target properties unspecified in the config. The CSV reader assumes Product follows JavaBean conventions and its properties are named the same as the defined CSV columns. Records disobeying the column definition are ignored. Consequently, I do not need to worry about the file's header and footer.

With the transformation configuration out of the way, I turned my attention to running the transformation on the CSV file from my Java code and process the Product objects as they are instantiated and bound by Smooks:

Trial II

A more complex transformation task I gave to Smooks was to load file records, holding a variable number of columns, into a database. As in the previous task, this file had a header as well as a footer:

You'll observe in the sample CSV file that records could be one of three types as denoted by the first column: TH, TB or TF. The CSV reader, as it transforms and pushes records to the XML stream, can be customised such that it renames the csv-record holder to the record's primary column:

As we'll see later, the above config permits Smooks to distinguish between the different record types. Given the sample file transactions.csv, the reader I've configured produces the following stream:

UNMATCHED elements represent the file's header and footer. A CSV record having TH in the first field will trigger the reader to create a TH element holding the other record fields. The same logic goes for TB and TF.

Database visitors load the records. However, since these visitors are limited to binding data from POJOs, I first must turn the XML mapped records from the stream into said POJOs. The CSV reader doesn't know how to bind variable field records to POJOs so I configure the mapping myself:

Given what we've learnt about Smooks, we can deduce what's happening here. The JavaBean visitor for lines 10 till 17 has a selector (i.e, createOnElement) for the element TH. A selector is a quasi XPath expression applied on XML elements as they come through the stream. On viewing TH, the visitor will:
  1. Instantiate a HashMap.

  2. Iterate through the TH fragment. If an element inside the fragment matches the selector set in a data attribute, then (a) a map entry is created, (b) bound to the element content, and (c) put in the map.

  3. Add the map to the Smooks bean context which is identified by the name set in beanID. The map overwrites any previous map in the context with the same ID. This makes sense since we want to prevent objects from accumulating in memory.
The database visitors reference the maps in the bean context:

The insert statements are bound to the map entry values and are executed after the element, the executeOnElement selector points to, is processed. The next step is to configure a datasource for the database visitors (lines 47-49):

Last but not least, the Java code to kick off the data load:

Trial III

The next challenge for Smooks makes the previous ones look like child's play. The goal: transform an XML stream to a CSV file that is eventually uploaded to an FTP server. The input:

The desired output:

Considering the CSV could be large in size, my requirement was for Smooks to write the transformed content to a PipedOutputStream. An FTP library would read from the PipedOutputStream's connected PipedInputStream, and write the streamed content to a file. To this end, I wrote the class running the transformation as follows:

My focus then turned to the XML-to-CSV mapping configuration. After deliberation, I reluctantly settled to use the FreeMarker visitor for writing the CSV. I considered as an alternative to develop a visitor specialised for this type of transformation but time constraints made this unfeasible. The FreeMarker visitor, like the database one, cannot read directly off the XML stream. Instead, it can read from DOM and POJOs. So I decide to use the DOM visitor such that it creates DOMs from record elements found within the input stream:

I then configured the FreeMarker visitor to apply the CSV template on seeing the element record in the stream:

Below is a simplified version of what I had in real life in account.ftl (note the last line of the template must be a newline):

An additional complexity I had to consider were the CSV's header and footer. Apart from being structured differently than the rest of the records, the header had to contain the current date whereas, for the footer, the total record count. What I did for the header was to bind the current date from my Java code to Smooks's bean context (lines 27-30 and 38):

The date is then referenced from the Smooks config (lines 9-12):

With respect to the above config, at the start of the XML stream, FreeMarker writes the header to the output stream (i.e., PipedOutputStream):

000000Card Extract   [current date]

<?TEMPLATE-SPLIT-PI?> is an embedded Smooks instruction that applies account.ftl to record elements after the header.

Adding the record count to the footer is just a matter of configuring the Calculator visitor to maintain a counter in the bean context and referencing that counter from the template:

Trial IV

The final challenge Smooks had to go against was to read from a java.util.Iterator of maps and, like the previous task, write the transformed output to a stream in CSV format. Unlike the InputStream that Smooks read from the other tasks, Smooks doesn't have a reader that is capable of writing a properly structured XML doc from an iterator of maps. So I'm left with writing my own reader:

The custom reader is hooked into Smooks as follows (line 5):

Finally, passing the iterator to Smooks for transformation consists of setting a JavaSource parameter, holding the iterator, on filterSource(...)  (line 27):

1: The Smooks version I used was 1.5.2.
2: You might be wondering how I know for certain the XML document shown is the one actually produced by Smooks. I know because of Smooks's HtmlReportGenerator class.

Wednesday, 19 February 2014

Safely Prevent Template Caching in AngularJS

AngularJS's $templateCache can be a pain in the ass. Sometimes we don't want templates to be cached. A quick Internet search to disable caching gives the following workaround:

But as I have learnt with the UI Bootsrap module, this may cause AngularJS modules that use $templateCache to break. A solution is to tweak the above workaround so that new cache entries are removed on route change instead of indiscriminately removing all entries:

Tuesday, 11 February 2014

Dynamically Create BitCoin Wallets & Payment Pages on Coinbase in Ruby

Last weekend, as part of my new year's resolution to dedicate some time to good causes, I participated in Hack4good: a global 48 hour hackathon aimed at bringing ideas for the social good into life. In Malta, our team brought forward a crowd funding solution for charitable fundraisers with minimal transaction fees. To this end, we selected BitCoin as the donation currency and Coinbase to host fundraise donations.

One requirement in our project was to have Coinbase automatically issue a BitCoin wallet to each fundraiser. To further complicate matters, we wanted to generate a Coinbase payment page that allows the donor to transfer his BitCoins to the fundraiser's wallet:

Coinbase's awesome API permitted us to do both things with very little effort. Since we developed the solution in Ruby on Rails 4, I'll show you the code of how we accomplished this using a forked version of Coinbase API's Ruby client [1]:

The create() controller action does numerous things so let's dissect it piece by piece. The action instantiates the Coinbase client with our API key: this key is created in Coinbase's account settings page. The client object's create_user(...) method is then invoked to make a wallet in addition to a Coinbase account for the fundraiser. The email address and password parameters are used by the end-user to access his fundraiser wallet on Coinbase. COINBASE_CLIENT_SECRET, linked to our API key, is passed as a parameter so that we can automatically grant ourselves merchant permissions on the created user account. These permissions are needed to dynamically generate the payment page on behalf of the user.

Making the call to Coinbase to generate the payment page requires that we follow the OAuth 2 protocol [2]. Fortunately, an OAuth 2 Ruby library exists. So we go ahead and use the library to instantiate an OAuth client, passing COINBASE_API_KEY and COINBASE_API_SECRET as parameters. Before we ask Coinbase to create a payment page on the user's behalf, an AccessToken object is constructed with the access token obtained from coinbase.create_user(...) and the OAuth client we have just instantiated. After this, we use the newly constructed oauth_token object to post a request to Note that JSON_CREATE_PAYMENT_PAGE's value is sent as the HTTP body.

All I need from the JSON response returned from the API call is the payment page code. This code lets Coinbase know which payment page to display. We persist this code along with the fundraiser details so that we can retrieve them later when we show the fundraiser to a potential donor:

Here is view associated with the above action:

The view gets the page code from @fundraiser.coinbase_page_code and sets the necessary HTML attributes with this value. button.js is a script provided by Coinbase that styles the anchor element and opens the fundraising donation page tied to the page code when the anchor is clicked:

The final step is to add the OAuth 2 and Coinbase dependencies to the project Gemfile:

1: We forked Coinbase's Ruby client because create_user(...) didn't support client ID.
2: You need to register your application on Coinbase before you can gain rights to manage user accounts through OAuth.