Monday, May 6, 2013

GWT Event-Bus in PUC


GWT Event-Bus in PUC


In 5.0 (SUGAR) we have finally made the upgrade from our homegrown listener system to using the GWT Event-Bus.  This was an easy thing to introduce, but it required a decent amount of work to migrate to once in place.  The problem with our existing listener system was that we had to write a new implementation each time we wanted to expose any kind of event from something.  Using the Event-Bus will allow us to cleanup a fair amount of code and greatly simplify how events are handled throughout the system.  We have previously had to write JSNI code in order to bridge any kind of event handling between GWT and JavaScript which was specific to each situation.

How GWT Event-Bus Works


An EventBus must be created, for example:

public static final EventBus EVENT_BUS = GWT.create(SimpleEventBus.class);

We use the same bus for the entire application, but we could actually create additional if needed.  Once we
have the EVENT_BUS available to us, we can fire events and add handlers to the bus:

EVENT_BUS.fireEvent(new MyEvent(x,y,z));

Anyone who has previously registered a handler with the EVENT_BUS for "MyEvent" will get called.  To add a handler to the bus:

EVENT_BUS.addHandler(MyEvent.TYPE, new MyEventHandler() {
  onMyEvent(MyEvent event) {
    // do something
  }
});

That's pretty much all there is to it.  If you are going to be creating your own event types, there is some additional work that you'll have to take care of.

Creating New Events


In PUC, we created about 10 new Event types, if you need to create any additional events, please put them in the org.pentaho.mantle.client.events package (for reasons explained later).  To create your own events, extend GwtEvent and implement two methods:  getAssociatedType and dispatch.  The following example is the event which is fired when the "recent" items list is changed.

public class RecentsChangedEvent extends GwtEvent<RecentsChangedEventHandler> {
  public static Type<RecentsChangedEventHandler> TYPE = new Type<RecentsChangedEventHandler>();
  public static final String TYPE_STR = "RecentsChangedEvent";

  public RecentsChangedEvent() {
  }

  public Type<RecentsChangedEventHandler> getAssociatedType() {
    return TYPE;
  }

  protected void dispatch(RecentsChangedEventHandler handler) {
    handler.onRecentsChanged(this);
  }
}

We have to create a handler interface as well (You can see the use of it in the Event itself).  The handler
interface for "recents" looks like this (these are general quite simple):

public interface RecentsChangedEventHandler extends EventHandler {
  void onRecentsChanged(RecentsChangedEvent event);
}

Bridging GWT + JavaScript


I created an EventBusUtil interface in the org.pentaho.mantle.client.events package for the purpose of bridging GWT and JavaScript.  It also has the actual EVENT_BUS we use throughout the application (PUC).  There interface looks like this:

public interface EventBusUtil {
  public static final EventBus EVENT_BUS = GWT.create(SimpleEventBus.class);
  public void addHandler(String eventType, JavaScriptObject handler);
  public void invokeEventBusJSO(JavaScriptObject handler, Object...params);
  public void fireEvent(String eventType);
}

At compile time, we use a generator to dynamically build an implementation for each of the methods on the interface.  This allows us to make available all of the events we know about to/from JavaScript from GWT.  The alternative would be to write and maintain a method which checks incoming JavaScript string types against known event names.  The result would be fragile, we could easily forget to add new events or break existing ones with trivial changes.  The generator guarantees that the system is always up-to-date and exposes everything we know about, which is every event in org.pentaho.mantle.client.events.  If your events are outside of this package, the generator is not going to pick them up and expose them through JSNI.  The generator itself is in the rebind package of PUC, called EventBusUtilGenerator.  It's fairly similiar to our CommandExecGenerator except that it is actually processing any fields which may exist on the Event itself (see x,y,z) in the "MyEvent" example above.  The fields are passed on down to the JSNI call for the handler.  Not everything is going to translate down very well, but we will pass everything we have.  As an example of this, here is a section of the generated Java source for the addHandler method in EventBusUtilImpl:

public void addHandler(final String eventType, final JavaScriptObject handler) {
..
else if(eventType.equals("UserSettingsLoadedEvent")){
 EVENT_BUS.addHandler(UserSettingsLoadedEvent.TYPE, new UserSettingsLoadedEventHandler() {
public void onUserSettingsLoaded(UserSettingsLoadedEvent event) {
 invokeEventBusJSO(handler, event.getSettings());
}
 });
    }
..
}

Notice event.getSettings() has been added to the call to invoke.  When addHandler is called with a String eventType (from JavaScript), we match it with the known events, such as "UserSettingsLoadedEvent." It is important to note that this is dynamically generated at compile time, we are not actually writing or maintaining code with hard-coded values like that.  We use reflection to grab class names, method names, signatures and return types.  When you add a JavaScript handler, it comes into GWT as a JavaScriptObject, in reality, these are just functions passed up from JavaScript.  When it comes time to invoke the handler, we dip back down into JSNI and call the handler with all of the parameters provided in JSON.

The actual JSNI call to invoke the handler is just two lines of code:

public native void invokeEventBusJSO(final JavaScriptObject jso, final String parameterJSON)
/*-{
  eval('var p = ' + parameterJSON);
  jso.call(this, p);
}-*/;

So these are the low-level details, to actually handle GWT events in JavaScript all you have to do is provide a handler:

mantle_addHandler("RecentsChangedEvent", function(paramJSON) {
 // do something 
});

Whenever GWT fires a RecentsChangedEvent, your handler function will get called.  If you would like to fire an event from JavaScript to GWT (such that GWT and any other JavaScript handlers will be made aware of) you would do it like this:

mantle_fireEvent("RecentsChangedEvent");

If you want to fire an event which has parameters, provide them as JSON (so that we can match the name of the parameter with the setter for the field on the event class itself).  For example:


mantle_fireEvent("GenericEvent", { eventSubType: "MDD", stringParam: "TestString" });

Events can now be shared between the two worlds of GWT & JavaScript with support for parameters.  We can only handle events we know about (at compile time), one way around this is to use the GenericEvent which has fields for each type of primitive you might want to set.  Use the eventSubType to distinguish the event from other types of events.  PUC does not have any uses of it, this would only be useful for any cross iframe situations or if an event bus or pub/sub model is not provided by whatever framework is being used.  An example handler in JavaScript might be something like this:

mantle_addHandler("GenericEvent", function(paramjson) {
  if (paramjson.eventSubType == "MDD") {
    // do something, this is the subType we care about
  }
});


CommandExec Revisted


While I was writing the EventBusUtilGenerator I revisited the CommandExecGenerator and added parameter support to it as well.  All of the commands that we used to ignore, which did not have a nullary constructor, are now available in JavaScript.  Pass parameters in the same manner as described above (as JSON).  We do not support complex types such as "FileItem" and going forward, discourage their use, try to find an alternative.  If you are passing a FileItem but only actually need the file's path, just use the path (as a String).  An example command execution with a parameter:

executeCommand("SwitchLocaleCommand", { locale : 'fr' });


Wednesday, January 2, 2013

Fun Week 2012 - Carte Mobile (Part 2)



This is the second part of my Funweek 2012 project.  My original goal during the week was to create a mobile application for Carte but I could not muster up the desire to do this with the current state of Carte servlets/XML output.  I basically did not care much for writing XML parsing code in JavaScript during Funweek when I know how great life can be if we had JSON.  So..  part one of my Funweek, which is already documented (LINK), was to add Jerser/REST (JAX-RS) to Carte.


Since I used, and enjoyed, Sencha Touch 2 for Pentaho Mobile and I only had 2 days left to write Carte Mobile I had to go with something I already knew.  I also had the confidence that I could use Sencha Touch to get the job done in the limited time without any major hurdles or snags.  I created a Sencha Touch application, eg:

Ext.application({
name: 'CarteMobile',

models: [
'Job', 'Transformation'
],

stores: [
'Jobs', 'Transformations'
],
launch: function() {
..
}
});

In the launch we add the code to login and then launch the primary UI of the application which is tabset showing status, jobs, transformations and config.



We use Sencha Touch's built-in List (Ext.List) widget and back it with a DataStore.  We have two different DataStores - jobs & transformations.  These stores are backed by the REST services.  Essentially, it is as simple as making an AJAX request to one of the jobs or transformation services, parsing the JSON and setting the JSON as the data for the DataStore.  For example:

Ext.Ajax.request({
url: 'http://' + localStorage.carteURL + '/api/carte/jobs/detailed',
success: function(response) {
var data = JSON.parse(response.responseText);
var store = Ext.getStore('Jobs');
store.setData(data);
}
});

The Ext.List widget supports Pull-to-Refresh if you enable it with the appropriate plugin, so I put this in as well.  Here's a code snippet showing how this was accomplished:

var list = Ext.create('Ext.List', {
plugins: [{
xclass: 'Ext.plugin.PullRefresh',
refreshFn: function(plugin) {
loadData(loadDataCallback);
}
}],
});

Once the application was written, it can be served by Carte itself.  More interesting is that if the resources (html, js, css, etc) are bundled up in a ZIP file (plus config file) they can be built using PhoneGap to provide mobile OS versions for publishing to the Apple App Store or Android Play Store.  The only thing I do in the mobile version different than the Carte served version is detect if we are running in PhoneGap to know if we have to prompt for the server name for Carte (we already know this information in the other case).  PhoneGap provides a cloud-specific build for most of the mobile platforms.  In this case, I was interested in the Android (APK) distribution.  See http://build.phonegap.com

Once I had the Android APK I uploaded it to the Android Play Store, where it can be downloaded and enjoyed today.  Here are a few screenshots from the mobile application on an Android phone.





Fun Week 2012 - Carte conversion to REST (Part 1)


During Pentaho Fun Week 2012 I converted Carte to using REST.
Carte fires up a Jetty server in WebServer.java.  This is the class we modify to mount all REST
endpoints under "/api".  To do this I used Jersey's PackagesResourceConfig with a Jersey ServletHolder.
The details of this code are as follows:

// setup jersey (REST)
ServletHolder jerseyServletHolder = new ServletHolder(ServletContainer.class);
jerseyServletHolder.setInitParameter("com.sun.jersey.config.property.resourceConfigClass", "com.sun.jersey.api.core.PackagesResourceConfig");
jerseyServletHolder.setInitParameter("com.sun.jersey.config.property.packages", "org.pentaho.di.www.jaxrs");
// mount all jersey REST under /api
root.addServlet(jerseyServletHolder, "/api/*");

The package "org.pentaho.di.www.jaxrs" is where all classes are scanned for JAX-RS annotations for example CarteResource.java.  This is a
general REST endpoint mounted to "/api/carte" and provides methods for getting system info, config details and a list of jobs or
transformations.  This file is really quite simple to understand and it replaces (or makes obsolete) several Carte servlets.  For example,
to get a list of transformations that are in Carte, you would invoke the url "http://carteserver:port/api/carte/transformations" with a
GET request (which means you can do this easily in a browser to preview the response).  The output is in JSON making it very easy for
JavaScript consumers to rehydrate.  XML can also be provided if so desired by putting this in the request header.  The code for Carte
REST to return a list of transformations is:

@GET
@Path("/transformations")
@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public List<CarteObjectEntry> getTransformations() {
List<CarteObjectEntry> transEntries = CarteSingleton.getInstance().getTransformationMap().getTransformationObjects();
return transEntries;
}

The @GET annotation specifies the HTTP method we use and the @Path annototation is the "add-on" to the service mount (The class itself
is mounted as "/carte"), so the combined path is "/carte/transformations" - the full path, since all resources in the "org.pentaho.di.www.jaxrs"
package or mounted in Jetty under "api" would be "/api/carte/transformations".  The @Produces annotation is a list of output types/formats
that can be provided by this service.  Jersey will handle the serialization from our List<CarteObjectEntry> to JSON/XML.

To keep things really simple, yet as complete as reasonable during a fun week timeframe, I created two additional REST endpoints which handle
job and transformation specifics.  JobResource.java and TransformationResource.java are the classes which provide the ability to add, remove,
pause, start, stop, get logs, and get status for jobs/transformations.  For example, to get the status of a job, you would invoke the url
"http://carteserver:port/api/carte/job/status/id" where id is the GUID for the Carte object (which can be obtained by listing from the
CarteResource.

There's still work to be done, I have not converted all of the CarteServlets to REST, for example, I don't have anything in the way of
slave server support or anything to list data services or server sockets.  This is not because it was difficult, in fact, it's probably
quite easy to do, I just didn't need it for the purposes of my funweek project.  Another thing that would be nice is to add the ability for
plugins to add REST services just as they can add Jetty servlets today.  I don't think this will be difficult either, I just did not have
time as I wanted to take what I had already written and write a mobile friendly UI for Carte.

Wednesday, August 1, 2012

Pull-to-Refresh with Sencha Touch NestedList

The NestedList widget in Sencha Touch is perfect for hierarchical navigation (tree structures).  Our solution repository
is just that, a database persistent file system.  Upon building our mobile application, this widget was a natural fit.
The NestedList is backed by a 'Store' - in our case I defined a Files store.  An AJAX call to our repository web service
returns a heap of XML, as much as I wish it to be JSON it's just not happening.  Not until SUGAR.  When SUGAR hits, we
have real REST services where I can specify the format I want it back in.  Until then I have XML and this Store really
seems to want JSON.  I wrote an xml2json converter, shown below.

Ext.define('PExt.json.XMLtoJSON', {

xml2Json : function(xml, isRoot) {

var jsonObj = {};

if (!isRoot) {
if (xml.nodeType == 1) {
jsonObj.file = {};
for (var i = 0; i < xml.attributes.length; i++) {
if (xml.attributes.item(i).nodeName == 'localized-name') {
jsonObj.file['localizedName'] = xml.attributes.item(i).nodeValue;
} else {
jsonObj.file[xml.attributes.item(i).nodeName] = xml.attributes.item(i).nodeValue;
}
}
} else if (xml.nodeType == 3) {
jsonObj = xml.nodeValue;
}
}

if (xml.hasChildNodes()) {
jsonObj.children = [];
for(var i = 0; i < xml.childNodes.length; i++) {
var child = xml.childNodes.item(i);
var visible = child.attributes['visible'].nodeValue;
if (visible === 'false') {
continue;
}
jsonObj.children.push(this.xml2Json(child, false));
}
}
return jsonObj;
}

});

It's not a generic converter, it cares about our repository/attributes it has.  But the beauty of it is that I get real
JSON that the Store can readily consume.  If only it were that easy, there are special attributes that the Store/NestedList
are looking for which we don't have.  Other attributes are provided as opposites, for example, the 'leaf' attribute the
NestedList is looking for is missing, we have an attribute to tell us if something is a folder (isDirectory).  The
solution, which was not at first obvious, was to define converters for each of the missing attributes.  These converters let us
write a function to perform any type of conversion or lookup to return whatever value we need.  For example:

 Ext.define('PentahoMobile.model.File', {
...
config: {
        fields: [
...
{name: 'leaf', type: 'boolean', mapping: 'file.isDirectory',
convert: function(value, record) {
// convert file.isDirectory into 'leaf'
// maybe check if a folder has children
...
}
},
]
}
...
});

After the store, mappings and converter function have been written, we have a NestedList which works against the
solution repository.  You can add markup to the HTML generated for each entry in the list, such as adding folder icons
or descriptions.

This all served its purpose quite well until our UX department asked about removing the refresh button in the toolbar
and doing the 'pull-to-refresh' that is becoming more common in mobile apps these days.  Fortunately, Sencha has such a
plugin, but I could not find any documentation on how to use it with a NestedList, only to the Ext.dataview.List.  After
a lot of reading and mostly trial/error I found a solution.  I had to 'use the source' in order to see how the NestedList
uses List internally, and maybe I could find some hint.

The key is a listConfig hidden within the config of the NestedList itself.  I basically used this config as if it were
literally the config for the internal List used by NestedList.  Thankfully this worked, after a few moments of wrestling
around with some improper refresh calls to my store I discovered that you can define your own refresh function for the
pull-to-refresh.  As a note for completeness, you can also override the text which displays when you pull/release the
control.  Here's the listConfig that I am using:

config: {
...
listConfig: {
plugins: [{
xclass: 'Ext.plugin.PullRefresh',
refreshFn: function(plugin) {
// refresh repository
}
 }]
},
...
}

Here is a screenshot of it in action.


Tuesday, April 17, 2012

Pentaho User Console now working with all REST services

Over the past week or so I've been busy rewriting parts of the Pentaho User Console (PUC aka Mantle) to talk with REST services and move away from GWT-RPC. The primary motivation to do this work was to make the system more accessible to system integrators and OEM developers.

I have broken MantleService into several logical REST end points.

Themes (/pentaho/api/theme)


A new service allowing you to list the available themes as well as set / get the current theme for the authenticated user.

User Settings (/pentaho/api/user-settings)


A general purpose user settings service API which lets you list settings (for the authenticated user) as well as get / set user settings.

Version (/pentaho/api/version)


Simple service to return the current product version and also provide a software updates list (if applicable).

System Refresh (/pentaho/api/system/refresh)


A collection of resource refresh/reload services for global actions, metadata, system settings, repository, mondrian and reporting.

User Console (/pentaho/api/mantle)


This was my dumping ground for services which I did not feel provided value to be separated into their own endpoint.  The service is able to return mantle settings (different than user settings), list mondrian cubes, and get/set the locale for the authenticated user.

This work has simplified the User Console from a development standpoint, we have less ivy dependencies and the debug configuration is much more simple. All of the services return XML or JSON (determined by request headers) except for the extremely simple services which just return a string (text/plain), such as getting the current theme. I had to change the client code in PUC to consume JSON rather than the convenience/luxury we had before with GWT-RPC. The approach I took was to create JSON overlay objects (which extend JavaScriptObject). These JSON overlays are returned from the JSON eval/parsing and we then re-gain all of the richness we once had, at the expense of rewriting our POJO mutators as JNI.

For example, in the simple case of themes, which have an id and a value we now have a JsTheme:

public class JsTheme extends JavaScriptObject {
  protected JsTheme() {
  }
  public final native String getId() /*-{ return this.id; }-*/; //
  public final native String getName() /*-{ return this.name; }-*/; //
}

Once we return a list of JsTheme from JSON, our GWT app can go on enjoying life pretty much the same way as before. As a point of completeness, here is how we return get the themes from the JSON string:

JsArray<JsTheme> themes = JsonUtils.safeEval(JsonUtils.escapeJsonForEval(response.getText()));

As of Friday, April 13, 2012 the entire use of GWT-RPC has been replaced with REST. Enjoy!

Monday, April 16, 2012

New modules in the Administration perspective



Thanks goes out to Ezequiel Cuellar for being my ghost writer for this post.

The Administration perspective just got more love from the Sugar team. In an effort to migrate all the functionality from the Enterprise Console into Sugar the following two administration modules have been created: Users/Roles and Email/SMTP Server.

The User/Roles new UI allows you to easy administer users by creating or editing them and change their password, you can also create new roles and assign them to a user either one by one or by performing a multi selection.



In a similar way the roles section allow you to assign users.



The Email/SMTP Server module allows you to administer the BI Platform internal email settings (email_config.xml), you can test your changes by sending test emails before saving them and it also features a new error notification mechanism.

Wednesday, February 29, 2012

Pentaho Admin/Enterprise Console in SUGAR

If you've grabbed a recent build of SUGAR you'll notice some obvious changes to the UI as well as a top-level directory structure which is totally missing. The "administration-console" (or "enterprise-console") folder has been removed. We are actively working to integrate PAC/PEC functionality in the Pentaho User Console as an Administration perspective.

The layout of the Administration perspective is defined in mantle.xul (pentaho/mantle/xul/mantle.xul). This will allow the possibility of changing the UI and/or removing UI elements. Presently, we have action-based security (ABS) and authentication (Pentaho/LDAP) implemented in this UI. In the upcoming weeks we'll be adding other missing parts from the admin console such as email/auditing.

Here's a recent screenshot of the Administration perspective:

How to Add New Admin Functionality

New functionality can be plugged into the admin perspective with a platform plugin. The plugin.xml of a platform plugin will have a XUL overlay section to add an item to the admin category tree. For example, if you want to add a new item to the "security" category you would do this to the plugin.xml:

<overlays>
    <overlay id="admin.perspective.overlay.ee" resourcebundle="content/my-admin/resources/messages/messages">
        <treechildren id="security">
            <treeitem command="mantleXulHandler.loadAdminContent('my-admin-panel', 'api/repos/myadmin/resources/my-admin.html')">
                <treerow>
                    <treecell label="${myadmin.label}" />
                </treerow>
            </treeitem>
        </treechildren>
    </overlay>
</overlays>

We are adding a new panel to the admin perspective with an ID of 'my-admin-panel' and we are specifying the location of the UI (by URL). At this point you have adding your content to the admin category tree. Just like any plugin, you can have back-end code in a JAR, eg my-admin/lib/my-admin-plugin.jar.
Even more interesting is the new capability of a platform plugin to easily register its REST services in the plugin.spring.xml. We're using Jersey (v1.12) for exposing these web services. For reference take a look at the echo-plugin.

Finishing the Job: JavaScript Integration

Whatever your choice of JavaScript library, you will be coexisting with PUC/admin perspective. While not required, you can improve the user experience by registering your UI for state changes, etc. To do this create an object with an id and activate/passivate methods. For example:

var myAdminPanel = {
    id : "my-admin-panel",
    activate : function() {
        refreshConfig();
    },
    passivate : function(passivateCompleteCallback) {
        if(isConfigDirty()) {
            passivateCallback = passivateCompleteCallback;
            dijit.byId("saveChangesDialog").show();
        } else {
            passivateCompleteCallback(true);
        }
    }
};

Now register this object with the admin perspective:
window.top.mantle_registerSysAdminPanel(myAdminPanel);

That's all there is too it, you will be notified when the user selects on/off of your panel so you can check for "dirty" and prompt for saving.

REST services

In order to support the new admin functionality added to PUC, we added several new REST services which might be generally useful to OEMs, integrators, and developers.

/pentaho/api/userrole/users
Using GET, will return a list of all users in the system.

/pentaho/api/userrole/roles
Using GET, will return a list of all runtime roles in the system.

/pentaho/api/userrole/roleAssignments
Using PUT, will set role bindings between roles and permissions (logical roles)

/pentaho/api/userrole/logicalRoleMap
Using GET, will return the list of roles and the permissions (logical roles) which are assigned to them

The following LDAP REST API calls are in the EE product

/pentaho/api/ldap/config/getAttributeValues
Using GET, returns all name/value pairs from applicationContext-security-ldap.properties, plus the current securityProvider

/pentaho/api/ldap/config/setAttributeValues
Using PUT, sets (merges) name/value pairs and saves them to applicationContext-security-ldap.properties as well as set the authentication type in pentaho-spring-beans.xml.

/pentaho/api/ldap/config/ldapTreeNodeChildren
Using GET, returns the list of

/pentaho/api/ldap/config/userTest
Using GET, simple test if a user can be found.

/pentaho/api/ldap/config/rolesTest
Using GET, tests if search for a user returns roleAttribute successfully.

/pentaho/api/ldap/config/userRolesTest
Using GET, will perform a populator test (check if granted authorities for the given user works)

/pentaho/api/ldap/config/providerTest