Wednesday, January 2, 2013
Fun Week 2012 - Carte conversion to REST (Part 1)
During Pentaho Fun Week 2012 I converted Carte to using REST.
Carte fires up a Jetty server in WebServer.java. This is the class we modify to mount all REST
endpoints under "/api". To do this I used Jersey's PackagesResourceConfig with a Jersey ServletHolder.
The details of this code are as follows:
// setup jersey (REST)
ServletHolder jerseyServletHolder = new ServletHolder(ServletContainer.class);
jerseyServletHolder.setInitParameter("com.sun.jersey.config.property.resourceConfigClass", "com.sun.jersey.api.core.PackagesResourceConfig");
jerseyServletHolder.setInitParameter("com.sun.jersey.config.property.packages", "org.pentaho.di.www.jaxrs");
// mount all jersey REST under /api
root.addServlet(jerseyServletHolder, "/api/*");
The package "org.pentaho.di.www.jaxrs" is where all classes are scanned for JAX-RS annotations for example CarteResource.java. This is a
general REST endpoint mounted to "/api/carte" and provides methods for getting system info, config details and a list of jobs or
transformations. This file is really quite simple to understand and it replaces (or makes obsolete) several Carte servlets. For example,
to get a list of transformations that are in Carte, you would invoke the url "http://carteserver:port/api/carte/transformations" with a
GET request (which means you can do this easily in a browser to preview the response). The output is in JSON making it very easy for
JavaScript consumers to rehydrate. XML can also be provided if so desired by putting this in the request header. The code for Carte
REST to return a list of transformations is:
@GET
@Path("/transformations")
@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public List<CarteObjectEntry> getTransformations() {
List<CarteObjectEntry> transEntries = CarteSingleton.getInstance().getTransformationMap().getTransformationObjects();
return transEntries;
}
The @GET annotation specifies the HTTP method we use and the @Path annototation is the "add-on" to the service mount (The class itself
is mounted as "/carte"), so the combined path is "/carte/transformations" - the full path, since all resources in the "org.pentaho.di.www.jaxrs"
package or mounted in Jetty under "api" would be "/api/carte/transformations". The @Produces annotation is a list of output types/formats
that can be provided by this service. Jersey will handle the serialization from our List<CarteObjectEntry> to JSON/XML.
To keep things really simple, yet as complete as reasonable during a fun week timeframe, I created two additional REST endpoints which handle
job and transformation specifics. JobResource.java and TransformationResource.java are the classes which provide the ability to add, remove,
pause, start, stop, get logs, and get status for jobs/transformations. For example, to get the status of a job, you would invoke the url
"http://carteserver:port/api/carte/job/status/id" where id is the GUID for the Carte object (which can be obtained by listing from the
CarteResource.
There's still work to be done, I have not converted all of the CarteServlets to REST, for example, I don't have anything in the way of
slave server support or anything to list data services or server sockets. This is not because it was difficult, in fact, it's probably
quite easy to do, I just didn't need it for the purposes of my funweek project. Another thing that would be nice is to add the ability for
plugins to add REST services just as they can add Jetty servlets today. I don't think this will be difficult either, I just did not have
time as I wanted to take what I had already written and write a mobile friendly UI for Carte.
Wednesday, August 1, 2012
Pull-to-Refresh with Sencha Touch NestedList
The NestedList widget in Sencha Touch is perfect for hierarchical navigation (tree structures). Our solution repository
is just that, a database persistent file system. Upon building our mobile application, this widget was a natural fit.
The NestedList is backed by a 'Store' - in our case I defined a Files store. An AJAX call to our repository web service
returns a heap of XML, as much as I wish it to be JSON it's just not happening. Not until SUGAR. When SUGAR hits, we
have real REST services where I can specify the format I want it back in. Until then I have XML and this Store really
seems to want JSON. I wrote an xml2json converter, shown below.
Ext.define('PExt.json.XMLtoJSON', {
xml2Json : function(xml, isRoot) {
var jsonObj = {};
if (!isRoot) {
if (xml.nodeType == 1) {
jsonObj.file = {};
for (var i = 0; i < xml.attributes.length; i++) {
if (xml.attributes.item(i).nodeName == 'localized-name') {
jsonObj.file['localizedName'] = xml.attributes.item(i).nodeValue;
} else {
jsonObj.file[xml.attributes.item(i).nodeName] = xml.attributes.item(i).nodeValue;
}
}
} else if (xml.nodeType == 3) {
jsonObj = xml.nodeValue;
}
}
if (xml.hasChildNodes()) {
jsonObj.children = [];
for(var i = 0; i < xml.childNodes.length; i++) {
var child = xml.childNodes.item(i);
var visible = child.attributes['visible'].nodeValue;
if (visible === 'false') {
continue;
}
jsonObj.children.push(this.xml2Json(child, false));
}
}
return jsonObj;
}
});
It's not a generic converter, it cares about our repository/attributes it has. But the beauty of it is that I get real
JSON that the Store can readily consume. If only it were that easy, there are special attributes that the Store/NestedList
are looking for which we don't have. Other attributes are provided as opposites, for example, the 'leaf' attribute the
NestedList is looking for is missing, we have an attribute to tell us if something is a folder (isDirectory). The
solution, which was not at first obvious, was to define converters for each of the missing attributes. These converters let us
write a function to perform any type of conversion or lookup to return whatever value we need. For example:
Ext.define('PentahoMobile.model.File', {
...
config: {
fields: [
...
{name: 'leaf', type: 'boolean', mapping: 'file.isDirectory',
convert: function(value, record) {
// convert file.isDirectory into 'leaf'
// maybe check if a folder has children
...
}
},
]
}
...
});
After the store, mappings and converter function have been written, we have a NestedList which works against the
solution repository. You can add markup to the HTML generated for each entry in the list, such as adding folder icons
or descriptions.
This all served its purpose quite well until our UX department asked about removing the refresh button in the toolbar
and doing the 'pull-to-refresh' that is becoming more common in mobile apps these days. Fortunately, Sencha has such a
plugin, but I could not find any documentation on how to use it with a NestedList, only to the Ext.dataview.List. After
a lot of reading and mostly trial/error I found a solution. I had to 'use the source' in order to see how the NestedList
uses List internally, and maybe I could find some hint.
The key is a listConfig hidden within the config of the NestedList itself. I basically used this config as if it were
literally the config for the internal List used by NestedList. Thankfully this worked, after a few moments of wrestling
around with some improper refresh calls to my store I discovered that you can define your own refresh function for the
pull-to-refresh. As a note for completeness, you can also override the text which displays when you pull/release the
control. Here's the listConfig that I am using:
config: {
...
listConfig: {
plugins: [{
xclass: 'Ext.plugin.PullRefresh',
refreshFn: function(plugin) {
// refresh repository
}
}]
},
...
}
Here is a screenshot of it in action.
is just that, a database persistent file system. Upon building our mobile application, this widget was a natural fit.
The NestedList is backed by a 'Store' - in our case I defined a Files store. An AJAX call to our repository web service
returns a heap of XML, as much as I wish it to be JSON it's just not happening. Not until SUGAR. When SUGAR hits, we
have real REST services where I can specify the format I want it back in. Until then I have XML and this Store really
seems to want JSON. I wrote an xml2json converter, shown below.
Ext.define('PExt.json.XMLtoJSON', {
xml2Json : function(xml, isRoot) {
var jsonObj = {};
if (!isRoot) {
if (xml.nodeType == 1) {
jsonObj.file = {};
for (var i = 0; i < xml.attributes.length; i++) {
if (xml.attributes.item(i).nodeName == 'localized-name') {
jsonObj.file['localizedName'] = xml.attributes.item(i).nodeValue;
} else {
jsonObj.file[xml.attributes.item(i).nodeName] = xml.attributes.item(i).nodeValue;
}
}
} else if (xml.nodeType == 3) {
jsonObj = xml.nodeValue;
}
}
if (xml.hasChildNodes()) {
jsonObj.children = [];
for(var i = 0; i < xml.childNodes.length; i++) {
var child = xml.childNodes.item(i);
var visible = child.attributes['visible'].nodeValue;
if (visible === 'false') {
continue;
}
jsonObj.children.push(this.xml2Json(child, false));
}
}
return jsonObj;
}
});
It's not a generic converter, it cares about our repository/attributes it has. But the beauty of it is that I get real
JSON that the Store can readily consume. If only it were that easy, there are special attributes that the Store/NestedList
are looking for which we don't have. Other attributes are provided as opposites, for example, the 'leaf' attribute the
NestedList is looking for is missing, we have an attribute to tell us if something is a folder (isDirectory). The
solution, which was not at first obvious, was to define converters for each of the missing attributes. These converters let us
write a function to perform any type of conversion or lookup to return whatever value we need. For example:
Ext.define('PentahoMobile.model.File', {
...
config: {
fields: [
...
{name: 'leaf', type: 'boolean', mapping: 'file.isDirectory',
convert: function(value, record) {
// convert file.isDirectory into 'leaf'
// maybe check if a folder has children
...
}
},
]
}
...
});
After the store, mappings and converter function have been written, we have a NestedList which works against the
solution repository. You can add markup to the HTML generated for each entry in the list, such as adding folder icons
or descriptions.
This all served its purpose quite well until our UX department asked about removing the refresh button in the toolbar
and doing the 'pull-to-refresh' that is becoming more common in mobile apps these days. Fortunately, Sencha has such a
plugin, but I could not find any documentation on how to use it with a NestedList, only to the Ext.dataview.List. After
a lot of reading and mostly trial/error I found a solution. I had to 'use the source' in order to see how the NestedList
uses List internally, and maybe I could find some hint.
The key is a listConfig hidden within the config of the NestedList itself. I basically used this config as if it were
literally the config for the internal List used by NestedList. Thankfully this worked, after a few moments of wrestling
around with some improper refresh calls to my store I discovered that you can define your own refresh function for the
pull-to-refresh. As a note for completeness, you can also override the text which displays when you pull/release the
control. Here's the listConfig that I am using:
config: {
...
listConfig: {
plugins: [{
xclass: 'Ext.plugin.PullRefresh',
refreshFn: function(plugin) {
// refresh repository
}
}]
},
...
}
Here is a screenshot of it in action.
Tuesday, April 17, 2012
Pentaho User Console now working with all REST services
Over the past week or so I've been busy rewriting parts of the Pentaho User Console (PUC aka Mantle) to talk with REST services and move away from GWT-RPC. The primary motivation to do this work was to make the system more accessible to system integrators and OEM developers.
I have broken MantleService into several logical REST end points.
This work has simplified the User Console from a development standpoint, we have less ivy dependencies and the debug configuration is much more simple. All of the services return XML or JSON (determined by request headers) except for the extremely simple services which just return a string (text/plain), such as getting the current theme. I had to change the client code in PUC to consume JSON rather than the convenience/luxury we had before with GWT-RPC. The approach I took was to create JSON overlay objects (which extend JavaScriptObject). These JSON overlays are returned from the JSON eval/parsing and we then re-gain all of the richness we once had, at the expense of rewriting our POJO mutators as JNI.
For example, in the simple case of themes, which have an id and a value we now have a JsTheme:
public class JsTheme extends JavaScriptObject {
protected JsTheme() {
}
public final native String getId() /*-{ return this.id; }-*/; //
public final native String getName() /*-{ return this.name; }-*/; //
}
Once we return a list of JsTheme from JSON, our GWT app can go on enjoying life pretty much the same way as before. As a point of completeness, here is how we return get the themes from the JSON string:
JsArray<JsTheme> themes = JsonUtils.safeEval(JsonUtils.escapeJsonForEval(response.getText()));
I have broken MantleService into several logical REST end points.
Themes (/pentaho/api/theme)
A new service allowing you to list the available themes as well as set / get the current theme for the authenticated user.
User Settings (/pentaho/api/user-settings)
A general purpose user settings service API which lets you list settings (for the authenticated user) as well as get / set user settings.
Version (/pentaho/api/version)
Simple service to return the current product version and also provide a software updates list (if applicable).
System Refresh (/pentaho/api/system/refresh)
A collection of resource refresh/reload services for global actions, metadata, system settings, repository, mondrian and reporting.
User Console (/pentaho/api/mantle)
This was my dumping ground for services which I did not feel provided value to be separated into their own endpoint. The service is able to return mantle settings (different than user settings), list mondrian cubes, and get/set the locale for the authenticated user.
This work has simplified the User Console from a development standpoint, we have less ivy dependencies and the debug configuration is much more simple. All of the services return XML or JSON (determined by request headers) except for the extremely simple services which just return a string (text/plain), such as getting the current theme. I had to change the client code in PUC to consume JSON rather than the convenience/luxury we had before with GWT-RPC. The approach I took was to create JSON overlay objects (which extend JavaScriptObject). These JSON overlays are returned from the JSON eval/parsing and we then re-gain all of the richness we once had, at the expense of rewriting our POJO mutators as JNI.
For example, in the simple case of themes, which have an id and a value we now have a JsTheme:
public class JsTheme extends JavaScriptObject {
protected JsTheme() {
}
public final native String getId() /*-{ return this.id; }-*/; //
public final native String getName() /*-{ return this.name; }-*/; //
}
Once we return a list of JsTheme from JSON, our GWT app can go on enjoying life pretty much the same way as before. As a point of completeness, here is how we return get the themes from the JSON string:
JsArray<JsTheme> themes = JsonUtils.safeEval(JsonUtils.escapeJsonForEval(response.getText()));
As of Friday, April 13, 2012 the entire use of GWT-RPC has been replaced with REST. Enjoy!
Monday, April 16, 2012
New modules in the Administration perspective
Thanks goes out to Ezequiel Cuellar for being my ghost writer for this post.
The Administration perspective just got more love from the Sugar team. In an effort to migrate all the functionality from the Enterprise Console into Sugar the following two administration modules have been created: Users/Roles and Email/SMTP Server.
The User/Roles new UI allows you to easy administer users by creating or editing them and change their password, you can also create new roles and assign them to a user either one by one or by performing a multi selection.
In a similar way the roles section allow you to assign users.
The Email/SMTP Server module allows you to administer the BI Platform internal email settings (email_config.xml), you can test your changes by sending test emails before saving them and it also features a new error notification mechanism.
Wednesday, February 29, 2012
Pentaho Admin/Enterprise Console in SUGAR
If you've grabbed a recent build of SUGAR you'll notice some obvious changes to the UI as well as a top-level directory structure which is totally missing. The "administration-console" (or "enterprise-console") folder has been removed. We are actively working to integrate PAC/PEC functionality in the Pentaho User Console as an Administration perspective.
The layout of the Administration perspective is defined in mantle.xul (pentaho/mantle/xul/mantle.xul). This will allow the possibility of changing the UI and/or removing UI elements. Presently, we have action-based security (ABS) and authentication (Pentaho/LDAP) implemented in this UI. In the upcoming weeks we'll be adding other missing parts from the admin console such as email/auditing.
Here's a recent screenshot of the Administration perspective:
<overlays>
<overlay id="admin.perspective.overlay.ee" resourcebundle="content/my-admin/resources/messages/messages">
<treechildren id="security">
<treeitem command="mantleXulHandler.loadAdminContent('my-admin-panel', 'api/repos/myadmin/resources/my-admin.html')">
<treerow>
<treecell label="${myadmin.label}" />
</treerow>
</treeitem>
</treechildren>
</overlay>
</overlays>
We are adding a new panel to the admin perspective with an ID of 'my-admin-panel' and we are specifying the location of the UI (by URL). At this point you have adding your content to the admin category tree. Just like any plugin, you can have back-end code in a JAR, eg my-admin/lib/my-admin-plugin.jar.
Even more interesting is the new capability of a platform plugin to easily register its REST services in the plugin.spring.xml. We're using Jersey (v1.12) for exposing these web services. For reference take a look at the echo-plugin.
var myAdminPanel = {
id : "my-admin-panel",
activate : function() {
refreshConfig();
},
passivate : function(passivateCompleteCallback) {
if(isConfigDirty()) {
passivateCallback = passivateCompleteCallback;
dijit.byId("saveChangesDialog").show();
} else {
passivateCompleteCallback(true);
}
}
};
Now register this object with the admin perspective:
window.top.mantle_registerSysAdminPanel(myAdminPanel);
That's all there is too it, you will be notified when the user selects on/off of your panel so you can check for "dirty" and prompt for saving.
/pentaho/api/userrole/users
Using GET, will return a list of all users in the system.
/pentaho/api/userrole/roles
Using GET, will return a list of all runtime roles in the system.
/pentaho/api/userrole/roleAssignments
Using PUT, will set role bindings between roles and permissions (logical roles)
/pentaho/api/userrole/logicalRoleMap
Using GET, will return the list of roles and the permissions (logical roles) which are assigned to them
Using GET, returns all name/value pairs from applicationContext-security-ldap.properties, plus the current securityProvider
/pentaho/api/ldap/config/setAttributeValues
Using PUT, sets (merges) name/value pairs and saves them to applicationContext-security-ldap.properties as well as set the authentication type in pentaho-spring-beans.xml.
/pentaho/api/ldap/config/ldapTreeNodeChildren
Using GET, returns the list of
/pentaho/api/ldap/config/userTest
Using GET, simple test if a user can be found.
/pentaho/api/ldap/config/rolesTest
Using GET, tests if search for a user returns roleAttribute successfully.
/pentaho/api/ldap/config/userRolesTest
Using GET, will perform a populator test (check if granted authorities for the given user works)
/pentaho/api/ldap/config/providerTest
The layout of the Administration perspective is defined in mantle.xul (pentaho/mantle/xul/mantle.xul). This will allow the possibility of changing the UI and/or removing UI elements. Presently, we have action-based security (ABS) and authentication (Pentaho/LDAP) implemented in this UI. In the upcoming weeks we'll be adding other missing parts from the admin console such as email/auditing.
Here's a recent screenshot of the Administration perspective:
How to Add New Admin Functionality
New functionality can be plugged into the admin perspective with a platform plugin. The plugin.xml of a platform plugin will have a XUL overlay section to add an item to the admin category tree. For example, if you want to add a new item to the "security" category you would do this to the plugin.xml:<overlays>
<overlay id="admin.perspective.overlay.ee" resourcebundle="content/my-admin/resources/messages/messages">
<treechildren id="security">
<treeitem command="mantleXulHandler.loadAdminContent('my-admin-panel', 'api/repos/myadmin/resources/my-admin.html')">
<treerow>
<treecell label="${myadmin.label}" />
</treerow>
</treeitem>
</treechildren>
</overlay>
</overlays>
We are adding a new panel to the admin perspective with an ID of 'my-admin-panel' and we are specifying the location of the UI (by URL). At this point you have adding your content to the admin category tree. Just like any plugin, you can have back-end code in a JAR, eg my-admin/lib/my-admin-plugin.jar.
Even more interesting is the new capability of a platform plugin to easily register its REST services in the plugin.spring.xml. We're using Jersey (v1.12) for exposing these web services. For reference take a look at the echo-plugin.
Finishing the Job: JavaScript Integration
Whatever your choice of JavaScript library, you will be coexisting with PUC/admin perspective. While not required, you can improve the user experience by registering your UI for state changes, etc. To do this create an object with an id and activate/passivate methods. For example:var myAdminPanel = {
id : "my-admin-panel",
activate : function() {
refreshConfig();
},
passivate : function(passivateCompleteCallback) {
if(isConfigDirty()) {
passivateCallback = passivateCompleteCallback;
dijit.byId("saveChangesDialog").show();
} else {
passivateCompleteCallback(true);
}
}
};
Now register this object with the admin perspective:
window.top.mantle_registerSysAdminPanel(myAdminPanel);
That's all there is too it, you will be notified when the user selects on/off of your panel so you can check for "dirty" and prompt for saving.
REST services
In order to support the new admin functionality added to PUC, we added several new REST services which might be generally useful to OEMs, integrators, and developers./pentaho/api/userrole/users
Using GET, will return a list of all users in the system.
/pentaho/api/userrole/roles
Using GET, will return a list of all runtime roles in the system.
/pentaho/api/userrole/roleAssignments
Using PUT, will set role bindings between roles and permissions (logical roles)
/pentaho/api/userrole/logicalRoleMap
Using GET, will return the list of roles and the permissions (logical roles) which are assigned to them
The following LDAP REST API calls are in the EE product
/pentaho/api/ldap/config/getAttributeValuesUsing GET, returns all name/value pairs from applicationContext-security-ldap.properties, plus the current securityProvider
/pentaho/api/ldap/config/setAttributeValues
Using PUT, sets (merges) name/value pairs and saves them to applicationContext-security-ldap.properties as well as set the authentication type in pentaho-spring-beans.xml.
/pentaho/api/ldap/config/ldapTreeNodeChildren
Using GET, returns the list of
/pentaho/api/ldap/config/userTest
Using GET, simple test if a user can be found.
/pentaho/api/ldap/config/rolesTest
Using GET, tests if search for a user returns roleAttribute successfully.
/pentaho/api/ldap/config/userRolesTest
Using GET, will perform a populator test (check if granted authorities for the given user works)
/pentaho/api/ldap/config/providerTest
Wednesday, November 16, 2011
Scheduling and Workspace in SUGAR
With the recent addition of perspectives to the Pentaho User Console (PUC) we opened up a whole new way to integrate with the BI platform. This will go a long way for customers and OEMs who want to add (or remove) functionality from PUC. Having said this, we are currently developing new perspectives for the SUGAR release. Sean Flatley has been developing a PDI admin perspective (based on CDF). There are also plans to create an admin perspective (or add to the PDI admin) to replace the admin console (PAC and PEC).
Recently, I have been developing a total replacement for the PUC workspace, which was in dire need of TLC. When PDI added scheduling capabilities against our DI server, this was against a brand new scheduling system. As of yet, we hadn't taken advantage of this in the BI server. All of this changes in SUGAR, the old scheduler is completely removed, the new scheduler has taken over! Rather than get our existing (pre-SUGAR) workspace to work against the new scheduler, we spent some time re-writing it. The new workspace makes all scheduler interactions using REST. This means that it will be easy for other developers to interact with the scheduler in their own interfaces.
Scheduling with REST
I mainly wanted to highlight the new workspace in this post, but I figured there might be a fair amount of outside interest in learning about scheduling + REST. We have held up our end of REST purity in that GET, POST and DELETE HTTP methods are used where appropriate. Simple results are returned as text/plain, while complex state (such as a list of jobs) can be returned as either XML or JSON. Whatever your client-side technology of choice is, you can set the "accept" HTTP header to instruct the server to return the desired type back. For example, myrequest.setHeader("accept", "application/json") will cause the scheduler REST service to return results back (if supported) as JSON.
The URLs listed in the examples below assume that your BI server is running on "localhost" port 8080.
Scheduler State
To get the state of the scheduler make a GET request to:
http://localhost:8080/pentaho/api/scheduler/state
The return type for this is text/plain and the result will be one of:
RUNNING, PAUSED or STOPPED
To control the state of the scheduler you must make a POST request. In order to start or resume the scheduler as a whole:
http://localhost:8080/pentaho/api/scheduler/start
To pause the scheduler:
http://localhost:8080/pentaho/api/scheduler/pause
To shutdown the scheduler (must be rebooted after a shutdown):
http://localhost:8080/pentaho/api/scheduler/shutdown
Remember, these are POST requests, you cannot just paste the URL in a browser and expect them to work (would be a GET request this way).
Listing Jobs
Since listing jobs does not change any state on the server, the request for getting the list of jobs is a GET.
The following URL can be used as GET request and will return XML or JSON.
http://localhost:8080/pentaho/api/scheduler/jobs
Job State
Like the scheduler itself, we can interact with the running state of an individual job. Getting the state of a specific job requires that you submit the jobId (which is given in the list jobs REST call). You can only get the state of a job that you created (own), unless you have administration privileges.
To get the state of a job, the REST url is:
http://localhost:8080/pentaho/api/scheduler/jobState
You must submit JSON or XML wrapping the jobId. For example, the JSON request payload:
{"jobId":"joe:1685214344:1321154720424"}
The request header for the "Content-Type" is also set: myrequest.setHeader("Content-Type", "application/json");
The return type for this is text/plain and the result will be one of:
NORMAL, PAUSED, COMPLETE, ERROR, BLOCKED or UNKNOWN
Altering the state of a job is not much different than getting the state except that a POST request must be made. The REST urls are:
http://localhost:8080/pentaho/api/scheduler/resumeJob
http://localhost:8080/pentaho/api/scheduler/pauseJob
Triggering a Job Immediately
To trigger the immediate execution of a job, you can invoke the triggerNow REST endpoint (POST) with the jobId wrapped with JSON or XML. You must be authorized to execute the job in order to trigger it.
Deleting a Job
To remove a job from the scheduler, you can invoke the removeJob REST endpoint (DELETE) with the jobId wrapped with JSON or XML. You must be authorized (job owner or admin) to delete the job from the scheduler.
Creating a New Job
This is the most complex part of interacting with the scheduler. In order to represent a new schedule, there are 3 "trigger" types, simple, complex and cron.
I'm just going to give some examples rather than document every possible combination. First, let's use a simple schedule, run the Inventory.prpt every 4 hours until December 31, 2012. The JSON payload would be:
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Inventory.prpt", "outputFile":null, "simpleJobTrigger":{"repeatInterval":14400, "repeatCount":-1, "startTime":"2011-11-16T00:00:00.000-05:00", "endTime":"2012-12-31T23:59:59.000-05:00"}}
The inputFile is the full path to the Inventory.prpt resource. We're using a simple trigger, meaning that we don't worry about special recurrence patterns, we just want to run every 4 hours until the "endTime" has been reached. The repeatInterval is 14400 seconds which equals 4 hours. If you want to repeat a specific number of times until the trigger is no longer fired (in lieu of endTime) you can give a repeatCount. A repeatCount of -1 means forever.
Next, let's imagine we want to schedule the Produce Line Sales.prpt every Sunday at 2am with no end date.
The REST endpoint is http://localhost:8080/pentaho/api/scheduler/createJob. The JSON payload would be something like this:
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Product Line Sales.prpt", "outputFile":null, "complexJobTrigger":{"daysOfWeek":["0"], "startTime":"2011-11-16T02:00:00.000-05:00", "endTime":null}}
Dissecting this, we can see the inputFile is set to the full path to the scheduled resource. We are creating a "complex" job trigger with a recurrence pattern of "daysOfWeek" including just "0" meaning Sunday, the days range from 0-6. If the trigger was going to be for multiple days of the week, this would be given as "daysOfWeek":["0","1"]" (for Sunday/Monday). All times are in ISO_8601 date format (this is true for dates coming out of the scheduler REST services as well). The startTime specifies the "from" date and endTime refers to the date at which the schedule will no longer be run. A null value for the endTime means it has no end.
Another example, "The last Friday of every month at 4am" would have a JSON payload of:
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Income Statement.prpt", "outputFile":null, "complexJobTrigger":{"weeksOfMonth":["4"], "daysOfWeek":["5"], "startTime":"2011-11-16T04:00:00.000-05:00", "endTime":null}}
Finally, a yearly schedule, "Every January 1st at midnight":
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Invoice Statements.prpt", "outputFile":null, "complexJobTrigger":{"monthsOfYear":["0"], "daysOfMonth":["1"], "startTime":"2011-11-16T00:00:00.000-05:00", "endTime":null}}
The Workspace
With all the REST details behind me now, I can finally cover some new UI work that I've been working on the past few weeks. As I said before, the new workspace interacts with the server exclusively through REST web services, meaning that it is possible for someone with better UI skills to replace it (by removing the default one from the default-plugin/plugin.xml).
The old workspace listed all content for each schedule, this was unbelievably unmanageable, it was also rather clunky when it came to starting/stopping/removing schedules and their output content. It also lacked the ability to manage the scheduler as a whole (start/stop).
The new workspace lists schedules (aka jobs), not content (output) from those jobs. You can start/stop the entire scheduler or pause/resume individual jobs. A human readable description of each schedule is provided. Each column in the table view can be sorted. If there are many schedules, the table will enter a "paging" mode. If there are still too many schedules to find what you are looking for you can easily add a filter. You can multi-select (with the help of CTRL or SHIFT keys) and manage many schedules at once. Selected jobs can be triggered to run immediately, paused, resumed or removed permanently. When you click on a cell in the file (resource) column you can view and manage (TBD) content from previous executions of that schedule.
Recently, I have been developing a total replacement for the PUC workspace, which was in dire need of TLC. When PDI added scheduling capabilities against our DI server, this was against a brand new scheduling system. As of yet, we hadn't taken advantage of this in the BI server. All of this changes in SUGAR, the old scheduler is completely removed, the new scheduler has taken over! Rather than get our existing (pre-SUGAR) workspace to work against the new scheduler, we spent some time re-writing it. The new workspace makes all scheduler interactions using REST. This means that it will be easy for other developers to interact with the scheduler in their own interfaces.
Scheduling with REST
I mainly wanted to highlight the new workspace in this post, but I figured there might be a fair amount of outside interest in learning about scheduling + REST. We have held up our end of REST purity in that GET, POST and DELETE HTTP methods are used where appropriate. Simple results are returned as text/plain, while complex state (such as a list of jobs) can be returned as either XML or JSON. Whatever your client-side technology of choice is, you can set the "accept" HTTP header to instruct the server to return the desired type back. For example, myrequest.setHeader("accept", "application/json") will cause the scheduler REST service to return results back (if supported) as JSON.
The URLs listed in the examples below assume that your BI server is running on "localhost" port 8080.
Scheduler State
To get the state of the scheduler make a GET request to:
http://localhost:8080/pentaho/api/scheduler/state
The return type for this is text/plain and the result will be one of:
RUNNING, PAUSED or STOPPED
To control the state of the scheduler you must make a POST request. In order to start or resume the scheduler as a whole:
http://localhost:8080/pentaho/api/scheduler/start
To pause the scheduler:
http://localhost:8080/pentaho/api/scheduler/pause
To shutdown the scheduler (must be rebooted after a shutdown):
http://localhost:8080/pentaho/api/scheduler/shutdown
Remember, these are POST requests, you cannot just paste the URL in a browser and expect them to work (would be a GET request this way).
Listing Jobs
Since listing jobs does not change any state on the server, the request for getting the list of jobs is a GET.
The following URL can be used as GET request and will return XML or JSON.
http://localhost:8080/pentaho/api/scheduler/jobs
Job State
Like the scheduler itself, we can interact with the running state of an individual job. Getting the state of a specific job requires that you submit the jobId (which is given in the list jobs REST call). You can only get the state of a job that you created (own), unless you have administration privileges.
To get the state of a job, the REST url is:
http://localhost:8080/pentaho/api/scheduler/jobState
You must submit JSON or XML wrapping the jobId. For example, the JSON request payload:
{"jobId":"joe:1685214344:1321154720424"}
The request header for the "Content-Type" is also set: myrequest.setHeader("Content-Type", "application/json");
The return type for this is text/plain and the result will be one of:
NORMAL, PAUSED, COMPLETE, ERROR, BLOCKED or UNKNOWN
Altering the state of a job is not much different than getting the state except that a POST request must be made. The REST urls are:
http://localhost:8080/pentaho/api/scheduler/resumeJob
http://localhost:8080/pentaho/api/scheduler/pauseJob
Triggering a Job Immediately
To trigger the immediate execution of a job, you can invoke the triggerNow REST endpoint (POST) with the jobId wrapped with JSON or XML. You must be authorized to execute the job in order to trigger it.
Deleting a Job
To remove a job from the scheduler, you can invoke the removeJob REST endpoint (DELETE) with the jobId wrapped with JSON or XML. You must be authorized (job owner or admin) to delete the job from the scheduler.
Creating a New Job
This is the most complex part of interacting with the scheduler. In order to represent a new schedule, there are 3 "trigger" types, simple, complex and cron.
I'm just going to give some examples rather than document every possible combination. First, let's use a simple schedule, run the Inventory.prpt every 4 hours until December 31, 2012. The JSON payload would be:
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Inventory.prpt", "outputFile":null, "simpleJobTrigger":{"repeatInterval":14400, "repeatCount":-1, "startTime":"2011-11-16T00:00:00.000-05:00", "endTime":"2012-12-31T23:59:59.000-05:00"}}
The inputFile is the full path to the Inventory.prpt resource. We're using a simple trigger, meaning that we don't worry about special recurrence patterns, we just want to run every 4 hours until the "endTime" has been reached. The repeatInterval is 14400 seconds which equals 4 hours. If you want to repeat a specific number of times until the trigger is no longer fired (in lieu of endTime) you can give a repeatCount. A repeatCount of -1 means forever.
Next, let's imagine we want to schedule the Produce Line Sales.prpt every Sunday at 2am with no end date.
The REST endpoint is http://localhost:8080/pentaho/api/scheduler/createJob. The JSON payload would be something like this:
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Product Line Sales.prpt", "outputFile":null, "complexJobTrigger":{"daysOfWeek":["0"], "startTime":"2011-11-16T02:00:00.000-05:00", "endTime":null}}
Dissecting this, we can see the inputFile is set to the full path to the scheduled resource. We are creating a "complex" job trigger with a recurrence pattern of "daysOfWeek" including just "0" meaning Sunday, the days range from 0-6. If the trigger was going to be for multiple days of the week, this would be given as "daysOfWeek":["0","1"]" (for Sunday/Monday). All times are in ISO_8601 date format (this is true for dates coming out of the scheduler REST services as well). The startTime specifies the "from" date and endTime refers to the date at which the schedule will no longer be run. A null value for the endTime means it has no end.
Another example, "The last Friday of every month at 4am" would have a JSON payload of:
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Income Statement.prpt", "outputFile":null, "complexJobTrigger":{"weeksOfMonth":["4"], "daysOfWeek":["5"], "startTime":"2011-11-16T04:00:00.000-05:00", "endTime":null}}
Finally, a yearly schedule, "Every January 1st at midnight":
{"inputFile":"/public/pentaho-solutions/steel-wheels/reports/Invoice Statements.prpt", "outputFile":null, "complexJobTrigger":{"monthsOfYear":["0"], "daysOfMonth":["1"], "startTime":"2011-11-16T00:00:00.000-05:00", "endTime":null}}
The Workspace
With all the REST details behind me now, I can finally cover some new UI work that I've been working on the past few weeks. As I said before, the new workspace interacts with the server exclusively through REST web services, meaning that it is possible for someone with better UI skills to replace it (by removing the default one from the default-plugin/plugin.xml).
The old workspace listed all content for each schedule, this was unbelievably unmanageable, it was also rather clunky when it came to starting/stopping/removing schedules and their output content. It also lacked the ability to manage the scheduler as a whole (start/stop).
The new workspace lists schedules (aka jobs), not content (output) from those jobs. You can start/stop the entire scheduler or pause/resume individual jobs. A human readable description of each schedule is provided. Each column in the table view can be sorted. If there are many schedules, the table will enter a "paging" mode. If there are still too many schedules to find what you are looking for you can easily add a filter. You can multi-select (with the help of CTRL or SHIFT keys) and manage many schedules at once. Selected jobs can be triggered to run immediately, paused, resumed or removed permanently. When you click on a cell in the file (resource) column you can view and manage (TBD) content from previous executions of that schedule.
Workspace View showing multi-select "pause" (notice state of selected items)
You can filter the list of jobs by file, state, user, schedule type and execution times
Selecting a file link will show past execution history and allow content to be viewed.
We're not done with the scheduling yet, but we've been making incredible progress. We still need to finish (WIP) parameter support and define (TBD) what content management can be done from the history (generated content dialog).
Tuesday, November 1, 2011
Plugin Overlays, PUC Layout
I was thinking about my previous post on PUC Perspectives and some of the things that were necessary to make that happen when it dawned on me that I hadn't really highlighted some really cool changes. There are two primary changes that are worth talking about: menubar and PUC layout.
Menubar
The PUC menubar was a GWT menubar, standing not much different than it did in the proof-of-concept that I did back in March 2008. James Dixon later extended upon this by adding the ability to define "menu customizations" through our plugin system. What we really needed though was a total rewrite of the menu system but the prospect of making such a drastic change was never a priority. Fortunately, we were able to justify the rewrite with the fact that without it, the capability of doing perspective overlays for the menubar were going to be pretty darn near impossible. That is, without adding hacks upon existing hacks. And so, the menubar was rewritten and XULified.
We now have a main_menubar.xul which defines the content and layout of the main menubar. This will make it MUCH easier for 3rd party/OEMs to add/remove/update any/all of the behavior of the menubar simply by tweaking the XUL file.
This means that plugin.xml which used to contain "menu-items" will now use an overlay. Most plugins which defined menu-items already had an overlay section to define toolbar tweaks. The same overlay section is used for both menu and toolbar changes, for example,
<overlay id="startup.analyzer" resourcebundle="content/analyzer/resources/messages">
There are a few subtle differences here with pre-SUGAR overlay definitions. I have fixed the annoying bug in the plugin system which required the nesting of an overlay inside of an overlay (simply due to XML parsing bug), for example:
Menubar
The PUC menubar was a GWT menubar, standing not much different than it did in the proof-of-concept that I did back in March 2008. James Dixon later extended upon this by adding the ability to define "menu customizations" through our plugin system. What we really needed though was a total rewrite of the menu system but the prospect of making such a drastic change was never a priority. Fortunately, we were able to justify the rewrite with the fact that without it, the capability of doing perspective overlays for the menubar were going to be pretty darn near impossible. That is, without adding hacks upon existing hacks. And so, the menubar was rewritten and XULified.
We now have a main_menubar.xul which defines the content and layout of the main menubar. This will make it MUCH easier for 3rd party/OEMs to add/remove/update any/all of the behavior of the menubar simply by tweaking the XUL file.
This means that plugin.xml which used to contain "menu-items" will now use an overlay. Most plugins which defined menu-items already had an overlay section to define toolbar tweaks. The same overlay section is used for both menu and toolbar changes, for example,
<overlay id="startup.analyzer" resourcebundle="content/analyzer/resources/messages">
<toolbar id="mainToolbar">
<toolbarbutton id="newAnalysisButton" image="../api/repos/xanalyzer/images/analyzer_toolbar_icon.png" onclick="mainToolbarHandler.openUrl('${tabName}','${tabName}','api/repos/xanalyzer/service/selectSchema')" tooltiptext="${openNewAnalyzerReport}" insertafter="dummyPluginContentButton"/>
</toolbar>
<menubar id="newmenu">
<menuitem id="new-analyzer" label="${openNewAnalyzerReport}" command="mainMenubarHandler.openUrl('${tabName}','${tabName}','api/repos/xanalyzer/service/selectSchema')" />
</menubar>
</overlay>
There are a few subtle differences here with pre-SUGAR overlay definitions. I have fixed the annoying bug in the plugin system which required the nesting of an overlay inside of an overlay (simply due to XML parsing bug), for example:
<overlay id="startup.analyzer" resourcebundle="content/analyzer/resources/messages">
<overlay id="startup.analyzer" resourcebundle="content/analyzer/resources/messages">
Another benefit is that overlays can have resource bundles associated with them, while the old menu-item section did not. This allows us to localize the display strings in the menu system.
PUC Layout
I was recently asked by James Dixon if it would be possible to update the entire layout of PUC with XUL. I said we needed to get the story on our sprint backlog, which I ended up doing (BISERVER-6693). Instead of using XUL, which might be a barrier to entry for some customers/OEMs, a much easier solution to the PUC layout actually exists: just use HTML. What if the PUC layout existed in HTML and we just inject into various id's? So, in SUGAR, I have done just this, the layout of PUC is based on DIV tags in the HTML (Mantle.jsp). When PUC loads, it no longer "takes over" the page, it now looks for certain elements by id, such as "pucMenuBar" or "pucPerspectives" and then injects the widget at that location. This will allow easier customization, for example, in SUGAR we have actually removed the "logo panel" from PUC itself. With the DIV-based layout, we can easily add a logo panel back into the product by editing the HTML. This is still a work in progress, the "pucContent" is very high-level and refers to the entire bottom section of PUC (explorer + content). The next phase will be to define the layout even further, but we've taken steps towards this direction and what has been done is beyond concept, it's committed.
The layout of PUC can be defined as something as simple as this:
Enjoy!
The layout of PUC can be defined as something as simple as this:
<div id="puc" style="height: 100%"> <div id="pucTopBar" style="background-color: black; height: 28px"> <div id="pucMenuBar" style="float: left"> </div> <div id="pucPerspectives" style="float: right;"> </div> </div> <div id="pucToolBar" style="clear; both; float: left; width: 100%"> </div> <div id="pucContent" style="clear: both; height: 100%; width: 100%"> </div> </div>
Subscribe to:
Posts (Atom)