Data Meet Process

Intalio is currently experiencing very rapid growth in emerging markets, especially in Latin America. Yesterday, our Spanish-speaking webinar attracted 79 participants, the highest turnout to date. In this particular market, we’re facing an interesting competitor: Aura Portal. We come across them more and more often in competitive situations, and I’ve been trying to understand why so many companies in Latin America were attracted to their proprietary product, which is a blend of BPM, ECM, Portal, and… CRM. After some more noodling, I think I finally came up with the answer: by integrating CRM into the mix, Aura Portal is blending Process and Data, which is the secret for a perfect recipe. It’s not the first time I write about this, but I’m now ready to act upon it.

Whether analyst firms like it or not, BPM is not a business application, it’s a business platform, and as such, it must support the development of enterprise applications in a process oriented way. For most new applications built against existing systems of records (ERP in particular), orchestration of services and support for human workflow is all you really need. But for more green-field environments (of which you’ll…

Intalio is currently experiencing very rapid growth in emerging markets, especially in Latin America. Yesterday, our Spanish-speaking webinar attracted 79 participants, the highest turnout to date. In this particular market, we’re facing an interesting competitor: Aura Portal. We come across them more and more often in competitive situations, and I’ve been trying to understand why so many companies in Latin America were attracted to their proprietary product, which is a blend of BPM, ECM, Portal, and… CRM. After some more noodling, I think I finally came up with the answer: by integrating CRM into the mix, Aura Portal is blending Process and Data, which is the secret for a perfect recipe. It’s not the first time I write about this, but I’m now ready to act upon it.

Whether analyst firms like it or not, BPM is not a business application, it’s a business platform, and as such, it must support the development of enterprise applications in a process oriented way. For most new applications built against existing systems of records (ERP in particular), orchestration of services and support for human workflow is all you really need. But for more green-field environments (of which you’ll find plenty in emerging markets), the development of custom data objects is necessary, and this is precisely what CRM systems are good for. To better understand this notion, forget about the content side of CRM (content as in business objects such as Customer or Opportunity). Instead, focus on the underlying infrastructure, which allows one to develop custom data objects through simple wizards. In other words, takes the Sales out of Salesforce.com, and focus on Force.com instead.

Much like BPM 2.0 tools let you build processes in a zero-code fashion, you’d like a true Business Process Platform (BPP) to let you build data objects without having to write code either. And not just that, but the platform should make these objects directly consumable by the process engine, through well-defined interfaces that could be generated automatically from the data model. Granted, there are many tools one could use to do that today, but they’re either too complex to use for business analysts, or too limited in their ability to support the more complex object models. This is precisely what Salesforce.com did a great job at (I’ve never tried Aura Portal, but I must assume it does a good job at it too), and we definitely need to add such a capability to our product. Here is how we’re approaching the problem.

First, we need to agree that we will use a Relational Database Management System (RDBMS) as repository for our data objects. While object-oriented databases are out of fashion and XML databases never really materialized, relational databases are available from a variety of sources, both commercial and Open Source, and nothing beats them in terms of reliability and dependability. Document-oriented databases like Apache CouchDB are intriguing, but they’re not optimized for structured data objects.

Second, we need to decide how to use the database itself. The first option is to store data objects in dedicated tables (one table per object). The second option is to create a set of generic tables (3 to 5 usually), and store all objects into it. The later is the option retained by Salesforce.com. When the Salesforce.com application was first designed ten years ago, virtualization did not really exist (at least in the way we understand it today), and the approach made perfect sense in order to support tens of thousands of customers in a multi-tenant environment. Unfortunately, the approach has its limits, for it forced Salesforce.com’s engineers to essentially re-invent the database on top of a database (Oracle in that particular example), making it very difficult to support complex queries with multiple levels of joins, and creating some severe scalability issues. Today’s virtualization technologies and low-cost databases (MySQL or PostgreSQL) dramatically change the equation, and deploying an off-the-shelf relational database on top of a grid of low-cost servers would give us scalability both horizontally (in terms of number of customers) and vertically (in terms of number of objects and users per customer).

Third, we need to select some kind of middleware to connect to the database. There are plenty of Java-based options there, from Hibernate to OpenJPA, but Java might not be the best language to start with. While we need to make it easy to create data objects without having to write code, we also need to support the ability to script data objects, without having to invent yet another language (like Salesforce.com did with APEX). When it comes to scripting, Ruby seems to get the most votes today, and Rails is exactly the kind of middleware we need to go from data model to database tables, without having to write much code, or any code at all for that matter.

Fourth, we need to develop a dead-simple user interface on top of the middleware. In this area, Salesforce.com should be used as ultimate benchmark. Its user interface makes it very easy to create new objects, define custom fields for them, set access control and visibility rules, and dynamically create forms and views for editing data objects. And when it comes to forms, simplicity should be the rule. Once you created a custom object with Salesforce.com, the application automatically creates a default “Page Layout” for it, which you can barely customize. All you can do is define sections which can have either one or two columns (of the same size), and move fields around with a drag-and-drop user interface. That’s it, and that’s all one should really need. And if you need something more custom, use a more advanced tool, such as Intalio|AJAX for example.

Fifth, we need to generate interfaces for these objects, and both REST and WSDL should be supported. REST because it’s easy to use, and WSDL because it’s all BPEL can natively consume (until we develop a RESTful version of BPEL that is). These interfaces should be totally canonical, and automatically generated by the database middleware. They should be registered into some kind of directory that could be looked-up through a dedicated REST interface (too bad UDDI never took off).

Sixth, we need to add some kind of service invocation tool to Intalio|BPP Business Edition, with a simple data mapping tool. This tool must be an order of magnitude simpler than the Data Mapper of Intalio|Designer, and should be built using the Google Web Toolkit (GWT, also used by the Business Edition). Nevertheless, the Data Object Builder described in this article should also be integrated with the other editions of Intalio’s platform, the Community Edition, Developer Edition, and Enterprise Edition.

Once we get all that done, we’ll get something pretty close to Coghead. At this point, it’s unclear whether we should add any business content to the platform or not. But should we decide to, Apache OFBiz looks like a good place to start from.

We’re currently kicking the tires with this project. If we decide to move forward with it, we will most likely develop it under the GNU Affero General Public License Version 3 (AGPL v3). If this sounds like fun, feel free to drop us a line.

In the meantime, we’ll create a D3 project for it.

Link to original post