an extract from a paper I wrote a while ago on BPM

[Note: some of this material comes from presentations I attended given by various Gartner analysts]

Background Material on BPM Technology

A Business Process Management System (BPMS) is:
• A development and runtime environment that enables process modeling and design, development and execution, and ongoing management and optimization;
• Automates and manages the end-to-end flow of work as it progresses across system and human boundaries;
• Unifies previously independent software infrastructure categories (such as workflow, EAI, document/content mgmt., mgmt., portals, Web servers and application servers);
• Supports SOA principles and XML Web services standards;
• Among many emerging technologies that can be used for creating SOA composite applications.

In general, as a management discipline, BPM is the ability to continuously optimize operational processes that most directly affects the achievement of corporate performance goals. When implemented in technology as an integrated business process management suite, business process architects and users can:
• Model the interactions among workers, systems and information that are necessary for accomplishing work;
• Consistently execute the optimal process;
• Coordinate and manage the handoff of work across boundaries;
• Adjust organizational structure and incentives to foster new behaviors;
• Monitor process outcomes to performance targets and seek continuous improvement.

There is no one technology that does all of the functions needed to support BPM. Enterprises will require some degree of integration across these technologies to enable consistency and reuse. Some will seek an integrated business process management suite from a single vendor (or its strategic partners that provide tight levels of integration). However, most will need to integrate tools from multiple vendors themselves.

The origin of BPM is in the Quality and BPR domains. In the 80s, the focus was on Quality. In the 1990s, focus shifted to BPR. In the 2000s, it has evolved to BPM. Thus, a focus on processes is not new. We have seen it before in the era of quality programs, followed by, and even more explicit, in business process re-engineering. The current phase is built on a stronger synergy of IT capabilities
and business understanding, but can equally be expected to follow the typical hype cycle. Gartner feels that we are still at an early stage in the current [BPM] wave. Management writers have been espousing a new found belief in process (Michael Hammer's book "TheAgenda" was published in 2001 and demonstrated a rebirth of process thinking from the ashes of BPR). Start-up companies have been pursuing BPM as a technology initiative (Gartner 2004 Magic Quadrant for BPM was constructed from consideration of 100 vendors — see "Magic Quadrant for Pure-Play BPM, 2Q04," June 2003, M-22-9774). Standardization initiatives have started and proliferated — Business Process Execution Language (BPEL), Business Process Modeling Language (BPML) and Web Service Choreography Interface (WSCI) — and according to Gartner are beginning to consolidate around BPEL. Gartner believes that the hype around BPM will continue to build before the industry will stabilize. Furthermore, Gartner believes that early adopters will gain their greatest benefit during the trough of disillusionment, which they feel we will see shortly.

Looking at figure 1, we are currently in the Service-Oriented era. In the future, industry will continue to evolve towards a more event-driven architecture and finally towards even a more adaptive and dynamic environment than will be available via the SOA. We will only briefly touch on the Event Driven Architecture in this report. The adaptive/dynamic era is beyond the scope of this paper.

The kinds of products that are included within the BPM market are shown in figure 2. BPMS contains core BPM-enabling tools: orchestration engines, business intelligence and analysis tools, rules engines; repositories (for process definitions, process components, process models, business rules, process data); simulation and optimization tools; integration tools.

A core component of BPM is modeling -- its importance cannot be understated. Good process models communicate how work is accomplished, reflecting the concerns of all of the stakeholders and participating functions. Process models are needed to help business and IT managers understand actual processes and enable them, by visualization and simulation, to propose improvements. Explicit process models are easily changed because non-technical managers understand them easily and they are independent of the underlying resources. Models provide a basis for cross-organizational collaboration between managers responsible for the separate tasks within a process, as well as with IT professionals on the implementation of the resulting design. The key elements to be identified in a process model are the business events that trigger actions, the sequence of steps, and the business rules used in and between those steps to support decision making and execution flow. Once this is done, IT professionals (architects and systems analysts) can begin to map the work tasks and information dependencies to existing logic, data and user interfaces. This kind of multilevel modeling effort identifies valuable existing IT assets to be leveraged in new process designs, and highlights those areas where business users want more control over process change. As in manufacturing (where a broken finished product can easily be fixed when its design is based on a component-assembly approach), so too can re-engineering to SOA turn existing IT assets into reusable services to achieve the desired flexibility. However, modeling is still just one of ten major components of a BPM platform!

Action Item: Modeling must become a business discipline — not a creative pastime.

With a BPMS, the full business process is made explicit in a graphical process flow model. This process model is completely autonomous from the resources performing the work steps, whether they are human or machine resources. By making it explicit and autonomous, changes to the process model can be made independently from changes in the resources.

This is the "loose coupling" principle familiar to many from earlier middleware forms and from SOA design guidelines.. Making process flow control explicit and decoupling it from the underlying technology allows processes to be changed more quickly. For example: Processes are easily changed since other system elements are not affected by flow control changes and need not, therefore, be retested. Some process changes are made by business professionals who need only limited knowledge of IT systems. In a growing number of cases, changes such as work item routing, business rule overrides, and parametric changes to approval levels, are made in real time to executing process. Near-real-time reporting of process steps deliver never-before-seen analysis of current operating conditions customized as appropriate to the organization. Tightly coupling business dashboards/cockpits/business activity monitoring (BAM) to the underlying runtime allows rapid and precise process change. Orchestration engines are the runtime environments. There will be different kinds of runtime environments — some for Web-service-based components, others for human workflow components, others for rules. The process model is simply an XML metadata description of how all the activities and events should be coordinated. And because it is XML, it can be made executable at runtime and the resources that perform the steps can be dynamically late-bound into the execution.

Figure 3 shows the traditional software development lifecycle along with that for BPM development (and integration). The traditional lifecycle model is shown simply t contrast it with today’s BPM-focused, highly agile one. We will focus on the BPM lifecycle (on the right) as most are quite familiar with that model shown on the left. (see next post for image)

Specifically, BPM consists of the following iterative phases:
• Definition (and/or discovery) identifies intricacies of how a process executes
• Modeling is valuable because it shows easy improvement opportunities, or at least the scale of the problem. Helps process owners collaborate on potential process improvements that will help achieve corporate goals.
• Simulation reveals bottlenecks not obvious during static modeling. Helps fine-tune adjustments in process model.
• Deployment creates detailed process execution scripts and makes required changes in systems. Training and facility changes must be coordinated; deployment usually involves integration with external systems and may include converting application code segments into sets of reusable web service components.
• Execution is where the main value of BPM is realized because its where the actual improvements are first seen.
• Monitoring collects information from executing processes in real time and helps facilitate immediate corrections.
• Analysis creates further value when key performance indicators based on process execution are linked directly to business objectives.
• Optimization is a fact-based approach to process scenario optimization, greatly reducing risk and eliminating guesswork.

Action Item: The separation between process design and process implementation and execution (a separation not found in conventional applications) will permit the emergence of a market for "process components" — a new kind of intellectual property. These components will include templates for specific horizontal and vertical processes, business content, and rule sets.

Up to now, this report has focused on the business side of BPM. Briefly addressing the information technology aspect of business management, SOA is the current best practice for defining service modularity and interoperability. SOA is a style of software system architecture in which certain discrete functions are packaged into modular, encapsulated, shareable elements ("services") which can be invoked in a loosely coupled manner by local or remote "consumer" parts of the system. Thus, SOA is an architectural style that is modular, distributable and loosely coupled. A service (or component) is a software process that acts in response to a request. Business services comprise a type of service; those that have semantic meaning in a business context. An SOA service refers to the combination of a provider service and its exposed interface to consumer services. The interface is the contract between consumer implementations and provider implementations. It is the consumer's view of a provider's capability and the provider's view of the consumer's responsibilities. Lastly, web services are software components that employ one or more of the following technologies — SOAP, WSDL and UDDI — to perform distributed computing.

Action Item: IT must decide on tools and approaches (wrapping or rewriting) to be used for migrating existing business capabilities into reusable services, and make them available to business analysts, process modelers and architects.

Service-oriented systems do not require the Web services suite of protocols, but the Web services protocol family provides a standard, widely supported foundation for intercommunication between SOA systems. Portal technology predates Web services, but has evolved to incorporate support for this distributed computing technology, along with other technology for application integration (for example, message queues and connectors). Another benefit of an SOA is that two subsystems can each pursue their own destinies or life cycles. As long as there is a contract that governs the interaction between the two, they can evolve separately. Thus, SOAs offer greater flexibility and agility. Organizations can recombine, reassemble and orchestrate their systems with other systems to create composites more quickly than they would by developing a monolithic system from scratch.

Figure 4 depicts how a business process platform that supported SOA might be visualized. SOA enables software to be defined as independent services that can be "composed" into operational systems. The composition process is driven largely by mapping the use of services within a business process (i.e., process orchestration). The services generally are assumed to already exist either as components delivered within new Service Oriented Business Applications (SOBAs) or accessed from an external source. In addition, established applications may be segmented and components wrapped to create service interfaces, so these legacy systems can be exploited within the composition platform, or new services may be developed from scratch to meet unique needs. Services will be managed and stored in a repository along with rules for maintaining their integrity (the repository may also point to external services, acting, in that case, as a registry). The composition process also looks out to the user experience and the way in which the functionality is delivered (typically via multiple channels and alternate device types). Management and security mechanisms must still span the repository, composition platform and user experience. The creation of this new framework for delivering applications is generating new families of integrated products and technologies from middleware, platform and application companies. In turn, this is creating new areas of competition and collaboration between technology suppliers. The result for businesses will be the reconstituting of their application software as a BPP delivering greater flexibility in running the business.

They equate Web services with SOA and distributed computing with SOA, or they combine events and services with SOA. This debate seems academic, but the position you take determines what your company gets out of your SOA initiative. Plain use of Simple Object Access Protocol (SOAP) enables client/server exchanges across the Internet (and through firewalls). This is a benefit for many applications, but users who expect additional SOA benefits from deploying a SOAP-based interchange will be disappointed. As users make advanced investment in their architectures, the benefits will include incremental engineering of software, technical or business reuse of software, business-to-business (B2B) opportunities, business activity monitoring (BAM), managed inter-application scalability and availability of software. Ultimately, the movement toward SOA and beyond promises to bring business-IT alignment closer — a long-time pursuit of the software industry. However, greater benefits require a greater investment. Declaring a distributed computing deployment an SOA will not deliver the full benefits of SOA. Adopting the vision of business semantics for services and adding integrated support of business events requires greater systematic effort, but it is essential to achieving the potential of the fully expressed architecture of business components.

Action Item: Seek an integrated platform, comprised of a business services repository and composition technologies, to enable the Business Process Platform.

SOA initiatives should be linked to BPM for mutual benefit! Although SOA is complimentary to BPM, BPM can be accomplished with or without SOA. The more the organization desires flexible processes with greater flexibility in the systems automated steps, the more important SOA becomes. Nevertheless, while the organization builds up a portfolio of reusable services, BPMS technology can be used to take early baby steps toward adaptive processes. The shift in control over processes from IT to line-of-business managers is best accomplished gradually, allowing everyone to gain confidence in the required skills and technologies. In this presentation, we've tried to present a number of pragmatic approaches to accomplishing high-value business initiatives leveraging existing IT assets. For those strategic solution areas, meant to differentiate the enterprise, and where the rate of change is expected to be high, re-engineering of automated business capabilities into SOA Web services is highly justified.

Action Item: To get the most from your SOA investment, think of SOA as long-term software architecture and engineering practice, not just as Web services or another tool.

Bottom Line: Migration to SOA is not an "all or nothing" proposition.

There are many reasons to use event-driven applications, rather than monolithic application architectures or even SOAs. Events can scale to higher volumes of business transactions and more users; they can simplify application development (AD) by reducing the complexity and amount of code in application programs; they can decrease the latency (response time) for a function and reduce the elapsed time for an entire end-to-end business process; they can facilitate data quality by being closer to real time; they can enable better auditability (track and trace); they can provide earlier warning of threats and earlier notice of emerging opportunities; and, perhaps most importantly, they provide the maximum application software "plugability" to facilitate agility through continuous adjustments to business processes.

Ultimately, the SOA is not as flexible or efficient as an event-driven design, but it is more flexible and agile than traditional tightly coupled or monolithic application architectures. SOA incurs the minimum possible lock-in among components in situations that involve collaboration. Companies that seek to maximize the effectiveness of their IT usage must invest in event-oriented design (as well as SOA). Their architects must have a thorough understanding of business events. They must identify and document business events in the earliest stages of business process design and follow through by implementing event-driven business components in the later stages of development and maintenance.
In the very long run, the Model Driven and Event Driven Architectures (closely related) will replace today’s SOAs.

Background Material on BPM Market

The BPM market is in tremendous flux and is rapidly growing and developing. Illustrating the rapid growth in this product arena, Gartner’s first BPM conference, held in 2005, had around 600 attendees. A year later, there was a 50% increase in the number of attendees at their next BPM conference. Along similar lines, Gartner conducted a survey among a representative population of CIOs in thirty-plus countries. Their survey results, summarized in a news story posted on www.bpm.com, contained a list of the absolute top business priorities in 2006. “Business process improvement” was the phrase that conveyed these representative CIO’s number one single priority. Similarly, among technology priorities, at least five of the top ten are those that concern areas which are directly addressed by BPM. These five in relative order of priority are: business intelligence applications, mobile workforce enablement, collaboration technologies, service oriented architecture, and workflow management.

The growth and excitement in the BPM space is important to consider when evaluating individual products. One of many reasons why it is important to consider this explosion of growth is that we should remember that it is common in the information technology field for a new technology to be initially pursued by numerous vendors before the actual leaders of the new technology area surface and show sustainability. Accordingly, in March of 2006, Gartner stated the following about the BPM market:

“The "BPM market" is really multiple markets or segments with over 160 software vendors vying for leadership. Most vendors are small, privately held companies with revenues under $50 million. With so many vendors, few have really enjoyed significant growth rates and few, if any, can legitimately claim leadership. As is typical of early markets, there will be multiple waves of consolidation. This will continue through 2009, as buyers become more sophisticated, and the technology matures into more-predictable sets of features and functionalities (0.7 probability)…Of the contenders in the BPMS market in 2005, we anticipate that only 25 will continue to compete beyond this time frame, with the others moving into potentially adjacent market spaces. There has already been some market consolidation through acquisition, although we expect the number of acquisitions to steadily rise through mid-2006, once IBM, Microsoft and Oracle start delivering more of their complete BPM architecture.”

Some time earlier, Gartner had also stated: “because many BPM pure-play vendors are small companies with limited resources, no more than 25 of the 140+ competitors of 2005 will transition to the emerging BPMS market even by 2008. The rest will transition to providing more specialized tools, become suppliers of packaged processes for specific industries, geographies or horizontal processes, migrate to alternative adjacent tool markets, be acquired or cease trading (0.8 probability).”

When one looks at the range of products that comprise the BPM product space, one sees a broad range of products. These include such diverse software products as those that provide workflow, enterprise application integration, business process modeling, business rules management, and business intelligence analysis, to name just a few. Ultimately, it is possible the future will consist of industry specific markets, rather than any specific and generic market, since it may ultimately prove to be easier to integrate all the elements for an industry rather than arrive at a generic, universal BPM suite solution.

According to Gartner, a comprehensive BPM suite needs to deliver 10 major areas of functionality:

• Human task support facilitating the execution of human-focused process steps (cross referenced (ref.) #1)
• Business process/policy modeling and simulation environment (ref. #2)
• Pre-built frameworks, models, flows, rules and services (ref. #3)
• Human interface support and content management (ref. #4)
• Collaboration anywhere support (ref. #5)
• System task and integration support (ref. #6)
• Business activity monitoring (BAM) (ref. #7)
• Runtime simulation, optimization and predictive modeling (ref. #8)
• Business policy/rule management support (heuristics and inference) (ref. #9)
• Real-time agility infrastructure supports (ref. #10)

In the 2005 Business Process Trends Report (also referred to as the 2005 BP Suites Report and available at www.bptrends.com), Derek Miers and Paul Harmon list ten BPM product areas. There is some similarity in this list to the preceding one. To highlight the similarities and differences, I have attempted to map these two lists. Thus, in the Gartner list, I have arbitrarily assigned reference numbers to which the BPM Suites Report subsequently cross references. Some of this mapping was subjective because product categories easily overlap and the difference between tools ultimately comes down to how practitioners use specific products. We will follow up these two lists with a report provided by Forrester. The reason I have chosen to include data by three different industry analyst organizations is to ensure completeness. The ten product areas listed in the BPM Trends report were:

• Process modeling tools (see ref. #1)
• Simulation tools (see ref. #8)
• Business rule management tools (see ref. #9)
• BPM applications (meaning complete turn-key enterprise resource planning (ERP) system with BPM elements which are found within a BPM suite)
• Business process monitoring tools (see ref. #7)
• Software development tools (not included by Gartner, and ref. #11)
• Enterprise Application Integration (EAI) tools (see ref. #6)
• Workflow tools (see ref. #1 and #4)
• Business process languages (not included by Gartner, and ref. #12)
• Organization and enterprise modeling tools (not included by Gartner, and ref. #13)