Wednesday, March 19, 2008

Friday, March 14, 2008

The SOA Consortium

I attended the SOA Consortium in Crystal City, Virginia on March 12 and 13. This group of people are identifying the major business-driven SOA activities and drafting advice on how to plan and execute those activities. The framework will be a publicly available, online resource. Their intent is to iterate content delivery over the course of 2008. The initial launch was targeted for March 2008. I am not sure if they will make this date since it is already the month of March. Initial launch will focus on the framework itself, a high-level description of the project and framework, and individual activity descriptions and their relevance to SOA. Throughout 2008, the group will incrementally add content for each framework activity.

The SOA Consortium is an advocacy group of end users, service providers and technology vendors committed to helping the Global 1000, major government agencies, and mid-market businesses successfully adopt Service Oriented Architecture (SOA) by 2010. Members of the SOA Consortium are listed below. The SOA Consortium is managed by the Object Management Group.

The Practical Guide to Federal Service Oriented Architecture (PGFSOA) Semantic Media Wiki (SMW) site is at ( http://smw.osera.gov/pgfsoa/index.php/Welcome ).

The members of the steering committee are listed at: ( http://www.soa-consortium.org/steering-committee.htm ).

Organizational membership to the consortium is $5,000 annually. The average time commitment for active members is 5 hours a month. This includes call participation and contribution to working group deliverables. The SOA Consortium is not a standards organization–it is an advocacy group.

I do think it is absolutely worthwhile to get involved with this consortium. They have quarterly meetings. Upcoming meetings are:

• June 25-26, 2008 in Ontario Canada
• September 24-25, 2008 in Orlando, Florida
• December 10-11 in Santa Clara, California

I hope to get some podcasts and powerpoint briefings from the conference.

Again, I definitely think this is something to pursue and I would be willing to commit to five hours a month plus quarterly meetings. It would be good exposure for Northrop Grumman.

Canonical Best Practices

“Canonical” is a typical IT industry buzzword - it is an overloaded term with multiple meanings and no clear agreement on its definition. There appears to be at least three uses for canonical modeling.

-- Canonical Data Modeling
-- Canonical Interchange Modeling
-- Canonical Physical Formats

One of the challenges at most large corporations is to achieve efficient information exchanges in a heterogeneous environment. The typical large enterprise has hundreds of applications which serve as systems of record for information and were developed independently based on incompatible data models, yet they must share information efficiently and accurately in order to effectively support the business and create positive customer experiences.

The key issue is one of scale and complexity and is not evident in small to medium sized organizations. The problem arises when there are a large number of application interactions in a constantly changing application portfolio. If these interactions are not designed and managed effectively, they can result in production outages, poor performance, high maintenance costs and lack of business flexibility.

Canonical Data Modeling is a technique for developing and maintaining a logical model of the data required to support the needs of the business for a subject area. Some models may be relevant to an industry supply chain, the enterprise as a whole, or a specific line of business or organizational unit. The intent of this technique is to direct development and maintenance efforts such that the internal data structures of application systems conform to the canonical model as closely as possible. In essence, this technique seeks to eliminate heterogeneity by aligning the internal data representation of applications with a common shared model. In an ideal scenario, there would be no need to perform any transformations at all when moving data from one component to another, but for practical reasons this is virtually impossible to achieve at an enterprise scale. Newly built components are easier to align with the common models, but legacy applications may also be aligned with the common model over time as enhancements and maintenance activities are carried out.

Canonical Interchange Modeling is a technique for analyzing and designing information exchanges between services that have incompatible underlying data models. This technique is particularly useful for modeling interactions between heterogeneous applications in a many-to-many scenario. The intent of this technique is to make data mapping and transformations transparent at build time. This technique maps data from many components to a common Canonical Data Model which thereby facilitates rapid mapping of data between individual components since they all have a common reference model.

Canonical Physical Format prescribes a specific runtime data format and structure for exchanging information. The prescribed generic format may be derived from the Canonical Data Model or may simply be a standard message format that all applications are required to use for certain types of information. The formats are frequently independent of either the source or the target system, and requires that all applications in a given interaction transform the data from their internal format to the generic format. The intent of this technique is to eliminate heterogeneity for data in motion by using standard data structures at run-time for all information exchanges.

Canonical techniques are valuable when used appropriately in the right circumstances. Some recommendations are:

-- Use canonical data models in business domains where there is a strong emphasis to “build” rather than “buy” application systems.
-- Use canonical interchange modeling at build-time to analyze and define information exchanges in a heterogeneous application environment.
-- Use canonical physical formats at run-time in many-to-many or publish/subscribe integration patterns. In particular in the context of a business event architecture.
-- Plan for appropriate tools to support analysts and developers. Excel is not sufficient for any but the simplest canonical models.
-- Develop a plan to maintain and evolve the canonical models as discrete enterprise components. The ongoing costs to maintain the canonical models can be significant and should be budgeted accordingly.

How can we minimize dependencies when integrating applications that use different data formats? Conventional thinking suggests that a Canonical Data Model, one that is independent from any specific application, is a best practice. By requiring each application to produce and consume messages in this common format, components in a SOA are more loosely coupled.

Here is what Gregor Hohpe has to say in his book Enterprise Integration Patterns:

The Canonical Data Model provides an additional level of indirection between application's individual data formats. If a new application is added to the integration solution only transformation between the Canonical Data Model has to be created, independent from the number of applications that already participate.
The promised benefits of canonical best practices generally include increased independence between components so that one can change without affecting other components and simplified interactions because all applications use common definitions for interactions. As a result solutions are expected to be lower cost to develop, easier to maintain, higher quality in operation, and quicker to adapt to changing business needs.

While canonical best practices can indeed provide benefits there is also a cost that must be addressed. The canonical model and interchanges themselves are incremental components which must be managed and maintained. Canonical data models and interchanges a) require effort to define, b) introduce a middleware layer (either at the build time or run time depending on which techniques are used), and c) incur ongoing maintenance costs. These costs can exceed the benefits that the canonical best practices provide unless care is taken to use the techniques in the right circumstances.

For a large scale system-of-systems in a distributed computing environment, the most desirable scenario is to achieve loose coupling and high cohesion resulting in a solution that is highly reliable, efficient, easy to maintain, and quick to adapt to changing business needs. Canonical techniques can play a significant role in achieving this ideal state.

The three canonical best practices are generally not used in isolation; they are typically used in conjunction with other methods as part of an overall solutions methodology. As a result, it is possible to expand, shrink, or move the “sweet spot” subject to how it is used with other methods. This posting does not address the full spectrum of dependencies with other methods and their resultant implications, but it does attempt to identify some common anti-patterns to be avoided.

Anti-Patterns:

Peanut Butter: One anti-pattern that is pertinent to all three canonical best practices is the “Peanut Butter” pattern which basically involves applying the methods in all situations. To site a common metaphor, “to a hammer everything looks like a nail”. It certainly is possible to drive a screw with a hammer, but it’s not pretty and not ideal. It is also possible to make a hole with hammer – but it might have some very jagged edges.

When, and exactly how, to apply the canonical best practices should be a conscious, well-considered decision based on a keen understanding of the resulting implications.

Canonical Data Model Best Practice

The sweet spot for a Canonical Data Model (or logical data model) is in directing the development of data structures for custom application development. This could apply for new applications built from scratch or modifications to purchased applications. This discipline dates back to the early 1990’s when the practice of enterprise data modeling came into the mainstream. It continues to be widely practiced and effective method for specifying data definitions, structures and relationships for large complex enterprise solutions. Vendors like SAP and Oracle use this method for their ERP solutions.

This technique, when applied effectively, tends to result in a high level of cohesion between system components. It is highly cohesive since its attributes are grouped into entities of related information. It also tends to result in tightly coupled components (since the components are very often highly interdependent) which are able to support very high performance and throughput requirements.

Anti-patterns:

Data model bottleneck: A Canonical Data Model is a centralization strategy that requires an adequate level of ongoing support to maintain and evolve it. If the central support team is not staffed adequately, it will become a bottleneck for changes which could severely impact agility.

Thursday, February 21, 2008

ESB sprawl

I attended a presentation on IBM's Data Power product. One of the three speakers gave a brief on GCSS-AF, which the speaker credited for being the largest IT structure within DoD with one million users and 120+ different locations, and how it is using DataPower. I was introduced to some new heuristics and reminded that XML was introduced in 1995 and web services were introduced two years later. Years later we are still trying to figure it all out. A new phrase I heard that resonated with me was ESB Sprawl. ESBs are the latest rage -- well, not really, these have been out for quite a while and are still misunderstood. Anyway, like anything, these components can proliferate madly unless one has a good handle on their IT environment; hence, the phrase.

Metcalfe's Law -- the value of a network increases as the number of nodes on it increase (specifically, the value of a network is proportional to the square of the number of users of the system (n²)

Reed's Law -- the utility of large networks (particularly social networks) can scale exponentially with the size of the network.

Conway's Law -- Any piece of software reflects the organizational structure that produced it (for example, if you have four groups working on a compiler, you are likely to get a 4-pass compiler). Another way of stating this is the components and interfaces of system tend to mirror the engineering groups and their interfaces.

Tuesday, February 12, 2008

just what we need -- another markup language

I learned of another markup language that I just need to be aware of although not much seems to be going on with it as of yet: Architecture Description Markup Language (AMDL) or Architecture Description Language (ADL). This language -- if/when developed -- will provide a common interchange format for exchanging data among architecture design tools and/or a foundation for the development of new architectural design and analysis tools.

Background: ADML, the Architecture Description Markup Language, is a standard XML-based mark-up language for describing software and system architectures. ADML provides a means of representing an architecture that can be used to support the interchange of architectural descriptions between a variety of architectural design tools. The standard makes possible the broad sharing of ADML models so that many present and future applications can manipulate, search, present, and store the models. Given the ongoing adoption of XML by industry, XML-based ADML models will be in a format that will not become orphaned. A standard, open representation will de-couple an enterprise's architectural models from vendors and enable the models to remain useful despite the rapid change in software tools. ADML leverages the work of academia and other essential organizations such as W3C and OMG. It provides a firm basis for a future where tools can share information more seamlessly, and where computer architecture can move towards the rigor we see already in the building industry. ADML is based on Acme, a software architecture description language developed at Carnegie Mellon University and the Information Sciences Institute at the University of Southern California.

ADML is part of a broader program of work being undertaken by The Open Group's Architecture Program. The goals of this program are to:

-- Define an industry standard Architecture Description Language for IT architecture tools, providing portability and interoperability of architecture definitions across different tools from different vendors, creating a viable market for open tools for IT architecture definition

-- Use this same language as the basis of a "Building Blocks Description Language", capable of defining open, re-usable architecture building blocks:
re-usable across customer IT architectures; fit for use in procurement (allowing real products to be conformance tested and procured to fulfil the defined functions), and catering for "fuzzy" definitions as well as tightly defined specification

-- Create an open repository in which to store such building block definitions (the "Building Blocks Information Base")

-- Develop testing and branding programs to verify conformance of vendor IT solutions to Building Block definitions

Status: Although we should be aware of the effort of The Open Group (TOGAF) and others in trying to develop a language so that artifacts can be shared amongst enterprise architecture tools, no one has of yet gotten very far. If you google ADML, a lot of the links date to the year 2000. We'll see what happens.

Tuesday, January 29, 2008

SOA for SOA's sake

Too many times, we’re engaging in SOA for SOA’s sake. There are many situations were SOA may simply not be necessary. A mainframe system that rapidly processes transactions may be well enough left alone. You’re not going to increase the speed of systems by putting a layer on top of it. SOA success means applying SOA where needed. But if it ain’t broke, don’t fix it. Last month I was at a DoD seminar and one of the speakers said that you don't want to wrap a nuclear bomb system in a layer of SOA. That's good. I am glad the speaker (and hopefully others within DoD) realize this.

What would an economic downturn mean for SOA?

On January 7th, Joe McKendrick posted an interesting question on his blog on zdnet. He asked what an economic downturn might mean for SOA. His post is at: ( http://blogs.zdnet.com/service-oriented/?p=1036 ). He says that with the recent hysteria of doom and gloom over the state of the economy, and while he believes that the economy is resilient, it’s only natural to see ups and downs in business cycles. However, he says that the current incarnation of SOA has only been around during a period of upward economic growth and he contemplates might be the impact of an economic downturn. You can read his post but the gist is that he feels that while some adjustments may need to be made, he and members of his SOA consortium believe that SOA efforts will continue to press on, especially if SOA is closely tied to business value. One industry practice leader said that: In a “downturn, people will have to be more creative, and will turn towards SOA.”

So, in summary, McKendrick suggests that the outlook for SOA for 2008 is quite bullish, no matter how rocky the economy gets. He does suggest, however, that the way SOA is sold to organizations -- as well as implemented -- may shift.

Tuesday, January 22, 2008

how to pick an ESB...

There is a great deal of debate and confusion around SOA and ESB products, so evaluating them for purchase is a challenging proposition. Many vendors, for example, argue that ESB is a product and others an architecture. Some vendors also argue that ESB is not product at all but is a marketing term. ESB is certainly a marketing term, but it does imply a set of functionality to expect, so the term does provide value in categorizing products.

Network Computing magazine recently completed an evaluation of ESB products that provides an interesting perspective. Their evaluation was largely based on development tool capabilities. They did not, for example, do any performance tests. Network Computing ranked BEA Systems Aqualogic Service Bus 2.1 first followed closely by Oracle SOA Suite. The other contenders included TIBCO Software, Fiorano, Cape Clear, IBM, Software AG and Sonic Software. Sonic Software placed last even though they were an early proponent of the ESB concept and marketing strategy.

Network Computing included integration in their evaluation and I agree with this approach. Integration is still a key criterion for evaluation since there are so many legacy systems to wrap as part of SOA. Not surprisingly, the vendors coming from the EAI space did well in this category.

Some of the products did very poorly, which is an indicator of the immaturity in the space. For example, some products had poor administration and debugging tools. Some did poorly in mapping XML to XML, which is surprising since this is such a fundamental feature.

As expected, the large platform vendors continues to place pressure on the pure-play vendors. BEA and Oracle has strong showings with their products. IBM was somewhat limited in the test by the test criteria. For example, they scored poorly in orchestration, but this feature is in a product that was not considered (WBI Process Server). Also, completeness of the product stack (a Gartner magic quadrant measure) was not considered. This measure could have helped IBM and TIBCO. Product performance is also a strong suit for IBM's Message Broker, which was not considered. The evaluation also did not play to Sonics strengths -- either performance or out-of-the-box fault tolerance. Also, the evaluation tested the 6.1 version of Sonic ESB, not the latest version which has had significant improvement. The Network Computing evaluation lacked some critical aspects of evaluation that should be considered, namely:

* The architectural fit with existing platforms and technology
* Completeness of stack - e.g. Business Process Management, Business Activity Monitoring, Portal (these are not ESB products but should be considered for enterprise SOA)
* Product performance
* Fault tolerance

So far, however, in multiple reports, by a variety of organizations, it seems that the strongest ESB product is BEA's. No wonder Oracle has decided to purchase BEA -- for its strong SOA foothold.

I am attending that ESB online conference in 2 days. Then I get to roll that knowledge into my ESB report and get back to real work. :-D

Sun's SeeBeyond product doesn't seem to be anywhere near the same level as all of these other ESB products. If Sun had an competitive ESB product it should be included in the study -- but it looks like they don't.

Cape Clear also seems mostly targeted towards smaller organizations.

Tuesday, December 18, 2007

some SOA facts

For some light humor, see: http://soafacts.com

Wednesday, December 12, 2007

Practical Guide for Federal Service Oriented Architecture (PGFSOA)

The American Council for Technology (ACT) / Industry Advisory Council (IAC) Enterprise Architecture (EA) Special Interest Group (SIG) (referred to as ACT/IAC EA SIG) just completed an internal review of the "Practical Guide for Federal Service Oriented Architecture (PGFSOA)", part of an IAC initiative. This document will be a product of the Federal CIO Council’s Architecture and Infrastructure Committee and the ACT/IAC's EA SIG and will provide guidance for federal agencies in implementing service oriented architecture. The draft was recently released for general comments and now approximately 300 comments are being resolved. The next version for broad comment is expected to be out in mid January. I was not able to gain access to the document itself so I volunteered my name for broad comment in January. About 35 persons have been actively supporting the writing. The objectives for the guide is to follow in the footsteps of "A Practical Guide to Federal EA" and build upon the Service-Component Based Architecture white papers that have been published. The collective group had wanted to rapidly create a document not too unlike the "Practical Guide to FEA", which provided practical guidance in support of agencies’ efforts to adopt SOA into their business, IT and EA practices.

Renee

Monday, November 5, 2007

some short notes from PegaWorld 2007

I am going to be brief on my notes from the conference as I will share all of the presentations with the team once they are available. Some assorted notes I captured are listed below.

Processes are the muscles; business rules are the nerves (quote by Ron Ross, Co-Founder and Principal of Business Rule Solutions, LLC, recognized internationally as being the "father of business rules”).

Applications are getting smaller. Different applications can share same rules and processes, bringing increased ability to rapidly respond to change.

Of the 20+ widely recognized workflow patterns (see http://www.workflowpatterns.com/vendors/documentation/BPMN-pat.pdf ), the patterns used 80% of the time are Sequential, Parallel Split and Join, and Decision Split (remember 80/20 rule?).

PowerPoint most widely used BPM tool, followed by Visio.

Business Activity Monitoring (BPM) is based on current (active) data; Business Intelligence (BI) is based on historical (and aggregated) data; Business Process Analytical (BPA) data is based on historical data for the intent of determining the future via statistical tools and predictive models. Examples of BI include Business Objects and Cognos. Examples of BPA include Hyperion, SAS and SPSS. BI is largely based on data warehousing.

PMML = Predictive Modeling Markup Language; PMML is a mark up language for statistical and data mining models. It can be used/generated by Pega.

SmartBPM = BPM + BI + SOA

Quote: “There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things.” ~Niccolo Machiavelli

Our brain (medulla oblongata) is genetically programmed to avoid/resist change (theorem of neurochemistry)

We resist change. Just for fun, the next time you shower, notice where you start lathering. The next day try somewhere new.

Strive for Business and IT fusion – not alignment. Alignment suggests trying to line up 2 or more things that are in motion? Ever try to deck a boat on a moving dock?

It is easy to buy technology pieces; it is hard to implement methodology.

What is the difference between a rule and a policy? The Ten Commandments are a policy; Thou Shalt Not Kill is a rule.

BPM is not just BPR – focus is on real time (or nearer real time)